COPERNICUS & MACHINE LEARNING:Collaboration of ESA and the AI community
Sašo Džeroski
Jožef Stefan Institute, Ljubljana, Slovenia
Visiting professor, Φ-lab, ESA/ESRIN, Frascati, Italy
“Copernicus and Artificial Intelligence” workshop28 January 2020
Brussels, Belgium
Jožef Stefan Institute (JSI)Founded in 1949, JSI is the largest research institute in SloveniaPhysics, chemistry, molecular biology, biotechnology, information technologies, energy and environmentStrongest fields: Nano-, Eco- and IC- technologies
About 1000 employees, most with PhD degreeMany, many EU projects
European Research Ranking: In top 10 research institutes in Europe“Copernicus and Artificial Intelligence”
28 January 2020
Department of Knowledge TechnologiesFounded in 2004Approximately 40 researchers
Knowledge technologies = Information technologies that support the acquisition, management, modelling & use of knowledge and data
• Knowledge Management• Decision Support• Language Technologies• Knowledge Discovery: Machine Learning and Data Mining
“Copernicus and Artificial Intelligence”28 January 2020
My research groupSince 2005, I lead a research group @DKT/JSI, now cca. 20 people
Development of Machine Learning Methods for• Mining Big & Complex Data/Predictive Modeling• Computational Scientific Discovery• Ontologies for Data Mining & Comp. Sci. Discovery
Applications of Machine Learning• Life Sciences (incl. drug development, medicine)• Environmental Sciences (incl. agriculture, ecology, forestry)• Earth Observation (incl. agriculture, archeology)
“Copernicus and Artificial Intelligence”28 January 2020
ESA’s Φ-lab Founded in 2017, the Φ-lab (Phi Lab) at the European Space Agency has the mission of accelerating the future of Earth Observation
Helping Europe's • earth observation• space researchers, and • companies Adopt disruptive technologies and methods
Disruptive technologies & methods = Artificial Intelligence
“Copernicus and Artificial Intelligence”28 January 2020
ESA’s Φ-lab Some of the Φ-lab activities include • Building bridges between the EO and AI communities• Promotion of the use of Artificial Intelligence within ESA• Organizing the Φ-week, a highly successful/popular event• Organizing thematic workshops that bring the EO and AI
communities together (Φ-lab thematic workshops), such as• Atmospheric Water Cycling in a Changing Climate• AI, Machine Learning and Deep Learning and Archaeology
• Producing white papers, such as “Towards a European AI for Earth Observation Research & Innovation Agenda”
“Copernicus and Artificial Intelligence”28 January 2020
ESA’s Φ-lab People
Visiting professors program • Work with the team of Research Fellows (providing them with advice, ideas
and connections) • Helping ESA in shaping a Research & Innovation agenda for a AI4EO and a
strategy for ESA EO• Develop partnerships with a rapidly growing AI ecosystem.
“Copernicus and Artificial Intelligence”28 January 2020
The Artificial Intelligence Challenge of COPERNICUSCOPERNICUS is generating massive amounts of remotely sensed images in different modalitiesHigh quality, high resolution, dense temporal coverageThe data are publicly available
The challenge: Derive added value from these dataFacilitate access to these dataUse artificial intelligence techniques (ML) to infer additional information
“Copernicus and Artificial Intelligence”28 January 2020
Artificial Intelligence includes, but is not limited to Machine LearningAI is the science and engineering of making intelligent machines, especially intelligent computer programs • Knowledge representation• Knowledge engineering• Reasoning• Planning• Natural language processing• Computer vision
“Copernicus and Artificial Intelligence”28 January 2020
Deep LearningAbility to learn many-layered neural networks from vast amounts of data
Machine Learning includes, but is not limited to Deep Neural Networks• Reinforcement learning• Computational scientific discovery• Learning from data (inductive learning)
• Unsupervised learning (clustering)• Supervised learning (predictive modeling)
• Classification • Regression
• Learning decision trees & tree ensembles• Learning (if-then) rules & rule ensembles• Learning logic programs (relational rules)• Understandable/explainable models• Explainable AI
“Copernicus and Artificial Intelligence”28 January 2020
Neural Networks, Shallow and Deep
“Copernicus and Artificial Intelligence”28 January 2020
Neural Networks and Image Classification
Neural networks learn classification models, see patterns, spot anomalies. They can handle datasets far larger and messier than humans can cope with. They are great for classifying images!
“Copernicus and Artificial Intelligence”28 January 2020
Neural Networks and Satellite Image ClassificationNeural networks are also great for classifying satellite images!E.g., land cover classification from SENTINEL-2 imagesCf. BigEarthNet (group of Begum Demir)http://bigearth.net/
Sumbul et al., IGARSS, 2019
“Copernicus and Artificial Intelligence”28 January 2020
End-to-End-Learning: The key to the success of Deep NNs
“Copernicus and Artificial Intelligence”28 January 2020
The Limitations of DNNs: Data hunger
Deep Neural Networks need lots of labeled training data!BigEarthNet has 590326 Sentinel2 image patches
But COPERNICUS provides an abundance of data! So what’s the problem? This data is unlabeled!
“Copernicus and Artificial Intelligence”28 January 2020
The Limitations of DNNs: Computing power hungerAn example from a presentation at a recent scientific event Comparing the results of DNNs and Random Forests on a task of land cover classification
DNN performance: 97% accuracy, 3300 hrs GPURFs performance: 96% accuracy, 36 hrs CPU
Electricity spent (rough estimate): 1KWh for RFs, 1 MWh for DNNs
“Copernicus and Artificial Intelligence”28 January 2020
Transfer Learning with Deep Neural Nets
“Copernicus and Artificial Intelligence”28 January 2020
To combat scarcity of labeled data• learn in one domain (where enough
labeled data available), then transfer the learned knowledge to another
Fine-tuning pre-trained DNNs• Cut-off final layers of pre-trained net• Retrain final layers with labeled data
Why we shouldn’t use ImageNet pre-trained DNNs for classifying RS images?Slide by Begum Demir, Φ-week, September 2019
“Copernicus and Artificial Intelligence”28 January 2020
Pre-training NNs with unlabeled RS images
Deep Clustering: Unsupervised Learning of Visual Features
Rather than pre-training with labeled ImageNet, pre-train with unlabeled RS images/ patches
“Copernicus and Artificial Intelligence”28 January 2020
Pre-training NNs with unlabeled RS images:Preliminary ResultsPreliminary results on BigEarthNetland cover classification are great!
“Copernicus and Artificial Intelligence”28 January 2020
Machine Learning for explainable AI
Machine learning for big and complex data• Predicting structured outputs• On-line learning from data streams • Semi-supervised learningMaking machine learning an open science• Ontologies for describing machine learning (data types, ML tasks,
ML methods)• Ontologies for describing application domain
“Copernicus and Artificial Intelligence”28 January 2020
Multi-target prediction on data streams
• The dataset is not available in its entirety ahead of time• Theoretically infinite data examples
• Cannot store them: Processed once and discarded
• Often, real- or near real-time response is needed• Temporal dimension may lead to distribution change
• “Concept drift”
“Copernicus and Artificial Intelligence”28 January 2020
Descriptive space Target space
… … …
… … … … …
Example n 1 TRUE 0.49 0.69 0.68 0.60 3.91Example n+1 4 FALSE 0.08 0.07 0.10 1.69 7.57
Example n+2 6 FALSE 0.08 0.07 0.08 0.77 8.86Example n+3 8 TRUE 0.00 1.00 0.11 3.51 2.50Example n+4 6 TRUE 0.00 0.00 0.43 2.10 8.09
Example n+5 6 TRUE 0.46 0.11 0.56 0.99 7.59
Hierarchical Multi-label Classification
“Copernicus and Artificial Intelligence”28 January 2020
Semi-supervised Learning: Using Unlabeled Data
“Copernicus and Artificial Intelligence”28 January 2020
AI Prototyping Environment for Earth Observation• Joint effort with ESA• Slovenian actors: Bias Variance Labs, JSI, ZRC SAZU• Will develop tools for
• Unsupervised pre-training of DNNs with different types of RS data• Enabling effective extraction of features/ variables describing images• Making the pre-trained networks available like ImageNet pre-trained networks
• Will also develop tools allowing • The use of features/ variables defined by pre-trained networks • With other machine learning algorithms , e.g., tree-ensembles • For more complex tasks of predictive modeling, e.g., hierarchical classification
“Copernicus and Artificial Intelligence”28 January 2020
EO (incl. COPERNICUS) Data is BIG!But is it FAIR?
• Findable• Accessible• Interoperable• ReusableFor data to be FAIR, it needs to be accompanied with some knowledge (annotations) using terms from ontologies
“Copernicus and Artificial Intelligence”28 January 2020
Ontologies, Open Data Science, Reproducible AI/ML Research• Both data and SW need to follow FAIR principles
(Findable, Accessible, Interoperable and Reusable)• Ontologies for describing the elements and processes of data
science (data mining)• Data• Data Mining Tasks• Data Mining Algorithms• Data Analytics Processes/Workflows
• Together with domain ontologies, this allows• Precise description of the data analyses performed• Matching algorithms and data• Automated construction of data analysis workflows
“Copernicus and Artificial Intelligence”28 January 2020
Ontology of Data Mining/ Data Science
“Copernicus and Artificial Intelligence”28 January 2020
Directions for developmentDevelop AI tools and environments integrating different and powerful machine learning methods
Make COPERNICUS data accessible and AI-ready • Infrastructural/hardware aspects • Semantic/ annotation aspectsConnect remotely sensed with ground data• ‘Ground truth’ data are needed• Labeling/annotation efforts
High-performance Artificial Intelligence (as a Service)“Copernicus and Artificial Intelligence”
28 January 2020
Take-home Messages for the Successful use of AI / Machine Learning to COPERNICUS data1. INTEGRATE different approaches to machine learning
• Unsupervised/Supervised• Deep Learning/Tree ensembles
2. ANNOTATE satellite imagery (connect to ground data)• Need more AI-ready COPERNICUS data• For a variety of application domains
3. REUSE data, as well as models in a FAIR fashion• Use ontologies describing data, remote sensing, application areas to
annotate datasets• Share and re-use NNs pre-trained with unlabeled satellite images
“Copernicus and Artificial Intelligence”28 January 2020
Thanks!
To the ESA Φ-lab, and especially Sveinung Loekken, Pierre-Philippe Matthieu, and Iarla Kilbane-Dawe To my team and especially Dragi Kocev, Pance Panov, Nikola Simidjievski, and Ivica DimitrovskiThe complete Ljubljana eco-system, incl. JSI, JSI IPS, BV Labs
“Copernicus and Artificial Intelligence”28 January 2020
Top Related