Sensing Urban Space: from Street View Recognition, Event...
18
太空與地球信息科學研究所 Institute of Space & Earth Information Science Sensing Urban Space: from Street View Recognition, Event Inference to Understand Urban Behavior http://www.iseis.cuhk.edu.hk/ ONTOLOGICAL APPROACHES TO SENSOR DATA ANALYSIS Fan Zhang 12 , Lezhi Li 23 , Hui Lin 1 May 30, 2016 1 Institute of Space and Earth Information Science, The Chinese University of Hong Kong 2 Senseable City Lab, Massachusetts Institute of Technology 3 Graduate School of Design, Harvard University
Transcript of Sensing Urban Space: from Street View Recognition, Event...
太空與地球信息科學研究所Institute of Space & Earth Information Science
Sensing Urban Space: from Street View Recognition,
Event Inference to Understand Urban Behavior
http://www.iseis.cuhk.edu.hk/
ONTOLOGICAL APPROACHES TO SENSOR DATA ANALYSIS
Fan Zhang12, Lezhi Li23, Hui Lin 1
May 30, 2016
1Institute of Space and Earth Information Science, The Chinese University of Hong Kong2Senseable City Lab, Massachusetts Institute of Technology
3Graduate School of Design, Harvard University
太空與地球信息科學研究所Institute of Space & Earth Information Science
Dataset – Street Level Images
Covering：More than 5 million miles of roads, 3,000 cities, 50 countries,
太空與地球信息科學研究所Institute of Space & Earth Information Science
Data Collection
6 km
100,000+ images collected from google street view API, within a 6 km radius.Images are extracted on a grid, 4 directions for each point.
太空與地球信息科學研究所Institute of Space & Earth Information Science
Objective
Way to discover Visual Patterns in Urban Area
Contents
Deep Learning Features
• Learning Features Intelligently• Greenery
• Building Area
• Entropy
• Colors (RGB)
Hand-Design Features
太空與地球信息科學研究所Institute of Space & Earth Information Science
Hand-Design Visual Features – Building Color
Fig. Building facade color map
太空與地球信息科學研究所Institute of Space & Earth Information Science
Fig. Image entropy: visual noise
Low entropy – industrial districts
Hand-Design Features –Entropy
太空與地球信息科學研究所Institute of Space & Earth Information Science
Image Entropy
Greenery
Openness
Building Color
Hand-Design Features
太空與地球信息科學研究所Institute of Space & Earth Information Science
Demo
http://lezhili.me/cv_streetview/
太空與地球信息科學研究所Institute of Space & Earth Information Science
Recognizing Cities/Neighborhoods from Street View images
Training
Testing
“Beacon Hill”
“Beacon Hill”
“Beacon Hill”
Image + LabelLabel
New Image Predicted Label
Deep Learned Visual Features
太空與地球信息科學研究所Institute of Space & Earth Information Science
Recognizing Cities/Neighborhoods from Street View
Deep Learned Visual Features
太空與地球信息科學研究所Institute of Space & Earth Information Science
Feature Visualization – Discovery the Hierarchical Feature Structure of Image sets
Deep Learned Visual Features
太空與地球信息科學研究所Institute of Space & Earth Information Science
Deep Learned Visual Features
• In order to evaluate visual similarity in a more comprehensive , hierarchical way, and an automated process is required for feature extraction.
• Deep Learning vs. Traditional Machine Learning • Deep Artificial Neural Networks (3 layers to N layers..)• Automatic Feature Extraction Process • Convolutional Neural Nets (CNN)
太空與地球信息科學研究所Institute of Space & Earth Information Science
“Cities Name”
“Neighborhood Name”
Label
“Populations”
“Crime”
“Income”
“Human Perception”
“Human Emotion”
What’s the visual feature of
What’s the visual feature with a high/low
What’s the visual feature with a high/low
Deep Learned Visual Features
Different Task to Discover Visual Features
太空與地球信息科學研究所Institute of Space & Earth Information Science
太空與地球信息科學研究所Institute of Space & Earth Information Science
“Cities Name”
“Neighborhood Name”
Label
“Populations”
“Crime”
“Income”
“Human Perception”
“Human Emotion”
What’s the visual feature of
What’s the visual feature with a high/low …
What’s the visual feature with a high/low
Image Spatial Features Images Temporal Features
Deep Learned Visual Features
Different Task to Discover Visual Features
太空與地球信息科學研究所Institute of Space & Earth Information Science
July 2008
August 2011
May 2015
From Image Semantic Recognition to Event Inference
Destruction in Onagawa, Japan after the 2011 earthquake
Earth Quake
太空與地球信息科學研究所Institute of Space & Earth Information Science
From Event Inference to Urban Development Behavior
太空與地球信息科學研究所Institute of Space & Earth Information Science
• Conclusion
• Street View Images have Infinite Dimension
• Deep Learning Helps Spatial knowledge discovery
Thank you!
