"Using Inertial Sensors and Sensor Fusion to Enhance the Capabilities of Embedded Vision Systems," a...
-
Upload
embedded-vision-alliance -
Category
Technology
-
view
23 -
download
3
Transcript of "Using Inertial Sensors and Sensor Fusion to Enhance the Capabilities of Embedded Vision Systems," a...
Kevin A. Shaw, Ph.D.
Chief Technology Officer
March 30th, 2014 Santa Clara, California, USA
2014 Embedded Vision Member Meeting [email protected]
R2
• Consumer games – More stable attitude
• Augmented Reality (AR) – Needs improvement, better accuracy
• Indoor Navigation – Need better accuracy; lower power
• Hyper photography – Super resolution; intraframe deblur
• Robotics – Visual odometry to detect egomotion
– Always need better accuracy
6/2/2014 2
• Construction equipment – Perimeter safety
• Context awareness – Understanding users better
• Change from mobile to wearables – Shift from mostly-pocket to always-visual
– Digital eyewear makes a big difference
• Natural interfaces – Using the same wealth of information as
humans do to understand the world
6/2/2014 3
• How to solve some of limitations of vision systems using some of these sensors
• Some limitations: – Lack of metric scale
• The Dollhouse problem
– Pose stability
• Feature point robustness
– Power consumption
• Suitable for mobile products?
6/2/2014 5
6/2/2014 6
Accelerometer
Gyroscope
Magnetometer
Barometer
Proximity
Amb. Light sensor
GPS
WiFi
Bluetooth
GSM/CDMA Cell
NFC
Camera (front)
Touch screen Camera (back)
20 sensors!
Humidity Colorimeter
CO2/VOC gas Microphones x 3
Fingerprint Thermal ambient
• What are they? – MEMS are tiny silicon structures
– MicroElectroMechanical Systems
– Leveraging semiconductor toolsets Bosch
6/2/2014 8
• Measures dynamic acceleration – Very versatile.
– Result: Vibration, tilt, & position
– Low power 1-10uA
6/2/2014 9
Hydrogen atom = 1Å
MEMS displacement Resolution ~ 0.1Å
• Used to measure rotation
• Absolute orientation reference for gyroscope
• Power is moderate: 300-1500uA
6/2/2014 10
• Gyros don’t measure angle! – They measure the rate of change
– Body rates: rotation about each axis
• Rates are relative to starting point – Depend on Accel/Mag for start
• Integrate to get angle
• Power is high: 1-5mA or more
Gyro SEM 𝜃= 𝜔 𝑡 𝑑𝑡 + 𝜃0
6/2/2014 11
• Measures air pressure
• Air pressure indicates altitude
• Not good for absolute
• Resolution of 1-2 feet
• Low power: 1-5uA
Melexis.com
6/2/2014 12
• We want to know Position and Attitude (pose). – Inertial and Vision systems can each help find this
6/2/2014 14
𝑃𝑜𝑠𝑒 = 𝑝 , 𝑞 = 𝑥, 𝑦, 𝑧, 𝑞0, 𝑞1, 𝑞2, 𝑞3,
• Position seems easy: double integrate
• Angle is only a single integration.
• No problem!
6/2/2014 15
𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 = 𝑣 𝑡 𝑑𝑡 + 𝑝0
𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦 = 𝑎 𝑡 𝑑𝑡 + 𝑣0
𝜃= 𝜔 𝑡 𝑑𝑡 + 𝜃0
• Noise Random walk – Integrating noise causes a linear walk.
6/2/2014 Sensor Platforms Proprietary and Confidential Information 16
Measured Acc error [m/s/s] for an accelerometer when sitting still.
• Noise Random walk – Integrating noise causes a linear walk.
6/2/2014 Sensor Platforms Proprietary and Confidential Information 17
dttatv )()(
Measured)( ta
6/2/2014 Sensor Platforms Proprietary and Confidential Information 18
dttvtp )()(
dttatv )()(
Measured)( ta
• Dead Reckoning – Over the past few years
significant progress has been made
– Stable solutions with consumer grade sensors
– Graph (right) uses stock sensors on Galaxy S3
– Pedestrian walking constraints aid solution
6/2/2014 20
Waypoints Measured Path
• Visual odometry – Visually tracking position (camera pose) through a space
• Tracking feature points is a powerful way to understand the world
6/2/2014 21
• Limitations – Can't tell size of objects: i.e. scale
• Doll house problem
– Hard to map points between frames over time
– Need cohesion over long time scales
– Need robustness in dark spaces & low-texture surfaces
– Need maintain vision lock (can get lost due to motion-blur)
– Enormous computational load (ready for mobile?!?)
• Can we aid the solution with more sensors?
6/2/2014 22
• Attitude estimates allow anticipated search space – Reduce computation for FP correspondence
• Power reduction with reduced/opportunistic frame rates – Can trust INS when not moving
– Or when spatial diversity is low
• Vision System can be turned on only when high resolution navigation/alignment is needed – PDR to Statue, VS for precise AR overlay
6/2/2014 23
• Need to find metric scale – Without it the world makes no sense
• Monocular / Binocular issues
6/2/2014 24
If you were a vision system, which one is real?
• Need some way to get extra scale information
– Binocular cameras (like humans do; monocular is cheaper)
– Reference object (hard to keep in sight; i.e. Ikea catalog)
– Location estimates (GPS is not available indoors)
– Mapped landmark (best but hard; humans do this)
– Inertial estimates (tend to drift, but commonly available)
– Depth cameras
6/2/2014 25
• State estimation from visual & inertial sources – Combined measurements & physical models
6/2/2014 26
],,,,ˆ,[ gak bbvqpx
• State estimation from visual & inertial sources
• Kalman filter – Recursive linear quadratic estimator
– Combined measurements & physical models
6/2/2014 27
],,,,ˆ,[ gak bbvqpx
– Closely coupled KF – Solve it all at once
• Computationally expensive (Order n2 or n3)
– Loosely coupled KF
• Estimate visual delta-pose
• Estimate inertial delta-pose
• Combine with KF
• But loose cross-correlations
6/2/2014 28
– Recursive vs Batch solution
• Kalman Filters are “recursive”; only one frame deep
– Batch solution:
• Compute solution across multiple frames
• Bundle Adjustment with well selected keyframes
• Much more stable, but computationally expensive
• Asynchronous to frame updates; non-uniform keyframes
6/2/2014 30
– Consumer sensors are cheap
• Free: they are already in place
• Contextual & Motional
– Need additional constraints
• Contextual constraints
– Are you moving?
» Easy for robots
» Harder for humans
» Not only for robots anymore
• Motion constraints
– Wheel constraints help (only on flat ground with no slippage)
– Pedestrian constraints (track steps, distance & direction)
6/2/2014 31
Demonstration 1: Ideal Vision & PDR Scenario
● Vision-only: 5%, PDR-only: 3%, Fused: 1.5%
● Use case: head-mounted AR, vision mapping
Initial scale estimate
Sensor Platforms, Inc. Confidential and Proprietary
32
End Start
Demonstration 2: Non-Ideal Vision/Ideal PDR
● Vision-only: 15%, PDR-only: 2%, Fused: 1%
● Use case: AR over large spaces
Vision Outage Initial scale estimate
Vision Outage
Sensor Platforms, Inc. Confidential and Proprietary
33
Start
End
Demonstration 3: Non-Ideal Vision/Non-Ideal PDR
● Vision-only: 8%, PDR-only: 15%, Fused: 5%
● Use case: intensive gaming
Initial scale estimate
PDR Outage
Sensor Platforms, Inc. Confidential and Proprietary
34
Start End
• Optical Image Stabilization (OIS) – Optical (in-lens) with inertial attitude tracking; gyro based
• Super resolution – Across multiple frames stabilized with pose tracking
• Deblurring – Within frame pose-detection and deconvolution
6/2/2014 36
– Across multiple frames stabilized with pose tracking
– Inertial data stabilizes the solution
6/2/2014 37