Post on 24-Feb-2016
description
776 Computer Vision
Jan-Michael Frahm, Enrique DunnSpring 2012
From Previous Lecture• Homographies• Fundamental matrix• Normalized 8-point Algorithm• Essential Matrix
Plane Homography for Calibrated Cameras
• In the calibrated case o Two cameras P=K[I |0] and P’ = K’[R | t]o A plane π=(nT,d) T
• The homography is given by x’=Hx
H = K’(R – tnT/d)K-1
• For the plane at infinityH = K’RK-1
The Fundamental Matrix F
P0m0
L
l1
M
m1
M
P1
Hm0
Epipole e1
F = [e]xH = Fundamental Matrix
1 1 0Tm I 1 0I Fm 1 0 0Tm Fm
1 0Te F
The eight-point algorithmx = (u, v, 1)T, x’ = (u’, v’, 1)T
Minimize:
under the constraintF33
= 1
2
1
)( i
N
i
Ti xFx
Essential Matrix(Longuet-Higgins, 1981)
Epipolar constraint: Calibrated case
0)]([ xRtx RtExExT ][with0
X
x x’
The vectors x, t, and Rx’ are coplanar slide: S. Lazebnik
Essential Matrix
Epipolar constraint: Calibrated case
0)]([ xRtx RtExExT ][with0
X
x x’
The vectors x, t, and Rx’ are coplanar slide: S. Lazebnik
1det( ) 0, ( ) 02
T TE EE E trace EE E 'FKKE T
Cubic constraint
Today: Binocular stereo• Given a calibrated binocular stereo pair, fuse it to
produce a depth image
Where does the depth information come from?
Binocular stereo• Given a calibrated binocular stereo pair, fuse it to
produce a depth imageo Humans can do it
Stereograms: Invented by Sir Charles Wheatstone, 1838
Binocular stereo• Given a calibrated binocular stereo pair, fuse it to
produce a depth imageo Humans can do it
Autostereograms: www.magiceye.com
Binocular stereo• Given a calibrated binocular stereo pair, fuse it to
produce a depth imageo Humans can do it
Autostereograms: www.magiceye.com
Real-time stereo
• Used for robot navigation (and other tasks)o Software-based real-time stereo techniques
Nomad robot searches for meteorites in Antarticahttp://www.frc.ri.cmu.edu/projects/meteorobot/index.html
slide: R. Szeliski
Stereo image pair
slide: R. Szeliski
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923
Anaglyphs
http://www.rainbowsymphony.com/freestuff.html
(Wikipedia for images)
slide: R. Szeliski
Stereo: epipolar geometry
• Match features along epipolar lines
viewing ray
epipolar plane
epipolar line
slide: R. Szeliski
Simplest Case: Parallel images
• Image planes of cameras are parallel to each other and to the baseline
• Camera centers are at same height
• Focal lengths are the same
slide: S. Lazebnik
Simplest Case: Parallel images
• Image planes of cameras are parallel to each other and to the baseline
• Camera centers are at same height
• Focal lengths are the same
• Then, epipolar lines fall along the horizontal scan lines of the images
slide: S. Lazebnik
Essential matrix for parallel images
RtExExT ][,0
0000
000][
TTRtE
R = I t = (T, 0, 0)
Epipolar constraint:
00
0][
xy
xz
yz
aaaa
aaa
t
x
x’
Essential matrix for parallel images
RtExExT ][,0
0000
000][
TTRtE
Epipolar constraint:
vTTvvTTvuv
u
TTvu
0
010
10000
0001
R = I t = (T, 0, 0)
The y-coordinates of corresponding points are the same!
t
x
x’
Depth from disparity
f
x x’
BaselineB
z
O O’
X
f
zfBxxdisparity
Disparity is inversely proportional to depth!
Depth Sampling Depth sampling for integer pixel disparity
Quadratic precision loss with depth!
Depth Sampling Depth sampling for wider baseline
Depth Sampling Depth sampling is in O(resolution6)
Stereo: epipolar geometry• for two images (or images with collinear camera
centers), can find epipolar lines• epipolar lines are the projection of the pencil of
planes passing through the centers
• Rectification: warping the input images (perspective transformation) so that epipolar lines are horizontal
slide: R. Szeliski
Rectification• Project each image onto same plane, which is
parallel to the epipole• Resample lines (and shear/stretch) to place lines
in correspondence, and minimize distortion
• [Loop and Zhang, CVPR’99]
slide: R. Szeliski
Rectification
BAD!
slide: R. Szeliski
Rectification
GOOD!
slide: R. Szeliski
Problem: Rectification for forward moving
cameras
• Required image can become very large (infinitely large) when the epipole is in the image
• Alternative rectifications are available using epipolar lines directly in the imageso Pollefeys et al. 1999, “A simple and efficient method for general
motion”, ICCV
Your basic stereo algorithm
For each epipolar lineFor each pixel in the left image
• compare with every pixel on same epipolar line in right image• pick pixel with minimum match cost
Improvement: match windows• This should look familar...
slide: R. Szeliski
Finding correspondences
• apply feature matching criterion (e.g., correlation or Lucas-Kanade) at all pixels simultaneously
• search only over epipolar lines (many fewer candidate positions)
slide: R. Szeliski
Matching cost
disparity
Left Right
scanline
Correspondence search
• Slide a window along the right scanline and compare contents of that window with the reference window in the left image
• Matching cost: SSD or normalized correlation
slide: S. Lazebnik
Left Right
scanline
Correspondence search
SSDslide: S. Lazebnik
Left Right
scanline
Correspondence search
Norm. corrslide: S. Lazebnik
Neighborhood size• Smaller neighborhood: more details• Larger neighborhood: fewer isolated mistakes
• w = 3 w = 20
slide: R. Szeliski
Matching criteria• Raw pixel values (correlation)• Band-pass filtered images [Jones & Malik 92]• “Corner” like features [Zhang, …]• Edges [many people…]• Gradients [Seitz 89; Scharstein 94]• Rank statistics [Zabih & Woodfill 94]• Intervals [Birchfield and Tomasi 96]• Overview of matching metrics and their
performance:o H. Hirschmüller and D. Scharstein, “Evaluation of Stereo Matching
Costs on Images with Radiometric Differences”, PAMI 2008
slide: R. Szeliski
Adaptive Weighting• Boundary Preserving• More Costly
Failures of correspondence search
Textureless surfaces Occlusions, repetition
Non-Lambertian surfaces, specularitiesslide: S. Lazebnik
Stereo: certainty modeling• Compute certainty map from correlations
• input depth map certainty map
slide: R. Szeliski
Results with window search
Window-based matching Ground truth
Data
slide: S. Lazebnik
Better methods exist...
Graph cuts Ground truth
For the latest and greatest: http://www.middlebury.edu/stereo/
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001
slide: S. Lazebnik
How can we improve window-based
matching?• The similarity constraint is local (each reference
window is matched independently)• Need to enforce non-local correspondence
constraints
slide: S. Lazebnik
Non-local constraints• Uniqueness
o For any point in one image, there should be at most one matching point in the other image
slide: S. Lazebnik
Non-local constraints• Uniqueness
o For any point in one image, there should be at most one matching point in the other image
• Orderingo Corresponding points should be in the same order in both views
slide: S. Lazebnik
Non-local constraints• Uniqueness
o For any point in one image, there should be at most one matching point in the other image
• Orderingo Corresponding points should be in the same order in both views
Ordering constraint doesn’t holdslide: S. Lazebnik
Non-local constraints• Uniqueness
o For any point in one image, there should be at most one matching point in the other image
• Orderingo Corresponding points should be in the same order in both views
• Smoothnesso We expect disparity values to change slowly (for the most part)
slide: S. Lazebnik
Scanline stereo• Try to coherently match pixels on the entire
scanline• Different scanlines are still optimized
independentlyLeft image Right image
slide: S. Lazebnik
“Shortest paths” for scan-line stereo
Left image
Right image
Can be implemented with dynamic programmingOhta & Kanade ’85, Cox et al. ‘96
leftS
rightS
correspondence
q
p
Left
occlu
sion
t
Rightocclusion
s
occlC
II
corrC
Slide credit: Y. Boykov
Coherent stereo on 2D grid• Scanline stereo generates streaking artifacts
• Can’t use dynamic programming to find spatially coherent disparities/ correspondences on a 2D grid
slide: S. Lazebnik
Stereo matching as energy minimizationI1 I2 D
• Energy functions of this form can be minimized using graph cuts
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001
W1(i ) W2(i+D(i )) D(i )
jii
jDiDiDiWiWDE,neighbors
221 )()())(()()(
data term smoothness term
slide: S. Lazebnik
Active stereo with structured light
• Project “structured” light patterns onto the objecto Simplifies the correspondence problemo Allows us to use only one camera
camera
projector
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002
slide: S. Lazebnik
Active stereo with structured light
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002
slide: S. Lazebnik
Active stereo with structured light
http://en.wikipedia.org/wiki/Structured-light_3D_scannerslide: S. Lazebnik
Kinect: Structured infrared light
http://bbzippo.wordpress.com/2010/11/28/kinect-in-infrared/ slide: S. Lazebnik
Laser scanning
• Optical triangulationo Project a single stripe of laser lighto Scan it across the surface of the objecto This is a very precise version of structured light scanning
Digital Michelangelo ProjectLevoy et al.
http://graphics.stanford.edu/projects/mich/
Source: S. Seitz
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
Source: S. Seitz
Laser scanned models
The Digital Michelangelo Project, Levoy et al. Source: S. Seitz
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
Source: S. Seitz
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
Source: S. Seitz
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
Source: S. Seitz
1.0 mm resolution (56 million triangles)
Aligning range images• A single range scan is not sufficient to describe a
complex surface• Need techniques to register multiple range
images
B. Curless and M. Levoy, A Volumetric Method for Building Complex Models from Range Images, SIGGRAPH 1996
Aligning range images• A single range scan is not sufficient to describe a
complex surface• Need techniques to register multiple range
images
• … which brings us to multi-view stereo