Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf ·...
Transcript of Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf ·...
![Page 2: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/2.jpg)
4/8/2014 [email protected] 2
![Page 3: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/3.jpg)
4/8/2014 [email protected] 3
![Page 4: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/4.jpg)
4/8/2014 [email protected] 4
![Page 5: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/5.jpg)
4/8/2014 [email protected] 5
![Page 8: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/8.jpg)
Image Composite Editor
4/8/2014 [email protected] 8
http://research.microsoft.com/en-us/um/
redmond/groups/ivm/ice/
![Page 9: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/9.jpg)
Deep Zoom & HD View
4/8/2014 [email protected] 9
http://research.microsoft.com/en-
us/um/redmond/groups/ivm/HDView/
![Page 13: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/13.jpg)
Homogeneous coordinates
4/8/2014 [email protected] 13
0 cbyax Ta,b,c
0,0)()( kkcykbxka TTa,b,cka,b,c ~
Homogeneous representation of lines
equivalence class of vectors, any vector is representative
Set of all equivalence classes in R3(0,0,0)T forms P2
Homogeneous representation of points
0 cbyax Ta,b,cl x , ,1x yTon if and only if
0l 11 x,y,a,b,cx,y,T 0,1,,~1,, kyxkyx
TT
The point x lies on the line l if and only if xTl=lTx=0
Homogeneous coordinates
Inhomogeneous coordinates Tyx,
, ,x y zT
but only 2DOF
![Page 14: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/14.jpg)
Points from lines and vice-versa
4/8/2014 [email protected] 14
l'lx
Intersections of lines
The intersection of two lines and is l l'
Line joining two points
The line through two points and is x'xl x x'
Example
1x
1y
![Page 15: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/15.jpg)
Ideal points and the line at infinity
4/8/2014 [email protected] 15
T0,,l'l ab
Intersections of parallel lines
TTand ',,l' ,,l cbacba
Example
1x 2x
Ideal points T0,, 21 xx
Line at infinity T1,0,0l
l22RP
tangent vector
normal direction
ab ,
ba,
Note that in P2 there is no distinction
between ideal points and others
![Page 16: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/16.jpg)
A model for the projective plane
4/8/2014 [email protected] 16
exactly one line through two points
exactly one point at intersection of two lines
![Page 17: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/17.jpg)
Duality
4/8/2014 [email protected] 17
x l
0xl T0lx T
l'lx x'xl
Duality principle:
To any theorem of 2-dimensional projective geometry
there corresponds a dual theorem, which may be
derived by interchanging the role of points and lines in
the original theorem
![Page 19: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/19.jpg)
Projective transformations
4/8/2014 [email protected] 19
A projectivity is an invertible mapping h from P2 to itself
such that three points x1,x2,x3 lie on the same line if and
only if h(x1),h(x2),h(x3) do.
Definition:
A mapping h:P2P2 is a projectivity if and only if there
exist a non-singular 3x3 matrix H such that for any point
in P2 reprented by a vector x it is true that h(x)=Hx
Theorem:
Definition: Projective transformation
3
2
1
333231
232221
131211
3
2
1
'
'
'
x
x
x
hhh
hhh
hhh
x
x
x
xx' Hor
8DOF
projectivity=collineation=projective transformation=homography
![Page 20: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/20.jpg)
Mapping between planes
4/8/2014 [email protected] 20
central projection may be expressed by x’=Hx
(application of theorem)
![Page 21: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/21.jpg)
Removing projective distortion
4/8/2014 [email protected] 21
333231
131211
3
1
'
''
hyhxh
hyhxh
x
xx
333231
232221
3
2
'
''
hyhxh
hyhxh
x
xy
131211333231' hyhxhhyhxhx
232221333231' hyhxhhyhxhy
select four points in a plane with know coordinates
(linear in hij)
(2 constraints/point, 8DOF 4 points needed)
![Page 22: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/22.jpg)
4/8/2014 [email protected] 22
![Page 24: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/24.jpg)
Transformation for lines
4/8/2014 [email protected] 24
Transformation for lines
ll' -TH
xx' HFor a point transformation
![Page 25: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/25.jpg)
Isometries
4/8/2014 [email protected] 25
1100
cossin
sincos
1
'
'
y
x
t
t
y
x
y
x
1
11
orientation preserving:
orientation reversing:
x0
xx'
1
tT
RHE IRR T
special cases: pure rotation, pure translation
3DOF (1 rotation, 2 translation)
Invariants: length, angle, area
![Page 26: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/26.jpg)
Similarities
4/8/2014 [email protected] 26
1100
cossin
sincos
1
'
'
y
x
tss
tss
y
x
y
x
x0
xx'
1
tT
RH
sS IRR T
also know as equi-form (shape preserving)
metric structure = structure up to similarity (in literature)
4DOF (1 scale, 1 rotation, 2 translation)
Invariants: ratios of length, angle, ratios of areas, parallel lines
![Page 27: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/27.jpg)
Affine transformations
4/8/2014 [email protected] 27
11001
'
'
2221
1211
y
x
taa
taa
y
x
y
x
x0
xx'
1
tT
AHA
non-isotropic scaling! (2DOF: scale ratio and orientation)
6DOF (2 scale, 2 rotation, 2 translation)
Invariants: parallel lines, ratios of parallel lengths, ratios of areas
DRRRA
2
1
0
0
D
![Page 28: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/28.jpg)
Projective transformations
4/8/2014 [email protected] 28
xv
xx'
vP T
tAH
Action non-homogeneous over the plane
8DOF (2 scale, 2 rotation, 2 translation, 2 line at infinity)
Invariants: cross-ratio of four points on a line (ratio of ratio)
T21,v vv
vv
sPAS TTTT v
t
v
0
10
0
10
t AIKRHHHH
Ttv RKA s
K 1det Kupper-triangular, decomposition unique (if chosen s>0)
![Page 29: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/29.jpg)
Overview Transformations
4/8/2014 [email protected] 29
100
2221
1211
y
x
taa
taa
100
2221
1211
y
x
tsrsr
tsrsr
333231
232221
131211
hhh
hhh
hhh
100
2221
1211
y
x
trr
trr
Projective
8dof
Affine
6dof
Similarity
4dof
Euclidean
3dof
Concurrency, collinearity,
order of contact (intersection,
tangency, inflection, etc.),
cross ratio
Parallellism, ratio of areas,
ratio of lengths on parallel
lines (e.g midpoints), linear
combinations of vectors
(centroids).
The line at infinity l∞
Ratios of lengths, angles.
The circular points I,J
lengths, areas.
![Page 30: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/30.jpg)
Line at infinity
4/8/2014 [email protected] 30
2211
2
1
2
1
0v
xvxv
x
x
x
x
v
AAT
t
000 2
1
2
1
x
x
x
x
v
AAT
t
Line at infinity becomes finite,
allows to observe vanishing points, horizon,
Line at infinity stays at infinity
![Page 31: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/31.jpg)
The line at infinity
4/8/2014 [email protected] 31
v1 v2
l1
l2 l4
l3
l∞
21 vvl
211 llv
432 llv
The line at infinity l is a fixed line under a projective transformation H if and only if H is an affinity
l
1
0
0
1t
0ll
A
AH
TT
A
![Page 32: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/32.jpg)
Affine properties from images
4/8/2014 [email protected] 32
projection rectification
APA
lll
HH
321
010
001
0,l 3321 llllT
![Page 34: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/34.jpg)
Projective 3D geometry
4/8/2014 [email protected] 34
vTv
tAProjective
15dof
Affine
12dof
Similarity
7dof
Euclidean
6dof
Intersection and tangency
Parallellism of planes,
Volume ratios, centroids,
The plane at infinity π∞
The absolute conic Ω∞
Volume
10
tAT
10
tRT
s
10
tRT
![Page 36: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/36.jpg)
Pinhole Camera Model
4/8/2014 [email protected] 36
TT ZfYZfXZYX )/,/(),,(
101
0
0
1
Z
Y
X
f
f
Z
fY
fX
Z
Y
X
101
01
01
1Z
Y
X
f
f
Z
fY
fX
![Page 37: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/37.jpg)
Pinhole Camera Model
4/8/2014 [email protected] 37
camX0|IKx
1
y
x
pf
pf
K
Principal point offset Calibration Matrix
![Page 39: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/39.jpg)
Camera rotation and translation
4/8/2014 [email protected] 39
C~
-X~
RX~
cam
X10
RCR
1
10
C~
RRXcam
Z
Y
X camX0|IKx
XC~
|IKRx
t|RKP C~
Rt
PXx
![Page 43: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/43.jpg)
4/8/2014 [email protected] 43
![Page 45: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/45.jpg)
4/8/2014 [email protected] 45
2 4 6
1 2 3
2 4 6
1 2 3
( )(1 )cos
( )(1 )sin
u x d x d d d
d y d d d
x c x c k r k r k r
y c k r k r k r
2 4 6
1 2 3
2 4 6
1 2 3
( )(1 )sin
( )(1 )cos
u y d x d d d
d y d d d
y c x c k r k r k r
y c k r k r k r
0 1 2
6 7 1
u up
u u
m x m y mx
m x m y
3 4 5
6 7 1
u up
u u
m x m y my
m x m y
![Page 46: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/46.jpg)
OpenCV
4/8/2014 [email protected] 46
http://opencv.willowgarage.com/documentation/camera_c
alibration_and_3d_reconstruction.html
![Page 47: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/47.jpg)
Camera Calibration Toolbox
4/8/2014 [email protected] 47
http://www.vision.caltech.edu/bouguetj/calib_doc/
![Page 50: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/50.jpg)
Robust Multi-camera Calibration
4/8/2014 [email protected] 50
http://graphics.stanford.edu/~vaibhav/projects/calib-
cs205/cs205.html
![Page 55: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/55.jpg)
Introduction
Computer vision is concerned with the theory behind artificial systems that
extract information from images. The image data can take many forms, such as
video sequences, views from multiple cameras. Computer vision is, in some
ways, the inverse of computer graphics. While computer graphics produces
image data from 3D models, computer vision often produces 3D models from
image data. Today one of the major problems in Computer vision is
correspondence search.
4/8/2014 55 [email protected]
![Page 56: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/56.jpg)
Epipolar Geometry
http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html
C,C’,x,x’ and X are coplanar
4/8/2014 56 [email protected]
![Page 59: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/59.jpg)
Epipolar Geometry
Family of planes p and lines l and l’
Intersection in e and e’
4/8/2014 59 [email protected]
![Page 60: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/60.jpg)
Epipolar Geometry epipoles e,e’
= intersection of baseline with image plane
= projection of projection center in other image
= vanishing point of camera motion direction
an epipolar plane = plane containing baseline (1-D family)
an epipolar line = intersection of epipolar plane with image
(always come in corresponding pairs)
4/8/2014 60 [email protected]
![Page 61: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/61.jpg)
The Fundamental Matrix F
4/8/2014 [email protected] 61
xHx' π
πl' e' x' e' H x Fx
H : projectivity=collineation=projective transformation=homography
![Page 64: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/64.jpg)
Three Questions
(i) Correspondence geometry: Given an image point x in the first
view, how does this constrain the position of the corresponding
point x’ in the second image?
(ii) Camera geometry (motion): Given a set of corresponding image
points {xi ↔x’i}, i=1,…,n, what are the cameras P and P’ for the two
views?
(iii) Scene geometry (structure): Given corresponding image points
xi ↔x’i and cameras P, P’, what is the position of (their pre-image)
X in space?
4/8/2014 64 [email protected]
![Page 65: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/65.jpg)
Parameter estimation
2D homography
Given a set of (xi,xi’), compute H (xi’=Hxi)
3D to 2D camera projection
Given a set of (Xi,xi), compute P (xi=PXi)
Fundamental matrix
Given a set of (xi,xi’), compute F (xi’TFxi=0)
4/8/2014 65 [email protected]
![Page 66: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/66.jpg)
The fundamental matrix F
4/8/2014 [email protected] 66
Algebraic Derivation
λCxPλX
PP'e'F
l' P'C P'P x
(note: doesn’t work for C=C’ F=0)
xP
λX IPP
πl' e' x' e' H x Fx
![Page 67: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/67.jpg)
The fundamental matrix F
4/8/2014 [email protected] 67
0Fxx'T
Correspondence Condition
The fundamental matrix satisfies the condition that for
any pair of corresponding points x↔x’ in the two images
![Page 69: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/69.jpg)
The Fundamental Matrix
4/8/2014 [email protected] 69
http://www.cs.unc.edu/~blloyd/comp290-089/fmatrix/
![Page 70: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/70.jpg)
Epipolar geometry: basic equation
4/8/2014 [email protected] 70
0Fxx'T
separate known from unknown
0'''''' 333231232221131211 fyfxffyyfyxfyfxyfxxfx
0,,,,,,,,1,,,',',',',','T
333231232221131211 fffffffffyxyyyxyxyxxx(data) (unknowns)
0Af
0f1''''''
1'''''' 111111111111
nnnnnnnnnnnn yxyyyxyxyxxx
yxyyyxyxyxxx
![Page 71: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/71.jpg)
the NOT normalized 8-point algorithm
4/8/2014 [email protected] 71
0
1´´´´´´
1´´´´´´
1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
f
f
f
f
f
f
f
f
f
yxyyyyxxxyxx
yxyyyyxxxyxx
yxyyyyxxxyxx
nnnnnnnnnnnn
~10000 ~10000 ~10000 ~10000 ~100 ~100 1 ~100 ~100
!
Orders of magnitude difference
Between column of data matrix
least-squares yields poor results
![Page 72: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/72.jpg)
the normalized 8-point algorithm
4/8/2014 [email protected] 72
Transform image to ~[-1,1]x[-1,1]
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1) (-1,1)
(-1,-1)
1
1500
2
10700
2
Least squares yields good results (Hartley, PAMI´97)
![Page 73: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/73.jpg)
The fundamental matrix F
4/8/2014 [email protected] 73
F is the unique 3x3 rank 2 matrix that satisfies x’TFx=0 for all x↔x’
(i) Transpose: if F is fundamental matrix for (P,P’),
then FT is fundamental matrix for (P’,P)
(ii) Epipolar lines: l’=Fx & l=FTx’
(iii) Epipoles: on all epipolar lines, thus e’TFx=0, x e’TF=0, similarly Fe=0
(iv) F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)
(v) F is a correlation, projective mapping from a point x to a line l’=Fx
(not a proper correlation, i.e. not invertible)
![Page 76: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/76.jpg)
Fundamental matrix for pure translation
4/8/2014 [email protected] 76
e'He'F RKKH 1
0101-00000
F T1,0,0e'
Example:
y'y 0Fxx'T
0]X|K[IPXx
ZxKt]|K[IXP'x'
-1
ZKt/xx'
ZX,Y,Z x/K)( -1T
motion starts at x and moves towards e, faster depending on Z
pure translation: F only 2 d.o.f., xT[e]xx=0 auto-epipolar
![Page 78: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/78.jpg)
Projective transformation and invariance
4/8/2014 [email protected] 78
-1-T FHH'F̂ x'H''x̂ Hx,x̂
Derivation based purely on projective concepts
X̂P̂XHPHPXx -1
F invariant to transformations of projective 3-space
X̂'P̂XHHP'XP'x' -1
FP'P,
P'P,F
unique
not unique
Canonical form
m]|[MP'0]|[IP
MmF
PP'e'F
![Page 79: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/79.jpg)
Projective ambiguity of cameras given F
4/8/2014 [email protected] 79
previous slide: at least projective ambiguity this slide: not more!
Show that if F is same for (P,P’) and (P,P’), there exists a
projective transformation H so that P=HP and P’=HP’ ~
]a~|A~
['P~ 0]|[IP
~ a]|[AP' 0]|[IP
A~
a~AaF
T1 avAA~
kaa~ k
lemma:
kaa~Fa~0AaaaF2rank
TavA-A~
k0A-A~
kaA~
a~Aa
kkIkH T1
1
v0
'P~
]a|av-A[v
0a]|[AHP' T1T1
1
kkkk
Ik(22-15=7, ok)
~
~ ~
![Page 80: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/80.jpg)
Canonical cameras given F
4/8/2014 [email protected] 80
F matrix corresponds to P,P’ iff P’TFP is skew-symmetric
X0,FPXP'X TT
F matrix, S skew-symmetric matrix
]e'|[SFP' 0]|[IP
000FSF
0Fe'0FSF0]|F[I]e'|[SF
TT
T
TTT
(fund.matrix=F)
Possible choice:
]e'|F][[e'P' 0]|[IP
Canonical representation:
]λe'|ve'F][[e'P' 0]|[IP T
![Page 81: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/81.jpg)
The Essential matrix
4/8/2014 [email protected] 81
~ Fundamental matrix for calibrated cameras (remove K)
t]R[RRtE T
0x̂E'x̂ T
FKK'E T
x'K'x̂ x;Kx̂ -1-1
5 d.o.f. (3 for R; 2 for t up to scale)
E is essential matrix if and only if two singularvalues are equal (and third=0)
T0)VUdiag(1,1,E
![Page 83: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/83.jpg)
Epipolar Geometry
4/8/2014 [email protected] 83
C1
C2
l2
P
l1 e
1
e2
0m m 1
T
2 F
Fundamental matrix
(3x3 rank 2 matrix)
1. Computable from corresponding points
2. Simplifies matching
3. Allows to detect wrong matches
4. Related to calibration
Underlying structure in set of
matches for rigid scenes
l2
C1 m1
L1
m2
L2
M
C2
m1
m2
C1
C2
l2
P
l1 e
1
e2
m1
L1
m2
L2
M l2 lT1
Canonical representation:
]λe'|ve'F][[e'P' 0]|[IP T
![Page 84: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/84.jpg)
3D reconstruction of cameras and structure
4/8/2014 [email protected] 84
given xi↔x‘i , compute P,P‘ and Xi
Reconstruction Problem:
ii PXx ii XPx for all i
without additional informastion possible up to projective ambiguity
![Page 85: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/85.jpg)
Outline of 3D Reconstruction
4/8/2014 [email protected] 85
(i) Compute F from correspondences
(ii) Compute camera matrices P,P‘ from F
(iii) Compute 3D point for each pair of corresponding points
computation of F
use x‘iFxi=0 equations, linear in coeff. F
8 points (linear), 7 points (non-linear), 8+ (least-squares)
computation of camera matrices
use ]λe'|ve'F][[e'P' 0]|[IP T
triangulation
compute intersection of two backprojected rays
![Page 86: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/86.jpg)
Reconstruction ambiguity: similarity
4/8/2014 [email protected] 86
iii XHPHPXx S
-1
S
λt]t'RR'|K[RR'λ0
t'R'-R' t]|K[RPH TTTT
1-
S
![Page 88: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/88.jpg)
Terminology
4/8/2014 [email protected] 88
xi↔x‘i
Original scene Xi
Projective, affine, similarity reconstruction
= reconstruction that is identical to original up to
projective, affine, similarity transformation
Literature: Metric and Euclidean reconstruction
= similarity reconstruction
![Page 89: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/89.jpg)
The projective reconstruction theorem
4/8/2014 [email protected] 89
If a set of point correspondences in two views determine the
fundamental matrix uniquely, then the scene and cameras may be
reconstructed from these correspondences alone, and any two such
reconstructions from these correspondences are projectively equivalent
i111 X,'P,P i222 X,'P,Pii xx
-1
12 HPP -1
12 HPP 12 HXX 0FxFx :except ii
theorem from last class
iiiii 22111
-1
112 XPxXPHXHPHXP
along same ray of P2, idem for P‘2
two possibilities: X2i=HX1i, or points along baseline
key result : allows reconstruction from pair of uncalibrated images
![Page 90: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/90.jpg)
4/8/2014 [email protected] 90
![Page 91: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/91.jpg)
Stratified reconstruction
4/8/2014 [email protected] 91
(i) Projective reconstruction
(ii) Affine reconstruction
(iii) Metric reconstruction
![Page 93: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/93.jpg)
Projective to affine
4/8/2014 [email protected] 93
iX,P'P,
TT1,0,0,0,,,π DCBA
TT 1,0,0,0πH-
π0|I
H (if D≠0)
theorem says up to projective transformation,
but projective with fixed p∞ is affine transformation
can be sufficient depending on application,
e.g. mid-point, centroid, parallellism
![Page 94: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/94.jpg)
Translational motion
4/8/2014 [email protected] 94
points at infinity are fixed for a pure translation
reconstruction of xi↔ xi is on p∞
]e'[]e[F 0]|[IP
]e'|[IP
![Page 95: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/95.jpg)
Scene constraints
4/8/2014 [email protected] 95
Parallel lines
parallel lines intersect at infinity
reconstruction of corresponding vanishing point yields
point on plane at infinity
3 sets of parallel lines allow to uniquely determine p∞
remark: in presence of noise determining the intersection of
parallel lines is a delicate problem
remark: obtaining vanishing point in one image can be sufficient
![Page 97: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/97.jpg)
Scene constraints
4/8/2014 [email protected] 97
Distance ratios on a line
known distance ratio along a line allow to determine point at
infinity (same as 2D case)
![Page 98: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/98.jpg)
The infinity homography
4/8/2014 [email protected] 98
∞
∞
m]|[MP ]m'|[M'P'
-1MM'H
T0,X~
X
X~
Mx X~
M'x'
m]Ma|[MA10aA
m]|[MP
-1-1MAAM'H
unchanged under affine transformations
0]|[IP e]|[HP
affine reconstruction
![Page 99: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/99.jpg)
One of the cameras is affine
4/8/2014 [email protected] 99
According to the definition,
the principal plane of an affine camera is at infinity
to obtain affine recontruction,
compute H that maps third row of P to (0,0,0,1)T
and apply to cameras and reconstruction
e.g. if P=[I|0], swap 3rd and 4th row, i.e.
0100100000100001
H
![Page 100: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/100.jpg)
Affine to metric
4/8/2014 [email protected] 100
identify absolute conic
transform so that on π ,0: 222 ZYX
then projective transformation relating original and
reconstruction is a similarity transformation
in practice, find image of W∞
image w∞ back-projects to cone that intersects p∞ in W∞
*
*
note that image is independent
of particular reconstruction
![Page 101: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/101.jpg)
Affine to metric
4/8/2014 [email protected] 101
m]|[MP ω
100AH
-1
given
possible transformation from affine to metric is
1TT ωMMAA
(cholesky factorisation)
m]|[MAPHP -1
M
proof:
TTT
MM
* MMAAMMω
T-T-1-1 AAMωM
000I*
![Page 102: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/102.jpg)
Orthogonality
4/8/2014 [email protected] 102
0ωvv 2
T
1
ωvl
vanishing points corresponding to orthogonal directions
vanishing line and vanishing point corresponding
to plane and normal direction
![Page 105: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/105.jpg)
Feature Matching
To match the points in one image to the points in the other
image by exhaustive search(to match one point in one image to
all the points in the other image) is a difficult and long process
so some constraints are applied. As geometric constraint to
minimize the search area for correspondence.
The geometric constraints is provided by the epipolar geometry
4/8/2014 105 [email protected]
![Page 106: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/106.jpg)
Harris Detector: Mathematics
Change of intensity for the shift [x,y]:
Intensity Shifted
intensity
Window function
or Window function w(u,v) =
Gaussian 1 in window, 0 outside
2
,
( , ) ( , )[ ( , ) ( , )]u v
E x y w u v I x u y v I x y
4/8/2014 106 [email protected]
![Page 107: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/107.jpg)
2
2
2
2
( , ) [ ( , ) ( , )] u x,v y
( , ) ( , ) [ ( , ) ( , )]
[ ( , ) ( , )]
( ( , )) ( , ) ( , )
(
w
x y
w
x y
w
x x y
w w
x
E x y I x y I x x y y
xI x y I x y I x y I x y
y
xI x y I x y
y
I x y I x y I x y
x yI x
2, ) ( , ) ( ( , ))
( , )
y y
w w
x
yy I x y I x y
xx y C x y
y
4/8/2014 107 [email protected]
![Page 108: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/108.jpg)
Harris Detector: Mathematics
( , ) ,u
E u v u v Mv
For small shifts [u,v] we have a bilinear approximation:
2
2,
( , )x x y
x y x y y
I I IM w x y
I I I
where M is a 22 matrix computed from image derivatives:
4/8/2014 108 [email protected]
![Page 109: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/109.jpg)
Harris Detector: Mathematics
( , ) ,u
E u v u v Mv
Intensity change in shifting window: eigenvalue analysis
1, 2 – eigenvalues of M
(max)-1/2
(min)-1/2
Ellipse E(u,v) = const
If we try every possible orientation n,
the max. change in intensity is 2
4/8/2014 109 [email protected]
![Page 110: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/110.jpg)
Harris Detector: Mathematics
1
2
“Corner”
1 and 2 are large,
1 ~ 2;
E increases in all
directions
1 and 2 are small;
E is almost constant
in all directions
“Edge”
1 >> 2
“Edge”
2 >> 1
“Flat”
region
Classification of image
points using
eigenvalues of M:
4/8/2014 110 [email protected]
![Page 111: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/111.jpg)
Harris Detector: Mathematics
Measure of corner response:
2
det traceR M k M
1 2
1 2
det
trace
M
M
(k – empirical constant, k = 0.04-0.06)
4/8/2014 111 [email protected]
![Page 112: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/112.jpg)
Harris Detector: Mathematics
1
2 “Corner”
“Edge”
“Edge”
“Flat”
• R depends only on eigenvalues
of M
• R is large for a corner
• R is negative with large
magnitude for an edge
• |R| is small for a flat region
R > 0
R < 0
R < 0 |R| small
4/8/2014 112 [email protected]
![Page 113: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/113.jpg)
Harris Detector
The Algorithm:
Find points with large corner response function R
Take the points of local maxima of R
4/8/2014 113 [email protected]
![Page 116: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/116.jpg)
Find points with large corner response: R>threshold
Harris Detector : Workflow
4/8/2014 116 [email protected]
![Page 119: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/119.jpg)
Correlation for Correspondence Search
j
i
0,0
7,7
Left image: 1. From the left feature point image we select one feature point.
2. Draw the window(N*N) around, with feature point in the center.
3. Calculate the normalized window using the formula
1
1)(1
1 1
1 11
),(),(
N ),(*),(
s
jiwjiw
windowofsizetheisjiwjiws
nor
N
i
N
j
4/8/2014 119 [email protected]
![Page 120: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/120.jpg)
Correlation Algorithm Select one feature point from the first image.
Draw the window across it(7*7).
Normalize the window using the given formula.
Find the feature in the right image, that are to be considered in the first image,(this should be done by
some distance thresholding)
After finding the feature point the window of same dimension should be selected in the second image.
The normalized correlstion measure should be calculated using the formula
1 1 1
1 1
11( )
1
( , )* ( , ) N
( , )( , )
N N
i j
nor
s w i j w i j is the size of window
w i jw i j
s
2 1 2
1 1
( , )* ( , )N N
i j
s w i j w i j
4/8/2014 120 [email protected]
2
2 2
1 1
( , )* ( , )N N
i j
scormat
w i j w i j
![Page 123: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/123.jpg)
RANSAC • Select sample of m points at
random
• Calculate model
parameters that fit the
data in the sample
4/8/2014 123 [email protected]
![Page 124: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/124.jpg)
RANSAC • Select sample of m points at
random
• Calculate model parameters
that fit the data in the sample
• Calculate error function
for each data point
4/8/2014 124 [email protected]
![Page 125: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/125.jpg)
RANSAC • Select sample of m points at
random
• Calculate model parameters
that fit the data in the sample
• Calculate error function for
each data point
• Select data that support
current hypothesis
4/8/2014 125 [email protected]
![Page 126: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/126.jpg)
RANSAC • Select sample of m points at
random
• Calculate model parameters
that fit the data in the sample
• Calculate error function for
each data point
• Select data that support
current hypothesis
• Repeat sampling
4/8/2014 126 [email protected]
![Page 127: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/127.jpg)
RANSAC • Select sample of m points at
random
• Calculate model parameters
that fit the data in the sample
• Calculate error function for
each data point
• Select data that support
current hypothesis
• Repeat sampling
4/8/2014 127 [email protected]
![Page 128: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/128.jpg)
RANSAC
RANSAC time complexity
k … number of samples drawn
N … number of data points
tM … time to compute a single
model
mS … average number of models
per sample
ALL-INLIER SAMPLE
4/8/2014 128 [email protected]
![Page 129: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/129.jpg)
Feature-space outliner rejection
Can we now compute H from the blue points?
No! Still too many outliers…
What can we do?
![Page 130: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/130.jpg)
Matching features
What do we do about the “bad” matches?
![Page 131: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/131.jpg)
RAndom SAmple Consensus
Select one match, count inliers
![Page 132: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/132.jpg)
RAndom SAmple Consensus
Select one match, count inliers
![Page 133: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/133.jpg)
Least squares fit
Find “average” translation vector
![Page 134: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/134.jpg)
RANSAC for estimating homography
RANSAC loop:
1. Select four feature pairs (at random)
2. Compute homography H (exact)
3. Compute inliers where SSD(pi’, H pi) < ε
4. Keep largest set of inliers
5. Re-compute least-squares H estimate on all of the
inliers
![Page 135: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/135.jpg)
RANSAC for Fundamental Matrix
135
Step 1. Extract features
Step 2. Compute a set of potential matches
Step 3. do
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 determine inliers
until (#inliers,#samples)<95%
Step 4. Compute F based on all inliers
Step 5. Look for additional matches
Step 6. Refine F based on all correct matches
(generate
hypothesis)
(verify hypothesis)
4/8/2014 [email protected]
![Page 138: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/138.jpg)
Example: Mosaicking with homographies
www.cs.cmu.edu/~dellaert/mosaicking
4/8/2014 138 [email protected]
![Page 142: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/142.jpg)
4/8/2014 [email protected] 142
9/28 10/5 10/12 10/19
Liu Chang photosynth create account
Take pictures
Construct
photosynth
Search others
photosynth works
Select one point in
DSU, take pictures
Markus Photomodeller Select one small
object
Make 3d model
data
Projector
Price?
Yang Yun PhotoCity Select one small
object
Make 3d model
data
Wang Ping PhotoCity Presentation file
How to install &
use
Cygwin, bunder,
…
Matlab CC
Matlab CC Matlab CC
Lab Image Data
SIFT
Fundamental
Matrix
Liu Pengxin Remove
Perspective
distortion
Load image
Choose feature
points
SIFT & ASIFT
OpenCV CC
OpenCV CC Remove PT
OpenCV CC PT
Zhu Zi Jian PhotoTourism
ARToolkit Camera
Calibration
PhotoTourism
ARToolkit Camera
Calibration
Fundamental
Matrix
Wang
Yuo
PhotoTourism
M C C – led lamp
![Page 145: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/145.jpg)
4/8/2014 [email protected] 145
![Page 146: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/146.jpg)
4/8/2014 [email protected] 146
![Page 147: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/147.jpg)
2014-04-08 147
![Page 148: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/148.jpg)
2014-04-08 148
![Page 151: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/151.jpg)
2014-04-08 151
![Page 153: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/153.jpg)
Total Immersion - D’Fusion Pro
4/8/2014 [email protected] 153
http://www.t-immersion.com/products/dfusion-
suite/dfusion-pro
![Page 157: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/157.jpg)
Scenarios
4/8/2014 [email protected] 157
Clear Vision - Denoising
Motion Deblurring
SuperResolution
Panoramic View
Background Modeling
![Page 161: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/161.jpg)
Projective transformations
4/8/2014 [email protected] 161
A projectivity is an invertible mapping h from P2 to itself
such that three points x1,x2,x3 lie on the same line if and
only if h(x1),h(x2),h(x3) do.
Definition:
A mapping h:P2P2 is a projectivity if and only if there
exist a non-singular 3x3 matrix H such that for any point
in P2 reprented by a vector x it is true that h(x)=Hx
Theorem:
Definition: Projective transformation
3
2
1
333231
232221
131211
3
2
1
'
'
'
x
x
x
hhh
hhh
hhh
x
x
x
xx' Hor
8DOF
projectivity=collineation=projective transformation=homography
![Page 162: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/162.jpg)
Mapping between planes
4/8/2014 [email protected] 162
central projection may be expressed by x’=Hx
(application of theorem)
![Page 163: Multiple View Geometry in Computer Visionkowon.dongseo.ac.kr/~lbg/cagd/MVG_bglee1207.pdf · 2016-05-25 · Projective transformations 19 lbg@dongseo.ac.kr 4/8/2014 A projectivity](https://reader030.fdocument.pub/reader030/viewer/2022040601/5e8f456b508e652d72511448/html5/thumbnails/163.jpg)
Removing projective distortion
4/8/2014 [email protected] 163
333231
131211
3
1
'
''
hyhxh
hyhxh
x
xx
333231
232221
3
2
'
''
hyhxh
hyhxh
x
xy
131211333231' hyhxhhyhxhx
232221333231' hyhxhhyhxhy
select four points in a plane with know coordinates
(linear in hij)
(2 constraints/point, 8DOF 4 points needed)