Institut für Elektrische Meßtechnik und Meßsignalverarbeitung … · 2017. 11. 7. · Institut...
Transcript of Institut für Elektrische Meßtechnik und Meßsignalverarbeitung … · 2017. 11. 7. · Institut...
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
1
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Image-based Measurement
• Human visual system
• Neurophysiology
• Color (perception, illumination, calibration, constancy)
• Image formation (CCD, CMOS sensors)
• Image acquisition• Projective geometry
• Camera models, camera calibration
• Fundamental algorithms• Salient point detection and description
• Linear algebra (RQ, SVD), Estimating H, P, F, Practical issues, Camera Pose
• Specific algorithms in detail• Camera calibration, F, PnP
• 3D scene structure (from stereo, motion, …), 4D Multibody structure and motion
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
2
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Calibration Issues
• Linear Models– Homography estimation H
– Epipolar geometry F, E
– Interior camera parameters K
– Exterior camera parameters R,t
– Camera pose R,t
• Algorithms– Interest Point Detection + Description
– Overdetermined systems of linear equations Error Minimization
– Direct Linear Transform – DLT
– Normalization
– Nonlinearities iterative error minimization, Levenberg-Marquardt
– Outliers Robustness, RANSAC
3
2
1
333231
232221
131211
3
2
1
'
'
'
'
x
x
x
hhh
hhh
hhh
x
x
x
x
x
H
MtMZ
Y
X
pppp
pppp
pppp
y
x
m
|
1
~
1 34333231
24232221
14131211
RKP
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
3
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Pinhole Camera
• “real” camera
image plane πi (x,y): Zcam = -f
x
y
Zcam
Xcam
Ycam
“principal”
point (x0,y0) “optical axis”
m(x,y)
M(Xcam,Ycam,Zcam)
f
“focal length” f
• 2D projection 3D scene
• m(x,y) ↔ line of sight = viewing direction
M’(Xcam’,Ycam’,Zcam’)
• “Pinhole” C … “center of projection”
Ccam
X Y
Z
M(X,Y,Z,1) M’(X’,Y’,Z’,1)
R,tm(x,y,1)
• “interior” camera parameters
– x0, y0, f, …
• “exterior” parameters
– camera pose
– R, t
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
4
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Pinhole Camera
image plane πi (x,y): Zcam = -f
x
y
Zcam
Xcam
Ycam“principal”
point (x0,y0) “optical axis”
f Ccam
X Y
Z
M(X,Y,Z,1)M’(X’,Y’,Z’,1)
R,tm(x,y,1)
MZ
Y
X
pppp
pppp
pppp
y
x
m
P
1
~
1 34333231
24232221
14131211P : 3 x 4 matrix
“camera projection matrix”
[Pollefeys p.24, eq. (3.8)]
Mm P~
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
5
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
The Basic Pinhole Model
Note: Figures taken from, notation according to [Hartley,Zisserman]
xZfYZfXZYXX TT ~)/,/(),,(~
~ … inhomog. coord.
XZ
Y
X
f
f
Z
fY
fX
xZ
Y
X
P
101
0
0
1
… homog. coord.
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
6
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
The Basic Pinhole Model
0~
|)1,,diag(
01
01
01
101
0
0
IP fff
f
f
f
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
7
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Principal Point Offset
camXZ
Y
X
yf
xf
Z
ZyfY
ZxfX
xZ
Y
X
P
101
0
0
1
0
0
0
0
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
8
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Principal Point Offset
camcam XXyf
xf
x
0~
|
01
01
01
1
0
0
IK
1
0
0
yf
xf
Kcamera calibration matrix
interior/internal parameters
interior/internal orientation
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
9
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Camera Rotation and Translation
CXX cam
~~~R
XC
X cam
10
~RR XCx
~| IKR
4 x 4
P3 P3
3 x 4
P2 P3
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
10
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Camera Rotation and Translation
C~
| IKRP 3 x 4 projection matrix P
9 degrees of freedom
3 “internal parameters” in K
3 rotation angles in R
3 translations in C~
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
11
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Camera Rotation and Translation
C~
| IKRP
Simplified notation: avoid explicit modeling of C
t|CttXXcam
RKPRR
~ ,
~~
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
12
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
• Pinhole– 3 parameters in K
• CCD– 4 parameters
• Finite projective camera– 5 parameters
– “skew” s
From Pinhole Real Cameras: K
1
0
0
yf
xf
K
yyxxy
x
fmfmy
x
, ,
1
0
0
K
mx
my
1
0
0
y
xs
y
x
K
3 x R, 3 x t
9
10
11
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
13
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Projective Camera
• Finite projective camera:– K is an upper triangular matrix
– KR is non-singular
• General projective camera:– P is an arbitrary 3 x 4 matrix of rank 3
– P has also 11 degrees of freedom
4
1|~
| pC MIMIKRP
34333231
24232221
14131211
pppp
pppp
pppp
P
But: We model real cameras
as finite projective cameras
(+ lens distortion)
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
14
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Camera Calibration in Practice (1)
• Take – 1 picture of a 3D calibration target,
– or several pictures of a planar calibration target
(take care so that all parameters can be recovered !)
• Establish point correspondences
• Calculate P– set of linear equations
• Decompose P
niXx ii 1 ,~~
t
|RKP
MZ
Y
X
pppp
pppp
pppp
y
x
m
P
1
~
1 34333231
24232221
14131211
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
15
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
3D Targets
[Hartley + Zisserman][Heikkilä]
Photogrammetry [Godding / Jähne]
• Many ways to build …
• Corners vs. circles (center of gravity) …
• Precision of building, attaching, …
• CNC measured points …
• EMT: coordinate measurement machine
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
16
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Simulating a 3D target by actuating a CMM
Camera calibration @ EMT using a CMM (coordinate measurement machine)
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
17
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
2D vs. 3D Targets
f = 28mm, z ~ 300mm f = 50mm, z ~ 470mm f = 84mm, z ~ 720mm
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
18
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
2D Targets
f = 28mm, z ~ 280mm f = 50mm, z ~ 470mm f = 84mm, z ~ 720mm
• arbitrary scaling !
– z/f ~ const.
– closeup of toy car vs. real car at a distance …
• but: subtle differences in image quality !
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
19
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Image Quality (1)
f = 28mm, z ~ 280mm f = 50mm, z ~ 470mm
lens distortion !
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
20
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Image Quality (2)
f = 28mm, z ~ 280mm f = 50mm, z ~ 470mm
“chromatic aberration”
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
21
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Lens Distortion Model
• Several ways to model
• Most common:– Scene >> lens system C is ok
– Radial lens distortion ki
– Tangential lens distortion tj
– Radial >> tangential
– Polynomial approximation up to varying order
(x0,y0)
,0
xxx ,0
yyy 22 yxr
)1)(2)2((' 2
32
221
64
2
2
31rtyxtrxtrxkrxkrxkxx
)1)(2)2((' 2
32
221
64
2
2
31rtyxtrytrykrykrykyy
r
x
y
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
22
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Camera Calibration in Practice (2)
• Take – 1 picture of a 3D calibration target,
– or several pictures of a planar calibration target
• Establish point correspondences
• Calculate P– set of linear equations
• Decompose P
A first
estim
ate
for
linear
Inte
rior
para
mete
rs (
K)
• Add nonlinear relationships (model ki, tj)
• Perform iterative optimization (w.r.t. some error)
• Enforce constraints (such as structure of K and R)
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
23
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
What can be measured with one calibrated camera?
• Directions (lines of sight)
• Angles between directions
• 3D reconstruction (“motion stereo”, SaM):– Camera is moved
– Several images are taken
– Point correspondences between the images are
established
• Special case: planar scene + known extrinsics– Angles, distances, areas
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
24
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
More Than One Camera
• Multiple views:– Epipolar geometry
– Uncalibrated stereo: “Fundamental” matrix F
– Calibrated stereo: “Essential” matrix E
– Stereo rig
– Camera motion
many views
tracking
Structure + Motion (S+M)
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
25
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Epipolar Geometry (1)
• Figures from [Hartley + Zisserman]
• C, C’, x, x’, X are co-planar (lie in the “epipolar plane” π)
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
26
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Epipolar Geometry (2)
• Assume that only C, C’, and x are known
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
27
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Epipolar Geometry (3)
• π projects on “epipolar lines” l and l’
• “baseline”: connects C, C’
• “epipoles”: e, e’
C C’
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
28
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Epipolar Geometry (4)
• When 3D position of X varies, π “rotates” about the baseline
• Family of planes – “epipolar pencil” – “Ebenenbüschel”
C C’
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
29
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Epipolar Geometry – Example 1:
Converging Cameras [Hartley+Zisserman]
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
30
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Epipolar Geometry – Example 2:
Forward Translation [Hartley+Zisserman], [Pollefeys]
e
e’
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
31
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
The Fundamental Matrix F (1)
We had an example:
Homography H
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
32
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
The Fundamental Matrix F (2)
• Transfer xi via Xi in π to xi’
• 2D homography Hπ maps each xi to xi’
''' xel
'''' xexe
0
0
0
' ,''
1
'
2
'
1
'
3
'
2
'
3
'
3
'
2
'
1
ee
ee
ee
e
e
e
e
e
xxelxx
FHH '' :'
“skew-symmetric”
matrix
xl
F'
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
33
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
The Fundamental Matrix F (3)
• F relates x in one image with its corresponding
epipolar line l’ in the other image (all X in R3 !):
• The corresponding point x’ must lie on l’:
• This relates to:
• How to estimate F?
xl
F'
0'' ,0'' lxlx T
0' xx T F
Point correspondences
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
34
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Two Calibrated Cameras: Essential Matrix E
• “Essential matrix” E– Similar to F
– Relates calibrated stereo rig
– Internal matrices K and K’ are known
tt T
RRRE R, t
xxxx T 1ˆ ,0ˆ'ˆ KE “normalized coordinates” x̂
FKKEEKKTTT xx ' 0'' 1
Institut für Elektrische Meßtechnik und Meßsignalverarbeitung
Professor Horst Cerjak, 19.12.2005
35
WS 2017/18Image-based Measurement 4 Calibration Axel Pinz
Calibration Issues
• Linear Models– Homography estimation H
– Epipolar geometry F, E
– Interior camera parameters K
– Exterior camera parameters R,t
– Camera pose R,t
• Algorithms– Interest Point Detection + Description
– Overdetermined systems of linear equations Error Minimization
– Direct Linear Transform – DLT
– Normalization
– Nonlinearities iterative error minimization, Levenberg-Marquardt
– Outliers Robustness, RANSAC
3
2
1
333231
232221
131211
3
2
1
'
'
'
'
x
x
x
hhh
hhh
hhh
x
x
x
x
x
H
MtMZ
Y
X
pppp
pppp
pppp
y
x
m
|
1
~
1 34333231
24232221
14131211
RKP