3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole...
Transcript of 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole...
![Page 1: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/1.jpg)
Camera model & calibration
Prof. Didier Stricker
Kaiserlautern University
http://ags.cs.uni-kl.de/
DFKI – Deutsches Forschungszentrum für Künstliche Intelligenz
http://av.dfki.de
3D Computer Vision
1
![Page 2: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/2.jpg)
2
f Z Z
fXx
X O
-x
Pinehole model - Perspective Projection.
Remember?
![Page 3: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/3.jpg)
• Finding camera’s internal parameters that effect the
imaging process.
1. Position of the image center on the image (usually not just
(width/2, height/2).
2. Focal length
3. Scaling factors for row and column pixels.
4. Lens distortion.
3
Camera Calibration
![Page 4: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/4.jpg)
Calibration: finding the function (defined by the camera) that maps 3D points to
2D image plane.
First thing required is to obtain pairs of corresponding 3D and 2D points,
(X1, Y1, Z1) (x1, y1), (X2, Y2, Z2) (x2, y2), ….
4
Calibration Procedure
![Page 5: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/5.jpg)
Calibration Target: Two perpendicular planes with chessboard pattern.
1. We know the 3D positions of the corners with respect to a
coordinates system defined on the target.
2. Place a camera in front of the target and we can locate the
corresponding corners on the image. This gives us the
correspondences.
3. Recover the equation that describes imaging projection and
camera’s internal parameters. At the same time, also recover the
relative orientation between the camera and the target (pose).
5
Calibration Procedure
![Page 6: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/6.jpg)
1. Corner detector
2. Canny Edge detector plus fitting lines to the detected edges. Find the intersections of the lines.
3. Manual input.
Matching 3D and 2D points (we know the number of corners) by counting.
This gives corresponding pairs
(world point) < --- > (image point )
(X1, Y1, Z1) (x1, y1),
6
Finding Corners
![Page 7: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/7.jpg)
7
World Coordinates and Camera Coordinates
![Page 8: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/8.jpg)
Rotation matrix
8
Camera Frame to World Frame
![Page 9: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/9.jpg)
9
Linear Algebra
Definition: a matrix R is a rotation matrix if and only
if it is a orthogonal matrix with determinant +1
Orthogonal Matrix: a square matrix with real entries
whose columns and rows are orthogonal vectors
with length 1.
That means: RRT = I
Or: R-1 = RT
Definition: 3D-Rotation
![Page 10: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/10.jpg)
10
Sequence not standardized! Many different conventions!
![Page 11: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/11.jpg)
Let = - Rt T
(C is the camera center)
11
Homogeneous Coordinates
![Page 12: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/12.jpg)
Use Homogeneous coordinates, the perspective projection becomes linear.
12
Perspective Projection
![Page 13: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/13.jpg)
13
Pixel Coordinates
![Page 14: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/14.jpg)
14
Calibration Matrix
![Page 15: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/15.jpg)
15
Putting Everything Together
![Page 16: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/16.jpg)
1. Estimate matrix P using scene points and their
images
2. Estimate the intrinsic and extrinsic parameters
Left 3x3 sub-matrix is the product of an upper
triangular matrix and an orthogonal matrix.
16
Calibration
![Page 17: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/17.jpg)
17
Computing the Matrix P
![Page 18: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/18.jpg)
18
Computing the Matrix P
![Page 19: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/19.jpg)
Rank 2
19
Computing the Matrix P
![Page 20: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/20.jpg)
20
Computing the Matrix P
![Page 21: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/21.jpg)
-
-
-
21
Computing the translation component
![Page 22: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/22.jpg)
Finding the camera orientation and internal parameters
KR (use RQ decomposition ~QR)
Q R =( )-1= -1 -1 Q R
(if only QR, invert)
A QR decomposition (also called a QR factorization) of a matrix is a
decomposition of the matrix into an orthogonal and a right triangular
matrix.
22
Camera matrix decomposition
![Page 23: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/23.jpg)
23
Project known 3D points on the image
Compare their projections with the
corresponding pixel coordinates of the
points.
Repeat for many points and estimate “re-
projection” error!
How should we estimate the accuracy
of a calibration algorithm?
![Page 24: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/24.jpg)
Further Improvement
Most popular non-linear minimization algorithm is the Levenberg-Marquart
minimization
LM is more robust to local minima than e.g. the Gauss–Newton algorithm
and the method of gradient descent)
24
Improvement: computing the matrix P
![Page 25: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/25.jpg)
25
Barrel and Pincushion
Tangential
Advanced Calibration: Nonlinear
Distortions
![Page 26: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/26.jpg)
26
tele wideangle
Barrel and Pincushion Distortion
![Page 27: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/27.jpg)
27
To model lens distortion Use above projection operation instead of standard projection matrix
multiplication
Modeling distortion
Apply radial distortion
Apply focal length translate image center
Project to “normalized”
image coordinates
Radial and Tangential Distortion
image plane
ideal actual
ideal actual
![Page 28: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/28.jpg)
28
Set k1k20, solve for undistorted case
Find optimal k1,k2 via nonlinear least squares
Iterate
Tends to generate good calibrations
Distorted Camera Calibration
![Page 29: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/29.jpg)
29
Image Rectification
![Page 30: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/30.jpg)
30
Examplary calibration set-ups
![Page 31: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/31.jpg)
31
Advantage:
Very simple to formulate and solve
Disadvantages:
Doesn’t tell you the camera parameters
Doesn’t model radial distortion
Hard to impose constraints (e.g., known focal length)
Doesn’t minimize the right error function
For these reasons, nonlinear methods are preferred
• Define error function E between projected 3D points and image positions
– E is nonlinear function of intrinsics, extrinsics, radial distortion
• Minimize E using nonlinear optimization techniques
– e.g., variants of Newton’s method (e.g., Levenberg Marquart)
Direct linear calibration
![Page 32: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/32.jpg)
Alternative: multi-plane calibration
32
Images courtesy Jean-Yves Bouguet, Intel Corp.
Advantage • Only requires a plane
• Don’t have to know positions/orientations
• Good code available online!
– Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/
– Matlab version by Jean-Yves Bouget:
http://www.vision.caltech.edu/bouguetj/calib_doc/index.html
– Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/
![Page 33: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/33.jpg)
33
Reduce the number of interactions
Pre-compute as much parameters as
possible: Usually all intrinsic camera parameters
More stable
Less user-input required
Calibration or Pose estimation using a
CAD model
![Page 34: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/34.jpg)
34
Calibration or Pose estimation using a CAD
model
![Page 35: 3D Computer Vision - - TU Kaiserslautern · 3D Computer Vision 1 . 2 Z f Z f x X X - O x Pinehole model - Perspective Projection. Remember? • Finding camera’s internal parameters](https://reader033.fdocument.pub/reader033/viewer/2022041419/5e1daf580a3c717d9f6cc07f/html5/thumbnails/35.jpg)
35
THANK YOU!