Depth Estimate and Focus Recovery
description
Transcript of Depth Estimate and Focus Recovery
![Page 1: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/1.jpg)
Depth Estimate and Focus Recovery
Presenter : Wen-Chih Hong Adviser: Jian-Jiun Ding
Digital Image and Signal Processing LaboratoryGraduate Institute of Communication EngineeringNational Taiwan University, Taipei, Taiwan, ROC台大電信所 數位影像與訊號處理實驗室
112/04/22 1DISP Lab @ MD531
![Page 2: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/2.jpg)
Outlines Introduction Binocular version systems
Stereo Monocular version systems
DFF DFD
Other method Conclusions References
112/04/22 2DISP Lab @ MD531
![Page 3: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/3.jpg)
Introduction Depth is an important information for
robot and the 3D reconstruction. Image depth recovery is a long-term
subject for other applications such as robot vision and the restorations.
Most of depth recovery methods based on simply camera focus and defocus.
Focus recovery can help users to understand more details for the original defocus images.
112/04/22 3DISP Lab @ MD531
![Page 4: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/4.jpg)
Introduction Categories of depth estimation
Monocular
Depth from defocus (DFD)
Depth from focus (DFF)
Binocular Stereo focus
112/04/22 4DISP Lab @ MD531
![Page 5: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/5.jpg)
Introduction Categories of depth estimation
Active : Sending a controlled energy beam Detection of reflected energy
Passive: Image-based
112/04/22 5DISP Lab @ MD531
![Page 6: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/6.jpg)
Introduction Geometric on imaging
1 1 1
lD F
F
D/2
F
us
2R : R>0
sensor
vBiconvex
112/04/22 6DISP Lab @ MD531
![Page 7: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/7.jpg)
Binocular version systems The flow chart to binocular depth
estimation. Depth map HVS modeling Edge detection Correspondence Vengeance control Gaze control Depth map
112/04/22 7DISP Lab @ MD531
![Page 8: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/8.jpg)
Binocular version systems Vengeance movement :
is some kind of slow eye movement that two eyes move in different directions.
But corresponding problem Gazing point(Corresponding point)
Baseline (B)
B/2B/2
Depth (u)
2 2 22 2
2 2
sin cos cos4sin sin
L R L R
L R L R
u B
112/04/22 8DISP Lab @ MD531
![Page 9: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/9.jpg)
Binocular version systems Complex model
RdiffLdiff
Corresponding point
Depth (u)
Right vision
Left vision Baseline (B)
(xR, yR)(xL, yL)
Figure 3.3 A more complete triangulation geometry for the binocular vision.We have to realize how much departure between the optical axis and the direction
of the
tan tan
tan tanLdiff Rdiff
Ldiff Rdiff
Bu
1tan tan / 2Rdiff Rx W
1tan tan / 2Ldiff Lx W
112/04/22 9DISP Lab @ MD531
![Page 10: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/10.jpg)
Binocular version systems Corresponding problem But more accuracy
112/04/22 10DISP Lab @ MD531
![Page 11: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/11.jpg)
Monocular version systems Depth from focus
Depth form defocus
112/04/22 11DISP Lab @ MD531
![Page 12: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/12.jpg)
Depth from Focus Taking pictures at different observer distance or object
distance We need an estimator to measure degree on focus
Using Laplacian operator
Such operator point to a measurement on a single pixel influence, a sum of Laplacian operator is needed:
( , ) ( , , )* ( , )kg x y h x y t f x y
2 22
2 2( , ) ( , )
( , ) k kk
g x y g x yg x y
x y
, ,j k i k
v j k u i k
n i j ML u v
112/04/22 12DISP Lab @ MD531
![Page 13: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/13.jpg)
Depth from Focus Gaussian interpolation
Figure 4.4 Gaussian interpolation to a measure curve, Nk≧ Nk-1, Nk≧ Nk+1
displacementdk
[SML]
NPFocus measure
Nk
Nk-1
dk-1dp
Measured curveIdeal condition
dk+1
Nk+1
112/04/22 13DISP Lab @ MD531
![Page 14: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/14.jpg)
Depth from Focus Range from focus
using
Take pictures along the axis Find the image having highest frequency Need more than 10 images (monocular)
1 1 1f D v
112/04/22 14DISP Lab @ MD531
![Page 15: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/15.jpg)
Depth from Focus We use Gaussian interpolation to form
a set of approximations The depth solution dp from above
Gaussian:
2 21 1
1 1 1
2 21 1
1 1 1
ln ln
2 ln ln ln ln
ln ln
2 ln ln ln ln
k k k kp
k k k k k k
k k k k
k k k k k k
N N d dd
d d N N N N
N N d d
d d N N N N
112/04/22 15DISP Lab @ MD531
![Page 16: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/16.jpg)
Depth from Defocus Due to geometric optics, the intensity inside the blur
circle should be constant. Considering of aberration and diffraction and so on, we
easily assume a blurring function: : diffusion parameter
Diffusion parameter is related to blur radius: derived from triangularity in geometric optics
For easy computation, we assume that foreground has equal-diffusion, background has equal-diffusion and so on
However, this equal-focal assumption will be a problem
2 2
2 2
1, exp2 2
x yh x y
112/04/22 16DISP Lab @ MD531
![Page 17: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/17.jpg)
Depth from Defocus Blurring model
Blurring radius
112/04/22 17DISP Lab @ MD531
![Page 18: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/18.jpg)
Depth from Defocus Blurring model
1 0
0
br rv v v
2 0
0
br rv v v
0v v v
112/04/22 18DISP Lab @ MD531
![Page 19: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/19.jpg)
Depth from Defocus Blurring model
1 0
0
br rv v v
2 0
0
br rv v v
0v v v
112/04/22 19DISP Lab @ MD531
![Page 20: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/20.jpg)
Depth from Defocus Blurring model
1 0
0
br rv v v
2 0
0
br rv v v
0v v v
112/04/22 20DISP Lab @ MD531
![Page 21: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/21.jpg)
Depth from Defocus Blurring model
when
blur radius is independent of the location of the point source on the object plane at depth
01 2 0 ( 1)b b
vr r r rv
0
01 2 0 ( 1)b b
vr r r rv
112/04/22 21DISP Lab @ MD531
![Page 22: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/22.jpg)
Depth from Defocus Blurring model
Using and
We get So diffusion parameter:
1 1 1
lv F D 0
1 2 0 ( 1)b bv
r r r rv
0 00
1 1 1( )bl
r r vF v D
1 1 1( )m m mlm m
r vF v D
112/04/22 22DISP Lab @ MD531
![Page 23: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/23.jpg)
Depth from Defocus Depth recovery
Eliminating D from m=1,2 we get
where and
1 2
1 1 1( )m m mlm m
r vF v D
1 1
2 2
rvr v
1 11 1 2 2
1 1 1 1( )l l
rvF v F v
112/04/22 23DISP Lab @ MD531
![Page 24: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/24.jpg)
Depth from Defocus Depth recovery
From Take F.T.:
The F.T. of Gaussian is Gaussian
( , ) ( , ) ( , )k kg x y h x y f x y
( , ) ( , ) ( , )k kG w v H w v F w v
2 2
2 2
1, exp2 2
x yh x y
2 2 2 211 2
2
( , ) 1exp[ ( )( )]( , ) 2
G w v w vG w v
112/04/22 24DISP Lab @ MD531
![Page 25: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/25.jpg)
Depth from Defocus Depth recovery
Take the log
Using the relationship between them
we get
1 2 2 2 2
2 2( 1) 2 C
2 2 11 2 2 2
2
( , )2 log( , )
G v Cv G v
2 2 2 211 2
2
( , ) 1exp[ ( )( )]( , ) 2
G w v w vG w v
112/04/22 25DISP Lab @ MD531
![Page 26: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/26.jpg)
Depth from Defocus Depth recovery
let apha=1 we obtain the value of sigma-2
Find out the depth D
2 2 22 2( 1) 2 C
1 1
2 2
1rvr v
1 1 1( )m m mlm m
r vF v D
112/04/22 26DISP Lab @ MD531
![Page 27: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/27.jpg)
Depth from Defocus The main sources of range errors in
DFD
Inaccurate modeling of the optical system. Windowing for local feature analysis. Low spectral content in the scene being
images. Improper calibration of camera parameters. Presence of sensor noise.
112/04/22 27DISP Lab @ MD531
![Page 28: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/28.jpg)
Depth from Defocus Block shift-variant blur model
Consider the interaction of sub-images Define the neighborhood function
Indeed, the image we observed is
compared with
{ , 1,..., ,..., 1, }in i J i J i i J i J
( ) ( )ni
i
i h in
g m f m a d
( , ) ( , ) ( , )k kg x y h x y f x y
112/04/22 28DISP Lab @ MD531
![Page 29: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/29.jpg)
Depth from Defocus Space-variant filtering models for
recovering depth Using complex spectrogram and P.W.D. Complex Spectrogram:
2 1( ) '( , ') ( ') 'x t h t t x t dt
*''( , ') ( ' ) '( , ) exp( ( '))2kh t t u t t H t j t t d
112/04/22 29DISP Lab @ MD531
![Page 30: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/30.jpg)
Depth from Defocus Space-variant filtering models for
recovering depth C.S.:
g_1/g_2
where
( , ) ( , ) ( , )ig f iC t C t H t
2 1( , ) ( , ) ( , )g gC t C t H t
2
1
( , )( , )( , )
H tH tH t
112/04/22 30DISP Lab @ MD531
![Page 31: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/31.jpg)
Depth from Defocus Space-variant filtering models for
recovering depth objective function:
Drawback: No consider the intersection of pixels there will
be interrupt in border. Regularized solution.
2 1
' 1 2 2 2 2
( ) 0
min ( ( ) ( ) exp( ( ) ( )))N
g i g is i k
C k C k k s i
2 22 1( ) ( ) ( )s i i i
112/04/22 31DISP Lab @ MD531
![Page 32: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/32.jpg)
Depth from Defocus No corresponding problem Less accuracy
S.V. > B.S.V. Blocking Trade-off
Blocking size Too large: less accuracy Too small: noise
112/04/22 32DISP Lab @ MD531
![Page 33: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/33.jpg)
Other method Structure from motion Shape from shading
ML Estimation of Depth and Optimal camera settings Recursive computation of depth from multiple images
112/04/22 33DISP Lab @ MD531
![Page 34: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/34.jpg)
Other method Structure from motion
Using the relative motion between object and camera to find out surface information
Corresponding problem (binocular) Find out what motion of camera
112/04/22 34DISP Lab @ MD531
![Page 35: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/35.jpg)
Other method Shape from shading
Need to know the reflectance Find the sliding rate and blindness
112/04/22 35DISP Lab @ MD531
![Page 36: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/36.jpg)
Focus recovery
SML measurement
Defocused image pair
Full focused image
Maximum value searching
Depth measurement of a point
Small aperture construction
Linear canonical transform based on constructed
optical system
focal point
Using the specific depth to retrieve imaging
distance
112/04/22 36DISP Lab @ MD531
![Page 37: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/37.jpg)
Conclusions Binocular stereo method
high accuracy Absolute depth information Complexity computation Corresponding problem
Structure form motion Nonlinear problem Corresponding problem
Shape from shading Very difficult method Active method
112/04/22 37DISP Lab @ MD531
![Page 38: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/38.jpg)
Conclusions Range from focus :
Slowly More than 10 images
depth from defocus : Easy method Less accuracy
112/04/22 38DISP Lab @ MD531
![Page 39: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/39.jpg)
References and future work1) Y. C. Lin, Depth Estimation and Focus Recovery, Master
thesis, National Taiwan Univ., Taipei, Taiwan, R.O.C, 20082) Subhasis Chaudhuri, A.N. Rajagopalan, ”Depth From Defocus:
A Real Aperture Imaging Approach. ” Springer-Verlag. New York, 1999.
3) M. Subbarao, “Parallel depth recovery by changing camera parameters,” Second International Conference on Computer Vision 1988, pp. 149-155, Dec. 1988.
4) M. Subbarao and T. C. Wei, “Depth from defocus and rapid autofocusing: a practical approach,” IEEE Conferences on Computer Vision and Pattern Recognition, pp. 773-776, Jun. 1992.
5) A. N. Rajagopalan and S. Chaudhuri, “A variational approach to recovering depth from defocused images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, pp. 1158-1164, Oct. 1997.
112/04/22 39DISP Lab @ MD531
![Page 40: Depth Estimate and Focus Recovery](https://reader034.fdocument.pub/reader034/viewer/2022051020/56815d8e550346895dcba01e/html5/thumbnails/40.jpg)
The end
112/04/22 DISP Lab @ MD531 40