Image-based modeling
description
Transcript of Image-based modeling
![Page 1: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/1.jpg)
Image-based modeling
Digital Visual Effects, Spring 2008Yung-Yu Chuang2008/5/6
with slides by Richard Szeliski, Steve Seitz and Alexei Efros
![Page 2: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/2.jpg)
Announcements
• Project #3 is extend to 5/16• Project #2 artifact voting results
![Page 3: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/3.jpg)
Honorable mention (7): 張子捷 游知澔
![Page 4: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/4.jpg)
Honorable mention (9): 胡傳牲 高紹航
![Page 5: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/5.jpg)
Third place (10): 吳懿倫 張哲瀚
![Page 6: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/6.jpg)
Second place (13): 賴韻芊 黃柏瑜
![Page 7: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/7.jpg)
First place (24): 游名揚 曾照涵
![Page 8: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/8.jpg)
Outline
• Models from multiple (sparse) images– Structure from motion– Facade
• Models from single images– Tour into pictures– Single view metrology– Other approaches
![Page 9: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/9.jpg)
Models from multiple images
(Façade, Debevec et. al. 1996)
![Page 10: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/10.jpg)
Facade
• Use a sparse set of images• Calibrated camera (intrinsic only)• Designed specifically for modeling
architecture• Use a set of blocks to approximate
architecture
• Three components:– geometry reconstruction– texture mapping – model refinement
![Page 11: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/11.jpg)
Idea
![Page 12: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/12.jpg)
Idea
![Page 13: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/13.jpg)
Geometric modeling
A block is a geometric primitive with a small set of parameters
Hierarchical modeling for a scene
Rotation and translation could be constrained
![Page 14: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/14.jpg)
Reasons for block modeling
• Architectural scenes are well modeled by geometric primitives.
• Blocks provide a high level abstraction, easier to manage and add constraints.
• No need to infer surfaces from discrete features; blocks essentially provide prior models for architectures.
• Hierarchical block modeling effectively reduces the number of parameters for robustness and efficiency.
![Page 15: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/15.jpg)
Reconstruction
minimize
![Page 16: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/16.jpg)
Reconstruction
![Page 17: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/17.jpg)
Reconstruction
nonlinear w.r.t. camera and model
![Page 18: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/18.jpg)
Results3 of 12 photographs
![Page 19: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/19.jpg)
Results
![Page 20: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/20.jpg)
![Page 21: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/21.jpg)
Texture mapping
![Page 22: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/22.jpg)
Texture mapping in real world
Demo movieMichael Naimark,San Francisco Museum of Modern Art, 1984
![Page 23: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/23.jpg)
Texture mapping
![Page 24: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/24.jpg)
Texture mapping
![Page 25: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/25.jpg)
View-dependent texture mapping
![Page 26: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/26.jpg)
View-dependent texture mapping
model VDTM
VDTMsingle
texturemap
![Page 27: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/27.jpg)
View-dependent texture mapping
![Page 28: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/28.jpg)
Model-based stereo
• Use stereo to refine the geometry
knownknowncameracamera
viewpointsviewpoints
![Page 29: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/29.jpg)
Stereo
scene pointscene point
optical centeroptical center
image planeimage plane
![Page 30: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/30.jpg)
Stereo
• Basic Principle: Triangulation– Gives reconstruction as intersection of two rays– Requires
• calibration• point correspondence
![Page 31: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/31.jpg)
Stereo correspondence
• Determine Pixel Correspondence– Pairs of points that correspond to same scene point
• Epipolar Constraint– Reduces correspondence problem to 1D search
along conjugate epipolar lines
epipolar planeepipolar lineepipolar lineepipolar lineepipolar line
![Page 32: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/32.jpg)
Finding correspondences
• apply feature matching criterion (e.g., correlation or Lucas-Kanade) at all pixels simultaneously
• search only over epipolar lines (much fewer candidate positions)
![Page 33: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/33.jpg)
Image registration (revisited)
• How do we determine correspondences?– block matching or SSD (sum squared differences)
d is the disparity (horizontal motion)
• How big should the neighborhood be?
![Page 34: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/34.jpg)
Neighborhood size
• Smaller neighborhood: more details• Larger neighborhood: fewer isolated
mistakes
w = 3 w = 20
![Page 35: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/35.jpg)
Depth from disparity
f
x x’
baseline
z
C C’
X
f
input image (1 of 2) [Szeliski & Kang ‘95]
depth map 3D rendering
![Page 36: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/36.jpg)
– Camera calibration errors– Poor image resolution– Occlusions– Violations of brightness constancy (specular reflections)– Large motions– Low-contrast image regions
Stereo reconstruction pipeline• Steps
– Calibrate cameras– Rectify images– Compute disparity– Estimate depth
• What will cause errors?
![Page 37: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/37.jpg)
Model-based stereokey image
offset image
warped offset image
![Page 38: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/38.jpg)
Epipolar geometry
![Page 39: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/39.jpg)
Results
![Page 40: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/40.jpg)
Comparisons
single texture, flat VDTM, flat
VDTM, model-based stereo
![Page 41: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/41.jpg)
Final results
Kite photography
![Page 42: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/42.jpg)
Final results
![Page 43: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/43.jpg)
![Page 44: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/44.jpg)
Results
![Page 45: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/45.jpg)
Results
![Page 46: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/46.jpg)
Commercial packages
• REALVIZ ImageModeler
![Page 47: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/47.jpg)
The Matrix
Cinefex #79, October 1999.
![Page 48: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/48.jpg)
The Matrix
• Academy Awards for Scientific and Technical achievement for 2000
To George Borshukov, Kim Libreri and Dan Piponi for the development of a system for image-based rendering allowing choreographed camera movements through computer graphic reconstructed sets.
This was used in The Matrix and Mission Impossible II; See The Matrix Disc #2 for more details
![Page 49: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/49.jpg)
Models from single images
![Page 50: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/50.jpg)
Vanishing points
• Vanishing point– projection of a point at infinity
image plane
cameracenter
ground plane
vanishing point
![Page 51: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/51.jpg)
Vanishing points (2D)
image plane
cameracenter
line on ground plane
vanishing point
![Page 52: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/52.jpg)
Vanishing points
• Properties– Any two parallel lines have the same vanishing
point v– The ray from C through v is parallel to the lines– An image may have more than one vanishing point
image plane
cameracenterC
line on ground plane
vanishing point V
line on ground plane
![Page 53: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/53.jpg)
Vanishing lines
• Multiple Vanishing Points– Any set of parallel lines on the plane define a vanishing point– The union of all of these vanishing points is the horizon line
• also called vanishing line– Note that different planes define different vanishing lines
v1 v2
![Page 54: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/54.jpg)
Computing vanishing points
• Properties– P is a point at infinity, v is its projection– They depend only on line direction– Parallel lines P0 + tD, P1 + tD intersect at P
V
DPP t 0
0/1
/
/
/
1Z
Y
X
ZZ
YY
XX
ZZ
YY
XX
t D
D
D
t
t
DtP
DtP
DtP
tDP
tDP
tDP
PP
ΠPv
P0
D
![Page 55: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/55.jpg)
Tour into pictures
• Create a 3D “theatre stage” of five billboards
• Specify foreground objects through bounding polygons
• Use camera transformations to navigate through the scene
![Page 56: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/56.jpg)
Tour into pictures
![Page 57: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/57.jpg)
The idea
• Many scenes (especially paintings), can be represented as an axis-aligned box volume (i.e. a stage)
• Key assumptions:– All walls of volume are orthogonal– Camera view plane is parallel to back of
volume– Camera up is normal to volume bottom– Volume bottom is y=0
• Can use the vanishing point to fit the box to the particular Scene!
![Page 58: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/58.jpg)
Fitting the box volume
• User controls the inner box and the vanishing point placement (6 DOF)
![Page 59: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/59.jpg)
Foreground Objects• Use separate
billboard for each
• For this to work, three separate images used:– Original image.– Mask to isolate
desired foreground images.
– Background with objects removed
![Page 60: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/60.jpg)
Foreground Objects
• Add vertical rectangles for each foreground object
• Can compute 3D coordinates P0, P1 since they are on known plane.
• P2, P3 can be computed as before (similar triangles)
![Page 61: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/61.jpg)
Example
![Page 62: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/62.jpg)
Example
![Page 64: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/64.jpg)
Zhang et. al. CVPR 2001
![Page 65: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/65.jpg)
Zhang et. al. CVPR 2001
![Page 66: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/66.jpg)
Oh et. al. SIGGRAPH 2001
![Page 67: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/67.jpg)
Oh et. al. SIGGRAPH 2001
video
![Page 68: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/68.jpg)
Automatic popup
Input
Ground
Vertical
Sky
Geometric Labels Cut’n’Fold 3D Model
Image
Learned Models
![Page 69: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/69.jpg)
Geometric cues
Color
Location
Texture
Perspective
![Page 70: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/70.jpg)
Automatic popup
![Page 71: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/71.jpg)
Results
Automatic Photo Pop-upInput Images
![Page 72: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/72.jpg)
Results
This approach works roughly for 35% of images.
![Page 73: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/73.jpg)
Failures
Labeling Errors
![Page 74: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/74.jpg)
FailuresForeground Objects
![Page 75: Image-based modeling](https://reader034.fdocument.pub/reader034/viewer/2022051316/56814ae4550346895db7f688/html5/thumbnails/75.jpg)
References
• P. Debevec, C. Taylor and J. Malik. Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach, SIGGRAPH 1996.
• Y. Horry, K. Anjyo and K. Arai. Tour Into the Picture: Using a Spidery Mesh Interface to Make Animation from a Single Image, SIGGRAPH 1997.
• L. Zhang, G. Dugas-Phocion, J.-S. Samson and S. Seitz. Single View Modeling of Free-Form Scenes, CVPR 2001.
• B. Oh, M. Chen, J. Dorsey and F. Durand. Image-Based Modeling and Photo Editing, SIGGRAPH 2001.
• D. Hoiem, A. Efros and M. Hebert. Automatic Photo Pop-up, SIGGRAPH 2005.