Constructing immersive virtual space for HAI with photos
description
Transcript of Constructing immersive virtual space for HAI with photos
Constructing immersive virtual space for HAI with photos
Shingo MoriYoshimasa Ohmoto
Toyoaki NishidaGraduate School of Informatics Kyoto University
GrC2011 2011/11/09
Abstract
• We automatically construct immersive virtual spaces for human agent interaction– Scenes are drawn by outside photo
images– Depth maps are reconstructed to
express occlusion– Rough 3D models are added for agent– Processing time is about 4.7 days to
reconstruct a 20m×20m virtual space
Introduction
• We want to observe HHI using HAI in a virtual space
• For example, sightseeing task:–we can select faraway place such as
foreign country–we can easily prepare environment to
observe
• Our Goal: creating system to construct environment to use such a task
Introduction
• To do sightseeing task and observe interaction, environment should be looked like the real world– virtual spaces should be immersive– outside scenes created by real world
photos is needed– spatial relationship between agent and
object should be correct– users walk freely on some level
• How to construct such a virtual space?
Related Work
• Model Based Rendering (MBR)– Reconstruct 3D models–Weak at trees or texture-less surfaces
• [1-3] are nice methods but,– [1] can’t use outside because of a
constraint of axis-aligned surface– [2,3] use high expensive equipments or
use a lot of time and effort
[1] Furukawa et al. 2010, Reconstructing build-ing interiors from images[2] Pollefeys et al. 2008, Detailed real-time urban 3d reconstruction from video[3] Ikeuchi et al, 2004, Bayon digital archival project
Related Work
• Image Based Rendering (IBR)– draw clearly complex structure such as
natural object–Weak at occlusion
• [4-5] have good image quality but,– They don’t consider agents–Movable space is restricted
[4]Google Street View[5] Ibuki , 2009, Reduction of Unnatural Feeling in Free-viewpoint Rendering Using View-Dependent Deformable 3-D Mesh Model (Japanese)
Our Method
• To make up immersive environment, we use IBR– because MBR has hole and low
resolution for a task– use panorama images and
omnidirectional display to show environment
Our Method
• To collect photo images– divide a space in into a 1-2m grid– shoot about 18 photos in each grid
• We use interpolation when move from one shooting point to another
obstacle
shooting point
shooting direction
1-2 meter
Our Method
• 3D geometry is needed for agents– use Structure from Motion and stereo
method in a similar way [1,5]– create depth map for occlusion between
objects and agents
• This information is used for better IBR– camera position & rotation– 3D position of a point cloud
System Pipeline
depth map
segmented image
camera parameter
Photos
Structure from Motion
Segmentation
Creating Depth Map
Show a Immersive
Virtual Space
interpolated image
Interpolation
: Input
: Process
:Output
panorama imagepanorama depth map
Creating Panorama
Use previous work
Tackle in this research
CMVS PatchesRough 3D model
Multi view Stereo
System of Constructing Virtual Space
Structure from Motion (SfM)
• Estimate camera parameter (translate matrix) from multi photos–we use Bundler[6]– it is robust and accurate camera position
photos points clout and camera positioncamera position[6] Snavely et al. 2006, Photo tourism: exploring photo collections in 3D
Multi view Stereo
• Reconstruct 3D geometry–we use CMVS[7] and Poisson Surface
Reconstruction[8]– get a point cloud (patches) and rough 3D
model
photos and translate matrix patches and rough 3D model[7] Furukawa et al. 2010, Towards internet-scale multi-view stereo[8] Kazhdan et al. 2006, Poisson surface reconstruction
Create Depth Map• Deal with holes and outliers• Using an assumption that the real
world is constructed by a planar surface– assume two points as same planar
surface if they are segmented to same area
– reconstruct surface from projected patches
raw image segmented image depth mapproject patches
Create Panorama Image
• To show a scene to an omnidirectional display, we create panorama images–we use Microsft ICE[9]– canonicalize direction of panorama
image from camera rotation
panorama image and depth map
[9] MicrosoftCorporation, Microsoft image composite editorhttp://research.microsoft.com/en-us/um/redmond/groups/ivm/ice.html.
Interpolation
project patchesto use as feature point
• To move freely, we create interpolated image between near panorama images• correct move direction and distance about
objecttwo raw panorama imagesabout 1-2m away from each other
find corresponding point
interpolate by morphing (medium point between raw images)
Demo
Processing time
• We experimented with 3 spaces• Most of processing time is SfM –We can drastically improve if we use
[10]
• Each shooting times are about one hour
[10] Agarwal et al.2009, Building rome in a day
Conclusion
• Conclusion– create a system to automatically construct
virtual spaces for HAI– unify various methods to create a system
• Future work– expand virtual spaces– research how natural and useful it for HAI