Notice

• This power point is made by Mitsuru Nakazawa, NOT an original author, for paper introduction of ECCV2012

1

Performance Capture of Interacting Characters with Handheld Kinects

Genzhi Ye1 Yebin Liu1 Nils Hasler2

Xiangyang Ji1 Qionghai Dai1 Christian Theobalt2

1: Deptartment of Automation, Tsinghua University

2: Graphics, Vision & Video Group, Max-Planck Institute for Informatics

2ECCV2012 paper introduction

Presenter: Mitsuru NAKAZAWA

Introduction movie

3

URL: http://media.au.tsinghua.edu.cn/yegenzhi/HandheldKinectsMocap_ECCV2012.jsp(Accessed on 26th Nov. 2012)

Related works

Multi-view motion capture approaches

Reconstruct a skeletal motion model & detailed dynamic surface geometry

Deal with people wide apparel

Require controlled studio setup (many number of sync video cameras)

4

Marker-less motion capture from a single range sensor

Estimate complex poses at real-time frame rates

Difficult to capture 3D, complex, detailed model

Objective

5

Reconstruct a skeleton motion & time varying surface geometry of humans in general apparel

Handle fast and complex motion with many self-occlusions & non-rid surface deformation

Not need studios with controlled lighting and many stationary cameras

Full performance capture of moving humans using

only 3 handheld, moving Kinects

Operator

Performer

freely move

Data capture

6

Capture environment Captured data from 3 Kinects

Asynchronous capture Use a start recording signal to all PCs connected through Wi-Fi

Intrinsic calibration Apply Zhang’s method

Alignment between the color image and the range data Use the OpenNI API

Operator

Performer

GND (r=3m)

Scene models at time t• Human model

– Laser scanner provide a static mesh with embedded skeleton of each performer

7

• 5,000 vertices of meshes• k-th performer’s Skeleton with

31 degrees of freedom: Ck

t

• Ground plane model (fixed)– Center of Environment– Planar mesh with circular boundary

• Camera extrinsic parameters of i-th Kinect– Translation, rotation: Lk

t

[*] F. Remondino: “3-D reconstruction of static human body shape from image sequence,” CVIU, Vol.93, No.1. pp.65-85

[*]

Geometric matching of Kinect point to vertices of a human model

Overview of the proposed method

8

Point Could Segmentation

Optimization of skeleton & camera pose

at timet

Non-rigid deformation of the human surface via Laplacian deformation

Optimization of skeleton and camera pose

9

Error function E is solved within iterative quasi-Newton minimization

• Human region error between model vertices and Kinect point cloud• Ground region error between model vertices and Kinect point cloud• Difference of matched SIFT feature positions between previous and current

time on background regions (SfM approach)

Result using t−1 & t−1 Result based on SfM approach Optimized result

Comparison with Multi-view Video TrackingMulti-view video trachking system with 10 calibrated cameras vs. Proposed method

10

“Rolling” with slow motion Similar results

“Jump” with fast motion proposed method gets better results

Performance capture results on a variety of Sequences

11

Conclusion

• Simultaneously marker-less performance capture system with several hand-held Kinects

– Iterative robust matching of tracked 3D models and input Kinect data

12

References

• Linear Blend Skinning (Accessed on Nov. 25th 2012 )

– http://bit.ly/RaijkQ

• Motion Capture Using Joint Skeleton Tracking and Surface Estimation (Accessed on Nov. 26th 2012)

– http://www.vision.ee.ethz.ch/~gallju/projects/skelsurf/index.html

13