High Resolution Facial Performance Capture and

Animation

Alex Ma, I-Chen Lin, Ming Ouhyoung

馬萬鈞， 林奕成， 歐陽明CMLab, Dept. of CSIE,

National Taiwan University

10/27/2008

My Face Generation Dreams我的十年大夢

虛擬實境 (Virtual Reality) 的極致金庸的小說，主角換成自己的朋友？熱門遊戲的角色中有自己的參與？

My Face Generation DreamsPhase 1: Talking Head

A Speech Driven Talking Head System Based on a Single Face Image, pp. 43-49, Proc. Pacific Graphics'99

Phase 2: Human Face Expression

Realistic 3D Facial Animation Parameters from Mirror-reflected Multi-view Video, IEEE CG&A, 2002

Phase 3: Try to pass the human face Turing Test , Facial Performance Synthesis using Deformation-Driven Polynomial Displacement Maps, SIGGRAPH ASIA 2008, ACM Trans. Graphics

VR Talking Head,TalkingShow (CL product)

Realistic 3D Facial Animation Parameters from Mirror-reflected Multi-view Video

Facial Performance Synthesis using Deformation-Driven Polynomial Displacement

Maps

Emily project: with Image Metrics Inc.

Emily demo: Turing test?

Demo video

Outline

Estimation of 3D facial motion trajectories

Synthesis of facial animation Results and conclusion

Introduction (cont.)

Estimating 3D facial motion parameters from mirror-reflected multi-view video.

Fig 1. (a) the capture equipment. (b) a mirror-reflected multi-view video clip. (55 markers: 10 for the lip contour, 12 for lip surfaces, 10 for mouths, 8 for cheeks, and 10 for the forehead)

What’s new

Previous tracking approach from mirror-reflected video:• E.C. Patterson et al.(Computer

Animation’91): Simply assumed that the mirror was vertical, and

the camera was also vertical.• S. Basu et al. (ICCV’98):

Position evaluation via R, t estimation of virtual camera.

The proposed method:• Taking advantages of nice properties of

mirrored images.• More robust, accurate, and simpler.

FAPs extraction

Demo1: a captured video and synthesized result (viseme “o”).

Estimation of 3D facial motion

The proposed procedure:

1)Semi-automatic tracking the markers’ projected 2D trajectories in videos.

2)Estimation of markers’ 3D trajectories.

3)Head motion removal.

Marker tracking in video

Fig 2. The flow chart of semi-automatic markers’ projected positions tracking in video.

Manually designate markers’ positions

(1st frame)

Manually designate markers’ positions

(1st frame)

Search the position with lowest cost(ith frame)

Search the position with lowest cost(ith frame)

(for i = 2nd~nth frame)

Adaptive Kalman filtering

Adaptive Kalman filtering

Markers’ 2D projected trajectories

Markers’ 2D projected trajectoriesestimated

positions

(ith frame)

Predicted positions in the (i+1)th frame Kalman

predictor

Kalman predictorDelayDelay GUI EditorGUI Editor

3D trajectory estimation Known: pi, pi’, f.

Unknown: mi, mi’, u, d. Calculating 3D positions

via mirror plane estimation.

Properties:– ax+by+cz=d, u=(a,b,c)t, |

u|=1. (1)– mi

’=mi + ku.(2)

– (mi’-)=Hu(mj - ), where Hu

is the Householder matrix.(3)

Fig 3. The geometric representation of physical point m, reflected point m’, and the projected point p and p’.

3D trajectory estimation (cont.)

Deduction from eq. (2),

(4) Calculate the mirror plane from

projected point p and p’ by the least square method.

(5)

3D trajectory estimation (cont.)

Depth of each marker i (zmi and zmi’)can be estimated by the least square method.

(6)

The 3D motion trajectories are reconstructed now.

Extraction of facial motion

marker’s motion = facial motion + head motion.

Head motion (R,t) estimation from specified rigid markers.

ri (j+1) = Rri j + t Remove the head motion.

dispi = R-1(vi(j+1) - t) – vij, where vij is the estimated 3D position of marker i at

time j.

Discussion Unknown: u (4 DOF) R,t (6 DOF) The proposed method can reach an

accuracy with much fewer calibration points.

Our proposed method is more robust in a noisy situation.

0

100

200

300

0 1 2 3

variance of noise (pixels)

the

abso

lute

err

or

proposed method

general purposedmethod via R,testimation

0

100

200

300

10 15 20 25 30

number of calibration points

the a

bso

lute

err

or

proposed method

general purposedmethod via R,testimation

Fig 3. The situation where normal-distributed noise perturbed the estimation of marker motion in video.

Synthetic face Deform a generic model to fit scanned

range image. Radial-basis scatter data interpolation

function. Animation: Morph the head according to

scaled facial motion trajectories.

Synthetic Face

Demo 2: Synthetic face.

Facial Animation

Demo 3: Facial Animation

Seems to be OK, but, how to improve the face

colors?

What’s wrong?

三種膚色： Henrik Jensen 的研究 (2006 EGSR)

皮膚色由：表皮， 真皮層 黑色素， 血紅素，油脂，決定

非洲人含有較多的 Eumelanin 暗沈黑色素，白種人則含有較多的 Pheomelanin 明亮的黑色素 , 而

亞洲人含有兩種相似等量的黑色素

Cm: 暗沈黑色素， Bm: 暗沈VS 明亮黑色素 , Ch: 血紅素

半透明材質： Translucent ，活人與死人的差異

鼻子附近皮膚的毛孔 : (a) 中性表情， (b)氣憤（拉扯皮膚）， (c) 真的照片。

Capture procedure flow diagram

Use a human face as a test 江任遠的臉

散射反射 VS. 似鏡面反射散射反射 (Diffuse Reflection)

與觀測位置無關 :COS (theta)

theta ： 光源與物體平面法向量的夾角

似鏡面反射 (Specular reflection)

與觀測位置有關， COSn(alpha),

alpha: 光源反射角與觀測向量的夾角 , n = 1 to 10000

Use a faked orange as a testleft to right: diffuse, specular,

hybrid, real photo

(a) 散射打光 (Diffuse only) (b) 似鏡面反射 (Specular)

不均勻反射（ Anisotropic reflection ） (a)(b)(c) 三種方向打光效果，注意下排的花

色變化 (I3D2006, 馬萬鈞 )

如何量測人臉的細微毛孔及縐摺？

粗燥皮膚：散射反射 ,

與觀測位置無關 :COS (theta)

theta ： 光源與物體平面法向量的夾角

油脂皮膚：似鏡面反射 (Specular),

與觀測位置有關， COSn(alpha),

alpha: 光源反射角與觀測向量的夾角

從 X ， Y ， Z 三方向產生漸進式亮度 (gradient) 的光源 , 及全

亮度

This would potentially allow to quickly compute the normals by taking only 4 photographs, divide the gradient images by the constant image, and then directly map the range to a normal direction.

Observed reflectance r(x) in X direction vs. normal n(x)

Observed reflectance r(x) in X-direction:The normalized halfway vector between w (light

reflected direction) and v (viewing direction) corresponds to the normal direction.

左側：純散射對法向量的估計產生的結果右側： Specular 反射對法向量的估計產生的結果

高度反光物體也可用此法

Future games? Turing Test

Demo video

The End

Q&A