Structured Face Hallucination

Chih-Yuan Yang Sifei Liu Ming-Hsuan Yang

Electrical Engineering and Computer Science

1

Outline• Motivation• Related work• Proposed method • Experimental results• Conclusions

2

Motivation• Generate high-quality face images

3

Algorithm

Challenges

4

• How to effectively model a face?– Landmark points

• How to preserve the consistency of details?– Transfer details of a whole component– Maintain consistency of edges in

upsampling– Exploit statistics of edge sharpness

Face Hallucination [Liu07]• PCA on intensities

– Global constraint• MRF on residues

– High-frequency details• Bilateral filtering as post-processing

– Suppress ghost effects

5

Sparse Representation [Yang08]• NMF on intensity

– Global constraint• Patch mapping through a pair of

sparse dictionaries– Restore the high-frequency details

6

Position Patch [Ma10]• No global constraint• Only local constraint by patch

position– Only use exemplar patches at the same

position– Weighted averaging exemplar patches

7

Proposed Approach

• Three classes– facial components

• Transfer the HR details from the whole region of a component– edges

• Preserve edge structures and restore sharpness by statistical prior– smooth regions

• Transfer the HR details from small patches

Aligning Component Exemplars• Exemplar images are labeled • Each component is aligned individually

9

Align high-resolution exemplar images: coordinates of landmark points

Generate low-resolution exemplar images

Search for the most similar exemplar

Insights• Consistency

– Consistent details because the whole component is transferred

– The pair of eyes is considered as one component, as well as the eyebrows

• Effectiveness– Landmark points enable the comparison for

a whole component– Effective for various shapes, sizes, and

positions

10

Preserve Edge Structures• Direction-Preserving Upsampling

11

Directional similarity in LR patches

Bilinear interpolation preserves the directional similarity in HR

Regularize theHR image

Restore Edge Sharpness

12

upsampled edge center mag. of grad. enlarged restored restored

Statistical priors

𝑑=0 𝑑=1 𝑑=√2

Smooth Regions• Approach

– Find the most similar LR patch and transfer the HR gradients

• Advantage– Highly adaptive

• Achieved by– PatchMatch

algorithm– Low computational

load

• Restriction– Consistency

– Accuracy

13

Patch only Component Exemplar

Patch only Edge Model and Priors

Generate Output Images

14

𝑤𝑐 𝑤𝑒 𝑈𝑒 𝑈𝑏𝑈𝑐𝑈

Merge gradient maps

Generate output images

Experimental Results

15

16

17

Conclusions• Structured face hallucination

– Effective whole component exemplars– Preserved edge structures and robust

statistical sharpness priors• Preliminary results

– Effective and consistent high-frequency details

– Robustness

18