以區域二元圖樣與部分比對為基礎之人臉辨識Face Recognition with Local Binary

Patterns and Partial Matching

Presenter: 施佩汝Advisor: 歐陽明 教授

1

Outlines

• Motivation• Implementation• Result• Conclusion

2

MOTIVATION

3

Motivation

4

Publication

• Che-Hua Yeh, Pei-Ruu Shih, Kuan-Ting Liu, Yin-Tzu Lin, Huang-Ming Chang, Ming Ouhyoung. A Comparison of Three Methods of Face Recognition for Home Photos. ACM Siggraph, poster, 2009.

5

Problem Statement

6

Main Contribution

• Improve Local Binary Patterns by using Partial Matching Metric

• Better Performance in Home Photos

7

IMPLEMENTATION

8

System Overview

Pre-Processing

Build Descriptor

Images Descriptors

ClusteringCalculate

LBPBuild

Descriptor

9

System Overview

Pre-Processing

Build Descriptor

Images Descriptors

ClusteringCalculate

LBPBuild

Descriptor

10

Pre-Processing

11

System Overview

Pre-Processing

Build Descriptor

Images Descriptors

ClusteringCalculate

LBPBuild

Descriptor

12

System Overview

Pre-Processing

Build Descriptor

Images Descriptors

ClusteringCalculate

LBPBuild

Descriptor

13

Local Binary Patterns [PAMI2006]

• An operator to encode the relationship of a pixel and its neighbors. Z1 Z2 Z3

Z8 Z0 Z4

Z7 Z6 Z5

58 99 21

54 54 86

67 12 13

1

otherwise ,0

0 if ,1

28

10

xxs

ZZsLBP p

pp

14

Local Binary Patterns [PAMI2006]

• An operator to encode the relationship of a pixel and its neighbors.

58 99 21

54 54 86

67 12 13

1 1

otherwise ,0

0 if ,1

28

10

xxs

ZZsLBP p

pp

Z1 Z2 Z3

Z8 Z0 Z4

Z7 Z6 Z5

15

Local Binary Patterns [PAMI2006]

• An operator to encode the relationship of a pixel and its neighbors.

58 99 21

54 54 86

67 12 13

1 1 0

otherwise ,0

0 if ,1

28

10

xxs

ZZsLBP p

pp

Z1 Z2 Z3

Z8 Z0 Z4

Z7 Z6 Z5

16

Local Binary Patterns [PAMI2006]

• An operator to encode the relationship of a pixel and its neighbors.

58 99 21

54 54 86

67 12 13

1 1 0

1

otherwise ,0

0 if ,1

28

10

xxs

ZZsLBP p

pp

Z1 Z2 Z3

Z8 Z0 Z4

Z7 Z6 Z5

17

Local Binary Patterns [PAMI2006]

• An operator to encode the relationship of a pixel and its neighbors.

58 99 21

54 54 86

67 12 13

1 1 0

1

0

otherwise ,0

0 if ,1

28

10

xxs

ZZsLBP p

pp

Z1 Z2 Z3

Z8 Z0 Z4

Z7 Z6 Z5

18

Local Binary Patterns [PAMI2006]

• An operator to encode the relationship of a pixel and its neighbors.

58 99 21

54 54 86

67 12 13

1 1 0

1

0 0

otherwise ,0

0 if ,1

28

10

xxs

ZZsLBP p

pp

Z1 Z2 Z3

Z8 Z0 Z4

Z7 Z6 Z5

19

Local Binary Patterns [PAMI2006]

• An operator to encode the relationship of a pixel and its neighbors.

58 99 21

54 54 86

67 12 13

1 1 0

1

1 0 0

otherwise ,0

0 if ,1

28

10

xxs

ZZsLBP p

pp

Z1 Z2 Z3

Z8 Z0 Z4

Z7 Z6 Z5

20

Local Binary Patterns [PAMI2006]

• An operator to encode the relationship of a pixel and its neighbors.

58 99 21

54 54 86

67 12 13

1 1 0

1 1

1 0 0

otherwise ,0

0 if ,1

28

10

xxs

ZZsLBP p

pp

Z1 Z2 Z3

Z8 Z0 Z4

Z7 Z6 Z5

21

Local Binary Patterns [PAMI2006]

• An operator to encode the relationship of a pixel and its neighbors.

58 99 21

54 54 86

67 12 13

1 1 0

1 1

1 0 0

otherwise ,0

0 if ,1

28

10

xxs

ZZsLBP p

pp

Z1 Z2 Z3

Z8 Z0 Z4

Z7 Z6 Z5

LBP = 1101001122

System Overview

Prepared-Works

Build Descriptor

Images Descriptors

ClusteringCalculate

LBPBuild

Descriptor

23

Facial Image Descriptor

• They use Spatially Enhanced Histogram in original Local Binary Pattern. [PAMI2006]

24

Local Patches

• We sample a patch for every s pixels.• There are S patches for one image.

mm

s s

25

Spatial Block [CVPR2007]

• We use three concentric circles to describe a patch.

26

Descriptor

• Build a descriptor for one face.

27

KKKK

K

K

fff

fff

fff

21

22221

11211

System Overview

Pre-Processing

Build Descriptor

Images Descriptors

ClusteringCalculate

LBPBuild

Descriptor

28

System Overview

Pre-Processing

Build Descriptor

Images Descriptors

Complete-Linkage Clustering

Calculate LBP

Build Descriptor

Compute all the similarities

Clustering

29

System Overview

Pre-Processing

Build Descriptor

Images Descriptors

Complete-Linkage Clustering

Calculate LBP

Build Descriptor

Compute all the similarities

Clustering

30

Similarity

• They use the weighted Chi-Square distance in original Local Binary Pattern. [PAMI2006]

•

j ijij

ijij

iiw MS

MSwMS

2

2 ,

31

Partial Matching [ICCV2009]

• Step1:– Compute the similarity of each patch from one

image with the nearby patches in another image.

,

Image 1: I(1) Image 2: I(2)

)2()1(||;|:|,

)1( ,min klijrslsjrsksilkij ffsimfd

32

Partial Matching [ICCV2009]

• Step2:– Sort the similarities of all patches.

– dαS is the similarity of I(1) to I(2). K

jiijsS fdSortdddd1,

)1(21 ,...,,...,,

SdIId )( )2()1(

33

Partial Matching [ICCV2009]

• Step3:– Calculate the similarity of I(2) to I(1)

)( )1()2( IId

34

Partial Matching [ICCV2009]

• Step4:– Use the maximum of two similarity

)(),(max),( )1()2()2()1()2()1( IIdIIdIID

35

System Overview

Pre-Processing

Build Descriptor

Images Descriptors

Complete-Linkage Clustering

Calculate LBP

Build Descriptor

Compute all the similarities

Clustering

36

Hierarchical Clustering

• Build a tree-based hierarchical taxonomy (dendrogram) from a set of documents.

Material Selected from Introduction to Information Retrieval, Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, 2008.

37

Hierarchical Clustering

• Clustering obtained by cutting the dendrogram at a desired level: each connected connected component forms a cluster.

Material Selected from Introduction to Information Retrieval, Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, 2008.

38

Hierarchical Complete-Linkage Clustering

• Similarity of the “furthest” points.• Makes “tighter,” spherical clusters that are

typically preferable.

Material Selected from Introduction to Information Retrieval, Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, 2008.

39

Performance Optimization

• 4 threads in Quad-Core system

Pre-Processing

Build Descriptor

Images Descriptors

Complete-Link Clustering

Calculate LBP

Build Descriptor

Compute all the similarities

Clustering

40

Performance Optimization

• 4 threads in Quad-Core system– 3 times faster than single thread.

• 73 minutes to 24 minute for 309 images.

Pre-Processing

Build Descriptor

Images Descriptors

Complete-Link Clustering

Calculate LBP

Build Descriptor

Compute all the similarities

Clustering

41

RESULT

42

FERET Result

• fa: gallery, 994 images• fb: alternative facial expression, 992 images• dup1: the photos taken after later, 736 images• dup2: the photos taken at least one year after

the gallery, 228 images

43

FERET Result

44

Accuracy fb dup1 dup2

LBP [PAMI2006] 95.67% 59.92% 45.61%

Our Result 98.89% 71.33% 68.42%

Time※ Register fb dup1 dup2

LBP [PAMI2006] 46 seconds 102 seconds 67 seconds 21 seconds

Our Result 4 minutes 224 minutes 168 minutes 5 minutes

※ The time results are computed in multithreads version.

Experiments

• Home Photo Dataset I– 309 images, 5 subjects

• Home Photo Dataset II– 838 images, 8 subjects

45

Evaluation

• Cluster Number• Unknown Number• Pair-wise Precision• Pair-wise Rand Index• Executing time

46

Unknown Number

• The number of clusters which contain only one component.

47

Precision/Rand Index

fptp

tpprecision

fntp

tprecall

tnfpfntp

tntpindexrand

48

Assigned

Same Cluster

Different Clusters

Ground Truth

Same Cluster tp fn

Different Clusters fp tn

Dataset-I Result

#Clusters #Unknown Precision Rand Index

Time ※

Picasa Web 94 73 99.92% 0.829816 10 seconds

Picasa PC 99 75 100% 0.865002 3 minutes

LBP [PAMI2006] 100 31 90.39% 0.811856 4 seconds

LID_PM

[ICCV2009]

99 37 98.78% - 11 minutes

Our Result 100 39 99.46% 0.816290 24 minutes

LID+PM

[Chang2010]

100 43 99.24% - 25 minutes

LBP: Local Binary Pattern, PM: Partial Matching, SB: Spatial Block

※ The time results are computed in multithreads version.49

Dataset-I Result

50

LBP Our Result

Dataset-I Result

• Wrong clustering result in LBP

• The clusters in our result

51

Dataset-II Result

#Clusters #Unknown Precision Rand Index

Time ※

Picasa Web 195 47 99.49% 0.876229 10 seconds

Picasa PC 253 150 100% 0.888492 10 minutes

LBP [PAMI2006] 253 45 91.70% 0.70261 17 seconds

LID_PM [ICCV2009] 253 79 97.88% - 51 minutes

Our Result 253 64 99.59% 0.871187 163 minutes

LID+PM

[Chang2010]

253 89 99.82% - 136 minutes

LBP: Local Binary Pattern, PM: Partial Matching, SB: Spatial Block

※ The time results are computed in multithreads version.52

Dataset-II Result

53

LBP Our Result

Dataset-II Result

• Wrong clustering result in LBP

• Clusters in our result

54

Demo

• Face Recognition with web camera

55

CONCLUSION

56

Conclusion

• LBP is an efficient algorithm for face recognition.

• Partial Matching is good for the different facial expression or different illumination in facial images.

• Our system has better performance than LBP.

57

Future Works

• Improve performance by GPU.

• Use other extension of LBP combined with Partial Matching.

58

Thank You for Your Attention!

59

Result

LBP: Local Binary Pattern, PM: Partial Matching, SB: Spatial Block

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20.700000000000001

0.750000000000001

0.800000000000001

0.850000000000001

0.900000000000001

0.950000000000001

1

Home Photo Dataset I

LBP LBP+SB LBP+PM LBP+LID+SB

Recall

Pre

cisi

on

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20.700000000000001

0.750000000000001

0.800000000000001

0.850000000000001

0.900000000000001

0.950000000000001

1

Home Photo Dataset II

LBP LBP+SB LBP+PM LBP+SB+PM

Recall

Pre

cisi

on

60