Jenn-Jier James Lien ()
Kernel Discriminant Analysis Based on Canonical Difference for Face Recognition in Image Sets Wen-Sheng Chu () Ju-Chin Chen () Jenn-Jier James Lien () Robotics Lab, CSIE NCKU CVGIP 2007 Motivation Challenges of face recognition
Facial variations Face recognition using image sets Surveillance Video retrieval illumination pose facial expression Why Multi-view Image Sets?
Multiple facial images contain more information than a single image. Person A Person A Person B Person B A or B? Single input pattern (Single-to-many) Multiple input patterns (Many-to-many) Training/Testing Data: Facial Expression
For subjecti Image Set 1 Training Image Set 2 Image Set 3 Image Set 4 Testing Image Set 5 More Training/Testing Data: Illumination (Yale B)
For subjectj Image Set 1 Training Image Set 2 Image Set 3 Image Set 4 Testing Image Set 5 System Overview T Pi Testing Data Testing image set Xtest
Training Data Testing Data Subject N Xm-2 Xm ... Xm-1 Subject 1 X3 X1 X2 Training image sets {X1,,Xm} Testing image set Xtest ... Testing Process Kernel Subspace Generation Ptest Training Process Kernel Subspace Generation (Total m subspaces) Kernel Discriminant Transformation (KDT) Pi Reference Subspace Reftest X T Reference Subspace: Refi=TTPi Output Identification result Training Process Pi Testing image set Xtest Training Data
Xi={ , ,, } 32 1 2 ni ni ~= 100 Training Data Subject N Xm-2 Xm ... Xm-1 Subject 1 X3 X1 X2 Training image sets {X1,,Xm} Testing image set Xtest ... Testing Process Kernel Subspace Generation Ptest Training Process Kernel Subspace Generation Pi (Total m subspaces) Kernel Discriminant Transformation (KDT) Pi Pi={ei1,,eid} Reference Subspace Reftest X Reference Subspace: Refi=TTPi Identification result Kernel Subspace Generation (KSG)
h ni Nonlinearmappingfunction 32 x 32 ni Kernel subspace of Xi (dImage Set Xi Kernel Matrix Kii Kernel Subspace Pi X1={ , ,, } 1 2 n1 K11 P1= ,d < ni Xm={ , ,, } nm 2 1 Kmm Pm= From the theory of reproducing kernels, Dimensionality may be ! i-th image set where SVD s-th image of i-th image set KPCA: B Scholkopf, A Smola, KR Muller - Advances in Kernel Methods-Support Vector Learning, 1999 Training Process: Kernel Discriminant Transformation (KDT)
Subject N Xm-2 Xm ... Xm-1 Subject 1 X3 X1 X2 Training image sets {X1,,Xm} Testing image set Xtest ... Testing Process Kernel Subspace Generation Ptest Kernel Subspace Generation (Total m subspaces) Kernel Discriminant Transformation (KDT) Pi Reference Subspace Reftest X T Reference Subspace: Refi=TTPi Identification result How to measure similarity?
KDT: Main Idea Based on the concept of LDA, KDT is derived to find a transformation matrix T. We proposed an iterative process to optimize T. Dimensionality of T is assumed to be w. d x w transformation matrix (KDT matrix) Within Subjects Subspace 32x32-dim d-dim w-dim KPCA 1 1 1 T KPCA How to measuresimilarity? Between Subjects 2 2 2 KDT: Canonical Difference (CD) Similarity Measurement
Kernel subspace P1 Canonical subspace C1 Kernel subspace P2 Canonical subspace C2 d1 d2 u1 u2 v1 v2 Capture more common views and illumination than eigenvectors. KDT: CD Canonical Vector v.s. Eigenvector (cont.)
eigenvectors B1 B2 B1B2 canonical vectors C1 C2 A similarity measurement of two subspaces C1C2 KDT: CD Canonical Subspace (cont.)
Consider SVD on orthonormal basis matrices B1 and B2: d-dimensinoal orthonormal basis matrices eigenvector SVD 0 Eigenvalue = cos2i 1 Similarity measurement canonical subspaces (also orthonormal) T.K. Kim, J. Kittler and R. Cipolla, Discriminative Learning and Recognition of Image Set Classes Using Canonical Correlations, IEEE Trans. on PAMI, 2007 KDT: KDT Matrix Optimization
Kernel Subspace Reference Subspace Canonical Subspace T Iterative learning Canonical Difference Based on LDA Kernel subspace Orthonormal basis matrices are required to obtain canonical subspaces Ci. Is Refi normalized? Usually not! KDT: Kernel Subspace Normalization
QR-decomposition is performed to obtain two orthonormal basis matrices. d d invertible upper triangular matrix w d orthonormal matrix SVD KDT: Formulation Canonical Subspace Derivation Qi = TTPiRi-1 SB, Sw
Form of LDA T KDT: Solution T={t1,,tq,,tw}
Contain the info of Dimensionality may be! T T={t1,,tq,,tw} Replace using kernel trick
Derivation Using the theory of reproducing kernels again: Replace using kernel trick Following similar steps, we can obtain KDT: Numerical Issues is solved by simply computing the leading eigenvectors of U-1V. To make sure that U is positive-definite, we regularize U by U (=0.001) where Training Process Refi = TTPi where each element is given by T Pi
Subject N Xm-2 Xm ... Xm-1 Subject 1 X3 X1 X2 Training image sets {X1,,Xm} Testing image set Xtest ... Testing Process Kernel Subspace Generation Ptest Kernel Subspace Generation (Total m subspaces) Kernel Discriminant Transformation (KDT) Refi = TTPi where each element is given by Pi Reference Subspace Reftest X T Reference Subspace: Refi=TTPi Identification result Testing Process T T Pi Testing image set Xtest
Subject N Xm-2 Xm ... Xm-1 Subject 1 X3 X1 X2 Training image sets {X1,,Xm} Testing image set Xtest ... Testing Process Kernel Subspace Generation Ptest Kernel Subspace Generation (Total m subspaces) T Kernel Discriminant Transformation (KDT) Pi X X Reference Subspace Reftest=TTPtest T Reference Subspace: Refi=TTPi Identification result Training List #individual (N) 32 #image set/individual 3
#image/set (ni) ~100 size of normalized template 32x32 dimensionality KMSM 30 KCMSM DCC 20 KDT of Gaussian kernel function 0.05 for regularization 10-3 Training: Convergence of Jacobian Value
J() tends to converge to a specified value under different initializations. Testing: Comparison with Other Methods
The proposed KDT is compared to 3 related methods under 10 randomly chosen experiments. KMSM (avg=0.837) KCMSM (0.862) DCC (0.889) KDT (0.911) Conclusions Canonical differences is provided as a similarity measurement between two subspaces. Based on canonical difference, we derived a KDT and applied it to a proposed face recognition system. Our system is capable of recognizing faces using image sets against facial variations. Thanks for your attention Related Works Mutual subspace method (MSM) Constrained MSM (CMSM)
Subspace V Subspace U project project Constrained Subspace c Uc Vc Discriminantive canonical correlation (DCC) Kernel MSM (KMSM), Kernel CMSM (KCMSM) Mutual Subspace Method (MSM)
Utilize the canonical angles for similarity. Subspace B1 Subspace B2 u1 Eigenvectors u2 v1 u 2 v2 v 1 K. Fukui and O. Yamaguchi, Face Recognition Using Multi-viewpoint Patterns for Robot Vision, ISRR 2003 Perform KDT on Subspace?
By KPCA, we can obtain s.t. Multiply T to both sides of equal sign, It can be observed that the kernel subspace of transformed mapped image sets is equivalent to applying T to the original kernel subspace. Using the theory of reproducing kernels again:
KDT Optimization Using the theory of reproducing kernels again: Following similar steps, we can obtain That is, T={t1,,tq,,tw} where Training: Dimensionality w of KDT V.S. Identification Rate
The identification rate is guaranteed to be greater than 90% after w > 2,200. Training: Similarity Matrix
32 1 Similarity matrix behaves better after 10-times iterative learning. 1 1 32 Similarity Id Number 1st iteration 10th iteration Kij KSG: Kernel Matrix Gaussian kernel function:
Kernel matrix Kij: the correlation between i-th image set and j-th image set. j-th image set ... r 1 2 nj i-th image set Kij ni nj Kernel trick s 1 2 ni