LOGO On Person Authentication by Fusing Visual and Thermal Face Biometrics Presented by: Rubie F. Vi...

LOGO On Person Authentication by Fusing Visual and Thermal Face

Biometrics

On Person Authentication by Fusing Visual and Thermal Face

Biometrics

Presented by: Rubie F. Viñas, 方如玉

Adviser: Dr. Shih-Chung Chen, 陳世中

www.themegallery.com

Company Logo

Outline

I. Abstract

II. Introduction

III. Method Overview

IV. Empirical Evaluation

V. Conclusion


Company Logo

I. Abstract

Recognition Algorithms Use data obtained by imaging faces in the thermal spectrum are

promising in achieving invariance to extreme illumination changes that are often present in practice.

Paper Analyze

• Performance– Recently: Proposed face recognition algorithm

» Combination: Visual and Thermal modalities by “Decision Level Fusion”. Examine:

• Effects of the proposed data preprocessing in each domain.• Contribution to improved recognition of different types of features• Importance of prescription glasses detection

– Recognition Vs. Verification» 1-to-N matching» 1-to-1 matching

Discuss:• Significance of the results

– Identify a number of limitations of the current state-of-the-art face recognition algorithm.

– Propose promising directions for future research.


Company Logo

II. Introduction

Optical Imager Sensor used in face biometric systems.

• Why?– Availability– Low-cost

Function• Captures the light reflectance of the face surface in

the visible spectrum. • Visible Spectrum

Problem• Ambient Light

sassyrubie

Provides features that depend only on surface reflectance. Thus, it is obvious that the face appearance changes according to the ambient light.Over the years a number of methods have been proposed to solve this problem [1] [4] [7] [19], but it still remains challenging in most practical circumstances (e.g. [3]).


Company Logo

II. Introduction (cont.)

Advantages of Thermal Spectrum Vs. Visible Spectrum Invariance Ambient Illumination changes

• Thermal Infrared Sensor – Measures heat energy radiation

» Emitted by the face

• Appearance-based face recognition algorithms – Applied to thermal IR imaging – Consistent Performance

» Under various lighting conditions and facial expressions.

Visual Spectrum

Thermal Spectrum

sassyrubie

Invariance to ambient illumination changes This is due to the fact that a thermal infrared sensor measures the heat energy radiation emitted by the face rather than the light reflectance. Appearance-based face recognition algorithms applied to thermal IR imaging consistently performed better than when applied to visible imagery, under various lighting conditions and facial expressions [9] [14] [16] [13]. Further performance improvements were achieved using fusion of different modalities [5] [12] [6] [8] [2].

sassyrubie

Invariance: Illumination changes have dramatic effects on data acquired in the visible spectrum (top row). In contrast, thermal imagery (bottom row) shows remarkable invariance.


Company Logo

III. Method Overview

System Overview

1. Data Preprocessing and Registration

2. Glasses Detection

3. Fusion

sassyrubie

In this section we briefly explain the system of Arandjelovic. Its main components are conceptually shown in Fig. 2. At the lowest level, image sets – rasterized (converted into pixels) images in either the visual or thermal spectrum – are matched using the cosine of first principal angle µ between the corresponding Principal Component Analysis (PCA) subspaces U1 and U2.The evaluated system consists of three main modules performing Data preprocessing and registrationGlasses detectionFusion of holistic and local face representations using visual and thermal modalities.


Company Logo

START

RASTER IMAGEVISUAL

SPECTRUM

RASTER IMAGE

THERMAL SPECTRUM

AFFINE WARPING

HOLISTIC FACE

APPEARANCE

LOCAL PATCHES

LOCATION DETECTION:

EYESMOUTH

OFFLINE GALLERY

CAMERA LIVE FEED

SET MATCHING

SET MATCHING

SET MATCHING

AFFINE WARPING

HOLISTIC FACE

APPEARANCE

LOCAL PATCHES

LOCATION DETECTION:

EYESMOUTH

OFFLINE GALLERY

USE GLASSES

YES

NO

USE ALL PARAMETE

RS

PATTERN MATCHING

PATTERN MATCHING

we=0

ESTIMATION ESTIMATION

END

sassyrubie

Feature fusionTwo types of features are used in set matching: Holistic face appearance is matched after affine warping using the locations of the two detected eyes and the mouth, see Fig. 3. Local patches used correspond to the same three detected regions, see Fig. 4.RegistrationOriginal image with detected facial features marked by yellow circles (left), result of affine warping the image to canonical frame (centre) and the cropped result.FeaturesIn both spectra, matching is done using holistic appearance and appearance of 3 salient facial features.


Company Logo

III. Method Overview (cont.)

Set-up Matching (Feature Fusion) Similarity Score

• Two images using only a single modality (visual or thermal)• Constant-Weighted Summation:

– Where:» pv visual score» pt thermal score» pm mouth score» pe eyes score» ph holistic score» wm mouth weighing constantVisual Spectrum0.0 Thermal Spectrum0.1» we=1-wh-wm eyes weighing constant» wh holistic weighing constantVisual Spectrum0.7 Thermal Spectrum0.8

Note: Choice of values are recommendations from the Arandjelovic System

. . (1 ).v h h m m h m ew w w w

. . (1 ).t h h m m h m ew w w w


Company Logo


Set-up Matching (Modality Fusion)

( ). (1 ( )).f v v v v v tw w Similarity Score • Visual and Thermal data• Joint Similarity

Where:• pf Fusion score

• wv=wv(pv) Visual weighing function

• wt Thermal weighing function


Company Logo


Visual Weighing Function Estimation wv=wv(pv)

i. Estimate – Probability that wv is the optimal weighting given

– Iterative Algorithm– Unknown Image vs. Offline Gallery

ii. Compute w(pv) – Maximum Posteriori

iii. Make an analytic fit – Obtain marginal Distribution

ˆ ( , )v vp w ˆvp


Company Logo


Prescription Glasses Thermal Image exact opposite of the Visual Image

• Thermal Spectrum – Dark patches

Practical Importance• US (year 2000) ~ 96 million people (34% total population)

– Wear prescription glasses

System• Appearance Distortion that glasses cause in thermal imagery

– Help recognize by detecting their presence.– Subject

» Not wearing glasses use all parameters » Wearing Glasses we=0 (thermal image)


Company Logo

IV. Empirical Evaluation

Old System to Evaluate Dataset 02: IRIS Thermal/Visible Face

Database• Subset of the Object Tracking and Classification

Beyond the Visible Spectrum (OTCBVS) database1

• Available for download – http://www.cse.ohio-state.edu/OTCBVS-BENCH/

• Contents– 29 individuals

» 11 roughly matching poses *Visual and Thermal spectra and large illumination

variations


Company Logo

IV. Empirical Evaluation (cont.)

Algorithm Trained

• Use all images– Single illumination– 3 Salient facial features could be detected

Result• 7-8 Visual Images• 6-7 Thermal Images

sassyrubie

Training setsHistograms of the number of images per person used to train the algorithm.


Company Logo


After evaluation Needs Performance improvement

• Solution– Band-pass and Self-Quotient Image Filters

Recognition Performance• Use

– Individual local features and fusion with holistic face appearance

Importance• Prescription Glasses

1 2* *BP w wI I G I G

2. / ( * )SQI BP wI I I G


Company Logo


Algorithm Performance Evaluate

• 1-to-N matching scenario– Test data

» One of the people in the training set– Recognition

» Associating it with the closest match

• 1-to-1 matching scenario (Verification)– To know if the person is the same– Performance was quantified

» Looking at the true positive admittance rate » Threshold 1 admitted intruder in 100


Company Logo


Results

Summary of 1-to-N matching results

Poor Performance

Improved Performance (35%)

Improved Performance (47%)

sassyrubie

Firstly, note the poor performance achieved using both raw visual as well as raw thermal data. The former is suggestive of challenging illumination changes present in the OTCBVS data set.The greatest power of the method becomes apparent when the two modalities, visual and thermal, are fused. In this case the role of the glasses detection module is much more prominent, drastically decreasing the average error rate from 10% down to 3%, see Tab. 1. Similarly, the true admission rate increases to 74% when data is fused without special handling of glasses, and to 80% when glasses are taken into account.


Company Logo


sassyrubie

Band-pass filterThe optimal combination of the lower and upper band-pass filter thresholds is estimated from a small training corpus. The plots show the recognition rate using a single modality, (a) visual and (b) thermal, as a function of the widths of the two Gaussian kernels. It is interesting to note that the optimal band-pass filter for the visual spectrum passes a rather narrow, mid-frequency band, whereas the optimal filter for the thermal spectrum is in fact a low-pass filter.


Company Logo


Holistic representations Receiver-Operator Characteristics (ROC)

Visual (blue) and thermal (red) spectra.


Company Logo


Low recognition rate of Raw Thermal Imagery Two main limitations of this modality

• Inherently lower discriminative power • Occlusions caused by prescription glasses

– Addition: Glasses Detection Module

• Little help

• Benefit

*Steering away from misleadingly good matches between any two people wearing glasses

• Limitation

* Discriminative region of the face is lost.

• Modest Improvement Optimal band-pass filtering in thermal imagery


Company Logo


Isolated local features Receiver-Operator Characteristics (ROC)Visual (blue) and thermal (red) spectra

sassyrubie

Neither the eyes or the mouth regions, in either the visual or thermal spectrum, proved very discriminative when used in isolation, see Fig. 9. Only 10-12% true positive admittance was achieved, as shown in Tab. 3. However, the proposed fusion of holistic and local appearance offered a consistent and statistically significant improvement. In 1- to-1 matching the true positive admittance rated increased for 4-6%, while the average correct 1-to-N matching improved for roughly 2-3%.


Company Logo


Glasses detection results

Inter- and intra- class similarities

sassyrubie

Finally, we note that the performance of the glasses detection module on this dataset was virtually perfect, incorrectly classifying an instance of a person without glasses only in a single case, see Fig. 10.


Company Logo

V. Conclusion Analyzed and Empirically evaluated

Recently proposed system • Personal identification

– Face biometrics from visual and thermal imagery

Results Filters can be used

• Greatly improve recognition accuracy in both domains

Little improvement is seen• Inclusion of local feature-based patches

Proposed Algorithm Detection of glasses• Very reliable across individuals and different imaging conditions

Fusion of visual and thermal imagery• Very promising in practical applications• Two Modes

– Offer discriminative power in complementary ways


Company Logo

V. Conclusion (cont.)

Verification Setup Evaluated method

• 80% correct admittance rate (1% intruder admittance)– Problem

» Much lower rate than that required in most practical applications

*Making the system useful only in the pre-screening process

Much better performance was achieved in 1-to-N matching evaluation

• High correct identification rate of 97% was obtained – Using

» Small number of training images 5-7

» Presence of large illumination changes


Company Logo

V. Conclusion (cont.)

Recommendations Further use of the thermal spectrum

• Segmenting out the glasses – Holistic appearance can still be used in matching

» Challenge: Presence of extreme poses

*Glasses merge with the background with more profile views

Different representation of local appearance• Possibly offer further benefit with large pose

changes


Company Logo

THANK YOU!!!

LOGO On Person Authentication by Fusing Visual and Thermal Face Biometrics Presented by: Rubie F. Vi...

Documents

Transcript of LOGO On Person Authentication by Fusing Visual and Thermal Face Biometrics Presented by: Rubie F. Vi...