Introduction to my Research

Research Review

Kuo-Yen Lo 羅国彦

ロ　コウエン

2013.4.18Sato Laboratory, University of Tokyo

Short Curriculum Vitae

• Personal Homepagehttp://www.hci.iis.u-tokyo.ac.jp/~kylo/

• Period of past yearsUniversity (2006 – 2010)Internship (2010 – 2011)Research Assistant (2012 – 2013)

• Two main research topic: 1. 2D-to-3D conversion2. Photo Aesthetics.

• Wrap up

Short Curriculum Vitae

• Language: English (TOEIC 935), JLPT(N1), Chinese(Native)

• Programming: C/C++, Matlab, Java(android)Technique: SIFT/SURF/HOG, K-means, GMM, kNN, SVM, PCA/LDA/ITML, bad-of-visual word, bilateral filter.

• Have traveled to: USA, Korea.Want to travel to: China, Thailand, Spain

• Why Japan-- historical and cultural connection-- camera companies and electronics maker here-- founded by Panasonic Scholarship

Visual Cues Low-levelMathematics

Machine LearningPsychology

Computer Vision

Overview

2006NTU

2007UPenn

2008OpenCV

2009RoboticsContest

GenderRecognition

Contest

2012ResearchAssistant

ICPR2012

ACCV_w2012

>>2010ISMAB

NTUGraduate

Internship@ TV corp.

2011Navy

National Taiwan University

2006NTU

2007UPenn

2008OpenCV

>>

People in Vision• Yi-Ping Hung(MM12, CVPR11, CHI11, UIST11)• Yung-Yu Chuang(CVPR12*3)• C.J. Lin (Libsvm)• H.T. Lin(ICML12, NIPS12, CVPR11

KDD12 Champion)• Winston H. Hsu(MM12*6)

33,0001928 B.C.

1.6World rank 80

Summer School in UPenn

2006NTU

2007UPenn

2008OpenCV

>>Summer Language ProgramUniversity of Pennsylvania

Join the Lab

2006NTU

2007UPenn

2008OpenCV

>>Biophotonics and Bioimaging Laboratory

Prof. Ta-Te LinOpenGLOpenCV

BorlandC++ Builder

Robotics Contest @ ASABE 2009

2009RoboticsContest

GenderRecognition

Contest

>>• Problem:

Detecting and Positioning the circular obstacle• Technique:

Graphical simulation(OpenGL), Sensor• Material:

Boe-Bot Toolkit, IR sensor, Ultrasonic sensor, Zigbee wireless communication

Robotics Contest @ ASABE 2009

2009RoboticsContest

GenderRecognition

Contest

>>

Please Visit the following link for viewing the video:

http://youtu.be/8EjON8Y2OJ0

Gender Recognition Contest

2009RoboticsContest

GenderRecognition

Contest

>>• Problem:

Recognize the gender with single face image• Technique:

Viola-Jones Face detector (OpenCV)Feature-based alignmentFalse-alarm check

Rotate the image 35 degreeto detect all possible tilt face


2009RoboticsContest

GenderRecognition

Contest

>>• Problem:



Use Eye detectorto wrap the face tountilt view.


2009RoboticsContest

GenderRecognition

Contest

>>• Problem:



Utilize Skin color model, Eye and Mouth detector to filter the false-positive result from the V-J face detector.


2009RoboticsContest

GenderRecognition

Contest

>>• Problem:



• Performance1.2 second per 480*320 image

• Result~85% face detection accuracy~75% gender recognition accuracyWin 3rd place among 20 teams (Taiwan and China). Bonus 60,0000yen.

Fish Recognition

2010ISMAB

NTUGraduate


>>• Problem:

Tuna species recognition for fishery conservation and management

• Task:Detection and Classification

Bigeye

YellowfinAlbacore

3 spices are considered2011Navy

Fish Recognition>>

• Problem:Tuna species recognition for fishery conservation and management


Fish Image are capturedin certain lighting condition with measurement plate.Body part is smooth,makes it reflect light well.

72%

2010ISMAB

NTUGraduate


2011Navy

Fish Recognition>>

• Problem:Tuna species recognition for fishery conservation and management


B Y AB 89 10 5Y 9 86 3A 10 9 81

84%

34% 72% 52% 58%

Confusion Matrix

(Head)(Abdomen)(Tail fin) (Tail)

Discriminate part!

2010ISMAB

NTUGraduate


2011Navy

Yeh, Graduation!>>

2010ISMAB

NTUGraduate


2011Navy

2D-to-3D conversion>>

2010ISMAB

NTUGraduate


2011Navy

• Problem:Generating 3D videofrom 2D content.

• Inspiration:3D information isrecovered by depthcues

Captured View + Depth


2010ISMAB

NTUGraduate


2011Navy

Reality Comfort

• Accurate depth map• Correct depth order• Real-time processing

• Clear boundary• Temporal smoothness• Visual impression

How people perceive depth?>>

2010ISMAB

NTUGraduate


2011Navy

1. Low-level cue 2. Scene Recognition


2010ISMAB

NTUGraduate


2011Navy

Video frame + motion estimation[ICCE 2009]

Approaches1. Depth map by motion2. Depth map by saliency 3. Depth map by prior

information fusion


2010ISMAB

NTUGraduate


2011Navy Video frame + Saliency map [SDA 2010]

Approaches1. Depth map by motion2. Depth map by saliency 3. Depth map by prior

information fusion

Introduction to Bilateral Filter>>

2010ISMAB

NTUGraduate


2011Navy

Bilateral Filter [Tomasi, ICCV98]:

f(x) h(x)

“Bi” lateral = Spatial term + Range term

Application of Bilateral Filter>>

2010ISMAB

NTUGraduate


2011Navy

“Bi” lateral = Spatial term + Range termSmooth Target Edge-Preserving Result

And this one?


2010ISMAB

NTUGraduate


2011Navy

Prior fusion [Siggraph 2009]1. Decide Geometric perspective2. Integrate Image and Depth map by Bilateral filter

One-year in Navy>>

2010ISMAB

NTUGraduate


2011Navy

Academia Sinica


ICPR2012

ACCV2012

>>Institute of Information Science(Central Research Academy)

People in Vision• Chu-Song Chen (CVPR12, CVPR11*2)• Mark H. Liao (MM12*2, MM11*2)• Y.-C. Frank Wang (ECCV12, CVPR12)• Yen-Yu Lin (CVPR13, MM12, TPAMI11)

Prof. Chen

PHOTO AESTHETICS CLASSIFICATIONPredicting the visual appealing quality of photos

good?>>

jcar@DPChallenge

good?>>

Mnet @ DPChallenge

Which one is better?>>

Voted by online photo community>>

Average: 5.088 votes

Average: 7.292 votes

Reason?>>

Reason?>>

BoundaryAlternating repetition(Texture)

Contrast Levels of scale

Roughness

Strong centers

Positive space

Local symmetries

The Void

Not-separateness

Good shape

GradientsEchoes

Simplicity and Inner Calm

Deep interlock and ambiguity

Color

Composition

HarmoniumRichness

Application>>

Image Search & Management Photo evaluation system

Embedded Camera system Media analysis

Photo Aesthetics


ICPR2012

ACCV2012

>>• Problem:

Recognition the appealing quality ofphoto by computational approaches.

• Technique:Image analysis, Pattern recognition,Crowdsourcing, Psychology, Photography

• Application

ICPR 2012


ICPR2012

ACCV2012

>>As a Pattern Recognition Problem…Comparison of feature 1. Edge distribution, Color histogram,

Hue, Saturation.. [Ke, CVPR06]2. SIFT + BOV [Marchesotti , ICCV11]3. Composition layout (Edge + HSV),

Color palette, contrast.. [Proposed]Result

Item Speed on PC Accuracy

CVPR06 0.2s 81%

ICCV11 4s 85%

Proposed 0.16s 84%

[ Photo aesthetics assessment with efficiency ]

Extraction of Color Information

Extract N Dominant colors

(we set N=5)K-Nearest Neighbor

(K=20)

List of Palettes

Dictionary

HQ Palettes Dictionary

LQ Palettes Dictionary

Palettes of Photo

Retrieved by Frequency

Retrieved by Kmeans(Cluster Center)

Proposed (Weighted Kmeans)

Finding the Dominant Colors

Video Demo


ICPR2012

ACCV2012

>>[ Intelligent Photographing Interface with On-Device Aesthetic Quality Assessment ]

Please Visit the following link for viewing the video:

http://youtu.be/o8mKuTfO6ao

Discussion 1 Device : On-line assistive camera system

• Contextual Information (Viewing angle)

camera < human• Feedback

from analysis to advice• Human behavior

What do people take?How do people take?

• Computation

Server-based v.s. Device• Market and Needs

Discussion 2 Algorithm: photo aesthetic value assessment

• Definition of photo aesthetics

Expert v.s. Volkswagen• Labeling process

Individual bias and variance.Absolute or Relative evaluationEffect of Labeling order

• Quantify photo aesthetic

Modeling, the Personalization

Thanks for your attention!

10 ratings5.00/7 average

5 ratings4.90/7 average

Introduction to my Research

Documents

Transcript of Introduction to my Research