視覚情報処理論

(Visual Information Processing )開講所属: 学際情報学府

水(Wed)5 [16:50-18:35]

Schedule• 9/ 26 Introduction (Prof. Oishi)• 10/3 Patch-based Object Recognition (1) (Dr. Kagesawa)• 10/10 Patch-based Object Recognition (2) (Dr. Kagesawa)• 10/17 Computer Vision basics (1)(Prof. Oishi)• 10/24 Computer Vision basics (2)(Prof. Oishi)• 10/31 Image and Video Inpainting (1) (Dr. Roxas) (※in English)• 11/7 Image and Video Inpainting (2) (Dr. Roxas) (※in English)• 11/14 (Cancelled)• 11/21 Vision for Robotics Applications (1) (Dr. Sato)• 11/28 Vision for Robotics Applications (2) (Dr. Sato)• 12/5 3D Data Visualization (1) (Dr. Okamoto)• 12/12 3D Data Visualization (2) (Dr. Okamoto)• 12/19 3D Data Processing (1) (Prof. Oishi)• 1/9 3D Data Processing (2) (Prof. Oishi)

Computer Vision Paradigm (Marr)

2.5D Image

2D Image

3D representation

Integration

Brightness Texture Line drawing Stereo Motion

Observer oriented

3D Feature Extraction(shape-from-x)

Object oriented 3D Model

Digital image processing (2D)

What is digital image?Analog information (Film, Painting, Real world)

Digital image• Digital camera• Smart phone• PC data, IT• Digital broadband

Discretization & Sampling

SamplingDiscrete segmentation of analog data

Analog data(Time and value are sequential)

Sampling data(Time is discrete)

Sampling interval

Sampling2D digital image

Image resolution is defined by sampling interval

What is pixel?Unit of 2D digital image Space sampling

0 1 N-1

0

1

M-1

columns

rows

Digital imageM x N pixels

n

m

Sampling-Resolution

320 x 240pixels

160 x 120pixels

80 x 60pixels

40 x 30pixels

QuantizationSampled values are discretized

Sampled data(Time line is discrete)

Quantization bit:3 bit = 8 level8 bit = 256 level

Digital data(Both time and value are discrete)

Quantization2-D digital image

Number of color depends on quantization bit

0

0

0

1

1

1

1

0

0

1

2

2

2

1

0

0

2

3

3

2

1

0

2

3

5

3

2

0

0

2

3

3

3

2

0

0

1

2

2

2

0

0

0

1

1

1

0

0

0

Color is represented by number

Color representationHow many colors do we need?

4colors(2bi)

16colors(4bit)

256colors(8bit)

16.7 millioncolors(32bit)

High Dynamic Range Imaging: HDRI

Exposure time - Intensity [Mathias Eitz, Claudia Stripf,High Dynamic Range Imaging, 2007]

Under Exposure Over Exposure

Dynamic range

Human

Camera

Multiple capturing

Camera response function

Exposure Exposure

Estimation of camera response functionCapturing multiple images with different exposure time

Computation of response curve

Zij = f (Eitj )f −1 (Zij ) = Eitj

ln f −1 (Zij ) = ln Ei + lntj

Log Exposure

Zij : Pixel valuef : Camera response functionEi : Radiancetj : Exposure time

Displaying HDRI

HDRI

LDRI

Tone mappingLinear mapping Logarithmic mappingGlobal Reinhard operator

L (x, y) = L(x,y) / 1+L(x,y)

Results of tone mapping

without tone mapping with tone mapping

HDRI Video [Kalantari et al. Patch-Based High Dynamic Range Video, TOG 2013]

Filtering

FilteringPre-processing for Computer Vision

• Noise reduction• Image enhancement• Feature extraction

FILTER ?

Spatial – Frequency filterProcessing in spatial domain

• Neighboring pixels

Processing in frequency domain• Using Fourier Transform

Image NoiseNoise source

• Capturing

• Compression/Transfer

Mean filterReplace value with mean of neighboring points

0 5

5

3

3

1

4

10

8

8

7

6

8

5

0

9

4

8

5

9

10

9

7

7

5 3 x 3(5 x 5)(7 x 7)

1 / 9

1 / 9

1 / 9

1 / 9

1 / 9

1 / 9

1 / 9

1 / 9

1 / 9

10 / 9

8 / 9

8 / 9

8 / 9

5 / 9

0 / 9

9 / 9

7 / 9

7 / 9

0 5

5

3

3

1

4

10

8

8

7

6

8

5

0

9

4

8

5

9

10

9

7

7

5

7

8

Mean filterWeighted average

0 5

5

3

3

1

4

10

8

8

7

6

8

5

0

9

4

8

5

9

10

9

7

7

5

4 / 16

2 / 16

2 / 16

2 / 16

1 / 16

1 / 16

2 / 16

1 / 16

1 / 16

40 / 16

16 / 16

16 / 16

16 / 16

5 / 16

0 / 16

18 / 16

7 / 16

7 / 16

0 5

5

3

3

1

4

10

8

8

7

6

8

5

0

9

4

8

5

9

10

9

7

7

5

8

6

Mean filter (Smoothing, Averaging)for Gaussian noise

Noise image(5% Gaussian)

Average Weighted average

Mean filterEx. Shot noise

Noise image(Random binary)

Average Weighted average

Non-linear filterMaximum filter

• Replace target value with maximum value in a window

Minimum filter• Replace target value with minimum value in a window

Median filter

1098887750

7859108780

ソート 中央値

Median filterReplace target value with median value in a window

0 5

5

3

3

1

4

10

8

8

7

6

8

5

0

9

4

8

5

9

10

9

7

7

5

0 5

5

3

3

1

4

10

8

8

7

6

8

5

0

9

4

8

5

9

10

9

7

7

5

8

8

Median filter3 x 3 Filter

Shot noiseGaussian noise

Edge detection

Edge typesStep edge

Roof edge

Peak edge

x

x

x

1-D edge differentialFirst and second order differentials

Fig. from Digital Image Processing (Springer)

Original signal

First order

Second order

Gradient-baseOperator of first order differential

Discrete difference equation

y

f

x

fyxf ,,

nmfnmfnmf

nmfnmfnmf

y

x

,1,,

,,1,

2 x 2 size

1,1,,

,1,1,

nmfnmfnmf

nmfnmfnmf

y

x

3 x 3 size

Strength and direction of edge

Gradient-baseOperators

• Roberts

• Prewitt

• Sobel

10

01

01

10\/ DD

111

000

111

101

101

101

yx DD

121

000

121

101

202

101

yx DD

Gradient-basePrewitt operator

Dx Dy

Laplacian operatorOperator of second order differential

Strength of edge is estimated

010

141

010

2

2

1

121222yx DD xxx DDD

2

010

141

0102

111

181

1112

4 direction 8 direction

yyy DDD

2

Laplacian operatorLaplacian operator

4 direction 8 direction

Laplacian of GaussianDifferential operation is weak to noiseGaussian filter (noise reduction) -> Laplacian operator

Laplacian of Gaussian 222 2/

2

1,G

yxeyx

222 2/2

22

42 2

2

1,G

yxeyx

yx

Laplacian of GaussianLOGオペレータ

1 2

Line drawing analysis

Line drawing extraction

Original image Differential image Line drawing image

3D Information form Line DrawingGiven

• Line drawing(2D)Find

• 3D object that projects to given lines

Find• How do you think it’s a cube,

not a painted pancake?

Line types

convex concave

occluding occluding

Labeling a Line Drawing

Easy to label lines for this solid→Now invert this in order to understand shape

Enumerating Possible Line Labeling without Constraints

•9 lines•4 labels each

→4ｘ4ｘ4ｘ4ｘ4ｘ4ｘ4ｘ4ｘ4= 262,144 possibilitiesWe want just one reality

must reduce surplus possibilities→Need constraints (by 3D relationship)

Huffman & Clows Junction DictionaryAny other arrangements

cannot ariseHave reduced configuration

from 208 to 12

• L-type － ６

• ARROW-type － ３

• FORK-type － ３

Constraints on LabelingWithout constraints-- 262,144 possibilitiesConsider →3ｘ3ｘ3ｘ6ｘ6ｘ6ｘ3＝ 17496 possibilitiesconstraints

We can reduce more bycoherency/consistency along line.

Labeling by Constraint Propagation“Waltz filtering”By coherence rule, line label constrains neighborsPropagate constraint through common vertexUsually begin on boundaryMay need to backtrack

Impossible objectsNo consistent labelingBut some do have a consistent labeling

• What’s wrong here?

Limitations of Line LabelingOnly qualitative; only gets topologySomething wrong

Color theory

Color Theory for Computer VisionColor in several domains:

• Physics• Human vision• Psychophysics• Perception• Computer Vision

Color problems in Computer Vision:• Color for segmentation• Color for reflection physics

Color spectrum

Intensity at each wavelength

RGB imageRGB color model

r=255g=5b=10

DSC(Digital Still Camera)

Spectrum is compressed to three color valuesResponse function

IlluminationSpectrum is richer than RGB

Are RGB enough?5900K light

MetamerismNatrium light

Standard illumination

D50 light

Spectral distribution measurement

Interference CameraSpectrum varies along the position

Interference filter

Y

Panoramic Multispectral Imaging SystemLCTF Capturing System

Automatic Pan/Tilt Platform

LCTF Capturing system

・・・

t (s)400nm ～ ～720nm

Target scene

LCTFMonochromatic CCD camera

400nm404nm408nm416nm・・・nm・・・nm・・・nm・・・nm・・・nm・・・nm・・・nm712nm716nm720nm

PC

[Tominaga et al. 00]

Tumulus and hill

In what condition painted?under sun-lightunder torch?

� U Tokyo / Topan / Kyushu National Museum

Simulation ResultsSimulation results suggest that

• Painted most likely under sun light• First paints, and then covers the tumulus

Torch Sun light

Point light source(Incandescent)

Spectral measurement sensor

Target object: Tomato

RGB camera

Data analysis

Spectral measurement of Aging process

Measurement time: every 12 hours in 14 days

0

0.05

0.1

0.15

0.2

0.25

380 480 580 680

波長(nm)

分光反

射率

Temporal variation

1st principal component proportion ： 61.1％Regressioncurve： -0.1996240.0153333t

3t0.00013939358t0.00000773 231

Y

2nd principal component proportion ： 23.3％Regressioncurve： 0.008506940.0720887t0.0225962t

0.002247t86t0.00008970-553t0.000001252

3452

Y

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

380 480 580 680

波長(nm)

主成分の係数

第一主成分

第二主成分

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0 5 10 15 20 25

日数

得点

第一主成分得点

第二主成分得点

第一主成分得点（回帰曲線）

第二主成分得点（回帰曲線）

Principal component analysis

Reflectance image

？

Color image Texture image Reconstructed image

3D model rendering

Human visionRetina

Retina has 4 cells

•“red” cone cell•“green” cone cell•“blue” cone cell• Rod cell Intensity

Color

Human vision

380nm 760nm

)(bC

)(gC

)(rC

Response of red cone =

Response of green cone =

Response of blue cone =

dECr )()(760

380 dECg )()( dECb )()(

Color spacered = green =blue =

dECr )()( dECg )()( dECb )()(

If we approximate spectral power distribution by vector, it’s a matrix multiplication.

)(E

red greenblue

=)(rC)(gC)(bC )(E

13 3 1

spectral space : infinitely many dimensionscolor space : 3 dimensions

Alternate color spaceother isomorphic color spaces formed by linear transforms

red greenblue

=)(rC)(gC)(bC )(E

define new axesABC

=red greenblue

=

=

)(rC)(gC)(bC )(E

)(a)(b)(c )(E

linear transform gives new axes

new response function

green

bluered

A

C B

Psychophysical color (X-Y-Z)international standard color space agreed upon byCommision Internationale de I’Eclairage (CIE)• particular linear transform of human cone responses• Two spectral distributions that result in the same values in the

space appear indistinguishable • all colors have positive x, y, zEach point in X-Y-Z is a different colorChromaticity

x = X / (X+Y+Z) ≒ R / (R+G+B)y = Y / (X+Y+Z) ≒ G / (R+G+B)z = Z / (X+Y+Z) ≒ B / (R+G+B)since x+y+z = 1, z = 1-(x+y). --- redundant usually plotted o x-y diagram

Each point is many XYZ colors

Chromaticity diagram

r = R / (R+G+B)g = G / (R+G+B)b = B / (R+G+B)

Color perceptionHow do people describe color ?NOT “X-Y-Z” nor “R-G-B” !People use cylindrical coordinates.hue, saturation, brightness

B H

S

blue

white

violetred

yellow

green

SH

One plane of constant brightness

hue+saturation form polar coordinates

relationship to red-green-blue

Hue-Saturation-Brightness (Value) Space

blue

black whitehue

Photometric properties

)()()( ESI

Observed color

ObservedSurfacereflectance

Illumination

Role of Color in Robot Vision1. Feature space for 2D segmentation

more features → be er discrimina�on2. Color physics of reflection

What physical information can color provide?

Color reflection physicssurface reflection and body reflection

bodyair

incident lightsurfacereflection

bodyreflection

internalpigment

Separating reflection components by colorPixel color vectors are

Make a histogram fit parallelogramProject each pixel onto vectorsDetermine everywhere

Klinker 88bbss CC

bs CC ,

bs ,

body reflection

surface reflection

b

ssC

bC

R

G

B

Color space analysis

dbLL

dgLL

drLL

dbL

dgL

drL

B

G

R

C

bs

bs

bs

)())()((

)())()((

)())()((

)()(

)()(

)()(

bbss

b

b

b

b

s

s

s

sbs

b

b

b

s

s

s

bs

bs

bs

CC

B

G

R

B

G

R

dbO

dgO

drO

dbI

dgI

drI

dbO

dgO

drO

dbI

dgI

drI

dbOI

dgOI

drOI

)()(

)()(

)()(

)()(

)()(

)()(

)()(

)()(

)()(

)()(

)()(

)()(

)())()((

)())()((

)())()((

body color vector in RGB spacesurface color vector in RGB spaceColor vector at a pixel is a linear combination of surface + body reflection color vector

Dichromatic Reflection Modelsurface reflection has SPD of incident lightbody reflection has SPD of body color

brightness reflected)( L

surface reflection body reflection

SPD of body colorSPD of incident light

Klinker et al.’s method

Steps:

1. Color segmentation

2. T-shape identification

Separation Results

Chromaticity-Intensity Space

a. Specular image c. Chromaticity Intensity space

a b c

b. Spatial Intensity space

96

Iteration Framework

Result: a single object

Input image Specular-free image

Separation Result

Diffuse reflection component

Specular reflection component

Separation using High Frequency Illumination

[S.K. Nayar et al. SIGGRAPH 2006]

Summary2D digital image processingEdge detectionLine drawing analysisColor theoryPhotometric properties