Scene illumination and surface albedo recovery via L1-norm total variation

minimization

Hong-Ming Chenhc2599@columbia.edu

Advised by: John Wright

2

Decomposition of a scene

= .*

scene Reflectance (albedo)

illumination

.* : Matlab element multiplication operation

3

Image Formation

=.*

scenereflectanceillumination

Sensor response (camera or eyes)

Light source power spectrum

Object reflectance

intensity response Sensor response

integration

Pixel i

signals

,

,

ˆ ,

ˆ ,

ˆ ,

ˆ , ,

ˆ , ,

ˆ , ,,

R

GG

B B

RR

B

G

i

i

E i

E i

E i

Sq N i d

q N i d

q N i

L i

L i

L ii

S

dS

ˆ ,L i , i RS

GS

BS

RE i GE i BE i

: shutter speed, aperture size, quantization factor etc

ˆkq

4

It is VERY HARD to directly model / simulate / solve this problem!

5

Narrowing down our target problem

• Simplification:– mean wavelength response (impulse response)

• Assumption (on surface reflectance):– Lambertian Surface (Perfect diffuse reflection, no

specular light) • Simulation (of light source model):– We need a formula to describe the behavior of the light

source – Blackbody radiation: parameterize the light source with:

• Light color (color temperature)• Light intensity

6

Problem formulation:

, , ,

, , ,

, ,

,

,

,

,B

RR

G

B B

G

R

G

R

B

G

i i

i i

i i

E i

E i

E i

q

L iI T i

L I T i

L iq

i

T

q

I i

,

,

ˆ ,

ˆ ,

ˆ ,

ˆ , ,

ˆ , ,

ˆ , ,,

R

GG

B B

RR

B

G

i

i

E i

E i

E i

Sq N i d

q N i d

q N i

L i

L i

L ii

S

dS

7

log

51 2( , , ) exp( / )L I T IC C T

Assume: λR λG λG are known

If there are N pixels in an image: 3N observations5N unknowns (I, T, ref )+ 3 quantize factors

underdetermined system!

, , ,

, , ,

, ,

,

,

,

,B

RR

G

B B

G

R

G

R

B

G

i i

i i

i i

E i

E i

E i

q

L iI T i

L I T i

L iq

i

T

q

I i

21

21

5

5

5 21

ln ln ln ln ln

ln ln ln ln ln

ln ln ln ln ln

RR

G

i

i

i

R i Rii

G i Gii

B i Bi

G

BBi

R

G

B

q IT

q IT

q I

CC

CC

CCT

Ax b

8

51

51

51

51

51

1

1

5

1

3

1

ln lnln lnln ln

ln lnln lnln ln

R

G

B

RN

N

B NN

G

CCC

bCCC

RGB

RGB

1

1

1

5 3

1

1

ln

lnln

ln

lnlnln

R

BN

N

N

R

G

B N

I

IT

Tqqq

x

3 5 3I T q N NA A A A A

3

111

111

111

I

N N

A

3 3

1

1N N

A

2

2

2

2

2

2 3

R

G

B

T

R

G

B N N

CCC

ACCC

3 3

11

1

11

1

q

N

A

Ax b

9

Recovering unknown x

• Previous approach– Introducing regularization terms into objective

function

• Current approach–Minimizing L1-norm total variation

22 ln lnmins

sp s px s C p

Ax b w

1

1

1

5 3

1

1

ln

lnln

ln

lnlnln

R

BN

N

N

R

G

B N

I

IT

Tqqq

x

10

Previous Approach

22 ln lnmins

sp s px s C p

Ax b w

1-D grayscale visualization

A segmentation-like result

A result of:Intrinsic images by entropy minimization , Finlayson, ECCV2004

ps

pp

ps

2

22s p

w

p p

spw e

0

255

11

Drawbacks of this approach

• There are at least 2 parameters (λ, σ) to be fine tuned.

• The results of Finlayson’s approach heavily affects the accurateness of our prior. – 1. Its Achilles heel: projection problem – 2. it is still an open problem to find the best

rotation angle. •

12

(λ =50 , σ = 10) (λ =10 , σ = 30)

(λ =120 , σ = 5) (λ =120 , σ = 8)

13

A brief review of Finlayson’ solution

• Its Achilles heel:

14

L1 norm Total Variation Minimization

Image From Wikipedia

b

ab

a

V f f x dx

1( ) n nn

TV x x x

15

L1 norm Total Variation Minimization

• Widely used in image denoise / Compressive sensing – E(x, y) + λTV(y).

1( ) n nn

TV x x x

21,2 n n

n

E x y x y

Image From Wikipedia

16

Current approach: L1 TV norm

• Applying L1-norm total variation on albedo term, • The L1-norm encourages a spiky result on

gradient–Which means: we want most of the albedo gradients

are 0 unless necessary => when albedo changes

•

1min , lni i i i

i

w st Ax b w D 1

1

1

5 3

1

1

ln

lnln

ln

lnlnln

R

BN

N

N

R

G

B N

I

IT

Tqqq

x

17

Results

Original image

Light color (temperature) imageLight intensity image

Albedo (reflectance) image

18

Results

Original image

Light color (temperature) imageLight intensity image

Albedo (reflectance) image

19

Results

Original image

Light color (temperature) imageAlbedo (reflectance) image

20

Results

Original image

Light color (temperature) imageAlbedo (reflectance) image

21

Editing

Original imageAverage T-1000 Average T+1000

Average T+2000 Average T+3000 Average T+4000

Average T = 3940

22

THANK YOU