Colouran algorithmic approach

Thomas Bangertthomas.bangert@eecs.qmul.ac.uk

PhD Research Proposal

Human Visual Sensor Array

physical sensor response

How the physical sensors respond to light

… actually a measure of pigment’s ability to absorb photons

virtual sensor responseimplied sensor response based on perceptual studies

…derived from colour matching studies

… using 3 primaries (700nm, 546nm, 436nm)

R=175G=200B=25

What is a colour matching study?

Subject is asked to adjust primaries until the colour of the 2 regions appears identical? to match one region with the other

Visual field divided into 2 regionsregion 1 illuminated by monochromatic lightregion 2 illuminated by primaries

R=200G=200B=50

R=0G=25B=25y=175

What is being proposed?

Subject is asked to adjust primaries until the colour of the 2 regions appears identical? to match one region with the other

4 primaries rather than 3:RGB + yellow

R=0G=0B=50y=200

… and using modern LCD technology

monochrome LCD

with modified backlighting

• one region lit by single spectrum source

• the second region lit by 4 primaries

Why?(the simple answer)

… to resolve the problem of negative primaries

ie. areas where colour matching with RGB fails

Amount of red needed to add to monochromatic stimuli to get a match

… but really, because the human brain is wired with 4 sensors in mind – organized into 2

opponent channels

How would the brain like to see its visual sensor input?

Colour information is packed into 2 ‘opponent channels’ (2 signed numbers).Driven by 4 sensors ideally, but otherwise what is available is used.

Senso

r V

alu

e

Wavelength(λ, in nm)400300 430 460 490 520 550 580 610 640 670 700

0.8

0.6

0.2

0.0

1.0

0.4

Why is this interesting?

July 2006. “What birds see”. Scientific American.

how a bird sees colour“… is difficult – impossible in fact – for humans to know”

370 nm 445 nm 508 nm 565 nm

700 nm330 nm 400 nm 500 nm 600 nm

1.0

0.5

0.0

Background

to colour

sensor array of natural visual systems

arrangement is random

note:very few blue sensors, none in the centre

Sensors we buildX

Y

The naïve approach:

Just measure

R

G

B

Opposite of what natural visual system do

http://www.cvl.iis.u-tokyo.ac.jp/~zhao/database.html

Human Perceputal Responseto luminance

350 400 450 500 550 600 650 7000

10

20

30

40

50

60

70

80

90

100

Wavelength (nm)

Abso

rpti

on

(%

)

RGB

Luminance Sensor IdealizedSe

nsor

Val

ue

Wavelength(λ)λ

0.8

0.6

0.2

0.0

1.0

0.4

λ+δλ−

note linear response in relation to wavelength

-

What does a light stimulus look like?Se

nsor

Val

ue

Wavelength(λ)λ

0.8

0.6

0.2

0.0

1.0

0.4

λ+δλ−

The sensor response is simple integration (summation across spectral range)

-

How do we code stimuli?Se

nsor

Val

ue

Wavelength(λ)λ

0.8

0.6

0.2

0.0

1.0

0.4

λ+δλ−

When spectral composition is approximately equal sensor response = luminous intensity

we assume intensity is equal throughout spectrum

-

Spatial Opponency

A Peculiarity of natural visual systems:Luminance is always measured by taking the difference between two sensor values.Produces: contrast value

Sens

or V

alue

Wavelength(λ)λ

0.8

0.6

0.2

0.0

1.0

0.4

λ+δλ−

Sens

or V

alue

Wavelength(λ)λ

0.8

0.6

0.2

0.0

1.0

0.4

λ+δλ−

 

Moving from Luminance to Colour

• Primitive visual systems were luminance only

• Night-vision remains luminance only

• Evolutionary Path– Monochromacy– Dichromacy (most mammals – eg. the dog)– Tetrachromacy (birds, apes, some monkeys)

• Vital for evolution: backward compatibility

Electro-Magnetic Spectrum

Visible SpectrumVisual system must represent light stimuli within this zone.

Low resolution – equal distribution is okHigh resolution – not!

Sens

or V

alue

Wavelength(λ)λ

0.8

0.6

0.2

0.0

1.0

0.4

λ+δλ−

spectral distribution is more complexsimple luminous intensity fails to describe stimuli correctly

-

Given a light stimulus within the visible range:

Sens

or V

alue

Wavelength(λ)λ

0.8

0.6

0.2

0.0

1.0

0.4

λ+δλ−

What information do we need to describe the stimulus fully?

1.

Lu

min

ou

s In

ten

sity

2. WavelengthIf we had a reference luminance we could calculate wavelength (by halves).

-

modify one sensor pair – shifting spectral sensitivity reference sensor:

Sen

sor

Valu

e

Wavelength

0.8

0.6

0.2

0.0

1.0

0.4

λ-δ λ λ+δ

RG

Roughly speaking wavelength is:

λ + ( R – G )One sensor can be used as a reference to measure intensity and the second to measure spectral position

the ideal light stimulusS

en

sor

Valu

e

Wavelength

0.8

0.6

0.2

0.0

1.0

0.4

λ-δ λ λ+δ

RGMonochromatic Light

Allows wavelength to be measured relative to a reference.

Problem:natural stimuli are often not ideal

Sen

sor

Valu

e

Wavelength

0.8

0.6

0.2

0.0

1.0

0.4

λ-δ λ λ+δ

RG

• Light stimulus might not activate reference sensor fully.

• Light stimulus might not be fully monochromatic.

Sens

or V

alue

Wavelength(λ, in nm)

400300 430 460 490 520 550 580 610 640 670 700

0.8

0.6

0.2

0.0

1.0

0.4

Solution:

A 3rd sensor is used to measure equiluminance.

Which is subtracted.

Then reference sensor can be normalized

This means a 3rd piece of information: 3. Equiluminance

Coding colour

With the assumption that a stimuli is monochromaticAny light stimulus (within the spectral range) can be represented exactly by 3 values:

• luminous intensity• wavelength• equiluminance

Wavelength is coded by taking a difference (or opponent) value of 2 sensors – simplest solution.

a 4 sensor opponent designS

en

sor

Valu

e

Wavelength(λ, in nm)

400300 430 460 490 520 550 580 610 640 670 700

0.8

0.6

0.2

0.0

1.0

0.4

2 opponent pairs• only 1 of each pair can be active• min sensor is equiluminance

,R G y B

350 400 450 500 550 600 650 7000

10

20

30

40

50

60

70

80

90

100

Wavelength (nm)

Abso

rpti

on

(%

)RGB

Pigment Absorption Data of human cone sensors

350 400 450 500 550 600 650 7000

10

20

30

40

50

60

70

80

90

100

Wavelength (nm)

Abso

rpti

on

(%

)

RGB

Red > Green

human colour representation is circular!

Which is not a new idea, but not currently in fashion.

540nm

620nm

480nm

Dual Opponency with Circularity

an ideal model using 2 sensor pairs

Senso

r V

alu

e

Wavelength(λ, in nm)400300 430 460 490 520 550 580 610 640 670 700

0.8

0.6

0.2

0.0

1.0

0.4

yellow - blueThe Primaries:

red - green

Defining Coloura working hypothesis

‘Colour’ means a stimulus is represented by 3 values

• luminous intensity• distance between 2 primaries• equiluminance

The primaries are fixed locations on the spectrum. The distance between primaries is measured.

Deliverables for 3 year Research Proposal

a 4-colour image standard

• luminous intensity value• dual opponent value• equiluminance value

Senso

r V

alu

e

Wavelength(λ, in nm)400300 430 460 490 520 550 580 610 640 670 700

0.8

0.6

0.2

0.0

1.0

0.4

method to produce 4 primary colours from 3 sensors

technique to create images in 4-colour format

3 sensor raw output of conventional technology may be used

algorithm is specific to device used

need to be able to translate sensor values to wavelength

Canon 400D

examine possibility of translating historical image archive to 4-colour

format

• photography• film

4-colour display prototype

adapt existing technology

if there is no direct hardware access – use pixels as sub-pixelsas long as each pixel is addressible

a diagnostic test to determine colour primaries in humans

• opponency means colours can be ‘tweaked’ by opposing complement

• human colour perception varies in the individual

• individuals with variation outside normal bounds are called ‘colour blind’

• ‘colour blindness’ can be ‘cured’ by ‘tweaking’ the primaries

a colour matching study to confirm approach

Does the 4 primary approach solve the ‘negative’ primary problem?

primariesR = 650nmG = 530nmB = 460nm

primariesR = 700nmG = 546.1nmB = 435.8nm

…using conventional lcd display technology for colour matching

Light Source

http://www.ccs.neu.edu/home/bchafy/monitor/crtlcd.html

… with modified backlighting – 2 light sources

light from 4 primaries

mono-chromatic light

(1) monochromatic light source

User selectable

(2) 4 primaries (red, green, blue, yellow)

high quality white light is already often produced by 4 primaries

each primary individually adjustable

Apparatus• monochrome LCD

display• spectrophotometer• monochromator• full spectrum light

source• 4-primary light

source

Further work:

• colour arithmetic• Transparency• implied objects

Why is understanding colour correctly important?

Colours are computed, not measured!Very important that colour information is in correct form!Starts with sensor information!

Colour is very useful for transparency

What is the colour?

Why do we need transparency?

otherwise we might have trouble with windows

… and difficulties with these kinds of tasks

Colour is very helpful in deciphering the layers

Aim: to reconstruct scenes with transparency

visual systems with 4 sensors

• Birds• Reptiles• Dinosaurs• Therapsids (our

dinosaur-like ancestor)

about 60nm between sensors

evenly spaced frequencies narrowed

370 nm 445 nm 508 nm 565 nm

700 nm330 nm 400 nm 500 nm 600 nm

1.0

0.5

0.0

The Ideal Sensor

• Equally spaced on spectrum

• Overlap with linear transition

0.8

0.6

0.2

0.0

−5 −3 1 3 5

1.0

−1 0 2 4−2−4

y

0.8

0.6

0.2

0.0400 460 580 640

1.0

520 550 610 670490430

λ

0.4

0.8

0.6

0.2

0.0400 460 580 640

1.0

520 550 610 670490430

λ

0.4

0.8

0.6

0.2

0.0400 460 580 640

1.0

520 550 610 670490430

λ

0.4

colour channel 1: R - Gcolour channel 2: yellow - B

• No overlap of opponent pairs

spectrum is shifted toward more even spacing

445

555600

020406080100380 420 460 500 540 580 620 660 700

LMS

Absorption

424 530 560

Actual Sensor Response

Sensor Response calculated from CIE perceptual data

460 530 640

CRT RGB Phosphorsspectrum is shifted more towards even spacing

HVS Sensor

+ yellow almost equal distribution

a yellow sensor + a few tweaksmakes human vision equivalent to bird vision

• even spacing• 60nm between

primary colours• response

narrowed• intermediary

colours at half-way points

0.8

0.6

0.2

0.0

1.0

y

0.4

400 460 580 640520 550 610 670490430

λrequires more processing, is less accurate, but is equivalent

Summary

• Colour is based on contrast• HVS has a circular model of spectrum• Colour is a code for where on spectrum• 2 colour channels, bi-polar 4 primary

colours• 2 channels 2-d colour space• Simple transform to circular representation• Single variable represents all colours• Purpose is to allow systematic colour

transforms colour computation

References

Poynton, C. A. (1995). “Poynton’s Color FAQ”, electronic preprint.http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html

Bangert, Thomas (2008). “TriangleVision: A Toy Visual System”, ICANN 2008.

Goldsmith, Timothy H. (July 2006). “What birds see”. Scientific American: 69–75.

Neitz, Jay; Neitz, Maureen. (August 2008). “Colour Vision: The Wonder of Hue”. Current Biology 18(16): R700-r702.

Questions?

Samples of simple colour

transforms

Blue-Yellow set to 0

Red-Green inverted

Blue-Yellow

inverted

playing with colour

is easy

these are simple

transforms

not touched by

hand