Camellia General

Wido Kruijtzer / Philips ResearchNovember 2003 CamelliaIST-2001-34410

Key Action IV.1.1

CAMELLIAGeneral overviewNovember 2003

Camellia ConsortiumContact:W.M. (Wido) KruijtzerPhilips ResearchProf. Holstlaan 4 (WDC31)[email protected]: +31 40 2742025

2

CamelliaWido Kruijtzer / Philips ResearchNovember 2003

IST-2001-34410Key Action IV.1.1

Contents

• Smart Cameras• Motivation• Camellia project

– Applications• Algorithms

– Co-processor requirements– Co-processor Architecture

• Prototyping• Conclusions

3



IEEE Computer September 2002

4



Smart Camera application areas

Consumer

Significantly growing demand for SI applicationsin several domains

Automotive Mobile

Surveillance

5



Ecole des Mines - Paris

University of Las Palmas

University of Hannover

Camellia partners

Research – EindhovenPS-CoC-VC – HamburgPS-NxMM – Nijmegen

Start: April 1, 2002Duration: 33 monthsEffort: 28 pyIST-34410

6



Goal

ARM 9xxCPU

embed.RAM &(boot)ROM

CCIR /CameraFrontend

DataIF

Video Compression IP

VideoInput

SyncIF

DataIF

SyncIF

DataIF

SyncIF

DataIF

MemoryController(Flash & DRAM

DataIF

ext.Flash

Synchronization Infrastructure

Peripherals*

DataIF

SyncIF

ext.SDRAM

Camellia SmartImaging Coprocessors

I/O Interface

DataIF

off-chipcommunication

* - Timers, Watchdog, Interrupt Controller etc.

I/D Cache

DataIF

SyncIF

DataIF

SyncIF

DataIF

SyncIF

DataIF

SyncIF

MotionEstimator

TextureCodec

SmartImagingCopro 1

SmartImagingCopro 2

BistreamGenerator

Data Communication Interconnect

DataIF

SyncIF

DataIF

SyncIF

SmartImagingCopro 1

SmartImagingCopro 2

HW assist

SW

Low-Cost and Low-Power Video Processor for Automotive and Mobile Applications with Intelligent Imaging Capabilities

7



Application selection

• Description of 7 automotive and 10 mobile applications– 1 paragraph general explanation– Application quotation based on:

• Customer interest• Compatibility

– With compression core (mainly motion estimation)

• Feasibility– Can we design it within the Camellia timeframe

• Benefits– How much value creation (Euros)

8



Applications selected for Camellia

Automotive• Low speed obstacle

detection (LSOD)• Pedestrian detection

Mobile• Image stabilization

• Face tracking

Reference:Camellia-D-2.1 “Applications”; November 2002

9



Application specifications

• Specification of each selected application– Functional requirements– Input/Output– Use cases

• E.g. face tracking:– Number of heads,– head coverage, size and orientation, – skin color, hairstyle, wearables, etc.

– Scenarios (combination of use cases)• Video sequences to reflect specific scenario

10



Algorithms

• General framework Particle Filtering*

– The theory behind it :• Bayesian approach of estimation“ A tutorial on Particle Filter for On-line Non-linear/Non-Gaussian

Bayesian Tracking”Sanjeev Arulampalam, Simon Maskell, Neil Gordon and Tim Clapp.

*Except for Image stabilization• motion estimation + motion field filtering

11



Particle FilteringGeneral architecture

ParticleFiltering

Output stage

ImageProcessing 1

Particles updating using likelihood functions

Feedback to image processing

ImageProcessing 2

ImageProcessing 3

12



Particle Filtering: Benefits

• Same architecture for several applications

• Scalability :– State space of targets

• Number of particles per target• Number of particles used for feedback

• Usability on target architecture:– No floating point operations– Light computation load with 100 particles– Can be computed in parallel with the image processing

operations (2-stages pipeline)

13



Particle FilteringApplied to obstacle detection

Edgedetection

Symmetrydetection

Motiondetection

ParticleFiltering

Output stage

Lightsdetection

Shadowdetection

14



Low level algorithms

• Basic pixel processing– image filtering, morphological operations, region analysis etc.

• Camellia Image Processing Library– Open Source– uses IplImage structure to describe images

• good replacement to the popular but discontinued Intel IPL library• good complement to the OpenCV library

– http://camellia.sourceforge.net

Reference:Camellia-D-3.1 “Report on Computer Vision Algorithms”; V4; July 2003

15



Example shadow detection

Road Histogram

020406080

100120140160180

1 21 41 61 81 101 121 141 161 181 201 221 241

Pixel value

Num

of P

ixel

s

Series1

Utilizes:

Histogramming, Thresholding, Erosion, Labeling+Blob

16



Application demos

Automotive application: Low speed obstacle detection (LSOD)

17



Particle FilteringApplied to face tracking

Face Colordetection

Face ovaldetection

ParticleFiltering

Output stage

Face featuresdetection

18



Application demos

Mobile application: Face tracking

19



Particle FilteringApplied to pedestrian detection

Edgedetection

LegsDetection

Motiondetection

ParticleFiltering

Output stage

BodyDetection

20



Application demos

Automotive application: Pedestrian detection

21



Application demos

Mobile application: Image stabilization

22



Algorithms results

• C++ code of all 4 applications available

– References:• Camellia-D-3.2b “Report on Initial algorithms 1”;

July 2003– Describes Image stabilization and LSOD

• Camellia-D-3.3b “Report on Initial algorithms2”; October 2003– Describes Pedestrian detection and Face tracking

23



SI Core architecture (1)

• SI is expected to be an essential differentiator for future products

• Successful introduction requires– Low cost (small Si area) and low power (esp. for mobile)– Easy integration into SoCs (small system interference)

• Architecture must support different application domains– No dedicated solutions !– Flexibility and Scalability are key

• Trade-off between Flexibility and Efficiency– Programmability vs. Adaptation to Algorithms

24



SI Core architecture (2)

• Hierarchy/structure of applications comprises three layers– Bottom layer: Set of core algorithms (Toolbox)– Common for all applications

• Core algorithms are basic pixel processing, image filtering, morphological operations, region analysis etc.

– Mid layer: Application-specific medium level algorithms• Uses core algorithms

– Top layer: Application-specific high level code • Based on particle filtering, uses medium level algorithms

25



SI Coprocessor architecture (1)• Selected Approach: Hierarchical Control

– Level 1: Micro-Instruction Level– Level 2: Macro-Instruction Level

• Macro-instruction initiates the execution of a sequence of micro-instructions for an image segment

• Only macro-instruction level is visible to the system

– Dedicated Macro-Instructions for each Algorithm mapped onto Coprocessor

• Implementation of macro-instruction functionality at design-time• Selection of processed image region at run-time

– Benefit: Reduced System Interference while sustaining Flexibility for a Variety of Applications

26



Memory

Datapath

Control

Data I/O

RegFile DataRAM 1a

DataRAM 1b

DataRAM 2

MacroControl

UnitMicro

ControlUnit

MacroMem

Data Bus

Ctrl Bus

SI Coprocessor

Arithmetic 1

Arithmetic 2

Accu Regs

SI Coprocessor Architecture (2)

27



Arithmetic Unit 2 (1 Arithmetic Stage)Support for processing of 4 pixels in parallel

Arithmetic Unit 1 (3 Arithmetic Stages)Support for processing of 4 pixels in parallel

Accu Regs

ADD/SUB SHIFT/CLIPControlWord (Arith2)

to local memory

Operands from local memory

ADD/SUB MIN/MAX MUL DIV SHIFT

INV/ABS

ADD/SUB

ControlWord (Arith1)

Coprocessor Datapath

28



SI Coprocessor Architecture

• Implementation of a synthesizable SystemC model of the SI coprocessor

• Performance analysis– 100 MHz target clock frequency, 32-bit arithmetic unit

Real-time processing of up to SDTV resolution (720x576 pixel, 25 Hz frame rate)

• Silicon area of SI Coprocessor (first results)– Logic synthesis for a 0.12 micron CMOS process

Coprocessor Area: around 1 mm2

Reference:Camellia-D-5.1a “Coprocessor specification 1”; October 2003

29



CoproSCModel

• SystemC-based reference implementation of SI-Core– Implemented in SystemC/C++ with graphical front end

• GUI uses WxWindows

– Used for the following tasks:• Definition of Architecture, micro- and macroinstructions• Performance Analysis• Architectural Refinement

– Thought as „Executable Specification“• Cycle-True Model of SI core• Reference for Implementation of architecture• Supplement to „Written Specification”

30



CoproSCModel

MemoryModel

CPUModel

SI-CoreModelSI-

Core

Handshake

Data, Address

Settings,Macro Ins.

Handshake

VideoIO

GUI

High-Level Ctrl

Data

31



CoproSCModel Demo

Downscale

Symmetry Detection

32



SI Core System integration

SI Core

SCI

SoC SyncWrapper

SoC DataWrapper

SI Coprocessor

Simple MI IF

Physical TTL

DTL

DTL

SoC Sync Infrastructure

SoC Data Communication Interconnect

33



Camellia System Simulation Environment (CSSE)

• System simulation environment– Implemented in SystemC/C++ with graphical front end

• Uses SystemC Master/Slave library• GUI uses WxWindows

– Investigation of selected system aspects• Obtain metrics of communication load and performance estimations• Make system and application optimizations

– Platform for mapping of applications• Check functional correctness

– Functional verification of SI Coprocessor

34



CSSE Architecture (TTL based)

CPU SMART IMAGING

CPU WRAPPER WRAPPER

MEMORY

Synchro bus arbiter

Data bus

Config bus

Ack / Req

SLAVE

MASTER

VIDEO

WRAPPER

Synchronization bus

Data bus arbiter

Coproc IF(TTL Layer)

35



CSSE Architecture (Logical view)

VIDEO

CPU

hlamla1

mla2

mlan

SIlla1

lla2

llan

macro_si_out port

macro_si_in port

macro_sichannel

Video_out port

Video_in port

video channel

memory_outport

memory_inport

memory_outport

memory_inport

memory_inport

memory_outport

Frame Memory

VIDEO

CPU

hlamla1

mla2

mlan

hlahlamla1mla1

mla2mla2

mlanmlan

SIlla1

lla2

llan

SIlla1

lla2

llan

lla1lla1

lla2lla2

llanllan

macro_si_out port

macro_si_in port

macro_sichannel

Video_out port

Video_in port

video channel

memory_outport

memory_inport

memory_outport

memory_inport

memory_inport

memory_outport

Frame Memory Synchronous channels

Asynchronous ports and channels

Synchronous ports

36



CSSE demo

37



Camellia prototyping system

• Build 2 prototypes• Prototype used in 2 modes

– Functional• Uses video sequences

– Field test• Automotive

– Integration with frame-grabber and CAN-bus controller

• Mobile– Integration with CMOS cameras and mobile display

• Prototype based on Avalon

38



Main AVALON features

• PCI board prototyping system• Contains 2 Altera’s PLDs (up to 808 I/O)

– One APEX 20KE device with 1.5M gates– One Altera’s Excalibur with 1M gates + ARM922T hard core (200 Mhz)

• Processor subsystem connection to 128Mb DDR SDRAM at 266Mhz• 144 pin SODIMM interface on each PLD• User I/O

– 430 signals between PLDs (32 via switch matrix)– 181 signals to expansion connectors– 48 signals to 68 pins bracket connectors– 96 signals trough advanced streaming port PrEDICT

39



Prototyping Hardware

Altera: Philips Avalon

40



AVALON plug-in board

41



AVALON - Excalibur inside

42



Main results

• A set of basic SI algorithms• A set of specific applications for automotive and mobile

domain• A low-cost low-power core for efficient acceleration of SI

algorithms to support the applications• 2 demonstrators showing the effectiveness of the applications

using the Camellia system prototype

• Explored advanced design methodologies for improving TTM• Contribution to sustaining the competitive advantage of

Europe in the mobile industry• Accelerated introduction of smart cameras in European cars

Camellia General

Technology

Transcript of Camellia General