Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf ·...
Transcript of Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf ·...
![Page 1: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/1.jpg)
IVSIAC 產學小聯盟成果發表會
Advanced ADAS Technology
Prof. Jiun-In GuoDept. of Electronics Engineering, National Chiao Tung University
Hsinchu, Taiwan
June 26th, 2015
1
![Page 2: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/2.jpg)
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
2
![Page 3: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/3.jpg)
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
3
![Page 4: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/4.jpg)
Design Trend for Car Safety: From Passive to Active
4
![Page 5: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/5.jpg)
Euro NCAP Regulation
5
![Page 6: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/6.jpg)
6
Technology for Self Driving
![Page 7: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/7.jpg)
Car Tech News in CES 2015
• Self-driving cars, Apple CarPlay and Android Auto take over CES 2015
• Delphi's computer chauffeur drives me around Las Vegas at CES 2015
• CES 2015: Cruise control gains self-steering with Valeo'sCruise4U
• Volkswagen's Trained Parking is a robotic valet for your home
• CNET's Connected Car panel explores the future of transportation at CES 2015
• Hands off with the Volkswagen Golf R Touch at CES 2015
• Cars of the future won't need drivers: CES 2015
7
![Page 8: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/8.jpg)
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
8
![Page 9: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/9.jpg)
Mobileye Technology
Features:1. Headway Monitoring and Warning2. Intelligent High-Beam Control
3. Daylight Pedestrian Collision Warning,
including Bicycle Detection
Vision SoC for Safety Driving
4. Forward Collision Warning,
both in Highway and Urban areas,
including Motorcycle Detection
5. Lane Departure Warning
6. Traffic light recognition
7. Speed limit indication
9
![Page 10: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/10.jpg)
TI TDA2x SoC for Advanced Driver Assistance Systems (ADAS)
1010
TI’s Automotive Technologies
![Page 11: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/11.jpg)
Advanced ADAS based on Machine Learning
• Nvidia Drive PX
11
![Page 12: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/12.jpg)
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
12
![Page 13: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/13.jpg)
Research Team
13
Dr. Jiun-In Guo
Dr. Ching-Lung Su
Dr. Ching-Wei Yeh
Dr. Kuan-Hung Chen
NCTUProfessor and Director of Institute of Electronics Department of Electronics Engineering
CCUProfessor
Department of Electronics Engineering
FCUProfessor
Department of Electronics Engineering
YuntechProfessor Department & Graduate school of Electronics Engineering
![Page 14: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/14.jpg)
ADAS Technology We Developed
14
Blind Spot Detection
Auto headlight control
Auto High Beam Control
Auto Windshield Wiper Control
Animated Car-Backing Track
Hands-free Trunk Lift gate
Inclement Weather Processing
Wide-view Video Stitching
HDR Night Vision(Single-CAM, Dual-CAM, Micro Camera Array)
Traffic Light Detection
Speed Limit Detection(Taiwan)
USA Driver Dangerous Behavior Detection
Car License Plate Detection
Pedestrian and Scooter Detection
(On IMX6)
Lane Departure Warning (Day, Night, Rain)
Forward Collision Warning(Day, Night, Rain)
LDWS+FCWS(Day, Night, Rain)
Stop & Go
2D/3D Hand Tracking (Single-CAM and Dual-CAM)
Hand Writing
![Page 15: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/15.jpg)
ADAS Technology Developed in NCTU
15
![Page 16: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/16.jpg)
Performance Summary
16
Function Feature
Performance
CPU Resolution FPS
Lane Departure Warning System (LDWS)
• Support variety of road lanes(Solid, dashed, cat-eyes)
• Support variety of weather conditions(Sunny, rainy, foggy days, nights, etc.)
ARM Cortex-A9 1GHz D1 30
Forward Collision Warning System (FCWS)
• Detection distance at day time: 0~50m• Detection distance at night time: 0~40m
ARM Cortex-A9 1GHz D1 30
Stop & Go System (working with FCWS)
• Working distance at day: 0~10m ARM Cortex-A9 1GHz D1 30
Pedestrian and Motorcycle Detection
System
• Obstacle type: pedestrian, motorcycle, bicycle
• Multiple ROI detection• Moving Object Detection IP
ARM Cortex-A9 1GHz D1 30
AMD A10-7850K GPU
D1 30
Accelerator (200MHz in TSMC 90nm)
D1 30
Blind Spot Warning System (BSWS)
• Detection region: outward 0~3m, backward 0~20m
• Object type: vehicle, motorcycleARM Cortex A9 1GHz VGA 30
Inclement Weather Processing Technology
• Dynamic local contrast enhancement • Automatic harsh environment judgment• Support a variety of inclement weather
conditions (nights, rainy, cloudy, foggy days, sandstorm)
ARM Cortex A9 1GHz D1 30
![Page 17: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/17.jpg)
Performance Summary
17
Function FeaturePerformance
CPU Resolution FPS
Traffic Light Detection System
• Vehicle speed: 0~100 km/h ARM Cortex A9 1GHz D1 30
Speed Limit Detection System
• Vehicle speed: 0~150 km/h• Support speed limit sign of North America,
Taiwan, Europe, and China
Intel Atom 1.6GHz (USA)
D1 50
ARM Cortex A9 1GHz (Taiwan, EU)
D1 15
Wide-view Video Stitching System
• Support variety of weather conditions (Sunny, rainy, cloudy, nights, etc.)
• Extending Panoramic to 360 degree surround view
ASIC4ch HD720@
178Hz 30
ASIC4ch HD1080@
202MHz30
Single-camera HDR Night Vision System
• Good for scenes with high variation in lighting (Tunnel, night view with strong light, etc. )
ARM Cortex-A9 1 GHz VGA 13
Dual-Camera HDR Night Vision System
• Good for scenes with high variation in lighting (Tunnel, night view with strong light, etc. )
ASIC HD1080 30
Driver Dangerous Behavior Detection
• To detect eating, phone calling and dozingTREK-668 (Intel Atom
1.6GHz)360x240 15
2D/3D Hand Tracking for Gesture Control
• Single-camera(2D)/Dual-camera(3D) vision based system
Intel Core i7-2600 3.40GHz (3D)
VGA 24
![Page 18: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/18.jpg)
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
18
![Page 19: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/19.jpg)
Highlight on Advanced ADAS Technology
19
Speed Limit Detection(Taiwan)
USA
Pedestrian and Scooter Detection
Lane Departure Warning
Forward Collision Warning
Cross Traffic Detection
![Page 20: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/20.jpg)
Vision-based system
Support resolution: Up to 720x480
Support variety of weather conditions(Sunny, rainy, foggy days, nights, etc.)
Support variety of road lanes(Solid, dashed, cat-eyes)
Lane Departure Warning System (LDWS)
20
Demo Video (Day, Night, Rain)
100 fps D1@ Intel Atom 1.6GHz (using one core)30 fps D1@ Cortex-A9 (Freescale iMX6 1GHz using one core)
Detected rate: 97.43%False alarm rate: 7.89%
![Page 21: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/21.jpg)
LDWS Flow chart
21
Image Capture
Adaptivethreshold
Integral Image
Line-thinning
Houghtransform
Departure
Right laneFSM
Left laneFSM
Line Collection
Lane detectprocess
line detectprocess
![Page 22: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/22.jpg)
Patent Comparison: LDWSMobileye
WO0180068A1、US6704621、US8254635、US2013141580A1、US2012069185A1
NCTU iVS LAB(Patent pending)
Process
Steps Methods Steps Methods
XImage pre-processing
ROI setting
Detect lane boundaries
(3-D)
1. Road skeleton estimation system
(1) Roadway is modeled as a circular/parabolic arc parallel to XZ plane in 3-D space
(2) Two-phase model(3) Value cost function
determine the line (Warped image)
Lane detection(2-D)
Line detection1. Pixel-based Adaptive Threshold
(PAT): dynamically distribute image into lane and non-lane part.
2. Sobel and Randomized Hough transform (RHT)
Lane definition1. Slope2. Vanishing point distance3. Previous frame lane
information4. Lane information finite state
machineFind road curvature XVehicle relationship
to laneX
Time to lane crossing (TLC)
Wheel-to-lane distanceDeparture warning
Image center to Lanes center distance
22
![Page 23: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/23.jpg)
Vision-based system
Detection distance at day time: 0~50m
Detection distance at night time: 0~40m
Support resolution: Up to 720x480
Forward Collision Warning System (FCWS)
23
Demo Video (Day, Night, Rain)
30 fps D1@ Intel Atom 1.6GHz (using one core) with LDWS30 fps D1 @ Cortex-A9 (Freescale iMX6 1GHz) with LDWS
Detected rate: 90.24%False alarm rate: 5.40%
![Page 24: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/24.jpg)
FCWS Flow chart
ROI setting
Image capture
Day or Night
Edge feature
Shadow Feature
Combination
Tracking
Distance estimation
Light feature
Edge feature
FCWSDay
FCWSNight
![Page 25: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/25.jpg)
Patent Comparison: FCWS
Mobileye (Note) NCTU-IVS
Detect Vehicle
Generate candidate regions of interest:1. Rectangular shaped region for all
position and all size2. Filter out candidates lack of texture
and in-compliance with perspective constrains on range and size.
Single frame classification:1. 2-stage classification algorithm.2. SVM + Adaboost.Multi-frame Approval Process:1. Additional information collected
over a number of frames are used in system.
Generate candidate regions of interest:1. Noon or Night judgement.2. Noon : Shadow feature extraction.3. Night : Light-pair feature extraction.Single frame classification:1. Vertical feature verification for noon
and night.Candidate Vehicle buffer:1. Accumulate feature frequency
according to spatial information.2. Select maximum temporal frequency
candidate as vehicle.Tracking state:1. Finite state machine and kalman filter.
Range estimation
Pinhole model and position-based geometry.
Horizontal line + position-based geometry.
Time to collision
Estimate TTC by Scale change and frame rate.
X
25
Note: Itay Gat, Meny Benady, and Amnon Shashua, “A Monocular Vision Advance Warning System for the Automotive Aftermarket,” SAE Technical Paper 2005-01-1470, 2005, doi:10.4271/2005-01-1470.
![Page 26: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/26.jpg)
Vision-based system using machine learning
Support resolution: Up to 720x480
30 fps D1 @ Intel Atom 1.6GHz (using single core)30 fps D1 @ Cortex-A9 (Freescale iMX6 1GHz)30 fps D1 @ AMD A10-7850K GPU Realization30 fps D1 @ Accelerator (200MHz in TSMC 90nm)
Obstacle type: pedestrian, motorcycle, bicycle
Multiple ROI detection
Pedestrian and Motorcycle Detection System
26
Demo Video (PC)
Demo Video (IMX6)
Detected rate: 85 %False alarm rate: 3.6 %
Detection distance: 0~40m (D1)
![Page 27: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/27.jpg)
Proposed Adaboost-based Detection Algorithm
27
Y
Y
Feature value
Calculation
Stage Sum
> thresholdReject
Accumulate
Weighting Value
for Stage Sum
Final Stage?
Choosing
“Left”or“Right”Weighting Value
Adjusted Threshold =
Variance * Feature Threshold
Start a New
Stage
Accept
Feature
Coordinate
Decoding
Integral & Square
Integral Image
Calculation
N
N
3m~5m (360x65)
1m~3m(180x55)
0.5m~1m (180x80)
ROI
Inner
Down-sample
Image
Original
Image
Choose Source
Image
No More
Feature
Y
N
Stage Number
from Classifier
Node number
from Classifier
1. Outside of 3m
2. Within 3m
(full-length)
3. Within 3m
(half-length)
Classifiers
Stage Threshold
from Classifier
Next stage
ROI
Dynamic setup
1. Classifier type
2. Classifier stage
Search size : 14x28~40x80
Next node
Off-line
14x28
40x80
Classifier Stages
Feature Node
12 types
![Page 28: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/28.jpg)
Moving Object Detection IP
Pedestrian and Motorcycle Detection in ASIC
28
Specification
Technology
Gate Count
On-Chip Memory
Capability D1@30fps
TSMC 90nm CMOS
173K Gates
35.7K Bytes
Memory Interface AXI 32bit Bus
Max Frequency 200MHz
Video Format YUV 4:2:0
Detection Rate 85% (based on classifiers)
SRAM (17.4KB)
Search
Window
Buffer
(14.5KB)
Mesh &
Integral pixel
Calculator
(48x88x21bit)
(48x88x8bit)
Feature
Coordinate
Calculator
Classifier 2
Classifier 1
Coordinate
Buffer
Object
Recognition
Engine
AXI Master AXI Slave
Register
Files
(feature number,
feature type)
Parameter Bus
Candidate
Window
DRAM
RISCSource Images
Unique Features
· Supporting pedestrian/motorcyclist detection
· Supporting close-range detection
· Dynamic ROI region
· Fully HW computation
Verification
· RTL simulation pass
· Altera Stratix4 FPGA testing pass
· ESL Aldec FPGA emulation pass
![Page 29: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/29.jpg)
Vision-based system
Support resolution: HD720/D1/VGA
Vehicle speed: 0~150 km/h
D1 23fps@Intel Atom 1.6GHz (Taiwan, EU)D1 50fps@Intel Atom 1.6GHz (USA)
Support speed limit sign of different countries
Speed Limit Detection System
29
Demo Video (Taiwan)
Demo Video (USA)
Detected rate: 94.34 % (USA), 94.52% (Taiwan, EU)False alarm rate: 5.66 % (USA), 5.48% (Taiwan, EU)
![Page 30: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/30.jpg)
Proposed Algorithm for USA Speed Limit Detection
1. Rectangle Detector
2. Achromatic Decomposition
3. Adaptive Threshold
4. Digit Segmentation
5. Digit Normalization
6. Digit Recognition
30
![Page 31: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/31.jpg)
• Behavior analysis• Eating detection
• Detect the distance between hands and mouth
• Phone calling detection
• Detect if hand tracking blob locate beside the face
• Dozing off detection
• The system can not detect eyes for more than 15 frames in the face region
• Dangerous driving warning• Eating detection
• Phone calling detection
• Dozing off detection
31
Driver Behavior Analysis System (DBAS)
![Page 32: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/32.jpg)
Alley View for Cross Traffic Detection
• Goal: • To show the left/right sides of view through view
transformation on a fish-eye camera
• Could be combined with object detection
32
Input: Fish-eye view Output: Alley view
![Page 33: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/33.jpg)
Multiple People Detection, Tracking, and Behavior Analysis
33
• Major Functions• People Detection
• People Tracking
• Behavior Analysis
• Challenges• Lightness variation
• Strong lightness variation by turn on/off the light indoors
• People keeping static• People will be updated to
background if they keep static
• People crossing• The labels often exchange
when people switch
• Moving objects• When objects move(like chair),
it will be detected
Functions
Behavior Analysis
Video Sequence
PeopleTracking
Falling Down
Detection
People detection
People Labeling
Posture Analysis
Applications
Virtual Fence
Trajectory Recording
Aware!
Falling down detection
Virtual fence
![Page 34: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/34.jpg)
Demo Video for Multiple People Detection, Tracking, and Behavior Analysis
34
![Page 35: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/35.jpg)
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
35
![Page 36: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/36.jpg)
ADAS for Fleet Management
• Concept
36
![Page 37: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/37.jpg)
ADAS Algorithms We Adopted
Lane Departure Warning System (LDWS)
Forward Collision Warning System (FCWS)
Driver Behavior Analysis System (DBAS)
Speed Limit Detection (SLD)
37
![Page 38: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/38.jpg)
ADAS for Fleet Management
• Dangerous driving behaviors
38
車道偏移警示
前車防撞警示
速限標示偵測
危險駕駛行為警示
左車道偏移
右車道偏移
前方車距
道路速限
手部偵測
臉部偵測
五官偵測
蛇行單位時間內,訊號交互
系統 功能 時間資訊 車速資訊 危險行為
單位時間內,維持短距 高速 逼車
<當前車速 超速
使用手機
進食
打瞌睡
左車道偏移警示
前車過近警示
方向燈訊息
右車道偏移警示
無左方向燈
無右方向燈
![Page 39: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/39.jpg)
ADAS for Fleet Management
•危險行為回傳資料
39
危險行為
回傳資料
車輛ID 時間 地點(GPS) 當前車速 規定速限 偏移資訊 距離資訊 駕駛行為
蛇行 ● ● ● ●
車道偏移 ● ● ● ●
逼車 ● ● ● ● ●
前車過近 ● ● ● ● ●
超速 ● ● ● ● ●
使用手機 ● ● ● ●
進食 ● ● ● ●
打瞌睡 ● ● ● ●
![Page 40: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/40.jpg)
Multiple ADAS Functions in Freescale i.MX6 with Multithreading
• Approaches• Multiple threads
are created• Captured thread
• Worker threads• LDWS, FCWS, SLD,
DBAS
• Display thread
• Thread pipelining• Synchronized by
FIFO in data transfer
40
pth_DLCE
pth_LDWS
pth_PDS
Write data packet to display
thread
Read data packet from the last
worker thread
Draw the boundaries of driving
lane and the marking windows of
detected pedestrians
Capture thread
Display thread
pth_ADAS
working loop
initialization
Create worker threads and FIFOs
working pipeline
pth_FCWS
Pth_LDWS
Pth_FCWS
Pth_SLD
Pth_DBAS
![Page 41: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/41.jpg)
Performance Summary
41
Freescale iMX6 Quad Core Cortex A9 (BD-SL-i.MX6)
(Automotive grade)
系統 平台 CPU 解析度 效能
A. 車道偏移警示
Freescale i.MX6 Cortex A9 1 GHz D1(720x480)
30fps
B. 前車防撞警示 30fps
C. 速限標誌偵測 24fps
D. 駕駛行為分析 15fps
E. A+B 30fps
F. A+B+C 24fps
G. A+B+C+D 15fps
![Page 42: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/42.jpg)
System Integration
42
• ADAS box (IMX6) + IDU-300 (3G) + Cloud Server
![Page 43: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/43.jpg)
Client Side
43
車隊管理 I.MX6 demo (camera on road)功能: FCWS + LDWS + SLD
系統: 40km/h + (-1~+1)km/h
資訊: 統計五分鐘內偏移次數、未保持安全車距、超速等行為
![Page 44: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/44.jpg)
Server Side
• GCP(receive message from IDU-300)
• GCP_AP(decode G2A data)
44
GCP
GCP_AP
Our waring message
![Page 45: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/45.jpg)
Vision Radar for ADAS
• LDWS+FCWS in traffic map
45
System Initial
Parameter Setting
Line model Vehicle model
Image capture
Not initial Initial already
Lane departure warning system(Multiple lanes detection)
Forward collision warning system(Multiple vehicles detection)
Lane width estimation
Locate vehicle x position
Build traffic map(vehicle position& departure
judgement)
![Page 46: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/46.jpg)
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
46
![Page 47: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/47.jpg)
Conclusion - On-going MOST Project (ViDAR) (2014/8~2017/7)
47
IVVS: Intelligent Vehicle ViDAR System
Multiple Object Identification Multiple Object Recognition Multiple Object Ranging Multiple Object Tracking Multiple Object Behavior Analysis HDR Night Vision 360/540 Degree Surround View
Multiple Object Identification
Multiple Object RecognitionMultiple Object Ranging
Multiple Object Tracking
Multiple Object Behavior Analysis
HDR Night Vision
360/540 Degree Surround View
![Page 48: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/48.jpg)
Conclusion - Roadmap
• Intelligent Vision Technology for Advanced ADAS Applications
48
ADAS (2013-2015) Advanced ADAS (ViDAR) with deep learning(2015-2017)
Advanced ADAS (ViDAR) with deep learning and 360 degree
surround view by SPADscanning
(2017-2019)
![Page 49: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/49.jpg)
Conclusion
• Intelligent Vision is an emerging topic in both academia and industry.• ADAS has become an attractive feature in the vehicles
developed by the major automobile companies in the world.
• Machine learning is also a hot topic in image/vision content analysis.
• Advanced ADAS is the trend for automotive applications.
• Sensor fusion is required (Vision + mmwave + Laser (SPAD))
• More and more intelligence is required and will be invented in the near future.• Technology always begins from Human.
49
![Page 50: Advanced ADAS Technologyivs.ee.nctu.edu.tw/iac/pdf/簡報檔-交大郭峻因教授20150626.pdf · Traffic light recognition 7. Speed limit indication 9. TI TDA2x SoC for Advanced](https://reader033.fdocument.pub/reader033/viewer/2022042023/5e7b98fb3582c622142c3084/html5/thumbnails/50.jpg)
50
Thank you very much for your attention !
http://ivs.ee.nctu.edu.tw/iac/
Our Vision, Your Intelligence !
Q&A