1. Active Safety Solutions DevCon 2012
Mark Feller, Field Application Engineer
0C09I

2. Mark Feller: FAE, Automotive
Education
Electrical and Computer Engineering, Purdue University
12 Years supporting Renesas solutions in Automotive applications
Powertrain
Infotainment
Now…ADAS!

3. Renesas Technology & Solution Portfolio
The wealth of technology you see here is a direct result of the fact that Renesas Electronics Corporation was formed on April 1, 2010 as a joint venture between Renesas Technology and NEC Electronics — Renesas Technology having been launched seven years ago by Hitachi, Ltd. and Mitsubishi Electric Corporation. There are four major areas where Renesas offers distinct technology advantage.
--The Microcontrollers and Microprocessors are the back bone of the new company. Renesas is the undisputed leader in this area with 31% of W/W market share.
--We do have a rich portfolio of Analog and power devices. Renesas has the #1 market share in low voltage MOSFET solutions.
--We have a rich portfolio of ASIC solution with an advanced 90nm, 65nm, 40nm and 28nm processes. The key solutions are for the Smart Grid, Integrated Power Management and Networking
--ASSP: Industry leader for USB 2.0 and USB 3.0. Solutions for the cell phone market
-- Memory: #1 in the Networking Memory market

4. ‘Enabling The Smart Society’
Challenge: “In the smart society, vehicles are incorporating more intelligence to detect threats, and then warn the driver and/or automatically respond (i.e. emergency braking). The computing solutions behind these active safety systems must balance high performance against requirements for integration and low power consumption.” 
Solution:
“Renesas focuses on adding specialized hardware to active safety solutions to add performance with lower power, allowing for higher integration.”
This is where our session, Active Safety Solutions, is focused within the ‘Big picture of Renesas Products’, Microcontroller and Microprocessors.

5. Microcontroller and Microprocessor Line-up
2010
2013 *
Up to 30K DMIPS, Performance
Cortex A7 or A15-based
Automotive, 28nm
32-bit
1200 DMIPS, Superscalar
Automotive & Industrial, 65nm
600µA/MHz, 1.5µA standby
1800 DMIPS, Performance
Automotive, 40nm
500µA/MHz, 35µA deep standby
500 DMIPS, Low Power
Automotive & Industrial, 90nm
600µA/MHz, 1.5µA standby
25 DMIPS, Low Power
Renesas is consolidating pre-merger product families, taking the best features and technologies. Renesas is also moving to ARM-based products for applications demanding the highest performance.
Industrial & Automotive, 150nm
190µA/MHz, 0.3µA standby
44 DMIPS, True Low Power
8/16-bit
Industrial & Automotive, 130nm
144µA/MHz, 0.2µA standby
10 DMIPS, Capacitive Touch
Industrial & Automotive, 130nm
350µA/MHz, 1µA standby
Wide Format LCDs
*tentative naming

6. Active Safety Solutions by Renesas*
Active Safety Solutions by Renesas
MP Start 　　　 　　　
System-on-Chip
ARM-based
Performance requirements continue to grow at high end
SH7766
w/ Signal Processing HW
Dot-Matrix
SoC
System-on-Chip
External Memory
Performance
IMAPCAR2
w/ Signal Processing HW
SH7766-L
w/ Signal Processing HW
Microcontroller
RH850-based
MCU
Internal Memory
For products targeting active safety solutions, Renesas relies on ARM-based products for applications demanding the highest performance. But process technology is advancing, and Renesas is able to squeeze enough performance into devices with emedded Flash and RAM to offer highly integrated solutions targeting some Active Safety applications.
As process technology advances, more performance can be packaged with internal memory
SH7455/6
200MHz
DSP by FPU
*tentative naming 6

7. Agenda Introduction to “ADAS” ADAS Systems Overview Renesas Solutions
SH7766 / TopView
Sensor Fusion
Smart Sensor
Ecosystem
Ethernet AVB in ADAS
Functional Safety

8. Introduction

9. ADAS Definition Key Term: ADAS “Advanced Driver Assistance Systems”
High level description of ADAS:
Monitor the vehicle and its environment
If a threat or other notable event is detected
Alert the driver
And/or take action
Goal: Provide the vehicle and driver greater ability to avoid accidents

10. ADAS Application Examples
Object detection and collision avoidance
Parking assistance
Back-up camera
Surround-view
Self-parking
Traffic Sign Recognition
Lane departure warning / lane keeping
Night Vision
Augmented reality
Headlight control (Automatic dimming of high beams)
Driver monitoring (distracted/drowsy driver)

11. ADAS Application Architecture
Today
Flash
DDR RAM
Complex PCB for high-speed
RAM access
DSP
Host
Control
Sensor Data
Car Network
Major steps
Collect sensor data on DSP (radar or camera)
Analyze data (object detection)
Communicate to other vehicle systems to take appropriate actions, as needed
Steering
Braking
Audio system (to warn driver)
etc.

12. High-End Vehicle Platforms Low-End Vehicle Platforms
ADAS in the Vehicle
High-End Vehicle Platforms
Multiple Cameras
Multiple Radars
Low-End Vehicle Platforms
Some ADAS content still required to meet regulations and insurance requirements
High-end vehicle platforms will offer a variety of “ADAS” systems supporting multiple cameras and radars. Even basic vehicle platforms will offer some “ADAS” capability to address government regulations or insurance industry safety requirements.

13. High-Level ADAS Trends
Strong growth is anticipated for ADAS applications
ADAS expertise is maturing across the Automotive industry
Technology costs are decreasing
Government/Insurance regulations increasing
…but adding sensors, networking, and processing capability adds weight and power consumption.
Reduces range of Electric Vehicles
Reduces mpg when government regulation pushes manufacturers to higher standards
Renesas goal: Offer solutions that enable higher integration and lower power

14. ADAS Application Growth
This slide illustrates the growth seen by application type as ADAS systems proliferate. Units are millions of systems, where there are possibly multiple systems per vehicle in some implementations.

15. Autonomous Driving as Long-term Target
Manufacturers and researchers are working on autonomous vehicle technology
Most of the building blocks are emerging for taking R&D to production
Where to go: Input by driver
How to get there: From the Navigation Unit
How to negotiate traffic, interchanges and road conditions: From ADAS
Understand position and obstacles through radar & cameras
Take action through steering, brakes, and throttle

16. Renesas Solutions for ADAS

17. Integration Target Today Target Car Network Sensor Data Car Network
Complex PCB for high-speed
RAM access
Flash
RAM
Integrate DSP, MCU & Memory
DSP
Host
Control
Flash
RAM
Simple PCB
Lower cost
Sensor Data
Car Network
DSP
Host
Control
Sensor Data
Car Network
Integration Advantages
Smaller footprint = lower weight, easier to place in vehicle
Lower power = less demand on electrical system
Lower cost PCB can be targeted
Advantages are multiplied with multiple units per vehicle

18. Computational Requirements for Image Recognition
MIPS
10,000
Eye Focus
Detection
Pedestrian
（Control）
Pedestrian Detection
（Warning）
Vehicle
Detection
（Control）
Camera Only
1,000
Driver Monitoring
Lane
Recognition
Detect
Rain
Fog
etc.
Sign Recognition
Vehicle
Detection
（Comfort）
100
Multi Sensors
Multi Sensors
It will be increasingly common for camera systems to integrate multiple functions, and these systems are pushing the performance of solutions targeting this space. 10
Requirements go up as camera technology advances in resolution and frame rates
Source: Hitachi Research Lab.

19. Power Strategy: Specialized Hardware
Power constraints impact how much performance can fit into a package
Strategy: Implement dedicated hardware for signal processing
Add more performance for less power versus faster CPU
Performance
Power
CPU 1
A dedicated image processor may not have the flexibility of a general-purpose CPU to handle multiple tasks, but when it comes to image processing, the dedicated hardware will be tremendously more efficient versus adding more CPU cores or increasing CPU frequency.
“work smarter, not harder”
CPU 1
CPU 0
CPU 0
General
Purpose
CPU
Dedicated
Image
Processor
General
Purpose
CPU
Dedicated
Image
Processor

20. ADAS Solution Evolution*
ADAS Solution Evolution
MP Start 　　　 　　　
R-Car/V
ARM CPU’s
w/ Signal Processing HW
Performance requirements continue to grow at high end
SH7766
w/ Signal Processing HW
Dot-Matrix
SoC
System-on-Chip
External Memory
Performance
IMAPCAR2
w/ Signal Processing HW
SH7766-L
w/ Signal Processing HW
RH850/V
RH850 CPU’s
w/ Signal Processing HW
As Renesas consolidates its roadmaps from the pre-merger products lines, the „V-series“ product families will encompass products targeting ADAS applications.
Power limits performance with integrated memory. New processes improve capability.
MCU
Internal Memory
SH7455/6
200MHz
DSP by FPU
*tentative naming 20

21. Building on ADAS Legacy*
Building on ADAS Legacy
The best technologies from pre-merger products are combined, with added support for new technology
2011
2013
R-Car/V
ARM CPU’s
w/ Signal Processing HW
Key Technologies
Signal Processing HW
Image Distortion Correction HW
Functional Safety
Etc.
SH7766
w/ Signal Processing HW
Performance
IMAPCAR2
w/ Signal Processing HW
SH7766-L
w/ Signal Processing HW
Key Technologies
Signal Processing HW
Functional Safety
High-performance Flash
Etc.
The V-series products will inherit and improve upon technologies from existing products that target vision and radar applications for Automotive
RH850/V
RH850 CPU’s
w/ Signal Processing HW
SH7455/6
200MHz
DSP by FPU
*tentative naming

22. SH7766 Solution Example

23. SH7766 – SoC for Top-View Park Assist
System
Package FCBGA 21x21 0.8bp
Interfaces
DMA
CPU
SH4A
533MHz
<1.0V + 1.8V + 3.3V Ta =-40°C to +85°C
2 x HSCIF
2 x CAN
4 x SPI
Timers
FPU
2 x HSPI
L1- I$ 32k / L1-D$ 32kB
3x Timer Unit
2x IIC
1x Watchdog Timer
4x PWM Unit
Vision/Graphics IP subsystem
Video interface
Memory interface
We will look at a recent product targeting the ADAS market as an example of how Renesas “works smarter, not harder” with solutions targeting ADAS. Here is the SH7766, targeting “surround view” applications. Included with this system-on-chip is hardware dedicated to combining multiple camera inputs into a birds-eye view of the vehicle and it surrounding environment, along with including capability to analyze the combined images for obstacle detection.
4x NTSC IN
2x Digital IN
SRAM I/F
44MHz
2D Gfx engine
Image Recognition
IMP-X2
4x IMR-LSX / 1x IMR
Image Correction Distortion
NOR Flash I/F tbc
66MHz
1x Video OUT
RGB666
DDR3 I/F
712 / 16-bit 23 23 23

24. Approach for Top-view System
1st step
Make top view picture in real time
2nd step
Parking lane, Pedestrian recognition
3rd step
Improve picture quality
Highlight concerns
Left camera
Right camera
Rear camera
Front camera !
Here, we break down the surround-view application into three basic steps, with some description of what parts of the SH7766 are active in performing each step.
4x IMR-LSX can do perspective correction in real-time
Parking lane, pedestrian recognition etc, works at real time with combination with CPU and IMP
Improve picture by
DRC (Dynamic Range Control) and draw graphics/icons with the 2D Graphic Engine

25. IMR: For Camera Perspective Correction
IMR: for Image data already in RAM
IMR-LSX: for Image data on Vin using line memory
SH-4A
533MHz
VIN （6ch :
4 Analog
2 digital）
Translated Image
IMR-LSX
IMP-X2
IMR-LSX
IMR-LSX （4ch）
IMR （1ch）
DDR3
IMR-LSX DU
IMR-LSX
Peripheral I/F and
modules
The IMR-LSX is a particularly efficient implementation of a MIPS-intensive process: performing the perspective correction transforming input from a camera pointed away from the vehicle into an output that is the “birds eye view”. This process is done in-line to the camera input, preserving memory bandwidth and lag versus having to first store the image in memory, read back from memory to do the correction, and then write the result back to memory. For flexibility, an IMR module is available for doing this type of operation on an image that is already in memory.
IMP-X2
IMR
Key Design Points:
IMR-LSX is in-line with video input, saving memory bandwidth
4-channel distortion correction is a ~10K MIPS operation. IMR- LSX is substantially more efficient than a general-purpose CPU

26. IMP: For Image Recognition Tasks
Computation volume of Normal image recognition applications
HIGH LEVEL
PROCESSING
Pattern Recognition
Tasks
10～30%
LOW LEVEL
PROCESSING
Fundamental
Image Processing:
Filtering on a whole image
Binarization, Histogram, etc
General
Purpose
Processor
70～90%
The IMP dedicated image processor is designed to churn through fundamental image processing tasks that are common to most types of applications. The general purpose CPU would then be responsible for high-level analysis, where more flexibility is required.
Dedicated
Processor (IMP)
from 10 to 20 times
Computation power
compared with CPU
General Purpose
Processor
General Purpose Processor +
Dedicated Image Processor

27. DRC: Dynamic Range Control
Dynamic range control can recover highlights and boost dark areas of images
Dynamic Range Control is implemented in hardware with support for multiple cameras
(No CPU needed!!)
(b)
(a)
Before
After
Luminance distribution
DRC is another function implemented in hardware much more efficiently than performing this type of operation with a general-purpose CPU
Before
After
Before
After
level
(a) Dark area control
(b) Bright area control

28. Next-gen Top-View: Moving to ARM
Similar concepts as SH7766
Focus on Digital Cameras, higher resolutions
Support Ethernet AVB Camera Networks or Direct Input
Use dedicated HW
IMP  next-gen
IMR-LSX2  next-gen
H.264 Decode
CPU CA15
TBD MHz x2
CPU
CA15
1.5GHz
IMP
2D Graphics
1Gb
EtherAVB
GMII
Gbit
Ether AVB
H.264/MJPEG
decoder
Main Bus 260MHz x 256bit x 2
The system concepts that make the SH7766 successful will move forward to next-generation products that will target digital cameras at higher resolutions and frame rates. Key technologies such as IMP and IMR-LSX will be improved to handle the additional processing requirements. OR
1280x1024
Camera Input x 5
VIN
0-4
IMR-LSX2
0-4

29. Next-gen Top View Example*
Next-gen Top View Example
4 to 5 Camera Support
Megapixel cameras
More processing throughput needed  improved IMP + faster CPU
Compress images as camera
H.264  Decode needed at ECU
Ethernet AVB to transport
Gbps Ethernet to receive multiple camera streams
Top
View
ECU
*tentative naming

30. Sensor Fusion & Smart Sensors

31. Sensor Fusion
Collect data from multiple sensors for a unified view of vehicle environment
Make decision based on the “big picture”
Fusion ECU
Compile
Sensor
Data
Analysis
&
Action
One of the applications in the ADAS space that may need some explanation: sensor fusion. The idea is to combine inputs from multiple sensors into a “big picture” of what is happening in and around the vehicle, to make a more informed decision about what actions to take.

32. Ethernet/CAN/Flexray
Sensor Fusion
For a vehicle platform with multiple sensors:
Use a distributed architecture
Process raw data in separate ECU’s
Top-View
ECU
Object Lists
Location
Direction
Velocity
Etc.
Steering
Front Cam
ECU
Braking
Fusion ECU
Front Radar
ECU
Ethernet/CAN/Flexray
CAN/Flexray
While integration is a target where possible, networking and processing considerations will limit what can go into one box.
A distributed architecture also makes it easier for Automotive manufactures to adjust vehicle content for different trim levels.
Throttle
Rear Radar
ECU
Action
Warning
HMI
Side Radar
ECU

33. Sensor Fusion Solutions*
Sensor Fusion Solutions
Update rate and number of sensors can drive CPU performance requirements for fusion
Break up processing tasks for more sensors
If output is to steering/braking/throttle, safety requirements are increased
Enable higher ASIL levels for decisions & communication
Compile
Analysis
Compile
Analysis
RH850/V
RH850 CPU’s
w/ Signal Processing HW
ASIL B
R-Car/V
ARM CPU’s
w/ Signal Processing HW
ASIL B
Compile
Analysis
Decision
Communicate
Decision
Communicate
Decision
Communicate
RH850/V
RH850 CPU’s
w/ Signal Processing HW
ASIL D
RH850/V
RH850 CPU’s
w/ Signal Processing HW
ASIL D
RH850/V
RH850 CPU’s
w/ Signal Processing HW
ASIL D
High End
Many Sensors
Low End
Few Sensors
Mid End
More Sensors
*tentative naming

34. Smart Sensor Solutions
Sensor fusion may require processing of raw radar or camera data “somewhere else”, outside of the fusion controller
Highly integrated solutions such as the RH850/V series make it easier to package the processing with the sensor

35. Smart Sensor – Camera Example
Power supply
1.25V
3.3V
RH850/V
Large SRAM
Flash
Image
Processor
CPU
RH850
perf
CPU
Flexray
PHYs
Video Input
CAN
Ethernet
High performance
Specialized image processing engine and MCU domains merged into single chip
Embedded memories
Local SRAM (2.5MB) and Flash (4MB) integration

36. Ecosystem

37. Customer Deployment Needs
Solutions
Lane Departure
Object Detection,
…etc.
Renesas is working with 3rd parties that can be solution providers or contractors.
Optimized Libraries
OpenCV
OpenVL
Libraries for common functions, including support for open API’s
Custom Development
Compilers, etc.
Tools for HW Accelerators: e.g. Smarter Compilers

38. Carmaker functionality Customer/Tier1 Application
Ecosystem Overview
3rd
Party
network
3rd party
network
Car
ECU
Middleware
Dvpt. Tools
Software
Dvpt. Tools Hardware
Hardware
Carmaker functionality
Infrastructure
Customer/Tier1 Application
System integration
Vision application SW
AUTOSAR
RTE
Demo PoC
Vision SDK
Video IF
Open std.
OpenCV
OpenCL
OpenGL
OpenVG
Customization
& porting
Renesas standard
MCU tools
ADTF – EB toolchain
MCAL
RTOS
Operating Systems
IDE : Eclipse
Debugger
Simulator
Compiler
linker
PRISM for
Multi-core
In addition to provide the normal set of tools for an MCU product, Renesas will work to develop software / middleware building blocks targeting ADAS applications. Renesas will also partner with 3rd parties to offer solution-level product offerings or contract development.
Hardware
debugger
Application board
Intermediate
devices
Companion chips
Device
PHY
bin
Power
Supply

39. Ethernet AVB in ADAS

40. Ethernet AVB in ADAS Ethernet is gaining traction in Automotive
Number of ECU’s and amount of data being transferred increasing
Infrastructure already exists in IT space, making it easier to adopt for Automotive versus developing new technology
Ethernet AVB adds guaranteed service and determinism for time-critical data
The bandwidth is necessary to move sensor data and results in ADAS applications
The determinism is critical: making a steering or braking decision based on out-of-date sensor data can be fatal!

41. Key Benefits for Ethernet / Ethernet AVB
Fast
100Mbps, 1Gbps, 10Gbps: in use. 40Gbps. 100Gbps: standardized
Fast, not expensive
Translate moving pictures at real time
Configurable/Controllable
Easy to add/update/remove equipments and sensors
Controllable including add/update/remove equipment and sensors
Configurable systems constructed with above equipment and sensors
Interoperability with external network
Can be connected with internet and cloud
Deploy enhanced Security concept
Ethernet driving down automotive cost of ownership
System Integration
MLB
INIC AV
Network
CPU
CPU
Debug
Diag AV
Network
Ether
Ether
AVB
Debug
Diag

42. Functional Safety

43. Functional Safety
As ADAS systems impact operation of critical vehicle systems, more effort is needed to verify that data is correct, decisions are acceptable, and commands are delivered
The amount of driving hours across all vehicles means that even relatively unlikely events can happen
Functional Safety defines design practices and device requirements to meet elevated reliability requirements
See the Functional Safety Session at DevCon for more detail

44. Renesas Functional Safety
RH850/P1x
RH850/V1x
V850E2/Px4
ISO26262
Compliant Development Flow & Safety Concept
ISO26262 Assessment
by TÜV Süd
"fit for purpose ASIL D"
V850E/PHO2
V850E/PHO3
V850E/PHC3
IEC61508 Assessment
by TUEV Süd
„fit for purpose SIL3“
Integr. Safety Concept
Redundant core in lockstep
Error Control Module
Multicore support
Guard concept
Dependent failure analysis
HW BIST (P-series only)
The key message here is that Renesas has experience in addressing industry requirements for functional safety. As functional safety requirements progress, Renesas is adding more capability to its products to keep pace.
Integr. Safety Concept
Redundant core in lockstep
HW BIST
Safety Guardian
ßIC measures
Safety Features
ECC on Flash
ECC/Parity on RAM
Clock Monitor
CRC hardware support
Core self test software
2008
2011
2015 44

45. Roadmap for Safety apps
LSDC: Lock step dual core
RH850/V1x Up to 4 MB Flash
Up to 2.5MB SRAM
LSDC + Perf
(w/ SIMD, FPU)
DSP engine
Autonomous Drive
Sensor Fusion
Smart camera
Radar
1800DMIPS
+ SIMD
+ DSP engine
ASIL D/B
RH850/P1x Up to 8 MB Flash
Up to 1MB SRAM
Up to 2x LSDC
(w/ FPU)
1350 DMIPS
ASIL D
Integrated Safety Unit
High-End Chassis Controller
Stability Control
EPS
Airbag with integrated sensors
670 DMIPS
450 DMIPS
ASIL D
The V-series is one of several families specifically targeting safety critical applications. One of the major differentiators for the V-series is the inclusion of the dedicated image/radar processing capability.
225 DMIPS
RH850/R1x 512 KB, 80 MHz, Single Core
ABS
Airbag
ASIL B
225 DMIPS 45 45 45 45

46. Questions?

47. ‘Enabling The Smart Society’ in Review…
This slide closes the loop on the Smart Society Challenge…
Challenge: “In the smart society, vehicles are incorporating more intelligence to detect threats, and then warn the driver and/or automatically respond (i.e. emergency braking). The computing solutions behind these active safety systems must balance high performance against requirements for integration and low power consumption.”  
“Renesas focuses on adding specialized hardware to active safety solutions to add performance with lower power, allowing for higher integration.”
Do you agree that we accomplished the above statement?

48. Please Provide Your Feedback…
Please utilize the ‘Guidebook’ application to leave feedback or
Ask me for the paper feedback form for you to use…