1. New Assessment & Test Methods
VMAD-02-09
New Assessment & Test Methods
Clarification of (new) tasks assigned to VMAD & way forward

2. Framework Document on AVs
The “Framework Document on Automated/Autonomous Vehicles” (WP ) introduced the work priorities for a new Informal Working Group named “New assessment / Test methods”
According to the Document, the group responsibilities are:
Assessment of Functional Safety
Audit
Simulation
Electronic System Compliance
Test Track
Real World Driving Evaluation
Digital Identity (not defined)
MULTI-PILLAR APPROACH as presented by OICA
(WP.29/2019/2)

3. Framework Document on AVs
The “Framework Document on Automated/Autonomous Vehicles” (WP ) introduced the work priorities for a new Informal Working Group named “New assessment / Test methods”
Group deadlines/deliverables: [March 2020] – NOT ENDORSED
General Deliverables
Specific Deliverables
Review of the existing and upcoming methods and a proposed way forward for the assessment for Automated / Autonomous Driving
The test and assessment method for Lane Keeping Systems of SAE levels 3/4 as New UN Regulation for Contracting Parties to the 1958 Agreement

4. Industry Proposal (Assessment)
A group led by the United Kingdom already developed amendments to the “CEL Annex” intended for ALKS with support from Germany, France, The Netherlands, Japan and OICA/CLEPA
The amendments proposed by the UK at GRVA 2nd session (GRVA/2019/4, as amended by GRVA-02-19), are deemed by industry as sufficient for the new UN Regulation on ALKS
If a Contracting Party to the 1958 Agreement would like to develop it further, given such short-term deadlines, Industry believes the most efficient way would be to rely on the existing UK Technical Group

5. Industry Proposal (Test Methods)
The same rationale applies to the development of test methods for ALKS, that should be dealt within the existing ACSF Informal Working Group
A complete new group would need some time to learn the detailed technical background of the ALKS requirements (not yet finalised), and the required “learning curve” would delay progress
Until GRVA organization is adopted and implemented, industry believes the “New assessment / Test methods” group should focus on the new certification concepts (e.g. the multi-pillars approach)
These proposals are guided by the willingness to secure the delivery of the ALKS new regulation in due time

6. Way Forward for the VMAD Informal Working Group

7. + The analysis should help to derive dedicated tasks for VMAD
Discussion Paper
Can Item x,y,z be reliably assessed through a “Classical Certification Approach”?
Safety Principles
(Framework Document)
Yes
Classical Certification Tests Methods No
Partially
Transition Phase
(e.g. ALKS)
Conventional Track Testing +
CEL Step 2
IWG ACSF
CEL UK
Increasing Level of Complexity
Higher Automation
New Assessment & Test Methods
Multi-Pillar Approach
(Simulations; Audit; Track Testing; Real World Driving)
VMAD
The analysis should help to derive dedicated tasks for VMAD

8. Conclusions
New Assessment & Test Methods (NATM) is a priority because conventional compliance test methods cannot capture the full range of automated driving challenges
NATM does not replace conventional testing; instead it complements testing such as done under classical type approval
NATM seeks methods that can be applied under the 1958 and 1998 Agreements
VMAD’s role should be to submit Technical Proposals for New Assessment & Test Methods to GRVA for further consideration

9. Backup
Background Information

10. Draft Agenda GRVA-03 (03-04 June 2019)
For the special GRVA session in June the following input connected to VMAD is expected according to the draft agenda:
5. Automated Lane Keeping System (ALKS)
(a) …
(b) New assessment / test method
GRVA may wish to receive information on the work done by the Validation Method for Automated Driving (VMAD) group that may be relevant specifically for ALKS.
(c) ….
(d) Functional safety, Complex Electronic (CEL) control system
GRVA agreed at its second session on the need to develop functional safety and Complex Electronic (CEL) control system provisions suitable for ALKS. GRVA may wish to review any proposal available on this matter, if available.
Documentation: (ECE/TRANS/WP.29/GRVA/2019/4 Informal document GRVA-02-19)

11. Overview of Amendments of “CEL Annex” for ALKS Outcome of UK ad-hoc group. Ref: ECE/TRANS/WP.29/ GRVA/2019/4, as amended by GRVA-02-19
Description and verification of the “control strategies”, which ensure “robust and safe operation of the function(s) of "The System" in response to a specific set of ambient and/or operating conditions…”).
Declaration of enabled and disabled safety related functions when the hardware and software are present in the vehicle at the time of production.
Functional safety, misuse and unauthorized modifications “The Technical Service shall perform an assessment of the application of the analytical approach(es). The assessment shall include:
(a) Inspection of the safety approach at the concept (vehicle) level with confirmation that it includes consideration of:
interactions with other vehicle systems;
Malfunctions of the electronic control system […];
Situations when a system free from faults may create safety critical risks (e.g. due to a lack of or wrong comprehension of the vehicle environment);
Potential misuse or simple unauthorised modification of the system.
This approach shall be based on a Hazard / Risk analysis […]
(b) Inspection of the safety approach at the system level (FMEA, fault tree) […]
(c) Inspection of the validation plans and results […]”