10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia AUTOSAR Conformance Tests - Feedback on their development and utilization Alain Feudjio-Vouffo, Fraunhofer Fokus (Germany), Alexandra Faucheux, Kereval (France), Mathieu Morvan, Kereval (France), Alain Ribault, Kereval (France), Ashish Jadhav, Patni Computer Systems (India), Philippe Robin, Technoveo (France), Facing Up to the Future of Testing

AUTOSAR conformance tests - feedback on their development and utilization 10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia Content of the presentation  Introduction: AUTOSAR CTSpecs release status  AUTOSAR CTSpecs & abstraction from hardware platform  Metrics on AUTOSAR CTSpecs TTCN-3 code and test environment  Combination of conformance test and source code coverage  Looking into the future Theoretical view on combination and reuse of TTCN-3 suites to create new ones From ATS for module to ATS for stack and integrated software platform  Example of deployment in the automotive industry  Conclusion 2

AUTOSAR conformance tests - feedback on their development and utilization 10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia Introduction - AUTOSAR CTSpecs release status  AUTOSAR conformance test specifications (CTSpecs) have been released on 15 April as part of Release4.0R0002 Conformance tests are available for 43 modules (including the RTE)  A CTspec contains the specification of the conformance test suite the TTCN-3 scripts for dynamic test cases the configuration sets to be used when executing the Conformance Test Suite  Number of test cases6820 Number of dynamic tests (TTCN3)2754 Number of static tests4066 3

AUTOSAR conformance tests - feedback on their development and utilization 10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia AUTOSAR CTSpecs & abstraction from hardware platform (1/2) 4 TTCN-3 files Adapters COM Module Under Test HW Specific COM Module Under Test Adapters Generator TTCN-3 Executable TTCN-3 Compiler KEREVAL' Platform ETS on PC Instrumented MUT on board  Adapters are generated  ETS is hardware independent HW Specific COM Module Under Test HW Specific COM Module Under Test HW Specific COM Module Under Test Execution of the CTS

AUTOSAR conformance tests - feedback on their development and utilization 10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia Metrics on AUTOSAR CTSpecs TTCN-3 code and test environment 55 CTSpec, TTCN-3 scripts and Configuration Set(s) Test Management System Reporting CAPITALIZED or GENERATED Test suite execution results Software, Hardware and Test design tool Test suite execution Attestation Configurations and environment Sources CUSTOMER MANUAL EFFORT BSW preparation (configuration) BSW integration Integration Reception Source Code Inspection Configuration Inspection Operation Signature Tests Dynamic Tests

AUTOSAR conformance tests - feedback on their development and utilization 10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia AUTOSAR CTSpecs & abstraction from hardware platform (2/2)  Test Memory Management allows Test Suite to access memory of the board 6 HW Specific COM Module Under Test Instrumented MUT on board Test Memory Management Adapters COM TTCN-3 Executable ETS on PC Allocate / Deallocate Read / Write / Compare /*... */ var PointerAddr_ v_AddrPointer; pt_TestMemoryAccess.call(TestMemoryAccess_Allocate:{v_Size}, nowait); alt { []pt_TestMemoryAccess.getreply(TestMemoryAccess_Allocate:{v_Size}) -> value v_AddrPointer { /*... */ } /*... */ pt_Fee.call( Fee_Read:{v_BlockNumber, v_BlockOffset, v_AddrPointer, v_Size}, nowait ); /*... */ TTCN-3 files module FeeJo

AUTOSAR conformance tests - feedback on their development and utilization 10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia Combination of conformance test and source code coverage 7  A Code Coverage Analysis tool (RTRT) is used during Conformance Test execution  Provides  An indicator of tests relevance according to structure  An effort estimation to go over functional validation from conformance testing  Information on dead code and unspecified code  Required in Safety Validation context  ISO and DO178B  Thanks to coverage measurement, conformance testing can be part of validation process

AUTOSAR conformance tests - feedback on their development and utilization 10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia Reuse of the TTCN-3 suites  Two types of reuse are envisaged for the module-level TTCN-3 ATS that are available for the 43 modules. 1) Reuse of the suites for creating the ATS for the later revisions of the AUTOSAR standard during the maintenance phase of the current release, and 2) Reuse of the suites for creating the ATS for the future releases of the AUTOSAR standard  The ATS has been created having maintainability as a major design criterion. TTCN-3 provides the high level artifacts in the ATS to enable the reuse.  Traceability between the specifications and the test cases and configuration parameters has been created to simplify the upgrade process. 8 RE-USE of Artifacts in the ATS AUTOSAR Rel 4.0 Rev 2 Rev 3 … AUTOSAR Rel 4.1 Rev 0 Rev 1 … …

AUTOSAR conformance tests - feedback on their development and utilization 10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia Combination of TTCN-3 suites to create new ones – extensibility  Different users of AUTOSAR might be interested in extending the existing ATS by having more tests that are vendor specific and do not exist in the current ATS for internal test compliance  The TTCN-3 suites are well structured and provide an ease for combining with other vendor specific test cases to extend the tests for internal test compliance  The TTCN-3 suite is well structured and provides ease of extensibility for test cases, configuration parameters and combinations of configuration parameters 9 AUTOSAR ATS Vendor Specific Module Tests Additional Tests Required by OEMs Extensibility of TTCN3 AUTOSAR ATS (for internal test compliance)

AUTOSAR conformance tests - feedback on their development and utilization 10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia Higher level Test Suites  The ATS is currently at module level  Using TTCN-3 provides ease of extensibility to combine test-cases and reuse code with additional test cases to obtain a test suite for a set of inter-connected modules  Some of the stubs and drivers would have to be replaced by actual modules and suites used for testing of stacks like the CAN stack, the Flexray stack, the communication stack, etc. can thus be prepared  In such type of higher level suites various AUTOSAR modules can be simultaneously tested  The test cases of the ATS can also be used to develop test suite for testing of a full AUTOSAR stack  The test cases of the ATS can also be used for developing test suites for testing of AUTOSAR compliant software components 10 Full AUTOSAR STACK ATS for Combination of Modules Module ATS

AUTOSAR conformance tests - feedback on their development and utilization 10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia Example of deployment in the automotive industry  Feedback on executions Average of 10 non conformities detected per Module Under Test at first CT execution  Typical non conformities detected on each module 11 Configuration data set Source Code inspection Static Compile Time Dynamic test suite Parameter container error Parameter range error Parameter type error Parameter missing Enum definition error Parameter type error include missing signatures or prototypes error parameters code generation error (from XML information) bad condition of function deactivation argument return unexpected API invoked expected API not invoked  Such non conformities have not been detected during validation of the module

AUTOSAR conformance tests - feedback on their development and utilization 10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia Non-regression tests for any changes in: - Configuration or Implementation, - HW platform Example of deployment in the automotive industry 12  Integrated with customers development plan Multi run conformance tests Unit Tests Coding Conformance checking according to AUTOSAR description files (non-TTCN-3 tests) Module Conformance checking Confirmation test and attestation delivery Functional design SW Architecture design Integration Tests Functional Tests

AUTOSAR conformance tests - feedback on their development and utilization 10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia Conclusions  Hardware dependent ETS is obtained from ATS and is used for testing the MUT, this ensures that the AUTOSAR CTSpecs are portable  The use of AUTOSAR CTSpecs TTCN-3 code and test environment by CTAs need a very low percentage of manual activities and up to 95% of activities can be tool-based  AUTOSAR Conformance testing can be combined with other tests like source code coverage so that independent test runs can be reduced resulting in overall reduction in test execution time  AUTOSAR ATS has maintainability and extensibility enabled by use of TTCN3 for efficient reuse in creating new ATS for future releases, ehancements and higher level testing  AUTOSAR Conformance testing can be integrated in the overall AUTOSAR development process so that testing can be performed in early stages of the development and can result in detecting non-conformances even at design stage which would improve the development efficiency 13