1. Group Truck Technology, Powetrain Engineering, Control Systems dept.
Model Based Testing in a seamless design process model – What is the business case?
Group Truck Technology, Powetrain Engineering, Control Systems dept.
Sébastien Ruzza

2. Model Based Testing in a seamless design process model / Objectives
In the Automotive industry :
The product complexity is increasing every day.
More and more requirements are delivered.
Verification activities duration are constantly challenged
Human resources & test equipment are rationalised
Evaluate Model Based Testing using a Control System study – case:
What are the benefits and the limitations?
What is the business model?
What is the process model?
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

3. Traditional Verification approach, the assumptions
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

4. Traditional Verification approach
Requirement driven verification
The main system verification approach used at Control Systems is “Requirement driven verification”
 Requirements are covered by Test Cases.
Specification
Verification
PXFS – REQ - 001
STC – A
PXFS – REQ - 002
PXFS – REQ - 003
STC – B
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

5. Traditional Verification approach
Limitations
If this approach has lots of advantages, some limitations also exist:
If the specification is not exhaustive, the verification will not be exhaustive.
The Verification activity is proportional to the number of requirements.
In case of Verification time reduction, which risk is taken, if a part of the verification is not executed?
Let’s illustrate this…
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

6. Traditional Verification approach
Traditional approach: effort is proportional the to number of requirements to cover
Satisfaction of the test objective T1
Limitations
Set of detected bugs
If the verification is stopped at T1, how can we ensure that the potential bugs are not in the set of requirements and associated in use cases we didn’t cover?
Testing effort
Actual verification duration
Required verification duration for a full coverage
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

7. Traditional Verification approach
Limitations
If this approach has lots of advantages, some limitations also exist:
If the specification is not exhaustive, the verification will not be exhaustive.
The Verification activity is proportional to the number of requirements.
In case of Verification time reduction, which risk is taken, if a part of the verification is not executed?
Selection of the test values is tricky.
Let’s illustrate this…
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

8. Traditional Verification approach
Limitations
PXFS- REQ-1: Engine shutdown request shall be sent if Engine Speed < 700 rpm
Equivalency Class technique is used to define test cases
(define in IEC61508 and ISO15998 and recommended by SPICE)
TRUE: Engine shutdown request is sent
FALSE: Engine shutdown request is NOT sent
Engine Speed
0rpm
700 rpm
In practice we select one value in the centre of the range. Is it enough? Often the issue is on ONE singular value. Which one to take?
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

9. Model – Based Testing
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

10. Test Data Distribution
Model – Based Testing
Why is it a Stochastic approach?
S1:
Engine is RUNNING
Usage model: a representation of how the system will be used under operating conditions.
P = 0.05
P = 0.95
S2:
Stall Engine
S3:
Stop & Start shutdown
S4:
Engine is OFF
Test Data Distribution
Use path Probability
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

11. + + Model – Based Testing 0.9 0.1 What are the input required?
Operational profiles & statistics of usage are implemented into the model. They are inputs to:
Improve the functional coverage,
Execute a Risk driven verification (the most probable)
Evaluate the reliability of the system (user oriented)
S1:
Engine is RUNNING
S2:
Stall Engine
S3:
Stop & Start shutdown
0.9
S4:
Engine is OFF
Specifications
FUNCTIONAL REQUIREMENT +
0.1
OPERATIONAL PROFILE +
Cope with the non completeness of the specification
USE CASE
USAGE MODEL
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

12. VERIFICATION ENGINEER AUTOMATIC TEST SEQUENCES
Model – Based Testing
What are the output?
MANUAL TESTING
Different algorithms are used depending on the objective: End – User oriented, reliability, risk driven, …
TEST PROCEDURE
VERIFICATION ENGINEER
Test Generator
USAGE MODEL
AUTOMATIC TEST SEQUENCES
TEST SEQUENCER
AUTO TESTING
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

13. MBT evaluation , Fact & Figures
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

14. Analysis & Results / Process Mapping, Activities
Req & design
Verification
MIL
Sys REQ
ANALYSIS
<test execution>
<test execution>
HIL
System Verif
Model Based Testing
SYSTEM LEVEL
<active review>
DESIGN
<usage model>
SIL
<test execution>
SW REQ
ANALYSIS
SIL
SW Verif
SW LEVEL
DESIGN
Model Based Design (Simulink/ASCET)
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

15. Analysis & Results / Process Mapping, Role
Req & design
Verification
MIL
Sys REQ
ANALYSIS
HIL
System Verif
Model Based Testing
SYSTEM LEVEL
DESIGN
Verif Engineer
SIL
SW REQ
ANALYSIS
SIL
SW Verif
SW LEVEL
DESIGN
Model Based Design (Simulink/ASCET)
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

16. Problem Report Distribution - Type
Analysis & Results
Some figures
Notes:
These results correspond to 2 verification phases for 2 different releases.
On the graph below, the aim is to show the different types of issues found using MBT.
Problem Report Distribution - Type
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

17. Problem Report Distribution – MBT vs. Traditional
Analysis & Results
Some figures
Notes:
On the graph below, the aim is to show the number of issues found with the traditional approach and the MBT approach.
In parallel to the MBT Verification, manual test was performed on the traditional way.
Problem Report Distribution – MBT vs. Traditional
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

18. Analysis & Results Some figures
3 test objectives: User oriented, Structural coverage, Reliability assessment
Covered System Requirements: 45
Generated Test procedure: 407 (xml and automatic test sequences)
Test Equipment: Hardware In the Loop
Auto Test Library: 17 test functions
The last results:
STATUS
Number %
Passed
250
61%
Failed
157
39%
Error 0%
TOTAL
407
100%
 Driving 19 Problem Reports
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

19. Model – Based Testing / Observation
Testing efficiency
Better efficiency at the beginning, when the most risky errors are discovered
Maximise the testing time on the most probable bugs from a customer prospective
Satisfaction of the test objective
Traditional approach
Model Based Testing
Testing effort
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

20. Cost for Test Creation* Cost for Test Execution*
Analysis & Results
Cost
What is the development cost to create the Test Procedures and execute them for 45 Systems Requirements?
Model Based Testing
Traditional approach
For Manual Execution
For Auto Execution
Test Procedure format
Textual test procedure
Auto test sequences
System Test Cases
Max nbre of Test Cases
Infinite
~45
Cost for Test Creation*
~10 days
~40 days
Cost for Test Execution*
~3 days
~0 (nightly execution)
(*): this is an estimation based on a first try and evaluation using one tool from the market.
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

21. Observations & Conclusions
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

22. Keys for success Human resources Technical scope
Train the people  This is a different way of testing
The generated test procedures for manual execution must not be executed by beginners High level procedures
Involve System experts to define the operational profile Secure the correct usage is verified
Technical scope
Select the appropriate system functions.
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

23. Keys for success Process Test Equipment
Map it as a V&V activity in the process
Identify clearly who shall deliver: Usage model & Test procedure
Test Equipment
Development of Automatic steps take more time than expected  Auto test library integration takes time
To test the reliability of a system, first, the test equipment must be reliable 407 Auto Test Case stimulate intensively the test equipments
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018

24. Question & Answers
Group Truck Technology, Powertrain Engineering, Sébastien Ruzza
12/5/2018