Presentation is loading. Please wait.

Presentation is loading. Please wait.

Software engineering and standardisation section (TEC-EME) 1 Automatic test case generation from a formal model LTG technology evaluation based on a B.

Similar presentations


Presentation on theme: "Software engineering and standardisation section (TEC-EME) 1 Automatic test case generation from a formal model LTG technology evaluation based on a B."— Presentation transcript:

1 Software engineering and standardisation section (TEC-EME) 1 Automatic test case generation from a formal model LTG technology evaluation based on a B model of a part of ATV onboard software Julien ORSAT – Stagiaire from the University of Franche- Comté (France) 08/03 – 23/07/2004 Supervisor: Philippe CHEVALLEY

2 2 Outline B modelling for Leirios Test Generator ATV software case study: Gyra functional unit Gyra B formal model LTG automatic test case generation Conclusion

3 3 Outline B modelling for Leirios Test Generator B formal notation LTG requirement traceability LTG syntax restrictions ATV software case study: Gyra functional Unit Gyra B formal model LTG automatic test case generation Conclusion

4 4 B formal notation Used as input language for LTG Other supported input format : Statecharts/Statemate, UML Part of the B method Code generation from the model Proof obligations to ensure correctness Several refinement steps from the model to the target code Created in the 90s by J-R Abrial Applied in railway transports and smart card industry LTG input: 1st step of the B method (abstract machine) Text format Statecharts and UML are graphically used

5 5 Some B concepts Sets ; could be considered as base types GYRO_ID = {G1, G2, G3, G4} GYRO_MODE = {COARSE, FINE} Relations ; to define sets of ordered pairs gyra_mode : GYRO_ID --> GYRO_MODE Substitutions ; to modify variable values gyra_mode := gyra_mode FINE)} Operations ; to describe system behaviour Used by LTG to compute test cases

6 6 B operation example A B operation is considered atomic <+ is the relation overriding operator cmd_report <-- setGyroMode(gyro, mode) = PRE /* precondition */ gyro : GYRO_ID & mode : GYRO_MODE & … THEN /* body */ IF condition THEN cmd_report := REJECTED ELSE cmd_report := ACCEPTED || gyra_mode := gyra_mode mode) } END Profile B reserved word Commentary Boolean expression Substitution

7 7 LTG requirement traceability Provided by LTG not part of B notation Uses tags in the model Based on B commentaries IF (...) THEN cmd_report := REJECTED ELSE... Allows automatic traceability report generation Test case -> covered requirements Requirement -> covering test cases Requirement coverage criteria for generated test case evaluation

8 8 LTG restrictions to the B notation Syntax Real numbers Difficulty to compute expected results Sequences list-like data structures Such as set description { element | predicates } Single B machine only No machine inclusion, promotion …

9 9 Outline B modelling for Leirios Test Generator ATV software case study: Gyra functional unit Gyra Functional Unit ATV software structure Case study choice explanation Gyra B formal model LTG automatic test case generation Conclusion

10 10 Gyra Functional Unit High level services to use a 4 GYRoscope Assembly (GYRA) Hardware control Monitoring algorithms Processing algorithms Designed and developed by EADS 89 requirements 13 Ada83 packages code lines Managed by an internal finite state machine (FSM)

11 11 ATV software structure Gyra Functional Unit is part of software layer 3 LN 2 library LN 1 library LN 3: … LN 3: Gyra FU Device #n Hardware Software layers 28 FUs … … … LN 3: … Device #1 Hardware interface Communications Applications

12 12 Case study choice explanation ATV spacecraft Uses current space software techniques (software under development) Gyra application Easy access to Gyra specification documents Focus on the FSM Heavily used in ATV: 28 FSMs, 989 transitions Gyra FSM has an average size (38 transitions) FSMs are difficult/expensive to test Transition testing is one of the most, if not the most, demanding tasks of the product validation EADS test plan Would be valuable to automate test case generation

13 13 Outline B modelling for Leirios Test Generator ATV software case study: Gyra functional Unit Gyra B formal model Modelled Finite State Machine Requirement traceability Automatic test case generation Conclusion

14 14 Modelled Finite State Machine OFF_N ON_N ON_4 ON_1 ON_2 ON_13 ON_14 ON_3 ON_24 ON_12 ON_23 ON_34 Transitions are activated by commands Redundant commands are accepted (requirement 16241) FSM modes are groups of FSM states (OFF mode, ON mode)

15 15 Modelled FSM key figures 12 states 22 graphically represented transitions + 14 different reflexive transitions would be 102 more transitions on the picture distinct from command redundancy transitions 750 line modelmodel NB: separated model for Survival modes (3 states, 2 transitions – 150 lines) ON_x (11 states) Example: GYRA_TR_MODE According to EADS: 1 transition Modelling choice: 11 different transitions ON_x (11 states)

16 16 Requirement traceability Traceable Gyra requirements: 43 / 49 Non traceable requirements (6) : Initial values of variables (3), No LTG tag for INITIALISATION part of the model High level requirements, refined by others (2) FSM mode shall be computed from FSM state (1) That is to say not stored in memory – code inspection is required

17 17 Outline B modelling for Leirios Test Generator ATV software case study: Gyra functional unit Gyra B formal model LTG automatic test case generation Test generation criteria Generated test cases Ada test program generation Conclusion

18 18 Test generation criteria Model coverage criteria: 4 possibilities Decision coverage (DC) Condition/Decision coverage (C/DC) Modified condition/Decision coverage (MC/DC) Multiple condition coverage (MCC) An LTG test case is made of 1. Preambule 2. Body + Identification 3. Postambule With Gyra, test programs may end in any FSM state no need for a postambule Software criticality

19 19 Generated test cases 179 functional abstract test cases generated = 179 operation activation sequences XML format (HTML and RTF translators) Comparison to EADS test plan Chaining of several tests (8 large test sequences) difficult to compare Traceable requirements covered by LTG: 37/43 Difficulty with generic requirements (5) Multiple tags for one requirement Were those requirements fully covered ? 1 requirement was not covered: some behaviours, related to 1 B operation, were not reached different generation criteria should solve the problem. HTML example

20 20 EADS test plan coverage Gyra FSM state and transition coverage OFF_N ON_N ON_4 ON_1 ON_2 ON_13 ON_14 ON_3 ON_24 ON_12 ON_23 ON_34

21 21 Coverage comparison State coverage EADS test plan: 9/12 LTG: 12/12 Graphically represented transition coverage EADS test plan: 14/22 LTG : 22/22 Reflexive transition coverage EADS: 13/14, tested on one state LTG: 13/14, tested on every state uncovered transition is related to uncovered B operation ON_x (11 states) Example: GYRA_TR_MODE In EADS test plan: 1 test case Our choice: 11 test cases

22 22 Required to generate test programs from test cases: Tagged test program pattern Relation mapping between B operations and Ada services Test program generation on a simple model Test program example Not enough time to generate test programs Ada test program generation

23 23 Outline B modelling for Leirios Test Generator ATV software case study: Gyra functional unit Gyra B formal model LTG automatic test case generation Conclusion

24 Test Cases LTG Test Case Generator Test Case Generation Test Generation Criteria LTG eXecutable Script Builder Test Execution Environment Test Driver Generation Test Driver Schemas 49 Gyra functional requirements B Model (750 lines) Functional Modeling LTG Model Animator Model Validation LEIRIOS Test Generator done to do 24

25 25 Conclusion Technology evaluation: B notation Particularly suited to model finite state machines Difficulty to model real time aspects like scheduling Added requirement traceability Some restrictions on the notation Automatically generated test cases on the case-study 15 hour computation on a desktop PC, for 179 test cases Coverage is similar or better compared to EADS test plan LTG product stability, documentation quality to be raised Transition from research prototype to commercial product is in progress

26 26 Perspectives Short-term perspectives Generate and execute Gyra FSM test programs 2 possibilities: by Estec or by EADS Structural coverage study Fault injection in the Gyra implementation to evaluate test efficiency Generate tests for Gyra algorithms Long-term perspectives LTG input format comparison for space projects B notation Statecharts UML

27 Software engineering and standardisation section (TEC-EME) 27 Automatic test case generation from a formal model LTG technology evaluation based on a B model of a part of ATV onboard software Julien ORSAT – Stagiaire from the University of Franche- Comté (France) 08/03 – 23/07/2004 Supervisor: Philippe CHEVALLEY


Download ppt "Software engineering and standardisation section (TEC-EME) 1 Automatic test case generation from a formal model LTG technology evaluation based on a B."

Similar presentations


Ads by Google