September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 1 SRM Reuse Presented by Thomas W. Otani Joint work with Doron Drusinsky, Bret Michael and Man-Tak Shing Naval Postgraduate School Monterey, CA

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 2 Disclaimer The views and conclusions in this talk are those of the author and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government The views and conclusions in this talk are those of the author and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 3 Outline Introduction Introduction Assertion Validation Testing Patterns Assertion Validation Testing Patterns SRM Reuse SRM Reuse Assertion Library Organization Assertion Library Organization Conclusions Conclusions

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 4 System Reference Model We advocate the use of System Reference Model (SRM) for the IV&V team to capture its understanding of the problem domain We advocate the use of System Reference Model (SRM) for the IV&V team to capture its understanding of the problem domain The SRM contains The SRM contains Use case scenarios, static and dynamic domain models (expressed in UML), and a set of formal assertions to model precisely the required behavior of the system Use case scenarios, static and dynamic domain models (expressed in UML), and a set of formal assertions to model precisely the required behavior of the system The formal assertions must be executable so the modelers can confirm the true meaning of the assertions via scenario simulations The formal assertions must be executable so the modelers can confirm the true meaning of the assertions via scenario simulations

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 5 System Reference Model (cont’d)

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 6 Statechart Assertions Visual, intuitive, and resemble UML statechart design models Visual, intuitive, and resemble UML statechart design models Reduce linguistic discontinuity Reduce linguistic discontinuity Turing equivalent Turing equivalent Statechart + Transition actions in Java Statechart + Transition actions in Java Executable Executable Simpler to validate and debug Simpler to validate and debug Support runtime execution monitoring Support runtime execution monitoring

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 7 Statechart Assertion Example Example 1: R1 The engine must not run for more than 10 minutes

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 8 Outline Introduction Introduction Assertion Validation Testing Patterns Assertion Validation Testing Patterns SRM Reuse SRM Reuse Assertion Library Organization Assertion Library Organization Conclusions Conclusions

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 9 Assertion Validation Formal assertions for sequencing behaviors often contains subtle errors Formal assertions for sequencing behaviors often contains subtle errors Possible causes of incorrect assertions Possible causes of incorrect assertions 1.Incorrect translation of the natural language to formal assertion (You said it correctly, but didn't formalize it correctly) 2.Incorrect translation of the requirements, as understood by the modeler, to natural languages (You understood, but didn't say it correctly) 3.Incorrect cognitive understanding of the requirements (You didn't understand)

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 10 Assertion Validation (cont’d) To ensure that the stated assertion correctly represents the intended behavior, we run validation test scenarios To ensure that the stated assertion correctly represents the intended behavior, we run validation test scenarios In our approach, test scenarios are implemented as JUnit test cases and executed in Eclipse-based StateRover tool In our approach, test scenarios are implemented as JUnit test cases and executed in Eclipse-based StateRover tool

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 11 Statechart Assertion Example Example 2: R2 An event Q must occur within 30 seconds of every event P The next three slides show the different statechart assertions that attempt to capture this requirement The next three slides show the different statechart assertions that attempt to capture this requirement

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 12 Assertion for R2: Version A The 30-second interval that begins at the first occurrence of P

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 13 Assertion for R2: Version B The 30-second interval that begins at the most recent occurrence of P

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 14 Assertion for R2: Version C Non-deterministic transition A new 30-second interval for each occurrence of P

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 15 Test Scenario 1 for R2 This scenario does not distinguish SA2a, SA2b and SA2c This scenario does not distinguish SA2a, SA2b and SA2c sa.P( ); sa.incrTime(20); sa.Q( ); assertTrue(sa.isSuccess()); √√ √

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 16 Test Scenario 2 for R2 This scenario distinguishes SA2b from SA2a and SA2c This scenario distinguishes SA2b from SA2a and SA2c sa.P( ); sa.incrTime(20); sa.P( ); sa.incrTime(20); sa.Q( ); assertFalse(sa.isSuccess()); X√√ Note that SA2a SA2c

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 17 Test Scenario Patterns We need a good test suite to differentiate the different interpretations of the natural language requirements in order to discover the correct meaning of the intended behavior We need a good test suite to differentiate the different interpretations of the natural language requirements in order to discover the correct meaning of the intended behavior Here is a list of test patterns that must accompany every assertion Here is a list of test patterns that must accompany every assertion Obvious Success Obvious Success Obvious Failure Obvious Failure Event Repetitions Event Repetitions Time-interval Repetitions Time-interval Repetitions Overlapping Time-interval Repetitions Overlapping Time-interval Repetitions

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 18 Obvious Success Example of a trivially successful case for R2: Example of a trivially successful case for R2: sa.P( ); sa.incrTime(20); sa.Q( ); assertTrue(sa.isSuccess());

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 19 Obvious Failure Example of a trivially unsuccessful case for R2: Example of a trivially unsuccessful case for R2: sa.P( ); sa.incrTime(35); sa.Q( ); assertFalse(sa.isSuccess());

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 20 Event Repetitions Example of test cases involving multiple events Example of test cases involving multiple events R3: An event Q must occur between an event P and an event R time PPPQR X√√√ PPQQR √√√√

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 21 Time-Interval Repetitions Example of an unsuccessful case involving repeating time-intervals for R2: Example of an unsuccessful case involving repeating time-intervals for R2: time PQPQ 30 X√

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 22 Overlapping Time-Intervals Example of an unsuccessful case involving overlapping time-intervals for Example of an unsuccessful case involving overlapping time-intervals for R4: An event Q may not occur within 30 seconds of any event P time P P Q 30 X

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 23 Outline Introduction Introduction Assertion Validation Testing Patterns Assertion Validation Testing Patterns SRM Reuse SRM Reuse Assertion Library Organization Assertion Library Organization Conclusions Conclusions

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 24 SRM Reuse l The availability of reusable SRMs can lessen the difficulties in developing correct formal assertions Benefits of SRM reuse include Benefits of SRM reuse include Accelerated development Accelerated development Reuse avoids “reinventing the wheel” efforts, saves time and reduces development cost Reuse avoids “reinventing the wheel” efforts, saves time and reduces development cost Increased quality of the products Increased quality of the products The SRM artifacts capture the experience base of expert analysts and contain domain knowledge necessary for the accurate understanding of the correct system behaviors The SRM artifacts capture the experience base of expert analysts and contain domain knowledge necessary for the accurate understanding of the correct system behaviors SRM reuse enables the transfer of that experience between practitioners SRM reuse enables the transfer of that experience between practitioners

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 25 Statechart Assertion Reuse Benefits Reduce the burden of the system analysts Reduce the burden of the system analysts The system designers no longer need to always specify assertions from scratch The system designers no longer need to always specify assertions from scratch The reusable statechart assertions can serve as a guidance on how best to use language features to solve a specific problem The reusable statechart assertions can serve as a guidance on how best to use language features to solve a specific problem Improves the quality of the formal assertions Improves the quality of the formal assertions The accompanied validation test scenarios clearly define the dynamic behavior of the reusable assertions and can serve as a checklist of the various potential alternative scenarios that needed to be examined by the analysts The accompanied validation test scenarios clearly define the dynamic behavior of the reusable assertions and can serve as a checklist of the various potential alternative scenarios that needed to be examined by the analysts

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 26 There are different levels of reuse From W. C. Lim, Managing Software Reuse, Upper Saddle River, N.J.: Prentice Hall, 1998, p Ad hoc Primarily directed at code leverage or reuse Short-term time horizon Focus on individual effort Systematic Domain- Oriented Strategy- Driven Time and effort Planned, orderly process Reuse of assets other than code Infrastructure to support reuse Emphasis on domain analysis to identify and design assets for reuse Collection of coherent assets for reuse Changes strategic and operational process/structure of organization Future markets and products partly determined by capacity to exploit reuse Value from reuse

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 27 Systematic Reuse versus Domain-Oriented Reuse Similarities Similarities Both have reuse in mind when developing software artifacts Both have reuse in mind when developing software artifacts Both have processes, methods, and organizational infrastructure in place to support reuse Both have processes, methods, and organizational infrastructure in place to support reuse Differences Differences Systematic reuse Systematic reuse Reusable artifacts are products of individual projects Reusable artifacts are products of individual projects Smaller startup cost Smaller startup cost Domain-oriented reuse Domain-oriented reuse Reusable artifacts are products of domain analysis and are designed/developed for the domain (instead of for individual projects) Reusable artifacts are products of domain analysis and are designed/developed for the domain (instead of for individual projects) Larger startup cost, bigger pay-off in the long run Larger startup cost, bigger pay-off in the long run

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 28 Types of Reuse Instantiation Instantiation A generic assertion is instantiated as a concrete assertion A generic assertion is instantiated as a concrete assertion E.g. Whenever P then Q within T becomes Whenever (signal turns green) then (signal turns yellow) within 30 secs. E.g. Whenever P then Q within T becomes Whenever (signal turns green) then (signal turns yellow) within 30 secs. Adoption Adoption A pre-existing concrete assertion is copied and used in another application (as is or with a slight modification) A pre-existing concrete assertion is copied and used in another application (as is or with a slight modification)

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 29 Instantiation Reuse To make the instantiation reuse possible, we need To make the instantiation reuse possible, we need A searchable library of generic assertions (along with the associated validation test scenarios) A searchable library of generic assertions (along with the associated validation test scenarios) A simple mechanism (tool) to instantiate a generic assertion as a concrete assertion A simple mechanism (tool) to instantiate a generic assertion as a concrete assertion A way to search validation test scenarios, independent of assertions A way to search validation test scenarios, independent of assertions

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 30 Outline Introduction Introduction Assertion Validation Testing Patterns Assertion Validation Testing Patterns SRM Reuse SRM Reuse Assertion Library Organization Assertion Library Organization Conclusions Conclusions

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 31 Library Organization To allow a ease and efficient lookup of the assertions catalog, we organize generic assertions into categories from simple to more complex ones To allow a ease and efficient lookup of the assertions catalog, we organize generic assertions into categories from simple to more complex ones Single events Single events Multiple sequencing events Multiple sequencing events Counting Counting

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 32 Single Events Never P Never P Eventually P Eventually P These are unbounded assertions so they are not particularly useful in practice. But they are important in showing the most basic form of writing assertions. These are unbounded assertions so they are not particularly useful in practice. But they are important in showing the most basic form of writing assertions.

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 33 Example 5 Test Scenarios: sa.P( ); assertFalse(sa.isSuccess()); assertTrue(sa.isSuccess()); Obvious Success Obvious Failure

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 34 Logical combination of single event assertions Eventually P and Eventually Q Eventually P and Eventually Q Eventually P or Eventually Q Eventually P or Eventually Q Eventually P xor Eventually Q Eventually P xor Eventually Q Eventually P and Never Q Eventually P and Never Q Eventually P or Never Q Eventually P or Never Q Eventually P xor Never Q Eventually P xor Never Q

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 35 Example 6

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 36 Test Scenarios for Example 6 assertFalse(sa.isSuccess()); sa.P(); assertFalse(sa.isSuccess()); Obvious Failure Obvious Success sa.Q(); sa.P(); assertTrue(sa.isSuccess()); Event Repetitions sa.Q(); assertFalse(sa.isSuccess()); sa.P(); sa.Q(); assertTrue(as.isSuccess());

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 37 Adding Timing Constraints to Single Events Never P within T Never P within T Never P after T Never P after T Eventually P within T Eventually P within T Eventually P after T Eventually P after T

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 38 Multiple Sequencing Events Whenever P then Eventually Q Whenever P then Eventually Q (Eventually Q after P) (Eventually Q after P) Whenever P then Never Q Whenever P then Never Q (Never Q after P) (Never Q after P) Whenever P then Eventually Q but before R Whenever P then Eventually Q but before R (Always Q between P and R) (Always Q between P and R) Whenever P then Never Q before R Whenever P then Never Q before R (Never Q between P and R) (Never Q between P and R)

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 39 Multiple Sequencing Events (cont'd) Never P before Q Never P before Q Always P before Q Always P before Q Always P and Q alternate Always P and Q alternate (Consecutive P's and consecutive Q's are not allowed) (Consecutive P's and consecutive Q's are not allowed)

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 40 Example 7

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 41 Test Scenarios for Example 7 sa.Q(); assertFalse(sa.isSuccess()); Obvious Failure Obvious Success sa.P(); sa.Q(); assertTrue(sa.isSuccess()); Event Repetitions sa.P(); sa.Q(); assertTrue(sa.isSuccess()); sa.Q(); assertTrue(sa.isSuccess());

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 42 Adding Timing Constraints to Sequencing Assertions Whenever P then eventually Q within T Whenever P then eventually Q within T Whenever P then never Q within T Whenever P then never Q within T Whenever P then eventually Q after T Whenever P then eventually Q after T Whenever P then never Q after T Whenever P then never Q after T

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 43 Example 8

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 44 Test Scenarios for Example 8 sa.P(); sa.incrTime(20); sa.Q(); assertFalse(sa.isSuccess()); Obvious Failure Obvious Success sa.P(); sa.incrTime(40); assertTrue(sa.isSuccess()); Event Repetitions sa.P(); sa.Q(); sa.incrTime(20); sa.Q(); assertFalse(sa.isSuccess());

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 45 Test Scenarios for Example 8 (cont’d) Time-Interval Repetitions sa.P(); sa.incrTime(40); sa.Q(); assertTrue(sa.isSuccess()); sa.incrTime(10); sa.P(); sa.incrTime(20); sa.Q(); assertFalse(sa.isSuccess()); Overlapping Time Intervals sa.P(); sa.incrTime(10); sa.P(); sa.incrTime(25); sa.Q(); //ok for 1 st P, but not the 2 nd P assertFalse(sa.isSuccess());

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 46 Counting Whenever P then < n Q before R Whenever P then < n Q before R Whenever P then ≥ n Q before R Whenever P then ≥ n Q before R Whenever P then < n Q within T Whenever P then < n Q within T Whenever P then ≥ n Q within T Whenever P then ≥ n Q within T < n Q within any T < n Q within any T ≥ n Q within any T ≥ n Q within any T Note: n > 0 for assertions Note: n > 0 for assertions

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 47 Example 9

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 48 Test Scenarios for Example 9 sa.P(); sa.incrTime(40); assertFalse(sa.isSuccess()); Obvious Failure Obvious Success as.P(); as.incrTime(10); as.Q(); as.incrTime(10); as.Q(); assertTrue(sa.isSuccess());

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 49 Test Scenarios for Example 9 (cont’d) Time-Interval Repetitions as.P(); as.incrTime(5); as.Q(); as.incrTime(5); as.Q(); as.incrTime(5); as.Q(); assertTrue(sa.isSuccess()); as.incrTime(10); as.P(); as.incrTime(10); as.Q(); as.incrTime(50); assertFalse(sa.isSuccess());

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 50 Conclusions l Effective reuse of domain specific assertions help reduce the burden of the system designers and increase the reliability and quality assurance of the system under test Effective assertion reuse requires Effective assertion reuse requires Library organization for effective assertion lookup Library organization for effective assertion lookup Generic assertions with adequate test coverage Generic assertions with adequate test coverage Short-term and long-term reuse strategy is needed to maximize the potential benefits of SRM reuse Short-term and long-term reuse strategy is needed to maximize the potential benefits of SRM reuse Systematic versus Domain-oriented Systematic versus Domain-oriented

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 51 Questions?

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 52 Backup Slides

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 53 Iterative Assertion Development

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 54 The Bandera Temporal Specification Patterns Occurrence patterns - require states/events to occur or not to occur in a given scope Order patterns – constrain the order of state/events within a given scope

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 55 Example of an bounded existence pattern R10: P can occur at most 2 times between Q and R The LTL assertion [ ] ((Q & <>R) -> ((!P & !R) U (R | ((P & !R) U (R | ((!P & !R) U (R | ((P & !R) U (R | (!P U R)))))))))) The Statechart assertion

September 8, 2008NASA IV&V Facility Workshop on Validation Morgantown, WV 56 Example of a chain precedence pattern R11: The event sequence (S, T) precedes P between Q and R The LTL assertion [ ] ((Q & <>R) -> (!P U (R | (S & !P & o(!P U T))))) The Statechart assertion