1 Jan Tretmans Radboud University Nijmegen (NL) © Jan Tretmans Radboud University Nijmegen together with: University of Twente Enschede (NL) TorX Automated Model Based Testing with Formal Methods

© Jan Tretmans Radboud University Nijmegen 2 TorX : A utomated Model Based Testing with Formal Methods Contents  Model based testing  Formal, model based testing with transition systems  Transition systems testing and ioco  A Tool for transition systems testing  TorX  Current and future research  What does it mean for ARTIST2

© Jan Tretmans Radboud University Nijmegen 3 Automated Model Based TestingAutomatedModel BasedTesting model IUT conf model TTCN test cases pass fail test tool test generation tool test execution tool IUT passes tests IUT conf model     soundexhaustive

© Jan Tretmans Radboud University Nijmegen 4 Model Based Testing  Testing with respect to a (formal) model / specification  SDL, CSP, Lotos, Promela, UML, state diagrams, Spec#,....  Precise, formal definition of correctness  good and unambiguous basis for testing  Formal validation of tests  Algorithmic derivation of tests  tools for automatic test generation  Maintenance of models / specifications, not of test suites  regression testing

© Jan Tretmans Radboud University Nijmegen 5 Model Based Testing with Transition Systems s  LTS i  IOTS i ioco s pass fail test tool der : LTS   (TTS) t || i i || der(s)  pass i ioco s    soundexhaustive pass fail model IUT conf model test tool test generation tool test execution tool IUT passes tests IUT conf model    soundexhaustive 

© Jan Tretmans Radboud University Nijmegen 6 Formal Testing with Transition Systems exec : TESTS  IMPS   (OBS) der : LTS   (TTS) T s  TTS s  LTS IUT  IMPS ioco i IUT  IOTS passes : IOTS  TTS  {pass,fail} Proof soundness and exhaustiveness:  i  IOTS. (  t  der(s). i passes t )  i ioco s Test hypothesis :  IUT  IMP.  i IUT  IOTS.  t  TTS. IUT passes t  i IUT passes t pass / fail

© Jan Tretmans Radboud University Nijmegen 7 i ioco s = def   Straces (s) : out (i after  )  out (s after  ) Implementation Relation ioco Correctness expressed by implementation relation ioco: p  p=  !x  L U  {  }. p !x out ( P )= { !x  L U | p !x, p  P }  {  | p  p, p  P } Straces ( s )= {   (L  {  })* | s  } p after  = { p’ | p  p’ }

© Jan Tretmans Radboud University Nijmegen 8 Algorithm To generate a test case from transition system specification s 0 compute T(S), with S a set of states, and initially S = s 0 after  ; 1end test case pass For T(S), apply the following recursively, non-deterministically: 2supply input !a T ( S after ?a   ) Test Generation Algorithm allowed outputs or  : !x  out ( S ) forbidden outputs or  : !y  out ( S ) 3observe output fail T ( S after !x ) fail allowed outputsforbidden outputs ?y  ?x

© Jan Tretmans Radboud University Nijmegen 9 Validity of Test Generation For every test t generated with algorithm we have:  Soundness : t will never fail with correct implementation i ioco s implies i passes t  Exhaustiveness : each incorrect implementation can be detected with a generated test t i ioco s implies  t : i fails t

© Jan Tretmans Radboud University Nijmegen 10 A Tool for Transition Systems Testing: TorX  On-the-fly test generation and test execution  Implementation relation: ioco  Mainly applicable to reactive systems / state based systems;  specification languages: LOTOS, Promela, FSP, Automata TorX IUT observe output offer input next input specification check output pass fail inconclusive user: manual automatic

© Jan Tretmans Radboud University Nijmegen 11 TorX Tool Architecture explorerprimerdriveradapter IUT spec. states transitions abstract actions concrete actions specification text TorX IUT specification

© Jan Tretmans Radboud University Nijmegen 12 TorX

© Jan Tretmans Radboud University Nijmegen 13 TorX Case Studies  Conference Protocol  EasyLink TV-VCR protocol  Cell Broadcast Centre component  ‘’Rekeningrijden’’ Payment Box protocol  V5.1 Access Network protocol  Easy Mail Melder  FTP Client  “Oosterschelde” storm surge barrier-control  DO/DG dose control  Laser interface academic Philips LogicaCMG Interpay Lucent LogicaCMG academic LogicaCMG ASML/Tangram

© Jan Tretmans Radboud University Nijmegen 14 What has been Achieved …… Sound and precise formal basis for model based testing  ioco test theory  proved test derivation algorithm Test tool TorX  prototype tool for model-based formal testing  “is at least as good as conventional testing”  supports test generation and test execution  more, longer, and provably correct test cases Applied successfully to different cases studies

© Jan Tretmans Radboud University Nijmegen 15 ? money ? button1 ? button2 ! coffee ! tea test case fai l ! money ! button2 ? tea fai l ? coffee pass  n: int  [ n  35 ] -> [ n  50 ] -> with data model and time and hybrid c := 0 c < 10 c < 15 [ c  5 ] -> c := 0 d V t / dt = 3 d V c / dt = 2 V c := 0 [V c = 10 ] -> V t := 0 [V t = 15 ] -> ?coin1 ?coin3 ?coin2 and action refinement ? Testing Transition Systems:StatusExtensions

© Jan Tretmans Radboud University Nijmegen 16 Current and Future Research Twente & Radboud  Testing real-time aspects  multi-channel real-time  Testing complicated data structures  transformational- + transition system based testing  Action refinement  when an abstract action is implemented as sequence of actions  What is a good test suite  test selection and test coverage  Test adapter and test interface  generic test environment  Compositionality and integration testing  differences diminish

© Jan Tretmans Radboud University Nijmegen 17 Current and Future Research Twente & Radboud  Hybrid testing  when continuous variables occur  Compositionality and integration testing  differences diminish  Testing stochastic and probabilistic properties  Multi-disciplinary  system testing  Relations between model checking, testing, static analysis, theorem proving, etc.  differences diminish .....

© Jan Tretmans Radboud University Nijmegen 18 Some Dutch Testing Projects  Côte de Resyste ( ) - Conformance Testing of Reactive Systems: TorX PhilipsTU Eindhoven (LogicaCMG) (KPN) LucentUni. of Twente (Interpay)  Atomyste -ATOm splitting in eMbedded sYStem TEsting Uni. of TwenteRadboud Uni. Nijmegen  Stress - Systematic Testing of Real-time Embedded Systems  Testing real-time propertiesUni. of Twente  Testing data-intensive systemsRadboud Uni. Nijmegen  Tangram - Model Based Testing and Diagnosis  Testing ASML Wafer Stepper machines - application oriented ASML, ESI, TUD, TUE, UT, RU, S&T, TNO

© Jan Tretmans Radboud University Nijmegen 19 ARTIST2 Activities Quantitative Testing & Verification 1. Theory for testing embedded systems a.real-time aspects b.data aspects c.extended conformance testing theories integrating a. and b. d.test action refinement 2. Verification and scheduling a.real-time schedulability analysis b.optimal control synthesis 3. Verification of stochastic systems a.model checking algorithms for CTMC, MDP b.integration of performance analysis into verification 4. Tool-oriented research a.data structures for real-time and stochastic modelling and analysis b.test interfaces and test adapters 5. Application of testing and verification tools in industrial settings a.collection of case studies b.comparison c.identification of links to industrial tools

© Jan Tretmans Radboud University Nijmegen 20 ARTIST2 Activities Quantitative Testing & Verification 1. Theory for testing embedded systems a.real-time aspects b.data aspects c.extended conformance testing theories integrating a. and b. d.test action refinement 2.Verification and scheduling a.real-time schedulability analysis b.optimal control synthesis 3.Verification of stochastic systems a.model checking algorithms for CTMC, MDP b.integration of performance analysis into verification 4.Tool-oriented research a.data structures for real-time and stochastic modelling and analysis b.test interfaces and test adapters 5. Application of testing and verification tools in industrial settings a.collection of case studies b.comparison c.identification of links to industrial tools