1. CS527 Topics in Software Engineering (Software Testing and Analysis) Darko Marinov September 15, 2011

2. Schedule First few lectures to help you select projects –Shared memory: CHESS, IMUnit, CAPP –Message passing: Setac –Comment analysis: iComment –Regression testing: survey paper –Today: Model-based testing –Next week two papers 3 votes: generation/evaluation of test assertions 2 votes: mock-based testing, GUI testing 1 vote: genetic algorithms, concolic testing, web testing Your project can be something we don’t cover

3. Paper Today Model-based quality assurance of protocol documentation: tools and methodology Wolfgang Grieskamp, Nicolas Kicillof, Keith Stobie and Victor Braberman (STVR 2010)Model-based quality assurance of protocol documentation: tools and methodology Journal paper –Not too long –Otherwise, we prefer conference papers Follows: personal slides on models for data, slides on MBT, questions Thanks to Wolfgang Grieskamp for MBT slides

4. Developed Two Kinds of Test Abstractions Declarative: describe what inputs look like –Two kinds of languages used Declarative language for properties of desired inputs [SOFTMC’01, ASE’01, FME’02, OOPSLA’02, SAT’03, MIT’03, J-ASE’04, SAT’05, LDTA’06, ALLOY’06] –Properties written in the Alloy modeling language –Uses Alloy Analyzer for generation of test inputs Imperative language for properties of desired inputs [ISSTA’02, TR’03, MIT’04, ICSE-DEMO’07, STEP’07, FSE’07] –Properties written in implementation language (Java, C#…) –Developed Korat for generation of test inputs Imperative: describe how to generate inputs –More recent work on ASTGen [FSE’07]

5. Korat at Microsoft Research Korat implemented in the AsmLT test tool in Foundations of Software Engineering group –Predicates in Abstract state machine Language (AsmL), not in Java or C# –GUI for setting finitization and manipulating tests –Korat can be used stand-alone or to generate inputs for method sequences –Some extensions (Controlled) non-exhaustive generation Generation of complete tests from partial tests Library for faster generation for common datatypes

6. AsmLT/Korat at Microsoft Used by testers in several product groups Enabled finding numerous errors –XML tools XPath compiler (10 code errors, test-suite augmentation) Serialization (3 code errors, changing spec) –Web-service protocols WS-Policy (13 code errors, 6 problems in informal spec) WS-Routing (1 code error, 20 problems in informal spec) –Others SSLStream MSN Authentication … Errors found in –Important real-world applications –Code already well tested using best testing practices

7. Some Comments from Microsoft Positive comments on AsmLT and Korat –“So far our stateless AsmL models are pretty successful.” –“AsmL parameter generation tool is quite convenient and powerful.” Negative comments on AsmLT not Korat –“Most of our testers prefer to write as much C# as possible.” –“Very difficult to debug AsmL.” Result: new tool for C# (.NET) SpecExplorer Korat is Korat

8. Questions for Discussion (1) Are there any open source tools equivalent to Spec Explorer? [KN] How would this methodology change if applied to something but protocols? [JT] In the case where the number of internal unit- tests that need to be tested, is it worth it at all to do MBT? [AY] –What's the right time to apply model-based testing? [XW] What other selection strategies exist but transition coverage? [AK]

9. Questions for Discussion (2) … possible to scale up MBT to make it work on large-sized systems? [AD] Does model based testing scale down to small projects? [DeG] NLP … to aid in … converting technical documentation into testing reqs? [CM] –Is it possible to reverse engineer a spec from a program or test suite? [SB] test models generated automatically? [YL] How to totally/partially automate slicing? [FS] –How does the slicing affect the success? [SO]

10. Questions for Discussion (3) integrate with regression testing? [KB] generate a test that checks to see if an action disallowed by the model program is in fact disallowed by the System Under Test? [DwG] Can Spec be used to generate proprieties for Chess? [SB] How are bugs in the model treated? [AK] Is the modeling language enough to model various types of applications? [SO]