Softeam’s testing process: a story Cyril Ballagny, SOFTEAM | ModelioSoft The 13th CREST Open Workshop London, England 13th of May 2011

2 Introduction to Softeam  A French SME (about 230 people)  More than 20 years of expertise in OO development and Modeling  Consulting, Training, Engineering activities  Software Publishing: Modelio and Objecteering  Strong R&D efforts since its creation –Innovator since 1989 (Modeling, MDA, UML) –Heavily involved in R&D cooperative projects (FITTEST, MADES, ENOSYS…) –Participates in the OMG standardization efforts since 1996 (UML profiles, SOAML, SysML…)

3 Introduction to Modelio  A CASE tool  For UML modeling  For MDA application  For Code generation (Java, C#, C++…)  For documentation generation (HTML, word…)  For team work and project sharing  And much more (BPMN, SysML, SOAML…)!

4 Modelio development process  Agile development with bootstrapping and continuous integration  20 people involved  1 million LOC (C++/Java)  A new release every 6 months

5 Modelio testing process (1)  Code quality management with Sonar: about 500 coding rules  Continuous integration with Hudson: report broken build (inconsistent API, missing source in the repository…)  Bug tracking with Mantis: everyone from the team can report bugs  Specification of test plans with Test Link (natural language): about 2000 procedures  Bootstrapping: Modelio is validated by developers

6 Modelio testing process (2)  A test case is either  A Mantis bug (bug 2123, bug 9478…)  A menu item (create, delete, mask…)  A diagram artifact (state, region, activity…)  Most test cases are UI-based and derived from general scenarios (e.g. apply command “xxx” to element in {a,b,c…} )  Test case execution is externalized to Eastern Europe

7 Our issues  Specification often changes  Writing operational test specification is too time consuming compared with natural language specification  Test code must be tested too  Mix of languages (C++/Java/Jython)  No tools for testing HMI combining Java and C++

8 Our expectations (1)  Reduce over-testing  Detect overlapping sections of test cases  Recombine sections to constitute smaller test cases with greater coverage  Apply regression tests only on parts of the system that have changed  Reduce under-testing  Detect system parts that are not covered by the test cases  Detect sequences of interactions allowed by the GUI that have not been tested

9 Our expectations (2)  Reduce Time-To-Market  Generate test cases instead of specifying them  Prioritize:  Identify the most used features  Identify the most frequent bugs

10 To sum up  We would like a scalable and pragmatic approach whose ROI has been demonstrated

Thank you for your attention! Andrey Sadovykh SOFTEAM | ModelioSoft SOFTEAM R&D Web Site: ModelioSoft Web Site:

M o d e l i n g s o l u t i o n s.