ETSI TTCN-3 Test Suites QUALITY IMPROVEMENT Anthony Wiles CTI Director

Validation of ETSI Test Suites ETSI makes every effort to validate Test Suites before publication The rules are described in “Guidelines for STF test suite validation” Validation Level 1 (Minimum) The test suite is compiled on at least one TTCN test tool (preferably more than one) but the tests are not executed against real products. Validation Level 2 (Best Effort) Same as Level 1 but the tests are also executed against a real product or Implementation Under Test (IUT). Level 3 validation (Rigorous) Similar to Level 2 validation but more rigorous in terms of the number of platforms and/or IUTs. All tests are compiled and executed on more than one test platform.

Is it enough? However, with limited validation resources/funding available to the STF often only level one is selected (and the minimum of level one!) Incompatibilities may arise due to: TTCN-3 version used vs. version implemented on the tool. E.g., Test suites checked with a version of the language/tools available prior to publication (i.e., tools and language spec may have moved on) => maintenance problem Misinterpretations of the TTCN-3 standard -> TC MTS Bad use of non-standard TTCN-3 syntax (to be avoided!) Problem: To ensure highest number of tool compatibility with the minimum amount of effort Ensure users understand extent and scope of validation done

Proposed ways to improve the situation Ensure the following information shall be written in the ToR for the STF (not always done): Level of validation to be used (and costs) Version of TTCN-3 to be used (all parts) Indication of the T3Q rule set to be used Provision of a ‘Continuous Integration Server’ for use by STFs and CTI Provision of validation/quality checklist for the STF (tied to contractual milestone) Ensure Validation Report is included in the published Test Suites Exact contents of the checklist/validation report still under discussion but could be: level of validation used, version of TTCN-3 used, results of the validation process (relevant to potential users) etc. Better information sharing with users, tool makers, TC MTS, ETSI CTI Web based tool for easy download of ETSI test suites together with bug reporting

Possible Improvement Process Libraries MANTIS TTCN-3 Code Release CTI and STF Coordination (Checklist) Continuous Integration Server Validation Report in published TS 1-n TTCN-3 tools (depending on validation level) Ease of use/efficiency for STF ETSI CTI

ES 201 873-9 ETSI has now 3 test tools with XML support STF405 has used all tools and observed different conversion results (conversion from XML to TTCN-3 types) Proposal to have conf calls between vendors to align the conversion Eventual bug reports/feature requests on –part 9 to be registered in Mantis (TTCN-3) Decision on how to proceed with actual STF405 work is to be taken by TC INT