1. World Class Standards Review of TC MTS (Methods for Testing and Specification) Dieter Hogrefe © ETSI 2009. All rights reserved Board#75, 3-4 November 2009

2. World Class Standards B65(08)xx 2 Description MTS is responsible for the identification and definition of advanced specification and testing methods, which take advantage of innovative techniques to improve the efficiency and economics of both the standard description and associated conformance and interoperability testing processes Methodologies are established, in co-operation with the CTI and relevant Technical Committees, for future use in the development of standards, and of testing specifications, including interoperability issues.

3. World Class Standards B65(08)xx 3 Terms of Reference Standardization of testing languages and methodology development Standardization of TTCN-3 Interoperability testing Methodology IP Testing Voip SIP IPv6 generic test suites Standards Engineering Standards Development Process Making Better Standards

4. World Class Standards B65(08)xx 4 Structure Plenary level TTCN-3 Interoperability methodology Standards Engineering WG IPT Voip, SIP, IPv6

5. World Class Standards B65(08)xx 5 Technical overview TTCN-3 - Extended standard series New language concepts e.g. module interfaces Updated and additional language mappings: ASN.1, IDL, XSD New documentation tags New extension packages: advanced parameterization, advanced types, static configurations Development of TTCN-3 test suites in many application domains VoIP testing, ETSI IPv6 testing, ETSI LTE testing, ETSI/3GPP Automotive software testing, AUTOSAR Mobile enabling services, OMA Mobile communication, TETRA

6. World Class Standards B65(08)xx 6 Technical overview Establishment of a TTCN-3 syllabus and certificate Cooperation of ETSI, GTB and iSQI Two accredited TTCN-3 training providers First exams Over 20 certificates issued Improvement of TTCN-3 tool interoperability Development of a TTCN-3 reference test suite based on a snapshot of 3GPP LTE benchmark tests 1st TTCN-3 Plugtest test with commercial tools The Plugtest has brought tool makers closer together No grey areas identified in TTCN-3 standards based on the compilation and execution of this LTE benchmark test suite

7. World Class Standards B65(08)xx 7 1 st ETSI TTCN-3 Tool Plugtest 2009

8. World Class Standards B65(08)xx 8 Technical overview Automated interoperability testing (IOT) Developed generic methodology for automated IOT that integrates conformance checks with pure IOT Extends ETSIs general approach to interoperability testing Independent of technology to be tested Independent of test language, i.e., TTCN-3 Defines generic IOT test architecture Applied concepts in development of highly reusable, library based TTCN-3 framework for automated IOT Specified 50 tests for IMS core network NNI IOT (TC INT) Developed first TCI based ETSI SIP/SDP codec Developed first TRI based ETSI IOT adapter Includes upper tester and PCAP capture reader adapter Used TTCN-3 test system in TC INTs 3 rd IMS Plugtest (Oct 09)

9. World Class Standards B65(08)xx 9 Technical overview Means of Interoperability Testing EUT 1 Interface Monitor Test Coordinator Configuration interface Application Support Nodes Monitoring interface SUT Test Oracle EUT 2 EUT N Interconnecting Network Stimulating interface Equipment User

10. World Class Standards B65(08)xx 10 STFs STF 370 Automated Interoperability Testing Funded by EC and ETSI Objective is to extend existing ETSI interoperability testing concepts for the purpose of automation and in context of distributed systems Main stakeholders ETSI TC Methods for Testing and Specification (owner of ETSI WIs) ETSI TC INT (IMS Network Testing) ETSI TC GRID B2B community (mainly around HL7) TETRA Association WiMax Forum (NWIIOT) Further signalled interest IPv6 community ITS community Testing labs

11. World Class Standards B65(08)xx 11 STFs STF 380 TTCN-3 extension and maintenance continuation Resolution of open CRs Cooperation with STF 160 on LTE testing Extensions for real-time and performance Real-time concepts and semantics Timing constraints Support for simulated time Support for performance tests and benchmarks Reference test suite (voluntary, based on inputs from tool vendors)

12. World Class Standards B65(08)xx 12 Major activities/deliverables Standardization of testing languages and methodology development TTCN-3, Edition 4 Automated IOT IP testing IPv6 generic test suites Standards Engineering MBS WEBSITE addition of Security pages to MBS in support of OCG SEC

13. World Class Standards B65(08)xx 13 Present problems the usual ones, i.e. workload, funding, travel restrictions of members, etc.

14. World Class Standards B65(08)xx 14 Successes/failures TTCN-3 Success Fast growing user community Increase of application domains Strong international conference series T3UC Growing number of TTCN-3 tools International TTCN-3 syllabus and certificate Shortcomings Limited number of freely available/open source tools Learning not well supported consider to establish TTCN-3 OS tools 20092009, Bangalore, India 20092009, Sophia-Antipolis 20082008, Madrid, Spain * 20072007, Beijing, China 20072007, Stockholm, Sweden 20062006, Berlin, Germany 20052005, Sophia-Antipolis 20042004, Sophia-Antipolis

15. World Class Standards B65(08)xx 15 Successes/failures MBS Success Cooperation with OCG-SEC to produce Security Standards process guidelines (TR 102 790) and security extensions to the Making Better Standards website Shortcomings Problem to provide the necessary methodological and validation documentation to support the ETSI Secretariat in their implementation of the MTS/OCG-IOP recommendations to The Board on improving the Standards Engineering Process.

16. World Class Standards B65(08)xx 16 Successes/failures Automated interoperability testing (IOT) Successes Automated IOT concepts successfully validated in IMS Plugtest Even event participants strongly valued conformance results Feedback on standards collected Demonstrated that TTCN-3 is independent of a specific compiler by working in all project phases with at least two commercial TTCN-3 tools Open source contributions to TTCN-3 community, i.e., TCI SIP/SDP codecs and TRI IOP test adapter Effort savings of more than 50% for IOP trace analysis in IMS Plugtest due to TTCN-3 based IMS IOT tool! Shortcomings Due to lack of IMS client availability no validation of TTCN-3 framework part dedicated to driving interoperability tests Due to funding shortage not all IMS IOT conformance checks could be automated, e.g., XML SIP message body & DNS message checks

17. World Class Standards B65(08)xx 17 How we operate ! Meetings Semi-annual plenaries Ad hoc meetings Electronic working Mailing lists Mantis for CR handling

18. World Class Standards B65(08)xx 18 Different to other TBs because… Horizontal issues Testing language TTCN-3 for use in other bodies and CTI Background material for use in other bodies and CTI Research oriented Responsible for the identification and definition of advanced specification and testing methods Methodologies are established, in co-operation with the relevant Technical Committees, for future use in the development of standards, and of testing specifications, including interoperability issues

19. World Class Standards B65(08)xx 19 Globalization aspects ITU-T TTCN-3 became Recommendation Z.16x Close collaboration with SG 17 on standardization methodology TTCN-3 User Conferences Asia (in the meantime 10000s of users, esp. China and India) Europe (same) North America coming TTCN3 usage all over the world is a big success

20. World Class Standards B65(08)xx 20 Where next ? TTCN-3 extensions Real time testing Performance testing Test Abstractions High Level Test descriptions Test Automation Model based?