1. World Class Standards Achieving Grid Interoperability The ETSI Approach Stephan Schulz, Ph.D. ETSI Centre for Testing and Interoperability

2. World Class Standards 2 Presentation Outline  On achieving interoperability  Conformance versus interoperability testing  ETSI test specification development  Requirement extraction  Test purposes  Conformance Tests  Interoperability Tests  Creation of test specifications at ETSI  Centre for Testing and Interoperability  Specialist Task Forces  On grid testing  Grid STF  Review of Byte IO and Grid RPC test specifications

3. World Class Standards 3 Interoperability is …  The ultimate aim of standardisation in Information and Communication Technology  The red thread running through the entire standards development process, it’s not an isolated issue  Not something to be somehow fixed at the end  ETSI approach  ETSI produces with the required degree of parallelism Base Standards and Test Specifications  Base standards should be designed with interoperability in mind Profiles can help to reduce potential non-interoperability  Two complementary forms of testing Conformance testing Interoperability testing (more formal IOT”)  Testing provides vital feedback into the standardization work

4. World Class Standards 4 About Conformance Testing  Is Black-Box testing  Stimulation and Response Test System Device Point of Control and Observation (PCO) Reqs.

5. World Class Standards 5 Conformance is Usually Tested Layer-by-Layer lat ii 123 456 789 * 8# latigid Test System SUT API MMI L3 L2 PHY SUT = System Under Test IUT = Implementation Under Test L3 Upper Tester Lower Tester Lower Adapter Upper Adapter L3 Test

6. World Class Standards 6 Characteristics of Conformance Testing  Tests IUT against requirements specified either in a base specification or profile (standard)  Gives a high-level of confidence that key components of a device or system are working as they were specified and designed to do  Usually limited to one requirement per test  High IUT control and observability  Can explicitly test error behaviour  Can provoke and test non-normal (but legitimate) scenarios  Can be extended to include robustness tests  Test execution can be automated and repeated  Requires a test system development (and executable test cases)  Can be expensive (e.g., radio-based test system)  Tests under ideal conditions  Tests are thorough and accurate but limited in scope  At level of detailed protocol messages, service primitives, or procedure calls

7. World Class Standards 7 Limitations of Conformance Testing  Does not prove end-to-end functionality (interoperability) between communicating systems  Conformance tested implementations may still not interoperate This is often a specification problem rather than a testing problem! Need for minimum requirements coverage or profiles  Tests individual system components  But a system is often greater than the sum of its parts!  Does not test the user’s ‘perception’ of the system  Standardised conformance tests do not include proprietary ‘aspects’  Difficult to test via proprietary interfaces, e.g., user APIs  Can however be done by a manufacturer with own conformance tests for proprietary requirements

8. World Class Standards 8 About Interoperability Testing  End-to-End Interoperability of devices  What is being tested?  Assignment of verdicts?  Need to distinguish between  More formal interoperability testing (= test specifications) versus  Informal interoperability events used to validate/develop technologies (e.g., ETSI Plugtests like events) Device 1 Device n Device 2

9. World Class Standards 9 SUT API MMI L3 L2 PHY API MMI L3 L2 PHY QE = Qualified Equipment EUT = Equipment Under Test ETSI Methodology for Interoperability Testing Interoperability Testing (of terminals) QEEUT Means of Communication (MoC)

10. World Class Standards 10 Characteristics of Interoperability Testing  It is system testing  Tests a complete device or a collection of devices  Shows that (two) devices interoperate  But within a limited scenario  Tests at a ‘high’ level (perception of users)  Tests the ‘whole’, not the parts or layers E.g., protocol stacks integrated with applications  Tests functionality Shows function is accomplished (but not how)  Does not usually involve test systems  Uses existing interfaces (standard/proprietary)  Interoperability testing looks at end-to end functionality  Less thorough but wide in scope  Gives a high-level of confidence that devices (or components in a system) will interoperate with other devices (components)

11. World Class Standards 11 Limitations of Interoperability Testing  Does not prove that a device is conformant  Interoperable devices may still interoperate even though they are non-conformant to a base specification or standard  Cannot explicitly test error behaviour or unusual scenarios  Invalid conditions may need to be forced (lack of controllability)  Has limited coverage (does not fully exercise the device)  Are executed manually (usually)  Difficult to automate, tests use more time to be prepared than executed.  Does not prove 100% interoperability with other implementations with which no testing has been done  ‘A’ may interoperate with ‘B‘ and ‘B’ may interoperate with ‘C’. But it doesn’t necessarily follow that ‘A’ will interoperate with ‘C’.

12. World Class Standards 12 Conformance and Interoperability Testing are Complementary  ETSI experience  As you move up a system stack the emphasis should change from conformance to interoperability testing  Lower layer protocols, infrastructure  Emphasis on conformance  Middleware, enablers  Combination of conformance and interoperability testing  Services, applications, systems  Emphasis on interoperability testing  Conformance testing can be viewed as a pre-requisite to interoperability testing Interop Testing Conformance Testing Interoperability

13. World Class Standards 13 Typical ETSI Standards Development Process (Unit) Conformance Testing Interoperability Testing Products mature from prototypes to commercial products ETSI: Development of base standards Certification Industry time Conformance Test Specifications Interoperability Test Specifications Iterative feedback Standards Body Testing provides vital feedback, e.g., holes, errors, or inconsistencies, in base standards

14. World Class Standards 14 Stages in ETSI Test Specification Development Base Standard or Profile Specification Requirement Catalogue Implementation Check List (ICS/IFS) Identification of Test Group Structure (TSS) Specification of Test Purposes (TP) Specification of Test Cases (TC) Validation of Test Cases Specification of Test Descriptions (TD)

15. World Class Standards 15 Requirements Cataloguing  Each requirement is identified, extracted, and catalogued as follows (example from IPv6):  Requirement type: Mandatory (MUST, MUST NOT, SHALL, SHALL NOT) Recommended (SHOULD, SHOULD NOT) Optional (MAY, MAY NOT, COULD)  Requirement role (tested entity): E.g., Host, Router, Node (Host or Router)  Requirement context  Requirement text  Base document citation and reference  Functional groupings E.g., Process Fragmented packet, Generate ICMPv6 Error Type etc.

16. World Class Standards 16 Test Purposes  Precise descriptions of the purpose of a test in relation to a particular (base standard) requirement  Meaningful to a larger audience than technical experts  Define the functionality being tested, i.e., WHAT?  Should be formulated using a similar level of information content and wording as the corresponding requirement description  Should only mention directly relevant aspects of interactions but not describe them in detail  Reference a test configuration (i.e., architecture)  It is possible to have several test purposes per requirement  Specified at ETSI in  Natural language, or  ETSI’s Test Purpose Language (TPLan) (www.tplan.info)www.tplan.info  But definitely not coded!

17. World Class Standards 17 TPLan Example from Digital Public Mobile Radio TP id : TP_PMR_0406_01 summary : 'Header frame acknowledges connect request' TP type : conformance RQ ref : RQ_001_0406 –- Catalogue Identifier IUT Role : CSF -- Configured Service Function (CSF) config ref: CF_dPMR_CSF_01 -- CSF Implementation Under -- Test (IUT) and TESTER with { IUT in standby } ensure that { when { IUT receives a Connection_Request } then { IUT sends an Acknowledgement_Frame }

18. World Class Standards 18 Interoperability Test Description  Specifies detailed steps to be followed to achieve stated test purpose, i.e., HOW?  State steps clearly and unambiguously without unreasonable restrictions on actual method:  Example: Answer incoming call NOT Pick up telephone handset  Written in a structured and tabulated natural language so tests can be performed manually  In theory it is also possible to increase automation in with test scripts, e.g., with TTCN-3  Proprietary nature of user APIs makes this in practice difficult  Requires additional adapter development from equipment vendors

19. World Class Standards 19 Example Interoperability Test Description Response to ‘ping’:

20. World Class Standards 20 Conformance Test Suite  A collection of detailed test cases or scripts that implement test purposes  Can be compiled and executed, e.g., when using TTCN-3  Specifies HOW to test  Implements also handling of, e.g., unexpected or background behaviour, or returning the SUT to the initial state after a test  Composition of test cases  Each individual test has a preamble, test body (i.e., implementation of the Test Purpose), and postamble  Test components may be used clearly separate interactions with different logical IUT interfaces during a test  Assigns test verdicts  Configurable at run-time, e.g., SUT address  A test suite is not a test systems, but  ETSI ensures all tests can be compiled  And validated by executing the tests against one or more real implementations on at least one test system

21. World Class Standards 21 What is TTCN-3?  Testing and Test Control Notation Version 3  Internationally standardized language developed specifically for executable test specification  Specified by ETSI MTS Technical Committee  Is independent of a specific IUT or IUT interfaces  Is independent of a test execution environment  Standard available at portal.etsi.org via ETSI programmeportal.etsi.org  Allows unambiguous implementation of tests  Look and feel of a regular programming language  Good tool support (today 6 commercial tools available)  Successfully deployed at ETSI and industry in a variety of application domains  e.g., telecom, automotive, software, etc.  Good text book available  http://www.wiley.com/legacy/wileychi/ttcn-3/ http://www.wiley.com/legacy/wileychi/ttcn-3/ www.ttcn-3.org

22. World Class Standards 22 Example TTCN-3 Test Case testcase TC_COR_0047_01() runs on Ipv6Node system EtherNetAdapter { f_cf02Up(); // Configure test system for HS->RT // No preamble required in this case f_TP_HopsSetToOne(); // Perform test // No postamble required in this case f_cf02Down(); // Return test system to initial state } function f_TP_HopsSetToOne() runs on Ipv6Node { var Ipv6Packet v_ipPkt; var FncRetCode v_ret := f_echoTimeExceeded( 1, v_ipPkt ); if ( v_ret == e_success and v_ipPkt.icmpCode == 0 ) { setverdict(pass);} else { setverdict(fail); } } function f_echoTimeExceeded(in UInt8 p_hops, out Ipv6Packet p_ípPkt ) runs on Ipv6Node return FncRetCode { var Ipv6Packet v_ipPacket; var FncRetCode v_ret; ipPort.send( m_echoReqWithHops(p_hops) ); alt { [] ipPort.receive( mw_anyTimeExceeded ) -> value p_ipPkt { return e_success } [] ipPort.receive { return e_error } } }

23. World Class Standards 23 Validation of (Conformance) Tests  Typical test suite validation requires  That TTCN-3 tests compile on several tools (5 at ETSI)  Tests executed on one or more platforms against various implementations of the standard (usually by ETSI members)  Requires very close co-operation with test tool suppliers, test platform providers, equipment manufacturers etc.  Strict and well-defined processes between Test lab (who execute tests[, certify] and provide feedback) And ETSI (who updates the tests)  Timely availability of stable base specifications, test platforms (tools) and implementations is essential  ETSI does not certify products!

24. World Class Standards 24 Requirements Catalogue Test Purposes Test Description Test Cases (TTCN3) Base Standard Database  Web interface, browsing by function, user-defined search and filter, traceability, document generation, …! CTI Test Specification Data Base

25. World Class Standards 25 Example: ETSI IP Testing Library http://www.ipt.etsi.org/STF295-ph1/

26. World Class Standards 26 Test Specification Development at ETSI  ETSI uses a variety of resources for test specification development  ETSI Technical Committee on Methods for Testing and Specifications (TC MTS) Composed of testing experts from major telecom companies, tool vendors research institutes and academia Works on guidelines and languages for ETSI specification development, e.g., TTCN-3, TPLan, IP Testing framework  ETSI Plugtests Service Organizes interoperability events  ETSI Centre For Testing and Interoperability (CTI)  ETSI Specialist Task Forces (STF)

27. World Class Standards 27  In-house team of experts providing direct support as a service to ETSI Technical Bodies (e.g., ETSI GRID TC) on the use of formal techniques in standards  Supports ETSI test specification development - conformance and interoperability (60%)  Assistance in development of protocol standards and profiles, e.g., application of modern specification techniques (35%)  ‘Standards Engineering’  Validation of standards (5%)  E.g., testbed activities (IPv6, SIP/H.323)  ETSI Plugtests™ Service  Validation of standards through interoperability events About ETSI CTI

28. World Class Standards 28 Areas ETSI CTI Involvement  Cellular: GSM and 3G (UMTS) terminals  WiFi: HiperMAN, HiperACCESS, WiMax  VoIP: H.323, SIP, SIGTRAN  Service Creation: OSA/Parlay (API, IDL)  IPv6: Core, Security, Mobility, v4-v6  Cordless phones: DECT  Radio communications: TETRA, DMR, dPMR  Access terminals: FSK, SMS  Broadband: ISDN, DSL  Smartcards: Readers, cards, security modules  Intelligent Transport Systems (ITS): DSRC  Early NGN/IMS  Future: More Security, GRID...

29. World Class Standards 29 About ETSI STFs  Team of experts seconded mostly from the ETSI membership  25-30 experts in 2006  Supported and often lead by CTI experts  Funded either by ETSI STF budget, European Union, or by commercial agreement  Can range from small (2 pm) to large (58pm over 2-3 years)  Approx. 2,7M€ of expert resource per year  Produce deliverables which are turned into freely and publicly available standards after approval from ETSI members  Requirement catalogue, test purposes, test descriptions, test cases  Frameworks, specification infrastructure, guidelines  Some validation  etc.

30. World Class Standards 30 Upcoming GRID STF  ETSI GRID Technical Committee has created an STF  Completely funded by EC DG ENTR  Purpose is to identify gaps in grid specifications and develop GRID testing framework  At this point focus is on analysis of available grid specifications  Review of other GRID testing frameworks, e.g., ETICS project  Actual tests will come are the next step Does not make sense to specify tests without clear scope  Call for experts is out (CL07_2532)  Application deadline May 27 th !!  Interview of candidates on June 11 th  Start of work expected in Q3  Any ETSI or OGF member can apply – do so today

31. World Class Standards 31 Results of GRID Test Specification Analysis  ETSI CTI has been asked by ETSI TC GRID and OGF members to review Byte IO and Grid RPC test specifications Dec 2006  Summary of observations  Lack of references to the specification, e.g., document clause or quotation of specification text  Test cases mix discussion of mandatory from optional features  Inconsistent use of terminology, e.g., “test case” Grid RPC corresponds to ETSI test purpose (Grid RPC “test code” corresponds to ETSI test case) Byte IO corresponds more to ETSI test description  Missing isolation of test purpose in Byte IO test specification makes understanding of test objective and verdict criteria difficult  Both documents from ETSI point of view specify conformance tests but claim to be interoperability tests (Byte IO specifies byte strings!)  Test cases should be more implementation language agnostic

32. World Class Standards 32 Thank You! Feel free to contact us:  e-mail Stephan.Schulz@etsi.orgStephan.Schulz@etsi.org  WWW www.etsi.org/ptcc