We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byJose Hutchison
Modified over 3 years ago
How to make better standards - Achieving Interoperable Standards Milan Zoric Centre for Testing and Interoperability (CTI) © ETSI All rights reserved Ankara, 6-7. May 2010 Turk Telekom - ETSI Workshop © ETSI 2010
Is Interoperability Important? We live in an interconnected world and interoperability is key to drive it forward Digital Home, Smart House M2M (embedded communication) Internet of Things, Intelligent Transport Systems etc. Users benefit from increased choice from multiple manufacturers Business, Governmental, Private Consumer And they expect stuff to work (Plug&Play) Manufacturers benefit from an increased market Economies of scale 2 Turk Telekom - ETSI Workshop © ETSI 2010
ETSI and Interoperability (IOP) Standardisation enables interoperability One main aim of standardisation is to enable interoperability in a multi-vendor, multi-network, multi- service environment IOP is the red thread running through the entire ETSI standards development process Interoperability is specified from the beginning Not something bolted on at the end ETSI philosophy Interoperability should be built-in! 3 Turk Telekom - ETSI Workshop © ETSI 2010
Typical Causes of Non-interoperable Standards Requirements not well identified or missing Ambiguous requirements Varying technical quality and use of language Inadequate handling of options Lack of clear system overview Loose definition of interfaces (reference points) Poor maintenance Using standards beyond their original purpose... 4 Turk Telekom - ETSI Workshop © ETSI 2010
Poor Interoperability Can be Expensive Bad publicity For the technology For the manufacturer Annoyance to the end customer Damage to brand name Loss of customer base Allegiances change rapidly May affect uptake of new technology Loss of investor confidence We can no longer afford to get it wrong! 5 Turk Telekom - ETSI Workshop © ETSI 2010
ETSI Support for Interoperability Technical Committee MTS Methods for Testing and Specification Standardised frameworks, methodologies, languages For protocol specification For testing Making Better Standards Centre for Testing and Interoperability (CTI) Direct support to ETSI Technical Bodies Application of protocol engineering and best practices Considerable contribution to the work of MTS Development of test specifications Standards validation, including interoperabilty events (Plugtests TM ) 6 Turk Telekom - ETSI Workshop © ETSI 2010
7 Project Experience Cellular: GSM, 3G UMTS (including IMS/SIP) WiFi: HiperMAN, HiperACCESS, WiMAX VoIP: H.323, IETF SIP, SIGTRAN Service Creation: OSA/Parlay (API, IDL, Java) IPv6: Core, Security, Mobility, v4-v6 Cordless phones: DECT Radio communications: TETRA, DMR, PMR Access terminals: FSK, SMS Broadband: ISDN, DSL Smartcards: Readers, cards, security modules Intelligent Transport Systems (ITS): DSRC TISPAN NGN: IMS Interoperability (interworking) Future: More Security, more NGN, GRID...
The ETSI Approach Turk Telekom - ETSI Workshop © ETSI SPECIFICATION VALIDATION TESTING
Turk Telekom - ETSI Workshop © ETSI Interoperability must be built in ! Unambiguous protocol specification Protocol engineering techniques Consistent and precise language Naming conventions, editing rules Where applicable use of standardized technical notations ASN.1, MSCs, SDL, UML, IDL, XML, etc. Specification of minimum requirements to achieve interoperability, at the same time leaving room for product differentiation Testability must be built in Making Better Standards:
CTI expert assistance CTI experts can be attached to a standardization group The sooner this is done, the better effects could be expected CTI experts can provide: Protocol engineering know-how Dedicated language and tool know-how Testing and test tools know-how Key assistance in planning interoperability related activities Project management In short – what, when and how At the same, CTI experts remain neutral and will respect the decisions of the standardization group Turk Telekom - ETSI Workshop © ETSI
Why Validate Standards? Validation reveals problems/errors in Standards and Products Validated standards give a higher chance of interoperable products Assurance that they provide the right functionality Gives manufacturers and operators confidence to implement and go to market Provides an opportunity to correct errors in a controlled manner Decreases time to market Late fixes in the product cycle are more expensive than early ones 11 Turk Telekom - ETSI Workshop © ETSI 2010
Validation of Standards Turk Telekom - ETSI Workshop © ETSI 2010 Development of Base Standards Peer Review Modelling and Simulation Prototyping Interoperability Events Products mature from prototypes to commercial products Time
Validation through IOP Events ETSI Plugtests events Open to members and non- members Aim is to validate standards Feedback (Change Requests) to relevant technical bodies A tool to develop and mature standards But testing and debugging are useful by-products Vendors validate their understanding of standards and their implementation Achieve in one week what would otherwise take months Promote technology and community Develop new ideas, confirm existing ones 13 Turk Telekom - ETSI Workshop © ETSI 2010
Series of IOP Events Maturity of the Standard Events Event 1 Event 2 Event nn Turk Telekom - ETSI Workshop © ETSI 2010
15 Plugtests can look like this…
… or this (Car2Car Interop) 16 Turk Telekom - ETSI Workshop © ETSI 2010
Typical ETSI IMS Bluetooth IPv6 Triple Play over xDSL SIM/Handset WLAN IRAP RFID STQ (Speech Quality) WiMAX SIGTRAN Femtocell OSA/Parlay (ParlayX) In operation since 1999 Over 100 events, more than 3000 engineers Technologies include: B2B (Business-to-Business) SIPiT J2ME HDMI Air Traffic Control (EUROCAE) Electronic Signature (XadES, CadES) Lawful Interception Optical Fibre (GPON) Power Line (PLT) Intelligent Transport Systems Femtocell Fixed Mobile Convergence (FMCA) 17 Turk Telekom - ETSI Workshop © ETSI 2010
Who attends Plugtests events? Participants are mostly ETSI members, but do NOT have to be ETSI members Plugtests are addressed to any company developing a product such as operators, vendors or equipment manufacturers, content or application providers Standardization Bodies, Fora or interest groups may also attend Plugtests also welcome Universities and Research Institutes 18 Turk Telekom - ETSI Workshop © ETSI 2010
ETSI Testing related activities Milan Zoric Centre for Testing and Interoperability (CTI) © ETSI All rights reserved Turk Telekom - ETSI Workshop © ETSI 2010
... and Testing 20 Turk Telekom - ETSI Workshop © ETSI 2010 Products mature from prototypes to commercial products Time Conformance Testing Development of Base Standards Peer Review Modelling and Simulation Prototyping Interoperability Events Certification (not done by ETSI) Development of Test Specs (Conformance & Interop)
Conformance Testing Tests a specific (part of a) product for compliance to requirements in a Base Standard 21 Turk Telekom - ETSI Workshop © ETSI 2010
Characteristics of Conformance Testing Gives a high-level of confidence that the standardised parts of a product are working as specified It is component (Black Box) testing Usually One requirement -> One tes t Requires a test system (i.e., executable test cases) Test execution is automated and repeatable (any time, any number of times) Tests in controlled conditions High degree of control and observation Always tests normal scenarios Can provoke and test non-normal (but legitimate) scenarios Can explicitly test error behaviour (robustness) Tests are thorough and accurate but limited in scope At level of detailed protocol messages, service primitives, or procedure calls Turk Telekom - ETSI Workshop © ETSI
Limitations of Conformance Testing Does not necessarily prove interoperability with other products Tests are focussed on part of a product A system is often greater than the sum of its parts! Does not test the users perception of the system Standardised conformance tests do not include proprietary features Test systems may be expensive But cost may be relative to size of the market For some technologies it may be the cheapest way to ensure interoperability For other it may be prohibitively expensive Turk Telekom - ETSI Workshop © ETSI
Case Study – 3GPP UE Testing 3GPP mobile protocol / signalling testing Project was started in companies involved A large extent of test industry involved 16 experts led by ETSI CTI Total budget > 90 person months / year (in 2009) 28 Test Suites More than 1400 Test Cases Running on 4 System Simulator platforms Delivery every 3 weeks Deployed by GCF / PTCRB for UE certification 24 Turk Telekom - ETSI Workshop © ETSI 2010
Interoperability Testing Tests end-to-end functionality between a collection of products 25 Turk Telekom - ETSI Workshop © ETSI 2010
26 Characteristics of IOP Testing Gives a high-level of confidence that products will interoperate with other products It is system testing Tests a complete product or a collection of products Is functional testing Tests can be performed manually Users operate the product via existing interfaces (standard/proprietary) Can also be automated with test drivers Testing includes perception of end users Exercises the whole product Less thorough than conformance testing but wider in scope Turk Telekom - ETSI Workshop © ETSI 2010
27 Limitations of IOP Testing Does not prove that a product is conformant Products may still interoperate even though they are non-conformant Requires availability of suitable interfaces Limited ability to trigger error behaviour or unusual scenarios Less controllability than in conformance testing Interoperability can be elusive! Configuration may be simplified (not a fully operational system, e.g., no billing, no load) Does not prove interoperability with other products with which no testing has been done A may interoperate with B and B may interoperate with C. Does not necessarily follow that A will interoperate with C With large number of products it may become prohibitively expensive Turk Telekom - ETSI Workshop © ETSI 2010
28 Conformance and IOP Testing are Complementary As you move up a system stack the emphasis should change from conformance to interoperability testing Lower layer protocols Mainly conformance testing Middleware, enablers, infrastructure Combination of conformance and interoperability testing Services, applications, systems Emphasis on interoperability testing Conformance testing should be a pre-requisite to interoperability testing Turk Telekom - ETSI Workshop © ETSI 2010
IOT with Conformance Checking 29 Turk Telekom - ETSI Workshop © ETSI 2010
TC MTS (Methods for Testing and Specification) Developed TTCN-3 ETSI Standard (ES) Key TTCN-3 Standards ES : TTCN-3 Core Language ES : TTCN-3 Runtime Interface (TRI) ES : TTCN-3 Control Interfaces (TCI) ES and upwards: ASN.1, XML, IDL, Code Documentation also all endorsed by ITU-T SG17 (Z.140 Series) New Extension Packages (drafts) Configuration and Deployment Support Performance Testing Real-time testing And others... Creator of the TTCN-3 Standard Turk Telekom - ETSI Workshop © ETSI
All test suite development done in TTCN-3 Some maintenance of legacy TTCN-2 test suites Test Suites developed by Specialist Task Forces (STF) At the request of the ETSI Technical Committees Experts recruited from the ETSI Membership Mostly under the technical management of CTI 15 to 20 testing STFs per year Use of TTCN-3 at ETSI Turk Telekom - ETSI Workshop © ETSI
Test Specification Development Standard Successive Levels of Abstraction Test Purposes Test Descriptions TTCN-3 Test Suite Executable Tests Compilation Requirements Catalogue and/or ICS/IXIT Test Case Parameterisation and Selection 32 Turk Telekom - ETSI Workshop © ETSI 2010
Where possible ETSI Test Specifications are validated prior to publication Minimum requirement is that they compile on at least one tool E.g., UMTS compiles on at least 4 platforms In many cases we execute the tests agianst live implementations In co-operation with partners (Test Labs and Vendors) E.g., UMTS tests executed against at least 2 different implementations Validation of Tests Turk Telekom - ETSI Workshop © ETSI
IPv6 (TC MTS) Core, Security, Mobility, Transitioning IMS (TC INT & TC TISPAN) Interoperability, Network Integration, ISDN Interworking, Supplementary Services WiMAX (TC BRAN & WiMAX Forum) Conformance (PCT, NCT), Interoperability Intelligent transport (TC ITS) Direct Short Range Communication (DSRC) LTE (3GPP) UE conformance DMR/DPMR (TC ERM) Terminal conformance SIP (TC MTS) RFC 3261 UA and Proxy conformance Some ETSI TTCN-3 Test Suites Turk Telekom - ETSI Workshop © ETSI
6 different TTCN-3 Development Environments and Compilers ETSI TTCN-3 Documentation Tool (T3D) ETSI TTCN-3 Code Quality Tool (T3Q) ETSI tools will be made available as Open source More information on TTCN-3 tools available at TTCN-3 Tools Used at ETSI Turk Telekom - ETSI Workshop © ETSI
CTI expert testing assistance The world of testing is complex in many ways Standard makers are not necessarily experts in testing themselves However, they know what they need to achieve, they know the technology being standardized, they understand the timing and financial constraints CTI experts can provide: Key assistance in deciding on the approaches appropriate for a particular technology Key assistance in planning test development and validation activities Testing techniques, languages and tool know-how Project management Co-ordination of many players involved in test development and validation Turk Telekom - ETSI Workshop © ETSI
Conclusions An interconnected world demands interoperability Standards enable interoperability Validation and Testing are cornerstones in the development of ETSI standards Validated standards mean interoperable standards Interoperable standards facilitate interoperable products Plan for validation and testing (early) Right mix of conformance and/or interop Synchronise testing activities with the development of the standard (and products!) Ensure feedback to the base standard Perform (regular) interoperability events Synchronise with availability of products 37 Turk Telekom - ETSI Workshop © ETSI 2010
38 Useful Links Achieving Interoperability - The ETSI approach: Plugtest: TTCN-3: TPLan: STF: Making Better Standards: IPR Policy:
Milan Zoric Centre for Testing and Interoperability (CTI) © ETSI All rights reserved THANK YOU! Turk Telekom - ETSI Workshop © ETSI 2010
Interoperability engineering Milan Zoric Centre for Testing and Interoperability (CTI) © ETSI All rights reserved Smart Grids.
SEM15-06 How to make better standards - Achieving Interoperable Standards Laurent Velez Centre for Testing and Interoperability (CTI)
1 The Protocol & Testing Competence Centre. 2 In Pursuit of Interoperability Ultimate aim of ICT standardisation is interoperability Likelihood of interoperability.
World Class Standards IMS Interoperability Testing - from Standards to Practice Giulio Maggiore ETSI TC INT Chairman © ETSI All rights reserved.
Philippe COUSIN Interoperability Service Manager Milan ZORIC ETSI PTCC (Protocol and Testing Competence Centre) In pursuit of Interoperability RFID and.
Halifax, 31 Oct – 3 Nov 2011Global Standards Collaboration ETSI ETSI Jørgen Friis, VP ETSI SES (Standards Enabling Services) Document No: GSC-16-PLEN-XX.
1 ETSI Plugtests The interoperability service Interop Events & Open Source Products 16 December Patrick GUILLEMIN ETSI Plugtests Technical Coordinator.
ETSI TC BRAN Milan Zoric Centre for Testing and Interoperability (CTI) © ETSI All rights reserved Ankara, 6-7. May 2010 Turk.
1 Achieving Technical Interoperability - the ETSI Approach Hans van der Veer ETSI Board Interoperability Champion.
World Class Standards Review of TC MTS (Methods for Testing and Specification) Dieter Hogrefe © ETSI All rights reserved Board#75, 3-4 November 2009.
Halifax, 31 Oct – 3 Nov 2011ICT Accessibility For All ETSI Conformance and Interoperability Testing Jørgen Friis VP ETSI SES (Standards Enabling Services)
ERM/MTS STF proposal on formal Phy measurement framework Friedbert Berens Vice Chair ERM TGUWB Presentation to ERM, July 2010.
©Ian Sommerville 2000Software Engineering, 6th edition. Chapter 29Slide 1 Configuration management l Managing the products of system change l Objectives.
Company Confidential 1© Nokia Siemens Networks / Multivendor Interoperability Introduction, Interoperability Verification at Nokia Siemens Janos Arrakoski,
1 ETSI Plugtests The interoperability Kick-off Meeting The acid test: Testing Interoperability 31 March - Patrick GUILLEMIN ETSI Plugtests.
Mastering Object-Oriented Analysis and Design with UML Module 3: Requirements Overview.
1 Why ETSI is the place to bridge EU and LA initiatives on e-administration Francisco Da Silva Chairman of the Kick Off Meeting Sophia Antipolis,
How does ETSI work? ETSI Seminar © ETSI All rights reserved.
1 Balloting/Handling Negative Votes September 22 nd and 24 th, 2009 ASTM Virtual Training Session Christine DeJong Joe Koury.
1 A Firm Step of 3GPP for UE Conformance & Interoperability Shicheng Hu ETSI PTCC 22 May 2003.
1 Introducing the Specifications of the Metro Ethernet Forum MEF 19 Abstract Test Suite for UNI Type 1 February 2008.
Balloting/Handling Negative Votes September 2012 ASTM Officers Training Workshop Dan Smith & Kelly Paul 1.
Setting Product Strategy Marketing Management, 13 th ed 12.
1 IEEE Media Independent Handoff Overview of services and scenarios for 3GPP2 Stefano M. Faccin Liaison officer to 3GPP2.
1 Week 2 The Object-Oriented Approach to Requirements.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 6 Slide 1 Chapter 6 Requirements Engineering Process.
PSSA Preparation. Question 1(no calculator) D Question 2 (no calculator)
TP#13 oneM2M testing discussion TP#13 Source: Laurent Velez, ETSI, Meeting Date:
1 Requirements Engineering Processes – 2. 2 Recap of Last Lecture - 1 We introduced the concept of requirements engineering process We discussed inputs.
14% of the exam 24 questions 1 Carol Pattyn 6/18/13.
Quality Manual for Interoperability Testing Morten Bruun-Rasmussen Presented by Milan Zoric, ETSI.
© 2010 Invensys. All Rights Reserved. The names, logos, and taglines identifying the products and services of Invensys are proprietary marks of Invensys.
18 Copyright © 2005, Oracle. All rights reserved. Distributing Modular Applications: Introduction to Web Services.
©Ian Sommerville 2004Software Engineering, 7th edition. Chapter 23 Slide 1 Software testing.
All rights reserved © 2006, Alcatel Grid Standardization & ETSI (May 2006) B. Berde, Alcatel R & I.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 11Slide 1 Chapter 11 Distributed Systems Architectures.
Fixed Mobile Convergence T Research Seminar on Telecommunications Business Johanna Heinonen.
Interoperability, why it is important Dr. Ghassem Koleyni FORUM ON NEXT GENERATION STANDARDIZATION (Colombo, Sri Lanka, 7-10 April 2009) Colombo, Sri Lanka,
©Ian Sommerville 2007Change Management Slide 1 Software change management.
HIGHLIGHTS BACKGROUND CHALLENGES IN DEVELOPING COUNTRIES NEGATIVE IMPACT THE PROPOSAL.
SEM02-10 The ETSI Secretariat ETSI Seminar © ETSI All rights reserved.
1 Balloting/Handling Negative Votes September 11, 2006 ASTM Training Session Bob Morgan Brynn Iwanowski.
1 Computer Networks TCP/IP Protocol Suite. 2 Protocols Cooperative action is necessary computer networking is not only to exchange bytes huge system with.
WIPO Patent Information Services Juneho Jang Senior Patent Information Officer October 2008.
DOCUMENT #:GSC15-PLEN-51 FOR:Presentation SOURCE:ITU AGENDA ITEM:Plenary 6.10 Conformance and Interoperability Testing.
All rights reserved, Copyright © INTAP Overview: Guide for applying RM- ODP with UML Profile for EDOC From Enterprise Model to Computational Model.
1 Contract Inactivation & Replacement Fly-in Action ( Continue to Page Down/Click on each page…) Electronic Document Access (EDA)
1 Chapter One Introduction to Computer Networks and Data Communications.
© UNCTAD ASYCUDA Overview … a summary of the objectives of ASYCUDA implementation projects and features of the software for the Customs computer.
Halifax, 31 Oct – 3 Nov 2011ICT Accessibility For All GTSC-9 Summary Glenn Parsons, GTSC-9 Chair, ISACC Document No: GSC16-CL-04 Source: GTSC-9 Contact:
© 2017 SlidePlayer.com Inc. All rights reserved.