Presentation is loading. Please wait.

Presentation is loading. Please wait.

Testing Techniques Conformance Testing Methodology and Framework ISO IS-9646.

Published byThomasine Preston Modified over 8 years ago

Similar presentations

Presentation on theme: "Testing Techniques Conformance Testing Methodology and Framework ISO IS-9646."— Presentation transcript:

1 Testing Techniques Conformance Testing Methodology and Framework ISO IS-9646

2 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 2 Overview lConformance Testing Methodology and Framework ISO IS-9646 Literature: “An Overview of OSI Conformance Testing”

3 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 3 Overview: This Lecture lProtocol conformance testing: why and what lStandardization lOverview Standard IS-9646: Conformance Testing Methodology and Framework nConformance: static and dynamic nOptions, protocol implementation conformance statement nTest generation: test purposes, abstract-, executable tests nTest methods, test architectures, nTest classification and -structuring nTest notation nTest implementation and -execution

4 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 4 Types of Testing

5 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 5 Protocol Conformance Testing distributed systems  exchange of information  rules for communication : protocol  standardized protocol specifications  conforming implementations  testing for protocol conformance

6 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 6 Protocol Conformance Testing International Standard ISO IS-9646 : Conformance Testing Methodology and Framework lHow to test protocol entity implementations for conformance with respect to their specifications lOriginally for OSI systems lNow more generally applied lDefines test language TTCN lStandard accepted by ITU-T, ETSI, ATM-Forum, ……

7 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 7 Protocol Conformance Testing lStarting point : a correct and valid (standardized) protocol specification lOriginal goal : standardization of test suites in TTCN nconsistency between specification and test suite ? lConformance testing  interoperability testing

8 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 8 The Conference Protocol

9 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 9 CPE UDP Layer CPE The Conference Protocol join leave send receive Conference Service

10 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 10 Conference Protocol Test Architecture CPE = IUT UT-PCO = C-SAP LT-PCO UDP Layer U-SAP LT-PCO Tester TorX BC A

11 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 11 (N)-ASPs (N)-service provider (N)-Protocol Entity (N)-PDUs (N-1)-ASPs (N-1)-service provider (N)-Protocol Entity protocol (N-1)-ASPs Protocol Conformance Testing vs. Interoperability Testing

12 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 12 Protocol Conformance Testing vs. Interoperability Testing lProtocol Conformance Testing: ncorrect protocol (design and validation) nconforming protocol entities (testing)  interoperability lInteroperability testing ntest directly whether each pair of protocol entities can work together (i.e. deliver (N)-service) lInternet: ninteroperability testing with (standard) reference entities

13 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 13 Standard Conformance Testing lTesting by: nmanufacturers nusers organizations ntelecom operators nindependent test laboratories lStandard IS-9646 for ncertification of implementations naccreditation of testing processes and test laboratories ninternational acceptance of test results

14 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 14 ISO IS - 9646 lPart 1 :Introduction and general concepts lPart 2 :Abstract test suite specification lPart 3 :TTCN Tree and Tabular Combined Notation lPart 4 :Execution of tests lPart 5 :Requirements on test laboratories and administrative conformance assessment process

15 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 15 test generation test execution test implementation standard protocol specification standard conformance test suite verdict implementation process protocol implementation IUT process of conformance testing executable test suite

16 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 16 Conforming Implementation lSpecifications in natural language lWhat is a conforming implementation ? l“ … exhibits conformance if it complies with the conformance requirements of the applicable specification standard …” lSatisfaction of conformance requirements lConformance requirements : “the implementation shall do ….. “ “the implementation shall not do …..” lComplication : options

17 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 17 Conforming Implementation l(P)ICS - (Protocol) Implementation Conformance Statement : nlist of implemented options nsupplied by manufacturer l(P)ICS proforma nfill-in questionnaire with all possible options nattachment to protocol standard nto be completed by manufacturer to make PICS lRestrictions on combinations of implemented options ndocumented in standard in static conformance requirements

18 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 18 Example : Static Conformance Requirement lFeature A is optional lFeature B is optional lBut A may only be implemented if B is also implemented

19 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 19 Conforming Implementation lRequirements on dynamic behaviour: dynamic conformance requirements lMessages : nstructure ncoding lOrdering : nsequence of sent and receive messages nrelation between information content of different messages lExample: nWhen in state Enable, upon receiving a Data-Request, the implementation shall send a Data-PDU or it shall enter the abort phase.

20 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 20 Conforming Implementation lA conforming implementation is one which satisfies both static and dynamic conformance requirements, consistent with the capabilities stated in the PICS lConformance testing: nchecking static conformance by reviewing nchecking dynamic conformance requirements by testing

21 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 21 Test Generation dynamic conformance requirement  test purpose(s)  generic test case  abstract test case  executable test case  test case execution

22 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 22 Test Generation lTest purpose : precise description (natural language) of what is to be tested in order to decide about satisfaction of a particular conformance requirement lExample: nWhen in state Enable, upon receiving a Data-Request, the implementation shall send a Data-PDU or it shall enter the abort phase. nCheck whether in state Enable, after receiving a Data-Request, a Data-PDU is sent. (but implementation may also enter the abort phase!)

23 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 23 Test Generation lGeneric test case: operationalization of test purpose; description of actions to be performed to reach test purpose lExample: nCheck whether in state Enable, after receiving a Data-Request, a Data-PDU is sent. nGo to state Enable, send a Data-Request, and observe a Data-PDU, and if not check that the implementation starts entering the abort phase

24 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 24 Test Generation lAbstract test case: take test method into account lTest method ( test architecture, test environment ) : abstract description of environment in which testing will be performed, in terms of PCO - Points of Control and Observation lExample: DoConnect-Request, check Connect-Response (now in state Enable), send Data-Request observe Data-PDU or observe Start-Abort

25 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 25 IUT LT UT (N)-ASPs (N)-PDUs PCO (N-1)-ASPs test coordination Local test method Test Methods

26 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 26 IUT LT UT (N)-ASPs (N)-PDUs PCO (N-1)-ASPs test coordination service provider test systemSUT Distributed test method Test Methods

27 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 27 Test Notation lAbstract test cases are written in standardized notation : TTCN : Tree and Tabular Combined Notation lSpecification language for Abstract Test Cases lBehaviour specified as tree of sequences of possible events lData specified in tables or in ASN.1 lISO IS-9646 Part 3 lWith:input ?, output ! (from tester perspective!), timers / time out, ordering of events, default behaviour, alternative behaviours, tree attachment (=subroutines), parallel component testers, verdicts: pass, fail, inconclusive

28 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 28 behaviour Test Case Dynamic Behaviour Test Case Name:Conn_Estab Group:transport/connection Purpose:Check connection establishment with remote initiative constraintsverdict +preamble LT ! T-PDU-connect-request UT ? T-SP-connect-indication UT ! T-SP-connect-response LT ? T-PDU-connect-confirm OTHERWISE LT ? T-PDU-disconnect-request OTHERWISE pass fail inconclusive fail Example : TTCN

29 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 29 Classification of Tests lBasic interconnection tests lCapability tests lBehaviour tests lConformance resolution tests

30 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 30 Structuring of Test Suites Test suite Test case Test step Test event PCO ? XPCO ! A Test group

31 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 31 Test Implementation lTransform abstract test suite into executable test suite by ncompilation ninterpretation na test slave ….. lSelect test cases from abstract test suite based on PICS lIncorporate (P)IXIT - (Protocol) Implementation eXtra Information for Testing : naddress information ntimer information nspecific parameter values n….. lPIXIT proforma supplied by test laboratory

32 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 32 Test Execution lExecute / interpret executable test suite lCollect and analyse the results based on verdicts pass, fail, inconclusive of each test case execution lCollect all results of nstatic conformance review ndynamic testing into Protocol Conformance Test Report

33 © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 33 implementation process static conformance review test execution analysis of results test selection test implementation standardized test methods test purposes generic test suite test notation verdict test report certification PIXIT proforma standard protocol specification PICS proforma dynamic conformance req’s static conformance req’s protocol implementation IUT PIXIT PICS (standardized) abstract test suite Basic interconn. capability behaviour executable test suite

Download ppt "Testing Techniques Conformance Testing Methodology and Framework ISO IS-9646."

Similar presentations

Introducing the Specifications of the Metro Ethernet Forum

Introducing the Specifications of the Metro Ethernet Forum
TTCN-3 & Conformance Testing Anitha Krishnamoorthy.

TTCN-3 & Conformance Testing Anitha Krishnamoorthy.
SDL+ The Simplest, Useful Enhanced SDL-Subset The documentation is the design, the design is the system! Copyright © 2003-2004 SDL Task Force Consortium.

SDL+ The Simplest, Useful Enhanced SDL-Subset The documentation is the design, the design is the system! Copyright © SDL Task Force Consortium.
Software Requirements

Software Requirements
Using TTCN-3 in Interoperability Testing for Real-time Communication Systems Zhiliang Wang, Jianping Wu, Xia Yin, Xingang Shi and Beihang Tian Department.

Using TTCN-3 in Interoperability Testing for Real-time Communication Systems Zhiliang Wang, Jianping Wu, Xia Yin, Xingang Shi and Beihang Tian Department.
Writing Quality Specifications July 9, 2004 Mark Skall Acting Director, Information Technology Laboratory National Institute of Standards and Technology.

Writing Quality Specifications July 9, 2004 Mark Skall Acting Director, Information Technology Laboratory National Institute of Standards and Technology.
Software Engineering COMP 201

Software Engineering COMP 201
Software Testing and Quality Assurance

Software Testing and Quality Assurance
Software Requirements

Software Requirements
Lecture 13 Revision IMS9001 - Systems Analysis and Design.

Lecture 13 Revision IMS Systems Analysis and Design.
Software Requirements

Software Requirements
Department of CIS University of Pennsylvania 1/31/2001 Specification-based Protocol Testing Hyoung Seok Hong Oleg Sokolsky CSE 642.

Department of CIS University of Pennsylvania 1/31/2001 Specification-based Protocol Testing Hyoung Seok Hong Oleg Sokolsky CSE 642.
Introduction To Networking

Introduction To Networking
Slide 1. Slide 2 Outline Protocol Verification / Conformance Testing TTCN Automated validation Formal design and specification Traffic Theory Application.

Slide 1. Slide 2 Outline Protocol Verification / Conformance Testing TTCN Automated validation Formal design and specification Traffic Theory Application.
Testing Framework TST 15 Source: Laurent Velez, ETSI, Meeting Date: 2015-01-15 TST-2015-0017-Testing_framework.PPT.

Testing Framework TST 15 Source: Laurent Velez, ETSI, Meeting Date: TST Testing_framework.PPT.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 5 Slide 1 Requirements engineering l The process of establishing the services that the.

©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 5 Slide 1 Requirements engineering l The process of establishing the services that the.
Software Testing and QA Theory and Practice (Chapter 10: Test Generation from FSM Models) © Naik & Tripathy 1 Software Testing and Quality Assurance Theory.

Software Testing and QA Theory and Practice (Chapter 10: Test Generation from FSM Models) © Naik & Tripathy 1 Software Testing and Quality Assurance Theory.
Doc.: IEEE 802.15-00/063r0 Submission March 2000 Matthew Graney, TelelogicSlide 1 Project: IEEE P802.15 Working Group for Wireless Personal Area Networks.

Doc.: IEEE /063r0 Submission March 2000 Matthew Graney, TelelogicSlide 1 Project: IEEE P Working Group for Wireless Personal Area Networks.
Testing RAVEN Helmut Neukirchen Faculty of Industrial Engineering, Mechanical Engineering and Computer Science University of Iceland, Reykjavík, Iceland.

Testing RAVEN Helmut Neukirchen Faculty of Industrial Engineering, Mechanical Engineering and Computer Science University of Iceland, Reykjavík, Iceland.
Software Testing Verification and validation planning Software inspections Software Inspection vs. Testing Automated static analysis Cleanroom software.

Software Testing Verification and validation planning Software inspections Software Inspection vs. Testing Automated static analysis Cleanroom software.

Similar presentations

Ads by Google