SIMPL-T: SDL Intended for Management and Planning of Tests By Qing Li, Robert Probert, William Skelton, Yiqun Xu SAM’04 Workshop, Ottawa, Canada June, 2004

Outline Background and Motivation Approach Assessment Contributions and Future Work SAM'04

Relative Error Correction Cost in a Software Life Cycle Background and Motivation Relative Error Correction Cost in a Software Life Cycle Specification is at the earliest stage in a software lifecycle Its correctness and completeness are critical to subsequent development It must be validated against user requirements Specification must be tested! SAM'04

No Existing Formal Language is Suitable for Testing SDL Specifications Background and Motivation No Existing Formal Language is Suitable for Testing SDL Specifications TTCN MSC UML URN/UCM LOTOS SDL SAM'04

Background and Motivation SDL Task Force The graphical representation, ensuring auto-layout is possible Test capabilities, such as SDL based test scripts ASN.1(1994) support, including encoding/ decoding of PDUs Associated methodology issues, such as maximum integration of tool chain SAM'04

Statement of Research Problem Background and Motivation Statement of Research Problem To define and investigate the applicability of a simple, useful and efficient language for describing tests of SDL specifications A simple test language for SDL specifications Defined as a minimal extension of existing SDL features To provide essential test specification functionality SIMPL-T -- SDL Intended for Management and Planning of Tests SAM'04

Approach Basic Testing Concepts Key Requirements Suitability of SDL for Test Specification SIMPL-T – SDL with Extensions SAM'04

SDL & TTCN Overlap Extensions to SDL Approach SDL Overlap TTCN TTCN SDL and TTCN has overlap TTCN is sufficient to test SDL Spec. Only part of TTCN is needed to test SDL Spec. After identify the SDL testing features, extensions to SDL can be defined Extensions to SDL TTCN SDL SAM'04

Test Architecture (ITU-T Z.500) Basic Testing Concepts Test Architecture (ITU-T Z.500) Basic testing concepts are defined in ITU-T Z.500 and ISO 9646 Key requirements of testing language can be derived from these standards SAM'04

Test Architecture (ISO 9646) Basic Testing Concepts Test Architecture (ISO 9646) Lower Tester Upper Tester U IUT L Underlying Service Provider * Test Configuration SAM'04

Test Architecture Tester – Run test suite IUT Connection – PCOs Basic Testing Concepts Test Architecture Tester – Run test suite IUT Connection – PCOs Communication Channels SAM'04

Test Case and Test Suite Basic Testing Concepts Test Case and Test Suite Test Suite Test Case Test Purpose Test Case Behaviour Sending a stimulus to the IUT Specifying expected response Store and Transfer data Take alternative actions Repeated test steps or actions SAM'04

Observations Check the responses Measure the timing of response Basic Testing Concepts Observations Check the responses Measure the timing of response Assign Verdict SAM'04

Key Requirements Test Architecture – Tester and SUT Approach Key Requirements Test Architecture – Tester and SUT Connection between the Tester and the SUT Communication between the Tester and the SUT Organization and Management of Tests Sending Stimuli to the IUT Receiving Response from the IUT Storing and Transferring Data Flow Control Test Step Repetition Checking Responses and Matching Mechanism Measuring the Timing of Responses. Assigning and Handling of Verdict SAM'04

Suitability of SDL for Test Specification Approach Suitability of SDL for Test Specification Based on the key requirements, this thesis investigated: Which SDL features can be used for testing How they can be used Identified what extensions are needed SAM'04

Extensions Organization and Management of Tests Checking Responses Approach Extensions Organization and Management of Tests Checking Responses -- “Input Via” and Matching mechanism Assigning and Handling of Verdicts New constructs for test suite, test group, test case, and test purpose are defined Existing “input” construct is enhanced to “input via” Matching mechanism is defined New constructs for Verdict is defined; Verdict handling mechanism is defined SAM'04

Organization and Management of Tests Approach Organization and Management of Tests Testsuite_Definition ::= “TESTSUITE” TestsuiteName “;” [ Gate_Definition ] [ Testsuite_Component ] “ENDTESTSUITE;” Gate_Definition ::= “GATE” GateName “;” [ In_Signal_List ] “;” [ Out_Signal_List ]”;” In_Signal_List ::= Signal_Identifier [ “,” In_Signal_List ] Out_Signal_List ::= Signal_Identifier [ “,” Out_Signal_List ] Testsuite_Component ::= ([Signal_Definition] [Signal_List_Definition] [Timer_Definition] …… [Test_Group_Definition] [Test_Case_Definition] ) [Testsuite_Component ] Test_Group_Definition ::= “TESTGROUP” TestGroupName “;” Test_Case_Definition_List Example syntax of “Testsuite” SAM'04

New INPUT VIA Construct Approach New INPUT VIA Construct STATE S1; INPUT A VIA Gate1; NEXTSTATE S2; SAM'04

Specifying Expected Values of Parameters inside INPUT Approach Specifying Expected Values of Parameters inside INPUT SAM'04

Matching Mechanism Unmatched Signal Handling: Disregard by default Approach Matching Mechanism Unmatched Signal Handling: Disregard by default Explicitly use “Save” construct when necessary SAM'04

Matching Mechanism Overlapped Signal Handling: Approach Matching Mechanism Overlapped Signal Handling: the same signal arriving from different gates/channels; -- > They are not considered as overlap in SIMPL-T (2) the parameters carried by the same signal have different values and the values have overlap -- > They are not allowed in SIMPL-T SAM'04

An Example of a SIMPL-T Test Case Approach An Example of a SIMPL-T Test Case SAM'04

An Example of a SIMPL-T Test Case (Cont.) Approach An Example of a SIMPL-T Test Case (Cont.) SAM'04

The Strengths and Limitations of SIMPL-T Comparing to TTCN Assessment The Strengths and Limitations of SIMPL-T Comparing to TTCN + Strength - Weakness = Same / Not needed To assess SIMPL-T, we compare it to the international standard testing language TTCN: SIMPL-T has many advantages over TTCN SIMPL-T also has weakness “/” – Many TTCN features are not needed in SIMPL-T, #11 and # 12 are examples Many features are equally supported by TTCN and SIMPL-T: Concurrency: TTCN -- Through Main Test component /Parallel Test Components (MTC/PTCs) SIMPL-T – Inherited from SDL SAM'04

Ordering Problem Two or more signals can arrive in arbitrary order Assessment Ordering Problem Two or more signals can arrive in arbitrary order The order is irrelevant, The test language does not have a mechanism to specify this situation SIMPL-T -- solve it using “save” construct SAM'04

Contributions Submitted to the SDL Task Force Contributions and Future Work Contributions Submitted to the SDL Task Force Defined a simple, easy to learn test language Create a potential for lower cost tools Lead to more interest in SDL and testing SAM'04

Future Work Concurrency Defaults Extensions for larger applications Contributions and Future Work Future Work Concurrency Defaults Extensions for larger applications Defaults” is not an essential feature in testing, so it is not discussed; but it makes a test language easier to use, so left as future work For larger applications, extensions may be needed SAM'04