© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:1 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Making TTCN-3 work! Issues and strategies for its use in product development Martin Botteck, Thomas Deiß, Colin Willcock Nokia Research Center Bochum, Germany

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:2 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work TTCN-3 adoption today: Generally, uptake is slower than anticipated. Speed-up could be possible through : Handling variations of the testing task Integrating TTCN-3 into existing test environments Using TTCN-3 in the hardware development process

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:3 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Overview TTCN-3 for testing hardware Conclusion/Summary TTCN-3 in existing test environments Variations in Test Systems

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:4 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Variations in Testing Within a set of test cases there often are several commonalities Example: 3G “Call” –several types of calls (voice, data,…) and calling routes –behaviour will in many cases be similar, but not the same –“intuitive” solution: copy/paste code from one test case to another “similar” one -> maintenance problem when the 3G specification is updated –alternative solution: add a parameter in the test case invocation -> code will become hard to read

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:5 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Types of variations Value Variants –Variations that may be expressed by altering a parameter inside a test case or message sent to the SUT –seems trivial but soon becomes difficult due to vast amount of parameters and combinations typically not all combinations make sense dependancies between the values is typically not documented and implicit –a language or coding style solution needs to be found for this

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:6 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Types of variations (ctd) Behavioural Variants –Variations that may be expressed by a different sequence of messages between the SUT and the tester several messages or sets of sequences may remain the same some are different –decomposition of the SUT may reveal common abstract test case structures with respect to the SUT components –but what about complex composition variants of such components? –-> Abstract Modelling of the tester? (UML testing profile, Test Purpose Language)

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:7 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Types of variations (ctd) Variants of the System under test –product customisation –evolution of conformance standards Might be expressed by altering a parameter (e.g. “feature_x=OFF”, “call_type=‘rich_call’”) in combination with reference to a set of message sequences –Typically, a vast amount of such combinations will be needed How to solve this without copy/paste of code but still avoiding combinatorial logic to the user?

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:8 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Overview TTCN-3 for testing hardware Conclusion/Summary TTCN-3 in existing test environments Variations in Test Systems

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:9 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work The dilemma of choice Nowadays most R&D departments already perform testing: Conformance testing, hardware testing, product testing, module testing, unit testing… Proprietary test systems (“home grown” or “tailor made”) Multitude of vendors with closed environments Trying out a system is a lot of effort. So, shortcomings of any choice being made show up too late for reversing the choice

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:10 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work The testing tool chain for each stage components from different vendors need to interoperate clearly defined interfaces are needed Testware Test cases Test System Codec Router Test Development Tools Editor Compiler Codec Generator … Test Execution Control Log Analysis Thomas Deiß: Usually, in the TTCN-3 world, the platform and the SUT adapter are considered to work side by side, not the PA on top of the SA. MBo: Better this way? It does not become clear to me, what are the stages: Tools, Testware (development), Test sytem (development?), Test execution. But 'tools' is not a stage. MBo: Indeed. Let´s talk Editorial comment: Test system Test Execution, note different capitalization of the second word Thomas Deiß: Usually, in the TTCN-3 world, the platform and the SUT adapter are considered to work side by side, not the PA on top of the SA. MBo: Better this way? It does not become clear to me, what are the stages: Tools, Testware (development), Test sytem (development?), Test execution. But 'tools' is not a stage. MBo: Indeed. Let´s talk Editorial comment: Test system Test Execution, note different capitalization of the second word Run Time System Executable Test System (ETS) Platform Adapter SUT Adapter SUT Codec

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:11 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Run Time System Executable Test System (ETS) Platform Adapter SUT Adapter SUT Codec The testing tool chain interfaces Testware Test cases Test System Codec Router Test Development Tools Editor Compiler Codec Generator … Test Execution Control Log Analysis TTCN-3 core notation language for exchange of test cases Run Time IF for Adaptors and Codecs Test Execution Control and logging through TCI TCI/Log TRI

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:12 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Further work on interfaces Tool vendors have a notion to “close shop” and not support public interfaces Codec interface part in TCI leads to non-interchangeable codecs realisation of a given specification in IDL depends on the target language. Java based tools and C based tools require different codecs performance problems when separating the codec to a separate executable Standardisation sometimes lags behind (e.g. TCI) It is also desirable to generate as many test cases and parts of the environment as possible directly from the behavioural description of the IUT -> modelling representations (e.g. XMI) need to be supported in the tools

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:13 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Overview TTCN-3 for testing hardware Conclusion/Summary TTCN-3 in existing test environments Variations in Test Systems

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:14 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Hw design flow Algorithmic description informal model, e.g. C source code Highest Level behavioural model (e.g. SystemC) Decomposition (Block diagram) Notion of “Time” Bit accurate Hardware model (e.g. VHDL) Notion of “Clock Cycles” Physical Model (e.g. RTL, Spice,…) Introduction of physics (layout, transistor, …) Description consistent through all stages/levels Automatic generation of tests (“Test vectors”) Back annotation integral part of the process

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:15 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Sw design flow use case description formal model, e.g. UML UC Highest Level behavioural model (e.g. UML sequence diagram, state chart) Decomposition (package diagram) Refinement, Introduction of “Signals” executable (e.g. binary, script,…) Introduction of programming language syntax (C, C++, Java,…) more detailed behavioural model (e.g. UML sequence diagram, state chart) Rather “novel” approach Automatic generation of tests (“Test vectors”) still in experimental phase Back annotation is not formally specified

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:16 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work “Traditional” HW-SW integration Algorithmic description informal model, e.g. C source code Highest Level behavioural model (e.g. SystemC) Decomposition (Block diagram) Notion of “Time” Bit accurate Hardware model (e.g. VHDL) Notion of “Clock Cycles” Physical Model (e.g. RTL, Spice,…) Introduction of physics (layout, transistor, …) use case description formal model, e.g. UML UC Highest Level behavioural model (e.g. UML sequence diagram, state chart) Decomposition (package diagram) Refinement, Introduction of “Signals” executable (e.g. binary, script,…) Introduction of programming language syntax (C, C++, Java,…) more detailed behavioural model (e.g. UML sequence diagram, state chart) SW release HW Prototype SW release HW prototype Integration Integration Tests

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:17 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Common nodes in the HW and SW design flow Integration tests are not determined by formal derivation from HW/SW high level models Integration typically is the 1 st stage where HW and SW meet Integration results feed back “somewhere” in the process in SW HW and SW tests only derive from HW and SW models respectively How about utilisation of SW models (e.g. High level behavioural) for generation of HW tests? including behavioural HW models (SystemC) in High level behavioural SW models? generation of tests directly from the model representation?

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:18 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Overview TTCN-3 for testing hardware Conclusion/Summary TTCN-3 in existing test environments Variations in Test Systems

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:19 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Summary/Conclusions Further deployment of TTCN-3 needs convincing advantages: –integration into existing test environments interfaces, replaceable components from heterogenous test systems –handling of variants through specific coding constructs –automated generation of test cases from abstract models –cover HW as well as SW through adequate representation in SW models

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:20 Nokia Research Center CAR/MEM/VTT Making TTCN-3 work Thank you! Questions? Acknowledgement: The authors would like to thank Mr. Risto Teittinen from Nokia Networks for his contributions to this topic