1. COTS testing
Torbjørn Skramstad

2. Some used approaches Component meta-data approach.
Retro-components approach.
Built-in test (BIT) approach.
The STECC strategy.
COTS

3. What is Meta-data
Meta-data are any data related to a component that is not the code. Commonly used meta-data are for instance:
State diagrams
Quality of Service information
Pseudo code and algorithm descriptions
Test logs – what has been tested?
Usage patterns – how has the component been used up till now?

4. Comments on Meta-data
Meta-data can take up a considerable amount of storage. Thus, they can be
an integrated part of the component
stored separately and have to be down-loaded when needed.

5. Testing info. done by provider
Component meta-data – 1
Component
Binary code
Call graphs,
Testing info. done by provider
Metadata

6. Component meta-data – 2 server Component functionality Meta DB
Metadata req
Metadata

7. Assessment based on meta-data
Round trip path tests based on state diagrams
Functional tests based on algorithms or pseudo code
Tests relevance assessment based on test logs
Previous usage patterns in accordance with our expected use

8. Retrospectors
A retrospector is a tool that records the testing and execution history of a component. The information is stored as meta-data.
A retro-component is a software component with a retrospector.

9. Retro-components approach
server
functionality
Meta DB
Metadata req + test and usage data
Metadata

10. Using the retrospectors
By collecting usage info we have a better basis for defect removal and testing
For COTS component users, the retrospector will give important info on
How the component was tested – e.g. instead of a test log
How the component has been used by others. This will tell us whether we are going to use the component in a new way => high risk
e.g. we plan to use it in a safety critical system

11. Built-In Test – BIT We will focus on BIT – RTT (Run Time Testability).
We need two sets of tests:
In the component, to test that its environment behaves as expected
In the component’s clients, to test that the component implements the semantics that its clients have been developed to expect

12. Testing the component The test consists of the following steps:
Bring the component to the starting state for the test
Run the test
Check that the
results are as expected
final state is reached as expected

13. BIT-description for a gearbox

14. Selecting tests We need to consider two issues:
The quality of the test – the more comprehensive the better. Unfortunately, more comprehensive => larger
The size of the test – the faster the better. Unfortunately, faster => smaller
The solution is to have several sets of tests that are run at different occasions.

15. BIT architecture – 1 We have the following components:
The component, with one or more interfaces and implementation of functionality
BIT-RTT provides support for the testing
External tester – runs the tests needed
Handler – takes care of errors, exceptions, fail-safe behavior etc.
Constructor – initialization of components such as external testers and handlers

16. BIT architecture – 2

17. BIT dead-lock testing

18. Disadvantages of BIT Static nature.
Do not ensure that tests are conducted as required by the component user
The component provider makes some assumptions concerning the requirements of the component user, which again might be wrong or inaccurate.

19. What is STECC STECC stands for “Self TEsting Cots Components”.
The method has much in common with BIT. The main difference is that
BIT is static – we rerun one or more already defined tests.
STECC is dynamic – we generate new tests based on a description. We may also interact with the tester.

20. STECC strategy Server query functionality Meta DB Metadata Req. Tester
Test generator

21. Benefits of STECC
The STECC addresses the needs of the component provider and the component user
The COTS provider might not want to disclose information, in particular source code
The COTS user might require this information for performing adequate and necessary testing
Existing approaches, such as BIT, do not appropriately tackle such a situation

22. STECC reference
S. Beydeda, The Self-Testing COTS Components (STECC) Method. PhD thesis, Universitat Leipzig, Fakultat fur Mathematik und Informatik, 2003

23. Assessing COTS – 1
When considering a candidate component, developers need to ask three key questions:
Does the component meet the developers’ needs?
Is the quality of the component sufficient? (quality factor, level)
What impact will the component have on the system?
It is practical to consider the answers to these questions for several relevant scenarios

24. Assessing COTS – 2

25. Black box test reduction using Input-output Analysis
Random Testing is not complete
To perform complete functional testing, the number of test cases can sometimes be reduced by Input-output Analysis
We can identify I/O relationships by using static analysis or execution analysis of program

28. Test reduction using Orthogonal Array Testing
Used when the number of input parameters is small and the values that each of the parameters may take are clearly bounded

29. L9 orthogonal array
One input at a time: (1,1,1,1), (2,1,1,1),…(1,1,1,3). Finds ”single mode faults”, but not faults when there exist logical relations between the variables
Exhaustive testing: 34 = 81 tests
L9 orthogonal: see table to the right.
Test strategy:
Detect and isolate all single mode errors
Detect all double mode errors (specific levels of two parameters occur together)
Multimode errors: some multimode errors can be found by L9 orthogonal array testing

30. Final test set