1. Acceptance Testing Strategy Guided by Evolutionary Operational Profiles
Authors: Maria de Fatima Mattiello-Francisco
Ana Maria Ambrosio
National Institute for Space Research
Edgar Toshiro Yano
Aeronautical Institute of Technology
Industry Track ISSRE2007

2. System’s stakeholders satisfaction Expected testing quality
Motivation
Acceptance Process
Service Profiles
Test Strategy
Results
The target is the Payload Embedded Software on board X-Ray Astronomy Satellite
System’s stakeholders satisfaction
Expected testing quality
Confidence on critical software operation
takes into account:
Stakeholders involved in
Development & Operation Processes
Principal Investigator
Quality is related
to Test Strategies
which includes
Test Techniques
Environmental facilities
Testing time duration

3. Acceptance Process Motivation Service Profiles Test Strategy Results
Mission Integration Stages
Instrument
ECSS- European Cooperation for Space Standardization
Subsystem
System
CDR – Critical Design Review QR – Qualification Review AR – Acceptance Review

4. Acceptance Process Acceptance Process Motivation Service Profiles
Test Strategy
Results
Mission Integration Stages
The target software is integrated with other mission's components in the three mission stages
Instrument
Subsystem
System
(Payload)
(Satellite Platform)
(Space & Ground Segment)
During each stage the payload software will be an encapsulated component being validated under the different tester’s point of view
Acceptance Process
A formal testing to determine whether or not a systems satisfies acceptance criteria based on the software problems reports resulting from verification and validation activities [IEEE-Std ]
In our approach the acceptance criteria is based on the number of faults revealed by the services execution during each integration stages.

5. Acceptance Process U1 U2 U3 U4 U2 U3 U4 U5 U2 U3 U4 U5 U6 U7
Motivation
Service Profiles
Test Strategy
Results
Testers involved in the Mission Integration Stages
Instrument
Subsystem
System
Payload
Satellite Platform
Space & Ground Segment
U1 U2 U3 U4
U2 U3 U4 U5
U2 U3 U4 U5 U6 U7
U1- Payload Software Engineer (focus on software requirement validation)
U2 – Payload Engineer (focus on interfaces performance)
U3 – Principal Investigator (Scientist focus on operational modes)
U4 – Software Engineer (Regression Test )
U5 – On Board Data Handling Computer Engineer (focus on communication)
U6 – Mission System Engineer
U7 – Ground Segment Engineer (Mission Center Operation)
Testers with different needs carry on the mission integration stages

6. Service Profiles Operational Profiles Services Motivation
Acceptance Process
Test Strategy
Results
Operational Profiles
External user-oriented test approach – specifies the intended usage of the system in terms of operations and their occurrence probabilities [MUSA93]
Services
Space system operations are service-oriented. A set of on-board performing functions is always related to one service purpose.
On-board embedded software is a component that collaborates on service provider by executing services activities, being destination of service requests and the source of service reports [ECSS-E ].
Operation problems can be anticipated by TESTERS performing service-oriented test cases during the acceptance testing

7. The proposed Evolutionary Operational Profile
Service Profiles
Motivation
Acceptance Process
Test Strategy
Results
The proposed Evolutionary Operational Profile
Aggregates to Musa’s approach the service operational profiles, adding the concept of service and the usage of system architectural elements
Such new approach aims at supporting a test strategy for embedded space software acceptance along the evolutionary integration stages
Denotes the growth in complexity and autonomy of the environment with which the embedded software interacts as far the integration stages evolves.
At each integration stage the environmental facilities are incrementally substituted by the real subsystems.
Software functionality does not change but service operation does due to the test scripts evolution

8. Identify software operational modes profile
Service Profiles
Motivation
Acceptance Process
Test Strategy
Results
SERVICE PROFILES
Is a concept which depends on software behavior in terms of operational mode and software architecture
Services are performed within system modes by component and their interfaces.
They are conceived taking into account the following steps
Identify software operational modes profile
Identify software architectural profile
List the SERVICES to be performed
Establish relationship between element and services concerning USAGE INTENSITY
Calculate the service operational profiles

9. Service Profiles Mode profile Motivation Acceptance Process
Test Strategy
Results
Mode profile
Is defined as the probability of occurrence of each operational mode in the system use.
A service might be provided by more than one software operational mode and its occurrence probabilities being different in each mode.
Software operations and services are performed within operational modes and by architectural elements.
Case Study: MIRAX Payload Software - SWPDC operational modes

10. Architectural Element profile
Service Profiles
Motivation
Acceptance Process
Test Strategy
Results
Architectural Element profile
Case Study: MIRAX Payload Software - SWPDC Architecture

11. Architectural Element profile
Service Profiles
Motivation
Acceptance Process
Test Strategy
Results
Architectural Element profile
Given a category of architectural element, one can attribute probabilities of usage of e in system operation lifetime as a whole. i
The occurrence probability estimation depends on high knowledge of the application and historical data from previous experience in similar projects
SERVICE USAGE MATRIX
The binary cell a indicates the relative use of the architectural element by each service ij

12. Service Profiles Calculation
Motivation
Acceptance Process
Test Strategy
Results
Service Profiles Calculation
Service profile is defined as the occurrence probability of each service in terms of using a particular element.
Where weight w represents the degree of usage intensity and/or criticality of the element e on such service q ij i j
MODE USAGE INTENSITY MATRIX

13. Test Strategy Test Strategy Motivation Acceptance Process
Service Profiles
Results
Test Strategy
Based on SERVICES PROFILES PSe, one selects the number of TEST CASES Ntc provided by U testers per service in order to match the previous total time T allocated for each mission integration stage Ek
There is a set of services Si that exercise better some elements Ek than other. Therefore, priority on performing particular test cases is recommended whenever reduction on testing duration are imposed to the testers Uj Uj
The test strategy effectiveness will be demonstrated in our case study with data from the mission integration stage at Instrument level provided by the testers. All test cases associated to the mentioned services were executed in order to evaluate this approach.
f ( S4,U2,E1) = ??? Si

14. Test Strategy Motivation Acceptance Process Service Profiles Results
Instrument
Ntc provided by tester per service
U U U U4 A
101
636 46 88
= 870
Considering that the average time of 2 test cases execution (including analyses) is 1 hour for group A, this stage will require 435 hours to be concluded.
If the available time is 220 hours (50%) only, our test strategy approach reduces respectively the number of test cases as showed in group B: 72
285 29 54
= 440 B
One observes that the reduced test set (60%) is not proportional to the schedule constraints (50%). There is a gain in terms of test set to be performed.
72%
45%
63%
61%
60%
Reduction Percentage

15. Results Motivation Acceptance Process Service Profiles Test Strategy
Faults detected by A and B
U U U U4 A
147 failures B
74 failures
7 bad configuration on test execution
2 source code malfunction
7 document inconsistencies
A particular system failure can be detected by several Test Cases provided by different testers. In such case study it was observed that 147 failures detected by the test set A were caused by the following faults:
The results showed that redundancies in terms of test cases coverage are given by the service-oriented test case generation. Such redundancy allows to choose test case which covers intrinsically more than one service, without losing test efficiency.
The 74 failures detected by the reduced test set B were caused by the same faults

16. Results Questions??? Motivation Acceptance Process Service Profiles
Test Strategy
Questions???
Financial support to the QSEE Project
Quality in Space Application Embedded Software