1. Embedded SoftwareTesting
CS 577b Software Engineering II -- Introduction
6 July 2018
Embedded SoftwareTesting
CS 577b Software Engineering II
Supannika Koolmanojwong
© USC Center for Software Engineering

2. General requirements for embedded software
Functional requirements
Non-functional requirements
Architectural requirements
Physical requirements
Project requirements
Business requirements
Reliability
Efficient memory usage
Performance at execution time
Efficient use of processing power
Power consumption
Cost effectiveness

3. General requirements for embedded software
Functional requirements
Non-functional requirements
Real-time constraints
-ilities: Reliability, Mobility, Security, Reconfigurability, Scalability, adaptability
Resources: power, CPU, memory, bandwidth
Architectural requirements
Physical requirements
Project requirements
Business requirements
Reliability
Efficient memory usage
Performance at execution time
Efficient use of processing power
Power consumption
Cost effectiveness

4. General requirements for embedded software
Functional requirements
Non-functional requirements
Architectural requirements
Software Oriented Design, Hardware Oriented Design, co-design
Physical requirements
Project requirements
Business requirements
Reliability
Efficient memory usage
Performance at execution time
Efficient use of processing power
Power consumption
Cost effectiveness

5. General requirements for embedded software
Functional requirements
Non-functional requirements
Physical requirements
Power consumption
Safety/reliability
Weight and size
Physical robustness: Thermal, ruggedized
Project requirements
Business requirements
Reliability
Efficient memory usage
Performance at execution time
Efficient use of processing power
Power consumption
Cost effectiveness

6. General requirements for embedded software
Functional requirements
Non-functional requirements
Architectural requirements
Physical requirements
Project requirements
Compatibility : Product-line, certification
Upgrades and sustainability
Business requirements
Cost, manufacturing time
Reliability
Efficient memory usage
Performance at execution time
Efficient use of processing power
Power consumption
Cost effectiveness

7. Level of software system
Software unit
Build
Software item
Subsystem
Element
Segment
System

8. Basic Embedded Software System Testing
Software unit testing
Data-driven testing, Scenario-based testing
Software integration testing
End-to-end testing; Interface Testing; white box testing
Software item qualification testing
Requirements tracing, black box testing
System unit testing
Software + hardware
System integration testing
Integrate all components
System validation testing
Functional and
non-functional testing

9. Developmental Test & Evaluation (DT&E) Operational Test & Evaluation (OT&E)
“ In God we trust, all others must bring data” - W. Edwards Deming “ Trust, but verify” - Ronald Reagan

10. Embedded Software and V-Model

11. Model-in-the-loop (MiL, MITL)
Model-based testing
Can be both manual or automated testing
Simulated model and environment; no physical hardware components
Testing at early stages
Functional models (physical models)
Robustness, performance
MATLAB/ Simulink, excel

12. Software-in-the-loop (SiL, SITL)
Simulation-based Software Evaluation
The software is tested within a simulated environment model without any hardware (virtual environment)
Controller is converted to code or the generated code from the model
Focus on the code and algorithm

13. Processor-in-the-loop (PiL, PITL)
Embedded controllers are integrated in the embedded devices
Similar to SiL but the generated code runs on a target board and target processor or target processor emulator or an evalutation board

14. Hardware-in-the-loop(HiL)
Plant simulation or Software-Hardware Integration Testing
Software runs on the final Electronic Control Unit (ECU), real piece of hardware, or simulated one
Must include electrical emulation of sensors and actuators
To verify the integration of hardware and software in a more realistic environment
Usually the most expensive and slowest testing

16. Human factors Focus on human factors requirements
Using, Controlling, Perform decision making
Could be on a mock up or real equipment
Human-in-the-loop
Has 2 meanings : during testing or during deployment
Testing: Interactive simulation, usability testing
Operating: human is required for an automated system to function
Human-on-the-loop
Generally refers to high level of automation in the operational environment
“Human-supervised weapon”
Compare to Human-in-the-loop
more automation, function with out human intervention

17. Verification Methods Inspection Analysis Demonstration Test
Visual examining the documentation or the product
Only for the product that can be easily observed and does not require execution of the software itself
Not for functional or performance requirements
E.g. code conforming to coding standards
Analysis
Determining the performance by checking the product or by calculations or extrapolating and interpolating empirical data
Demonstration
Exercising or operating the product
Test
Exercising or operating the product using instruments or test equipment that is not part of the system

18. Test Verification Matrix

19. Software Qualification Testing
Functionality testing as in operational environment
Interface requirement testing
Specialty requirement testing
Supportability, testability, safety, security
Fault tolerance, recovery
Stress testing, worst-case scenarios
Resource utilization
CPU, memory, storage, bandwidth
Including COTS
Use combination of verification methods

20. Test Metrics Test Progress Test coverage Test status (pass, fail)
Resources
Utilization: CPU, I/O, memory
Response time
Defect list
Defect status
Defect Density

21. Test Like You Fly “Days in the life”, “Weeks in the life”
“Scenario Testing”
“End-to-End Test”

22. Test Life You Fly Acquisition and Systems Engineering Process
Risk analysis, buy/make decision
Schedule control, project management
Automation, security, resiliency upfront
Assessment Process
Mission Assurance and Validation tool
Test technique
Mission readiness test
Not focus on requirements verification, but overall scenario testing
© 2016 USC-CSSE

23. Test Life You Fly
Consider all limitations and constraints upfront; rainy day scenarios
Try to uncover defects or flaws
Compliment to other tests; not replacement
Focus on mission performance, not individual requirement
Focus on demonstrating that the system is able to perform the mission
© 2016 USC-CSSE

24. Scenario Testing Black box testing, soap opera testing
Ways to develop Scenario Testing
Benefits-driven: how will they do to achieve X?
Sequence-driven: what are the common sub-tasks to achieve X
Transaction-driven: What are the steps, information needed, output, display to perform x?
Real life examples
Could reuse your user scenario
Somewhat similar to BDD
© 2016 USC-CSSE

25. Steps in Scenario Testing
Pick more than one persona
Think about a day / a week in life in their lives
Think about how these persona are going to use your system
Write the soap opera in steps with specific info
Not just “location X”, but “Staples Center”
Not just “take picture”, but “take a picture of Kobe Bryant”
Thank about the expected output or how do you expect the system to react to you
© 2016 USC-CSSE

26. Behavior-Driven Development
© 2016 USC-CSSE

27. Examples of scenario testing
© 2016 USC-CSSE

28. Examples of Scenario Testing - post picture on facebook
Leonardo DiCarpio just received his first oscar, he wants to post his selfie picture on facebook
Leo took his selfie picture with his oscar
Leo clicks facebook app icon on his iphone
Leo clicks “Photo” icon on top of the screen
Leo selects picture from the “Camera Roll”
Leo clicks “Done”
Leo types “My first Oscar” and “Add Activity” clicks “Feeling Fabulous”
Leo click “Post”
Kate Winslate sees the post on her timeline ..
…..
© 2016 USC-CSSE

29. Workshop For each team, separate into 2 groups
First group: based on your requirements, identify our Test verification matrix
Second group: pick 5 major use cases, write user scenarios
Prepare for presentation
Off-campus: develop TLYF / Scenario testing for your project
Due : Monday March 7, 2016
© 2016 USC-CSSE