1. Lecturer: Dr. AJ Bieszczad Chapter 99-1 Causes of faults during development

2. Lecturer: Dr. AJ Bieszczad Chapter 99-2 System testing process Function testing: does the integrated system perform as promised by the requirements specification? Performance testing: are the non-functional requirements met? Acceptance testing: is the system what the customer expects? Installation testing: does the system run at the customer site(s)?

3. Lecturer: Dr. AJ Bieszczad Chapter 99-3 Steps in testing process

4. Lecturer: Dr. AJ Bieszczad Chapter 99-4 Techniques used in system testing Build or spin plan for gradual testing Configuration management –versions and releases –production system vs. development system –deltas, separate files and conditional compilation –change control Regression testing

5. Lecturer: Dr. AJ Bieszczad Chapter 99-5 Test team Professional testers: organize and run the tests Analysts: who created requirements System designers: understand the proposed solution Configuration management specialists: to help control fixes Users: to evaluate issues that arise

6. Lecturer: Dr. AJ Bieszczad Chapter 99-6 Function testing A test should: –have a high probability of detecting faults –use a test team independent of the designers and programmers –know the expected actions and output –test both valid and invalid input –never modify the system to make testing easier –have stopping criteria

7. Lecturer: Dr. AJ Bieszczad Chapter 99-7 Cause-and-effect graphs (1) INPUT: The syntax of the function is LEVEL(A,B) where A is the height in meters of the water behind the dam, and B is the number of centimeters of rain in the last 24- hour period. PROCESSING: The function calculates whether the water level is within a safe range, is too high, or is too low. OUTPUT: The screen shows one of the following messages: 1. “LEVEL = SAFE” when the result is safe or low. 2. “LEVEL = HIGH” when the result is high. 3. “INVALID SYNTAX” depending on the result of the calculation.

8. Lecturer: Dr. AJ Bieszczad Chapter 99-8 Cause-and-effect graphs (2) Causes:

9. Lecturer: Dr. AJ Bieszczad Chapter 99-9 Cause-and-effect graphs (3) Effects: Intermediate nodes:

10. Lecturer: Dr. AJ Bieszczad Chapter 99-10 Notation for cause-and-effect graph

11. Lecturer: Dr. AJ Bieszczad Chapter 99-11 Additional graph notation

12. Lecturer: Dr. AJ Bieszczad Chapter 99-12 Cause-and-effect graph

13. Lecturer: Dr. AJ Bieszczad Chapter 99-13 Test 1Test 2Test 3Test 4Test 5 Cause 1IIISI Cause 2IIIXS Cause 3ISSXX Cause 4SISXX Cause 5SSIXX Effect 1PPAAA Effect 2AAPAA Effect 3AAAPP Decision table for cause-and-effect graph

14. Lecturer: Dr. AJ Bieszczad Chapter 99-14 Performance tests Stress tests Volume tests Configuration tests Compatibility tests Regression tests Security tests Timing tests Environmental tests Quality tests Recovery tests Maintenance tests Documentation tests Human factors (usability) tests

15. Lecturer: Dr. AJ Bieszczad Chapter 99-15 Software Reliability, Availability and Maintainability Reliability –the probability that the system will operate without failure under given conditions for a given time interval Availability –the probability that a system is operating successfully according to specifications at a given point in time Maintainability –the probability that, for a given condition of use, a maintenance activity can be carried out within a stated time interval and using stated resources and procedures

16. Lecturer: Dr. AJ Bieszczad Chapter 99-16 Example: Inter-failure times (read left to right, in rows)

17. Lecturer: Dr. AJ Bieszczad Chapter 99-17 Example: Inter-failure graph

18. Lecturer: Dr. AJ Bieszczad Chapter 99-18 Measuring Reliability, Availability and Maintainability Mean Time to Failure (MTTF) R = MTTF/(1 + MTTF) Mean Time to Repair (MTTR) M = 1/(1+MTTR) Mean Time between Failures (MTBF) MTBF = MTTF + MTTR A = MTBF/(1+MTBF)

19. Lecturer: Dr. AJ Bieszczad Chapter 99-19 Reliability function f(t) failure probability density function from statistics F(t) failure distribution function

20. Lecturer: Dr. AJ Bieszczad Chapter 99-20 Uniform probability density function

21. Lecturer: Dr. AJ Bieszczad Chapter 99-21 Example: Inter-failure times (read left to right, in rows) for reliability prediction Predictions: t 4 =71.5, because t 2 =30 and t 3 =113 t 5 =97, because t 3 =113 and t 4 =81 etc.

22. Lecturer: Dr. AJ Bieszczad Chapter 99-22 Reliability prediction graph Averaging previous two failure times to predict the third.

23. Lecturer: Dr. AJ Bieszczad Chapter 99-23 Acceptance tests Benchmark test: testing on pre-defined sets of typical (“important”) test cases Pilot test: install on experimental basis Alpha test: in-house test Beta test: customer pilot Parallel testing: new system operates in parallel with old system

24. Lecturer: Dr. AJ Bieszczad Chapter 99-24 Test documentation Test plan: describes system and plan for exercising all functions and characteristics Test specification and evaluation: details each test and defines criteria for evaluating each feature Test description: test data and procedures for each test Test analysis report: results of each test

25. Lecturer: Dr. AJ Bieszczad Chapter 99-25 Documents produced during testing

26. Lecturer: Dr. AJ Bieszczad Chapter 99-26 Parts of a test plan

27. Lecturer: Dr. AJ Bieszczad Chapter 99-27 EXAMPLE: Test SORT INPUT DATA: Input data are to be provided by the LIST program. The program generates randomly a list of N words of alphanumeric characters; each word is of length M. The program is invoked by calling RUN LIST(N,M) in your test driver. The output is placed in global data area LISTBUF. The test datasets to be used for this test are as follows: Case 1: Use LIST with N=5, M=5 Case 2: Use LIST with N=10, M=5 Case 3: Use LIST with N=15, M=5 Case 4: Use LIST with N=50, M=10 Case 5: Use LIST with N=100, M=10 Case 6: Use LIST with N=150, M=10 INPUT COMMANDS: The SORT routine is invoked by using the command RUN SORT (INBUF,OUTBUF) or RUN SORT (INBUF) OUTPUT DATA: If two parameters are used, the sorted list is placed in OUTBUF. Otherwise, it is placed in INBUF. SYSTEM MESSAGES: During the sorting process, the following message is displayed: “Sorting... please wait...” Upon completion, SORT displays the following message on the screen: “Sorting completed” To halt or terminate the test before the completion message is displayed, press CONTROL-C on the keyboard.

28. Lecturer: Dr. AJ Bieszczad Chapter 99-28 Example: Test script Step N:Press function key 4: Access data file. Step N+1:Screen will ask for the name of the date file. Type ‘sys:test.txt’ Step N+2:Menu will appear, reading * delete file * modify file * rename file Place cursor next to ‘modify file’ and press RETURN key. Step N+3:Screen will ask for record number. Type ‘4017’. Step N+4:Screen will fill with data fields for record 4017: Record number: 4017X: 0042 Y: 0036 Soil type: clayPercolation: 4 mtrs/hr Vegetation: kudzuCanopy height: 25 mtrs Water table: 12 mtrsConstruct: outhouse Maintenance code: 3T/4F/9R Step N+5:Press function key 9: modify Step N+6:Entries on screen will be highlighted. Move cursor to VEGETATION field. Type ‘grass’ over ‘kudzu’ and press RETURN key. Step N+7:Entries on screen will no longer be highlighted. VEGETATION field should now read ‘grass’. Step N+8:Press function key 16: Return to previous screen. Step N+9:Menu will appear, reading * delete file * modify file * rename file To verify that the modification has been recorded, place cursor next to ‘modify file’ and press RETURN key. Step N+10:Screen will ask for record number. Type ‘4017’. Step N+11:Screen will fill with data fields for record 4017: Record number: 4017X: 0042 Y: 0036 Soil type: clayPercolation: 4 mtrs/hr Vegetation: grassCanopy height: 25 mtrs Water table: 12 mtrsConstruct: outhouse Maintenance code: 3T/4F/9R

29. Lecturer: Dr. AJ Bieszczad Chapter 99-29 Problem report forms Location Timing Symptom End result Mechanism Cause

30. Lecturer: Dr. AJ Bieszczad Chapter 99-30 Testing safety-critical systems Design diversity: use different kinds of designs, designers Software safety cases: make explicit the ways the software addresses possible problems –failure modes and effects analysis –hazard and operability studies Cleanroom: certifying software with respect to the specification

31. Lecturer: Dr. AJ Bieszczad Chapter 99-31 Known causeUnknown cause Known effectDescription of system behaviorDeductive analysis, including fault tree analysis Unknown effectInductive analysis, including failure modes and effects analysis Exploratory analysis, including hazard and operability studies Perspectives for safety analysis

32. Lecturer: Dr. AJ Bieszczad Chapter 99-32 Cleanroom The cleanroom approach addresses two fundamental principles: –to certify with respect to the specifications, rather than wait for unit testing to find faults –to produce zero-fault or near-zero-fault software Blending of numerous techniques