1. 10- Verification and Validation

2. What is a computer bug? In 1947 Harvard University was operating a room-sized computer called the Mark II. –mechanical relays –glowing vacuum tubes –technicians program the computer by reconfiguring it –Technicians had to change the occasional vacuum tube. A moth flew into the computer and was zapped by the high voltage when it landed on a relay. Hence, the first computer bug! –I am not making this up :-)

3. Bugs a.k.a. … Defect Fault Problem Error Incident Anomaly Variance Failure Inconsistency Product Anomaly Product Incidence Feature :-)

4. Defective Software We develop programs that contain defects –How many? What kind? Hard to predict the future, however… it is highly likely, that the software we (including you!) will develop in the future will not be significantly better.

5. Sources of Problems Requirements Definition: Erroneous, incomplete, inconsistent requirements. Design: Fundamental design flaws in the software. Implementation: Mistakes in chip fabrication, wiring, programming faults, malicious code. Support Systems: Poor programming languages, faulty compilers and debuggers, misleading development tools.

6. Sources of Problems (Cont’d) Inadequate Testing of Software: Incomplete testing, poor verification, mistakes in debugging. Evolution: Sloppy redevelopment or maintenance, introduction of new flaws in attempts to fix old flaws, incremental escalation to inordinate complexity.

7. Adverse Effects of Faulty Software Communications: Loss or corruption of communication media, non delivery of data. Space Applications: Lost lives, launch delays. Defense and Warfare: Misidentification of friend or foe.

8. Adverse Effects of Faulty Software (Cont’d) Transportation: Deaths, delays, sudden acceleration, inability to brake. Safety-critical Applications: Death, injuries. Electric Power: Death, injuries, power outages, long-term health hazards (radiation).

9. Adverse Effects of Faulty Software (Cont’d) Money Management: Fraud, violation of privacy, shutdown of stock exchanges and banks, negative interest rates. Control of Elections: Wrong results (intentional or non-intentional). Control of Jails: Technology-aided escape attempts and successes, accidental release of inmates, failures in software controlled locks. Law Enforcement: False arrests and imprisonments.

10. Bug in Space Code Project Mercury’s FORTRAN code had the following fault: DO I=1.10 instead of... DO I=1,10 The fault was discovered in an analysis of why the software did not seem to generate results that were sufficiently accurate. The erroneous 1.10 would cause the loop to be executed exactly once!

11. Military Aviation Problems An F-18 crashed because of a missing exception condition: if... then... without the else clause that was thought could not possibly arise. In simulation, an F-16 program bug caused the virtual plane to flip over whenever it crossed the equator, as a result of a missing minus sign to indicate south latitude.

12. Therac-25 Radiation “Therapy” In Texas, 1986, a man received between 16,500-25,000 rads in less than 1 sec, over an area of about 1 cm. He lost his left arm, and died of complications 5 months later. In Texas, 1986, a man received at least 4,000 rads in the right temporal lobe of his brain. The patient eventually died as a result of the overdose.

13. Therac-25 Radiation “Therapy” (Cont’d) In Washington, 1987, a patient received 8,000-10,000 rads instead of the prescribed 86 rads. The patient died of complications of the radiation overdose.

14. AT&T Bug: Hello?... Hello? In mid-December 1989, AT&T installed new software in 114 electronic switching systems. On January 15, 1990, 5 million calls were blocked during a 9 hour period nationwide.

15. AT&T Bug (Cont’d) The bug was traced to a C program that contained a break statement within an switch clause nested within a loop. The switch clause was part of a loop. Initially, the loop contained only if clauses with break statements to exit the loop. When the control logic became complicated, a switch clause was added to improve the readability of the code...

16. Bank Generosity A Norwegian bank ATM consistently dispersed 10 times the amount required. Many people joyously joined the queues as the word spread.

17. Bank Generosity (Cont’d) A software flaw caused a UK bank to duplicate every transfer payment request for half an hour. The bank lost 2 billion British pounds! The bank eventually recovered the funds but lost half a million pounds in potential interest.

18. Making Rupee! An Australian man purchased $104,500 worth of Sri Lankan Rupees. The next day he sold the Rupees to another bank for $440,258. The first bank’s software had displayed a bogus exchange rate in the Rupee position! A judge ruled that the man had acted without intended fraud and could keep the extra $335,758!

19. Discussion … Have you heard of other software bugs? –In the media? –From personal experience? Does this embarrass you as a future software engineer?

20. Specification “if you can’t say it, you can’t do it” You have to know what your product is before you can say if it has a bug. A specification defines the product being created and includes: –Functional requirements that describes the features the product will support. E.g., on a word processor Save, print, check spelling, change font, … –Non-functional requirements are constraints on the product. E.g, Security, reliability, user friendliness, platform, …

21. A software bug occurs when at least one of these rules is true The software does not do something that the specification says it should do. The software does something that the specification says it should not do. The software does something that the specification does not mention. The software does not do something that the product specification does not mention but should. The software is difficult to understand, hard to use, slow …

22. Most bugs are not because of mistakes in the code … Specification (~= 55%) Design (~= 25%) Code (~= 15%) Other (~= 5%)

23. Relative cost of bugs “bugs found later cost more to fix” Cost to fix a bug increases exponentially (10 x ) –i.e., it increases tenfold as time increases E.g., a bug found during specification costs $1 to fix. … if found in design cost is $10 … if found in code cost is $100 … if found in released software cost is $1000

24. Bug Free Software Software is in the news for the wrong reason –Security breach, Mars Lander lost, hackers getting credit card information, etc. Why can’t software engineers develop software that just works? –As software gets more features and supports more platforms it becomes increasingly difficult to make it create bug-free.

25. Goal of a software tester … to find bugs … as early in the software development processes as possible … and make sure they get fixed.

26. Customer requirements The software development team must determine what the customer wants. How can you do this? –Guess? –Collect detailed information from surveys? –Get feedback from a previous version of the software? –Read reviews in magazines? –Get information about the competition? –Other ways? The collected data is used to guide the specification effort.

27. Specification “If you don't know where you're going any road will take you there” The specification takes the data from the customer requirements and other sources and defines: –The features of the software (functional requirements). –The constraints on these features (non-functional requirements). Specifications can be: –formal (e.g., aerospace industry), rigid –informal (e.g., a.com start up), on a cocktail napkin or a whiteboard.

28. Schedules The goals of scheduling are to know: –What work needs to be completed? –How much work is left to do? –When will the work be finished? –Who will finish each task? –Other measurable queries. A Gantt chart is a popular type of bar chart that illustrates a project schedule.

29. Gantt Chart

30. Design Before coding begins on non-trivial software projects, a set of design documents are created to serve as blueprints. –Software Architecture –Data flow diagram –State transition diagram –Flowchart –Commented code

31. Source code … of course The ultimate specification of the software! ‘Code is king’ philosophy is still prevalent. Many programming languages and tools to choose from.

32. Test documents Test plan –Quality objectives, resource needs, schedules, assignments, methods, etc. Test cases –Inputs and expected outputs. Bug reports –E.g., the Bugzilla web-based bug tracker. Test tools and automation Metrics, statistics, and summaries –Number of unresolved bugs, mean time to repair a bug, etc.

33. Software Project Staff Project managers –Write product specification, manage the schedule, critical decision tradeoffs Software architects, system engineers –Design the software, work closely with programmers Programmers, developers, coders –Write code, fix bugs Testers, quality assurance staff –Find bugs, document bugs, track progress of open bugs Technical writers –Write manuals, on line documentation Configuration managers, builders –Packaging and code, documents, and specifications

34. Software Development Lifecycles Code and Fix Waterfall Spiral

35. Assuring that a software system meets a user’s needs Verification: “Are we building the product right” –The software should conform to its specification. Validation: “Are we building the right product” –The software should do what the user really requires. Verification vs Validation

36. Is a whole life-cycle process - V & V must be applied at each stage in the software process. Has two principal objectives: –The discovery of defects in a system. –The assessment of whether or not the system is usable in an operational situation. The V & V Process

37. Dynamic V & V Concerned with exercising and observing product behavior (testing). Static verification Concerned with analysis of the static system representation to discover problems. Dynamic and Static Verification

38. Static and Dynamic V&V

40. Example

41. Testing – Terminology Test case –Individual test Test suite –Collection of test cases Test harness –Program that executes a series of test cases Test framework –Software that facilitates writing & running tests –Example – JUnit

42. Testing – Terminology Test driver –Program to create environment for running tests –Declares variables, creates objects, assigns values –Executes code and displays results of tests Tester (Quality Assurance) –Person devising and / or performing tests –More effective if 2nd person writes tests Walkthrough –Programmer explains code to 2 nd person

43. Types of Testing Program testing can be used to show the presence of bugs, but never their absence.” Static: Done without actually executing program –Code inspections –Walkthroughs Dynamic: Done by executing program or its parts –Module or unit testing –Integration testing –System testing

44. Who Tests the Software? developer independent tester Understands the system but, will test "gently" and, is driven by "delivery" Must learn about the system, but, will attempt to break it and, is driven by quality

45. Testing Principles All tests should be traceable to customer requirements. Tests should be planned before testing begins. 80% of all errors are in 20% of the code. Testing should begin in the small and progress to the large. Exhaustive testing is not possible. Testing should be conducted by an independent third party if possible.

46. Software Defect Causes Specification may be wrong. Specification may be a physical impossibility. Faulty program design. Program may be incorrect.

47. Types of Errors Algorithmic error. Computation & precision error. Documentation error. Capacity error or boundary error. Timing and coordination error. Performance error. Recovery error. Hardware & system software error. Standards & procedure errors.

48. Test Strategies Black-box or behavioral testing –knowing the specified function a product is to perform and demonstrating correct operation based solely on its specification without regard for its internal logic White-box or glass-box testing –knowing the internal workings of a product, tests are performed to check the workings of all possible logic paths

49. Black-box Testing Focus: I/O behavior. If for any given input, we can predict the output, then the module passes the test. –Almost always impossible to generate all possible inputs ("test cases") Goal: Reduce number of test cases by equivalence partitioning: –Divide input conditions into equivalence classes –Choose test cases for each equivalence class. (Example: If an object is supposed to accept a negative number, testing one negative number is enough)

50. Black-box Testing (Continued) Selection of equivalence classes (No rules, only guidelines): –Input is valid across range of values. Select test cases from 3 equivalence classes: Below the range Within the range Above the range –Input is valid if it is from a discrete set. Select test cases from 2 equivalence classes: Valid discrete value Invalid discrete value Another solution to select only a limited amount of test cases: –Get knowledge about the inner workings of the unit being tested => white-box testing

51. White-box Testing Focus: Thoroughness (Coverage). Every statement in the component is executed at least once. Four types of white-box testing –Statement Testing –Loop Testing –Path Testing –Branch Testing

52. White-box Testing (Continued) Statement Testing (Algebraic Testing): Test single statements (Choice of operators in polynomials, etc) Loop Testing: –Cause execution of the loop to be skipped completely. (Exception: Repeat loops) –Loop to be executed exactly once –Loop to be executed more than once Path testing: –Make sure all paths in the program are executed Branch Testing (Conditional Testing): Make sure that each possible outcome from a condition is tested at least once

53. Unit testing –testing of individual components Module testing –testing of collections of dependent components Sub-system testing –testing collections of modules integrated into sub-systems System testing –testing the complete system prior to delivery Acceptance testing –testing by users to check that the system satisfies requirements. Sometimes called alpha testing Testing Stages

54. The Testing Process

55. Unit Testing Program reviews. Formal verification. Testing the program itself. –black box and white box testing.

56. Types of Testing Unit Testing: –Individual subsystem –Carried out by developers –Goal: Confirm that subsystems is correctly coded and carries out the intended functionality Integration Testing: –Groups of subsystems (collection of classes) and eventually the entire system –Carried out by developers –Goal: Test the interface among the subsystem

57. System Testing System Testing: –The entire system –Carried out by developers –Goal: Determine if the system meets the requirements (functional and global) Acceptance Testing: –Evaluates the system delivered by developers –Carried out by the client. May involve executing typical transactions on site on a trial basis –Goal: Demonstrate that the system meets customer requirements and is ready to use Implementation (Coding) and testing go hand in hand

58. Eaxample:Test Cases Test case 1 : ? (To execute loop exactly once) Test case 2 : ? (To skip loop body) Test case 3: ?,? (to execute loop more than once) These 3 test cases cover all control flow paths

59. Generating Test Data Ideally want to test every permutation of valid and invalid inputs Equivalence partitioning it often required to reduce to infinite test case sets –Every possible input belongs to one of the equivalence classes. –No input belongs to more than one class. –Each point is representative of class.

60. Regression Testing Check for defects propagated to other modules by changes made to existing program –Representative sample of existing test cases is used to exercise all software functions. –Additional test cases focusing software functions likely to be affected by the change. –Tests cases that focus on the changed software components.

61. Integration Testing Bottom - up testing (test harness). Top - down testing (stubs). Modified top - down testing - test levels independently. Big Bang. Sandwich testing.

62. Top-Down Integration Testing Main program used as a test driver and stubs are substitutes for components directly subordinate to it. Subordinate stubs are replaced one at a time with real components (following the depth-first or breadth-first approach). Tests are conducted as each component is integrated. On completion of each set of tests and other stub is replaced with a real component. Regression testing may be used to ensure that new errors not introduced.

63. Bottom-Up Integration Testing Low level components are combined in clusters that perform a specific software function. A driver (control program) is written to coordinate test case input and output. The cluster is tested. Drivers are removed and clusters are combined moving upward in the program structure.

64. Acceptance Testing Making sure the software works correctly for intended user in his or her normal work environment. Alpha test –version of the complete software is tested by customer under the supervision of the developer at the developer’s site Beta test –version of the complete software is tested by customer at his or her own site without the developer being present

65. System Testing Recovery testing –checks system’s ability to recover from failures Security testing –verifies that system protection mechanism prevents improper penetration or data alteration Stress testing –program is checked to see how well it deals with abnormal resource demands Performance testing –tests the run-time performance of software

66. Performance Testing Stress test. Volume test. Configuration test (hardware & software). Compatibility. Regression tests. Security tests. Timing tests. Environmental tests. Quality tests. Recovery tests. Maintenance tests. Documentation tests. Human factors tests.

67. Testing Life Cycle Establish test objectives. Design criteria (review criteria). –Correct. –Feasible. –Coverage. –Demonstrate functionality. Writing test cases. Testing test cases. Execute test cases. Evaluate test results.

68. Testing Tools Simulators. Monitors. Analyzers. Test data generators.

69. Test Team Members Professional testers. Analysts. System designers. Configuration management specialists. Users.

70. Test Documentation Needed Requirement being tested. Design verification methodology. Code verification methodology.

71. Document Each Test Case Requirement tested. Facet / feature / path tested. Person & date. Tools & code needed. Test data & instructions. Expected results. Actual test results & analysis Correction, schedule, and signoff.

72. Test Case Design "Bugs lurk in corners and congregate at boundaries..." Boris Beizer OBJECTIVE CRITERIA CONSTRAINT to uncover errors in a complete manner with a minimum of effort and time

73. Debugging Debugging (removal of a defect) occurs as a consequence of successful testing. Some people better at debugging than others. Is the cause of the bug reproduced in another part of the program? What “next bug” might be introduced by the fix that is being proposed? What could have been done to prevent this bug in the first place?

74. The Debugging Process

75. Debugging Approaches Brute force –memory dumps and run-time traces are examined for clues to error causes Backtracking –source code is examined by looking backwards from symptom to potential causes of errors Cause elimination –uses binary partitioning to reduce the number of locations potential where errors can exist

76. End of lecture