1. 1/38 2013 REU Program at ECU Software Testing - Foundations, Tools, and Applications Lecture 1 May 21, 2013 Introduction to Software Testing Dr. Sergiy Vilkomir

2. 2/38 Dr. Sergiy Vilkomir Instructor’s office: S & T Building, C-111 Email: vilkomirs@ecu.edu http://www.cs.ecu.edu/reu/index.html http://core.ecu.edu/vilkomirs http://core.ecu.edu/STRG/

3. 3/38 Software life cycle concept requirements design implementation (coding) testing installation operation and maintenance testing Is testing important ?

4. 4/38 Software Errors Software errors can be found and eliminated during software testing “Software bugs, or errors, are so prevalent and so detrimental that they cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 percent of the gross domestic product, according to a newly released study commissioned by the Department of Commerce's National Institute of Standards and Technology (NIST)”. (NIST News, June 38, 2002 )

5. 5/38 Software Errors Buggy Software May Have Crashed Mars Polar Lander (March 2000) The software problem crashed the Mars Polar Lander into the Red Planet’s frozen ground The most likely cause of the lander’s failure, investigators decided, was that a spurious sensor signal associated with the craft’s legs falsely indicated that the craft had touched down when in fact it was some 130-feet (40 meters) above the surface. This caused the descent engines to shut down prematurely and the lander to free fall out of the martian sky. The error was traced to a single bad line of software code.

6. Causes of software defects Software Bugs with Fatal Consequences 1985 - 1986 few patients were killed in a hospital in USA because the medicine dose was calculated wrongly (software bugs in Therac-25 treatment machine lead to radiation overdoses, approximately 100 times the intended dose). 6/38

7. 7/38

8. Causes of software defects In1996 the Ariane 5 rocket of the European space agency was destroyed because they used the software from Ariane 4 rocket (was self-destroyed 40 seconds after takeoff; more than $370 million was lost ). 8/38

9. Causes of software defects Software Bugs with Fatal Consequences In 1962 NASA lost their 80 Million Dollar "Mariner 1" because of a missing "hyphen" in the program code. 9/38

10. Causes of software defects Software Bugs with Fatal Consequences During the Gulf War in February, 1991, a Patriot missile system operating in Dhahran, Saudi Arabia, failed to track and intercept an incoming Scud. The Iraqi missile impacted into an army barracks, killing 38 U.S. soldiers and injuring another 98. 10/38 The cause of the missile system failing to defend against the incoming Scud was traced back to a bug in Patriot’s radar and tracking software. The bug occurs in the calculation of the next location of the incoming target. That modified software reached Dhahran the day after the attack.

11. Computers and Software Errors 11/38 On Tuesday, June 3, 1980, at 1:26 a.m., the display system at the command post of the Strategic Air Command (SAC) near Omaha, Nebraska, indicated that two submarine-launched ballistic missiles (SLBMs) were headed toward the United States. The SAC duty controller directed all alert crews to move to their B- 52 bombers and to start their engines Land-based missile crews were put on a higher state of alert, and battle-control aircraft prepared for flight Fortunately, there were a number of factors which made those involved in the assessment doubt that an actual attack was underway The cause of these incidents was eventually traced to the failure of a single integrated circuit chip in a computer which was part of a communication system

12. Causes of software defects Software Bugs with Fatal Consequences Software bugs in a Soviet early-warning monitoring system nearly brought on nuclear war in 1983. The software was supposed to filter out false missile detections caused by Soviet satellites picking up sunlight reflections off cloud-tops, but failed to do so. Disaster was averted when a Soviet commander, based on what he said was a '...funny feeling in my gut', decided the apparent missile attack was a false alarm. The filtering software code was rewritten. 12/38

13. Software Errors 13/38 Investigation found that the satellite in question had picked up the sun's reflection off the cloud tops and somehow interpreted that as a missile launch.

14. CNN Money - August 9, 2012 In less than an hour, Knight Capital's computers executed a series of automatic orders that were supposed to be spread out over a period of days. Millions of shares changed hands. The resulting loss, which was nearly four times the company's 2011 profit, crippled the firm and brought it to the edge of bankruptcy. 14/38 Computer glitch that set fire to $440 million of Knight Capital Group's funds

15. Top Software Failures Of 2011 (from http://www.businesscomputingworld.co.uk) 1.Financial services giant fined $25 million for hiding software glitch that cost investors $217 million 2.Computer system bugs cause Asian banking facilities’ downtime 3.Cash machine bug benefits customers by giving them extra money 4.22 people wrongly arrested in Australia due to failures in new NZ $54.5 million courts computer system 5.38,380 cars recalled after airbag-related software glitch 15/38

16. Top Software Failures Of 2011 (from http://www.businesscomputingworld.co.uk) Conclusions Software failures are costing companies and consumers large amounts of money The worst software failures have damaged reputations, impacted negatively on financials and caused stress to users. Each of the top 2011 software failure examples could easily have been avoided through an effective quality management and proper software testing 16/38

17. 17/38 Software testing Software errors can be found during software testing Testing can help but it is not for free The cost of software testing could be around 40% of the overall costs of software development

18. 18/38 Terminology Software testing Error Fault Failure Input domain Oracle Regression testing Unit testing Functional testing Verification Validation V&V

19. 19/38 Terminology Unfortunately, SE terminology in textbooks is often contradictory Two different textbooks could provide two different definitions for the same term What to do in this situation? Solution: use terminology from standards

20. 20/38 IEEE (Institute of Electrical and Electronics Engineers) standard IEEE Std 610.12 1990 IEEE Standard Glossary of Software Engineering Terminology

21. 21/38 New standard IEEE Std 610.12 1990 has not been updated for many years Some terms are obsolete Instead: ISO/IEC/IEEE 24765:2010 “Systems and software engineering – Vocabulary”

22. 22/38  S. Vilkomir, 2010 From IEEE Std 610.12 1990 testing - the process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component Contrast with: Static Analysis, Inspection

23. 23/38  S. Vilkomir, 2010 From IEEE Std 610.12 1990 Static Analysis - the process of evaluating a system or component based on its form, structure, content, or documentation. Inspection - a static analysis technique that relies on visual examination of development products to detect errors, violations of development standards, and other problems.

24. 24/38  S. Vilkomir, 2010 From IEEE Std 610.12 1990 Fault - an incorrect step, process, or data definition in a computer program. Note: In common usage, the terms "error" and "bug" are used to express this meaning. Failure - the inability of a system or component to perform its required functions within specified performance requirements. The result of the fault.

25. 25/38  S. Vilkomir, 2010 From IEEE Std 610.12 1990 test case - a set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.

26. 26/38  S. Vilkomir, 2010 Input domain – the set of all possible inputs to a program test oracle - a source to determine expected results to compare with the actual result of the software under test. An oracle may be the existing system (for a benchmark), a user manual, or an individual’s specialized knowledge, but should not be the code

27. 27/38 Y= a X + b – it is very easy to determine expected results expected results ?

28. 28 Goal of software testing Test case that did not find an error Test case that discovered a new error - successful Increasing the quality and reliability of the program after removing the error Increasing our confidence on the quality and reliability of the program

29. 29 Goal of software testing Dual Goal To find undiscovered errors AND to get confidence on the quality and reliability of the program

30. 30 Lifecycle phase in which testing takes place © Aditya P. Mathur 2005

31. 31 From IEEE Std 610.12 1990 black-box testing -Testing that ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions. Syn:. functional testing Contrast with: structural testing. white-box testing -Testing that takes into account the internal mechanism of a system or component. Types include branch testing, path testing, statement testing. Syn: structural testing, glass-box testing Contrast with: functional testing

32. Is testing always done in the same way? Types of softwareConsequences of defects Program A Program B Program C Program D Safety-critical software Personal software Company’s internal software Commercial software internal problems nothing serious financial losses, damaged reputation loss of human life 32/38

33. Testing is done differently in different contexts. –Type of software –Recourses (constraints): human recourses, financial recourses, time –Regulatory requirements Conclusion: testing is context dependent 33/38

34. Causes of software defects Human mistake (error) Defect (bug, fault) in software Failure during software execution 34/38

35. How much testing is enough? Exhaustive testing is impossible Testing everything (all combinations of inputs and preconditions) is not feasible Example: The exam has 30 questions and four possible answers for each question. What is a number of all possible combinations of answers? If we can test 1000 combinations per second, how long will we test all combinations? (An hour? A day? A month?) Total number of combinations: 4  4  4  ….  4= 4 30  1.000.000.000.000.000.000 30 times 1 year  38.000.000 sec. We need  38.000.000 years for testing 35/38

36. How much testing is enough? Not possible to say in general Depends on each specific situation Pressures on a project include time and budget Risk assessment to decide how much testing to do. Important: prioritization (do the most important tests first) setting criteria when to stop testing Take into account technical and business risk Take into account project constraints. 36/38

37. Testing principles Testing shows presence of defects Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness 37/38 “Program testing can be used to show the presence of bugs, but never to show their absence!” 1970, famous quote of ?

38. Testing principles Testing shows presence of defects 38/38 “Program testing can be used to show the presence of bugs, but never to show their absence!” 1970, famous quote of Dijkstra Edsger Dijkstra (1930 – 2002) was a Dutch computer scientist. Winner of the 1972 Turing Award; creator of "structured programming."