1. Introduction to Software Testing Chapter 9.3 Challenges in Testing Software Test Criteria and the Future of Testing Paul Ammann & Jeff Offutt www.introsoftwaretesting.com

2. Introduction to Software Testing (Ch 9.2), www.introsoftwaretesting.com © Ammann & Offutt 2 Chapter 9 Outline 1.Testing for Emergent Properties: Safety and Security 2.Software Testability 3.Test Criteria and the Future of Software Testing Test Criteria and the Future of Software Testing

3. Introduction to Software Testing (Ch 9.2), www.introsoftwaretesting.com © Ammann & Offutt 3 Testing in the 1980s and 1990s Software engineering was very different – Low quality software was normal and expected – The cost of building better software outweighed the benefits Software was smaller – Most software was bundled, shrink-wrapped, contracted – Industry became dominated by one monopolistic vendor About a dozen books – A few broad survey books, one or two on testing, design, requirements, etc In the early 1980s, one semester course was enough to learn most of the knowledge in software engineering Safety critical software was a tiny part of the industry Testing, especially high-end testing, simply was not very important

4. Introduction to Software Testing (Ch 9.2), www.introsoftwaretesting.com © Ammann & Offutt 4 Testing in the 2000s The field has dramatically changed Today’s software market : – is much bigger – is more competitive – has more users – used in more places The web offers a new deployment platform – Very competitive and very available to more users Enterprise applications means bigger programs and more users Embedded software is now ubiquitous … check your pockets ! Paradoxically, free software increases our expectations ! Security has now become essential …

5. Introduction to Software Testing (Ch 9.2), www.introsoftwaretesting.com © Ammann & Offutt 5 Problems Leading to Security Vulnerabilities 1980 : Security was mostly about math – Cryptography 1990 : Security was mostly about protecting the database 2000 : Security was mostly about bullet-proofing the network Today : Most security vulnerabilities are a result of faults in the software Software testing is fast becoming essential to an essential problem in the software industry Software Security

6. Introduction to Software Testing (Ch 9.2), www.introsoftwaretesting.com © Ammann & Offutt 6 Testing Research 1980s : Criteria and algorithms for unit testing 1990s : Test criteria for other technologies and “levels” 2000s : More automation for test criteria and more technologies A major point of this book is that we have enough criteria Test criteria are uniform across different software artifacts – A graph is a graph is a graph … – 36 criteria on 4 structures So are we done with testing research ? – Can I retire now …

7. Introduction to Software Testing (Ch 9.2), www.introsoftwaretesting.com © Ammann & Offutt 7 Testing Research – Are We Done ? We need more work on how to test modern technologies – The ideas in chapter 7 are still developing – OO, web, GUIs, real-time, embedded, … – How to create the structures and how to adapt the criteria ? Which criteria to use when ? – Cost / benefit tradeoffs among criteria are not known Which structure should be used when ? Technology transition to industry – How best to automate the testing research ideas ? – How to insert new testing techniques into the development process ? – How to effectively and efficiently balance research theory with practical needs ? No !

8. Introduction to Software Testing (Ch 9.2), www.introsoftwaretesting.com © Ammann & Offutt 8 Testing Research – Going Forward Automatic test data generation is one of the hardest problems – But also the core of testing – getting values to satisfy criteria – Research started in the 1970s – major advances in the 1980s, 1990s and 2000s – Still very little help from industry-quality tools Testability, reliability, and other related areas are wide open We know a lot about testing new software – much less about regression testing – Which happens to account for most of the effort in industry … Research in software testing is increasing – Currently lots of funding in Europe and Asia – from both government and industry sources – IEEE International Conference on Software Testing, Verification and Validation (ICST) in its first year (2008), had 224 papers submitted Including 65 from industry ! This may be an IEEE Software Engineering record for an “area” conference

9. Introduction to Software Testing (Ch 9.2), www.introsoftwaretesting.com © Ammann & Offutt 9 Summary We are clearly entering a golden age for software testing With more attention from – Industry – Research – Education … Finally we have the knowledge, resources, and motivation to make testing a technical discipline