1. Introduction to Software Testing Maili Markvardt

2. We will learn… What is (software) testing? What is the role of a tester? What are the testing main principles? Why, what, how (can be tested)?

3. What is (software) testing?

4. Testing? A myriad of definitions Checking whether or not “something” conforms to requirements Executing a program with the aim to find bugs (defects in the program) Checking whether or not the program behaves expectedly

5. Testing – ISTQB definition The process consisting of all life cycle activities, concerned with planning, preparation and evaluation of software products and related work products to determine: that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects

6. It is possible to test… everything Requirements can be defined for every single outcome (artifact) in software development Everything can be tested: Analysis documentation (must describe customer’s needs, must be sufficient for further development) Design documentation (must conform to analysis, must be efficient and effective) Installation, maintenance guides (must be sufficient) The software (code) itself (stable, functional) The same software running on different configurations, different platforms (stable, functional)

7. Testing in SW development process Different timing in different processes (agile methods, V-model, RUP) Same Input Artifact to be tested (program, document) Requirements (specification) Same output Test plans, test cases Test reports, bug reports/descriptions

8. Lessons learned from poor testing Therac (1985 – 87) – 6 lives lost Main causes for failure Insufficient check of preconditions Insufficient exception handling No 3rd-party quality control/testing involved (just developer’s testing) Poor documentation (especially what concerns alternative behaviour (exceptions))

9. Lessons learned from poor testing (2) Mars satellite (1999) – 125Mil USD loss Measurement system not agreed – poor quality of requirements Institute of Standardisation of USA survey: Losses caused by software bugs for US economy: ~60billion USD a year Can be reduced by 1/3 employing better testing!

10. Discussion 1 What requirements can be defined for the following software development outcomes (articats)? Project plan Analysis documentation Program code Test case documentation Automated tests (test scripts) How would you test the above-mentioned artifacts?

11. The role of a Tester

12. A tester “serves” many clients Project manager Software developer Technical writer Techincal support Marketing Management End users

13. More about tester’s role Since a tester focuses to defects his clients can focus to success Find important defects fast Evaluate the product tested Use certain methods and tools Educate your clients about testing Educate yourself Do your best to make your clients happy, but face the fact and let others know – all bugs can not be found!

14. Discussion 2 Your experiences on interacting with testers Based your experience, describe Information You -> Tester Information Tester -> You Any problems?

15. Testing main principles

16. Testing Principles (1) Testing can demonstrate only the presence of defects and not their absence Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness. Exhaustive testing is impossible Exhaustive testing (all combinations of inputs and preconditions) is not feasible except for trivial cases. Instead of exhaustive testing, risk analysis and priorities should be used to focus testing efforts.

17. Testing Principles (2) Early testing is important Testing activities should start as early as possible in the software or system development life cycle and should be focused on defined objectives. Defects are clustering A small number of modules contain most of the defects discovered during pre-release testing, or are responsible for the most operational failures.

18. Testing Principles (3) Pesticide paradox If the same tests are repeated over and over again, eventually the same set of test cases will no longer find any new defects. To overcome this “pesticide paradox”, the test cases need to be regularly reviewed and revised, and new and different tests need to be written to exercise different parts of the software or system to potentially find more defects. Testing is context dependent Testing is done differently in different contexts. For example, safety-critical software is tested differently from an e-commerce site.

19. Testing Principles (4) Absence-of-errors fallacy Finding and fixing defects does not help if the system built is unusable and does not fulfill the users’ needs and expectations.

20. Types of testing

21. Types of Testing Several (overlapping) classifications of testing Who is testing? What is tested? How it is tested?

22. Quality Attributes of ISO/IEC9126 Functional testing Everything concerning activities performed with the software Non-functional testing Reliability Recoverability Usability Efficiency -> Performance, Load, Stress testing Portability, Installability, Scalability

23. Granularity of the system Module/unit testing Integration testing System testing

24. Who is testing? Alpha and Beta testing Acceptance testing In-house testing (eat-your-own-dogfood) Subject-matter expert testing …

25. How it is tested? Static vs dynamic testing Manual vs automated testing Black box vs white box testing Regression testing Smoke testing Exploratory testing Guerilla testing Paired testing Ad-hoc testing …

26. What data is used? Equivalence partitions (input and output) Coverage criteria Random testing Statistical testing Function testing Menu tour