1. IBM Haifa Research Lab IBM Haifa Labs An Open Source Simulation Model of Software Testing Shmuel Ur Elad Yom-Tov Paul Wernick http://qp.research.ibm.com/concurrency_testing

2. IBM Haifa Labs 2 What Are the Development Tasks?  Writing Code Creating lines of code, creating bugs  Testing Creating tests, running tests  Reviews Searching the code for bugs  Debugging Looking for the bug root cause  Fixing Replacing some code that has a bug  Putting new bugs in?  We will ignore specification, design, maintenance

3. IBM Haifa Labs 3 Simulation Program Development  A program of certain size with certain quality need to be created Size is measured in LOC Quality is measured in # of bugs  A developer may spend time doing of the development tasks, The task may result in artifacts being created Writing code create new line objects, one for each line, which may contain bugs Writing tests, results in tests each of which can test a subset of the line objects. Automated tests are more expensive. Running a test takes some time (depends if automated) and may find bugs in specific line objects. Running a test a second time is not as effective …

4. IBM Haifa Labs 4 Default Simulation Values  # Code lines per hour = 30  Bugs per line = 0.01  Debugging time Correction = 1 + (LOC since bug)/2000  Hours to fix bug = base 2, multiply by 2 if a month passed, multiply by 3 if two months passed  Probability discovered by inspection = 0.5 : Laitenberger and DeBaud suggest that 70% is achievable  # Lines reviewed per hour = 200  Cost of testing = 1.14 hour per one hundred lines to do unit testing Other measures exist, cost of creating tests, cost of running tests  Probability of finding bugs in test =.5 Probability of finding bug when re-running the test = 0

5. IBM Haifa Labs 5 Simulation Cycle Write Test Write code Run Test Check Size (b) Code building cycle: Test First Done Debug Test /inspect Debug Check Quality (c) System test Done Write code Test / Inspect Debug Check Size (a) Code building cycle : iterative Done

6. IBM Haifa Labs 6 Sample Simulation

7. IBM Haifa Labs 7 Sample Simulation

8. IBM Haifa Labs 8 Overnight Regression – Chaotic behavior

9. IBM Haifa Labs 9 Overall Productivity when Code Creation Rate Increase

10. IBM Haifa Labs 10 What do Developers Really Do?

11. IBM Haifa Labs 11 Impact of Length of Iteration in Iterative Software Development

12. IBM Haifa Labs 12 Is Pair Programming a Good Idea? Depends!!

13. IBM Haifa Labs 13 Automating Testing May be Overrated

14. IBM Haifa Labs 14 Future Direction  Encapsulation Reduce cost of debugging, review …  Code complexity  Planning tool Quality requirement impact on methodology Not one size fit all