1. High Volume Automated Testing for Software Components Harry RobinsonDoug Szabo Microsoft

2. High Volume Test Automation The essence of HVTA techniques is automated execution and evaluation of large numbers of tests, for the purpose of exposing functional errors that are otherwise hard to find. Experiments with High Volume Test Automation, 2004 McGee & Kaner

3. Subtle Bugs Only weak bugs have a logic to them and are amenable to exposure by strictly logical means. Subtle bugs have no definable pattern - they are wild cards. Software Testing Techniques 2 nd Edition, 1990 Beizer

4. Bounded Exhaustive Testing Bounded exhaustive testing (BET) is a verification technique in which software is automatically tested for all valid inputs up to specified size bounds. Software assurance by bounded exhaustive testing, 2005 Coppit et al

5. Typical Biases Developers – “happy paths” Testers – “a nose for bugs”

6. CodeProject: Sorting Algorithms in C#

7. The Sorting Algorithms

8. Popular …

9. Recommended …

10. … and it looks cool!

11. … but dark clouds appear on the horizon

12. ...

13. Reasonable Unit Tests

14. … Even Some Randomized Unit Tests

15. Pre-Oracled Data 012345 250314 012345 shuffle sort compare

16. The Loop

17. Shearsort demo #1

18. Shearsort demo #2

19. Shearsort bug

20. Shearsort bug - fixed

21. Heapsort bug

22. Heapsort bug - fixed

23. OddEvenTransport bug

24. OddEvenTransport bug - fixed

25. Distribution of Buggy Array Sizes

26. Bug is not detected at array size 1000

27. Number of Random Arrays to Detect a Bug

28. Generated Unit Tests