Presentation is loading. Please wait.

Presentation is loading. Please wait.

Property-Based Testing A Silver Bullet? John Hughes December 2009.

Similar presentations


Presentation on theme: "Property-Based Testing A Silver Bullet? John Hughes December 2009."— Presentation transcript:

1 Property-Based Testing A Silver Bullet? John Hughes December 2009

2 Software testing: most famous quote Program testing can be used to show the presence of bugs, but never to show their absence! – E.W.Dijkstra

3 $60 billion

4 $240 billion

5 50%

6 Money spent on testing Cost of remaining errors

7 Testing in Practice? Human effort? Test automation

8 Large-Scale Test Automation Nightly runs provide rapid feedback New test cases added for each error found Test Server Software under test Automated test cases Report of test case failures 1,5MLOC Erlang, 2MLOC C++ 700KLOC Erlang

9 Typical Large Projects Design team Test team

10 Bug Detection Rate

11 Developer Testing Why wait until system testing to use test automation? – Why not automate developers own testing? Unit testingone module in isolation – A key element of agile development methods such as XP

12 Claims for Unit Testing Immediate discovery of errors – bug fixing is cheap! Confidence in refactoring – cleaner code! TDD: write tests first, then just enough code to make them pass – KISS! No wasted effort! Tests serve as a specification – So keep test code clean and elegant! – Not too many… one test for each thing!

13 TDD with HUnit in Haskell Problem: implement a key-value store -- Type signatures empty :: Store k v store :: Ord k => k -> v -> Store k v -> Store k v find :: Ord k => k -> Store k v -> Maybe v remove :: Ord k => k -> Store k v -> Store k v

14 Step 1: Tests for find testFindEmpty = "find empty" ~: find 1 (Nothing :: Maybe Int) A test case is a definition Attach a name to a test case An assertion where left side is unknown, right side is expected value testFind1 = "find with one element" ~: find 1 (store 1 2 Just 2 testFind2 = "find with two elements" ~: do let s = store 1 2 (store 3 4 empty) find 1 Just 2 find 3 Just 4 find 5 Nothing Can combine several assertions and IO actions in one test case

15 Hunit Glue import Test.HUnit main = runTestTT findTests findTests = "find tests" ~: [testFindEmpty, testFind1, testFind2]

16 Step 2: Run the tests import Test.HUnit main = runTestTT findTests findTests = "find tests" ~: [testFindEmpty, testFind1, testFind2] data Store k v = Store find = undefined store = undefined remove = undefined empty = undefined *Main> main ### Error in: find tests:0:find empty Prelude.undefined ### Error in: find tests:1:find with one element Prelude.undefined ### Error in: find tests:2:find with two elements Prelude.undefined Cases: 3 Tried: 3 Errors: 3 Failures: 0 Counts {cases = 3, tried = 3, errors = 3, failures = 0} *Main> A message from each failing test A summary of the test results

18 Step 4: Repeat the tests *Main> main ### Failure in: find tests:2:find with two elements expected: Just 2 but got: Nothing Cases: 3 Tried: 3 Errors: 0 Failures: 1 Counts {cases = 3, tried = 3, errors = 0, failures = 1} testFind2 = "find with two elements" ~: do let s = store 1 2 (store 3 4 empty) find 1 Just 2 find 3 Just 4 find 5 Nothing

19 Step 5: Debug the code store k v Nil = Node k v Nil Nil store k v (Node k' v' l r) | k'<= k = Node k' v' (store k v l) r | k' > k = Node k' v' l (store k v r) k <=k' k > k'

20 Step 6: Rerun the tests All the tests passnow we write more tests! *Main> main Cases: 3 Tried: 3 Errors: 0 Failures: 0 Counts {cases = 3, tried = 3, errors = 0, failures = 0}

21 Next Iteration: tests for remove removeTests = "remove tests" ~: [testRemoveEmpty, testRemove1, testRemove2] testRemoveEmpty = "remove empty" ~: remove 1 (empty :: Store Int Int) testRemove1 = "remove with one element" ~: remove 1 (store 1 2 empty testRemove2 = "remove with two elements" ~: do let s = store 1 2 (store 3 4 empty) remove 1 store 3 4 empty remove 3 store 1 2 empty remove 5 s

22 Run the tests main = runTestTT allTests allTests = "all tests" ~: [findTests, removeTests] *Main> main ### Error in: all tests:1:remove tests:0:remove empty Prelude.undefined ### Error in: all tests:1:remove tests:1:remove with one element Prelude.undefined ### Error in: all tests:1:remove tests:2:remove with two elements Prelude.undefined Cases: 6 Tried: 6 Errors: 3 Failures: 0 Counts {cases = 6, tried = 6, errors = 3, failures = 0}

23 Implementation of remove k,v nk,nv

24 Code for remove remove k Nil = Nil remove k (Node k' v l r) | k == k' = case r of Nil -> l _ -> let (nk,nv) = leftmost r in Node nk nv l (remove nk r) | k < k' = Node k' v (remove k l) r | k > k' = Node k' v l (remove k r) leftmost (Node k v Nil _) = (k,v) leftmost (Node _ _ l _) = leftmost l

25 Last step: rerun the tests No failures, so were done! *Main> main Cases: 6 Tried: 6 Errors: 0 Failures: 0 Counts {cases = 6, tried = 6, errors = 0, failures = 0} …or are we???

26 Test Coverage All tests passbut how good are our tests? Source code coverage tools tell us how much code we tested When tests pass, check coverage!

27 Using Haskell Program Coverage C:\Users\John Hughes\Desktop> ghc -fhpc Store.hs -- make C:\Users\John Hughes\Desktop> Store.exe Cases: 6 Tried: 6 Errors: 0 Failures: 0 C:\Users\John Hughes\Desktop> hpc markup Store.exe Writing: Main.hs.html…

28 Marked-up source code Conditions which were always true Code which was never executed!

29 Just… one… more… test… testRemoveNonEmptyRightBranch = "remove with non-empty right branch" ~: remove 1 (store 3 4 (store 1 2 store 3 4 empty

30 But… This last test has nothing to do with a specification It cannot be written first Test cases written just to get coverage are often bad test cases Many many tests are neededboring! Does TDD really cut the mustard?

31 Which Unit Tests to Write? You should test things that might break Kent Beck Not too few, not too many Partition the cases into classes with similar behaviour Write one test per partition

32 Example: insertion into an ordered list Partitions: – Empty list/non-empty list – Insert at beginning/middle/end Test boundary values and middle values – Element already present/not present

33 Partition tests insertNonEmpty covered by other cases insertAbsent covered by other cases Note: expected values play a major rôle! insertEmpty = "insert empty" ~: insert 1 [1] insertStart = "insert start" ~: insert 1 [1,2,4] insertMid = "insert mid" ~: insert 3 [2,3,4] insertEnd = "insert end" ~: insert 5 [2,4,5] insertPresent = "insert present" ~: insert 1 [1,1]

34 Sum or Product of Partitions? Given several ways to partition inputs, should we – Write one test for each partition? – Write one test for each combination of partitions? E.g. Non-empty/Beginning/Present, Non- empty/Beginning/Absent, … – (Can be smart and cover all pairs of partitions, or all triples…)

35 Property Based Testing Generate test cases instead of inventing them – Automate the boring bit! – Reduce size of test code Focus on properties true in all cases, not single tests – A true specification Minimize failing test cases to speed debugging

36 Generating Stores How to generate, how to shrink instance (Ord k, Arbitrary k, Arbitrary v) => Arbitrary (Store k v) where arbitrary = do (k,v,s) <- arbitrary elements [empty, store k v s, remove k s] shrink Nil = [] shrink (Node k v l r) = [l,r] ++ [Node k v l' r | l' <- shrink l] ++ [Node k v l r' | r' <- shrink r] ++ [Node k v' l r | v' <- shrink v]

37 Model-based testing What does a store represent? – A set of key-value pairs! model s = List.sort (contents s) contents Nil = [] contents (Node k v l r) = (k,v):contents l ++ contents r Sorted so we can compare them with ==

38 Properties: Agreement with the model prop_find k s = find k s == lookup k (model s) where types = s :: Store Int Int prop_store k v s = model (store k v s) == List.insert (k,v) (model s) where types = s :: Store Int Int prop_remove k s = case find k s of Just v -> model (remove k s) == model s List.\\ [(k,v)] Nothing -> remove k s == s where types = s :: Store Int Int

39 Testing the Properties We forgot to consider duplicate keys! *Main> quickCheckWith stdArgs{maxSuccess=10000} prop_find *** Failed! Falsifiable (after 95 tests and 1 shrink): 1 Node 1 1 (Node 1 (-1) Nil Nil) Nil prop_find k s = case [v | (k',v) <- model s, k==k'] of [] -> find k s == Nothing vs -> find k s `elem` map Just vs where types = s :: Store Int Int

40 Testing remove Were not removing the duplicate key…??? *Main> quickCheckWith stdArgs{maxSuccess=10000} prop_remove +++ OK, passed tests. *Main> quickCheckWith stdArgs{maxSuccess=10000} prop_remove *** Failed! Falsifiable (after 2 tests): 0 Node 0 1 Nil (Node 1 1 (Node 1 0 Nil Nil) Nil) *Main> let s = Node 0 1 Nil (Node 1 1 (Node 1 0 Nil Nil) Nil) *Main> find 0 s Just 1 *Main> remove 0 s Node 1 0 Nil (Node 1 0 Nil Nil)

41 The Bug remove k Nil = Nil remove k (Node k' v l r) | k == k' = case r of Nil -> l _ -> let (nk,nv) = leftmost r in Node nk nv l (remove nk r) | k < k' = Node k' v (remove k l) r | k > k' = Node k' v l (remove k r) Removes nk with the wrong value

42 How are we doing for coverage?

43 Hunit vs QuickCheck HunitQuickCheck Bugs found after TDD01 Source code coverage95%100% Lines of test code3526 QuickCheck – Finds more bugs – With better coverage – In less time – With less test code – And a clearer specification

44 OK 3G Radio Base Station Setup OK Reject

45 Media Proxy Multimedia IP-telephony (IMS) Connects calls across a firewall Test adding and removing callers from a call Add Sub Add Sub Add Sub Call Full

46 Property Based Testing is Great! Improves quality! – Finds more bugs, achieves better coverage Reduces cost! – Less test code, shrinking speeds diagnosis And its actually fun! – Please can I write some tests today?

47 How do we know? Case studies in industry +Real software development + Professional software developers - Unrepeatable - Difficult to control Experiments in universities + Focus on a single question + Carefully controlled – Student volunteers – Unrealistically small

48 Test Driven Development Case Studies – YES, quality is improved – NO, cost is not reduced (costs rise about 20%) Experiments – YES, code is developed faster – NO, quality is not improved (it drops)

49 Property Based Testing Case studies + Property-based testing does increase quality + Property-based testing does reduce cost – PBT during system testing actually reduces the quality of conventional unit testing done earlier

50 Our Experiment Hypothesis: – Property-based testing is more effective than conventional unit testing Effective? – Quality (better quality in the same time) number of bugs, number of tests failed, subjective judgement – Test quality code coverage, subjective judgement – Design quality Size of code, size of test code, subjective judgement

51 Isnt it obvious? Well… – Property-based testing requires writing generators… additional work! – Property-based testing requires formalizing the specification, unit testing only needs examples – Property-based tests are selected at random, unit tests are carefully crafted… better at provoking errors?

52 A comparison of Hunit and QuickCheck – Solve two problems, one with Hunit, one with QuickCheck Time: 13:1516:00, Wednesday 8 December Place: lab rooms 3507, 3354 and 3358 Info: sign up at experiment experiment Next week, in a lab room near you…

53 Exercises Get the HUnit tests for insert running Add test cases to cover all combinations of partitions Write and test functions to extract the base name and extension from a file name: – baseName Foo.hs == Foo – extension Foo.hs == hs


Download ppt "Property-Based Testing A Silver Bullet? John Hughes December 2009."

Similar presentations


Ads by Google