Black box (Finite State Machine) testing Design for testability Coverage measures Random testing Constraint-based testing Debugging and test case minimization Using model checkers for testing Coverage revisited (“small model property”) Topics in Testing We’ll Cover

Testing Project Start with: interface and some definitions You – Implement (w ith intentional bugs) – Provide some (possibly VERY minimal) unit tests – Exchange programs with two other students – Submit a test plan for colleagues’ code – Present & defend test plan (bonus points) – Build a tester for your colleagues’ code Must include some random testing or SPIN testing by final checkin – Test! (and submit bug tickets) – Submit bug fixes, evaluations, mini-reports at milestones – Turn in final test report

Testing Project Your submitted code should contain two “subtle” bugs – Make sure your unit tests do not find the bug – Mail me: test cases and descriptions for the bugs Subject MUST be: CS362W11 BUGS [onidid] – You get to debug programs in lots of classes and in real life – in this class you get to bug a program intentionally, for once in your life I will see which testers find which bugs (over a sample of submitted programs)

Testing Project Grading criteria – Design/implementation project/tester – Effectiveness of the tester – Presentation of test plan (extra credit) – Quality of test/bug reports Can I figure out how you tested the system? Can I figure out what wasn’t tested? – Coverage, plus information on input restrictions Can I figure out how reliable you think the code is? What are key strengths/weaknessess? – How “interesting” and hard-to-find your intentional bugs are

Testing Project Expectations You can program in C You can figure out possibly poorly written specifications (a key software engineering skill) Or know when to ask someone who can! You can (learn to) use makefiles / build system “he who learns to play the harp learns to play by playing it” - Aristotle, Metaphysics, Book IX

Testing Project Oh, right. What are we implementing and testing? A card game Well defined notion of correctness “Specifications” not intended for implementation, but intended to be unambiguous Games are a good example of formal specifications: no one wants to have to “make up” the rule or interpret ambiguity in the middle of playing a game

Testing Project What are we implementing and testing? Dominion (+ expansions) Card game published by Rio Grande games Players start with their own draw decks Buy cards to add to decks (using cards) Some cards worth points if in deck Play this cards & you can draw a card and gain two actions (playing a card is an action)

Schedule Jan 23: First code checkin – Initial dominion implementation, expected to perhaps be buggy/incomplete; not frozen Feb 6: Test plan due Feb 22: First test report (brief) due Mar 7: Second test report (brief) due Mar 14: “Late Midterm” Mar 19: FINAL TEST REPORT DUE, PROJECT COMPLETED, FINAL CODE FREEZE, TESTER CODE DUE

Grading & Other Admin Stuff Project: 75% Late midterm: 15% (Possibly-in-class) quizzes/exercises/homework: 10% My office hours: Fri, 11-12:30 KEC 3067 TAs: Amin Alipour, Anirban Roy office hours Fri 10:00- 11:00, KEC Atrium office hours Wed 11:30-1pm, KEC Atrium