1. 1 “Embedded” Dominion NO GUI, NO NETWORK, NO NOTHING NO REAL USER INTERFACE You may want to build a limited UI for testing A simple text interface is in the directory Only need to implement cards in the enum If we’re missing state needed for other cards, don’t worry about it Assume “customer” use is for simulation of various strategies, rather than real game play Caller will be other code, and cards with “interactive” decisions are not included

2. 2 “Embedded” Dominion Support 2-4 players (ignore Intrigue’s support for more players) Use cards played to determine what phase someone is in: While playing actions, in Phase A Buying a card indicates moving to B Calling endTurn() ends turn Return -1 if their choice is illegal AND DO NOT CHANGE THE STATE All supply piles are ‘face up’ – only deck is face down

3. 3 “Embedded” Dominion Player 0 is always starting player Over multiple games, we’ll need to reassign player numbers, if following the rules – responsibility of calling code, not our code Means we have enough info to compute tiebreakers… Right? Deck is sorted before shuffle, so as long as you have the right cards in the discard pile (then move them to deck before shuffling) it doesn’t matter what order things go in discard pile (or played card area)

4. 4 Question Time, Inspect the Code

5. 5 How to Test Test Plans Can be text documents or slides Can include a bit on what you’ve already done (getting the thing to compile, very basic bugs, letting developer finish a few more cards…) Should outline your future plans Broad enough that it doesn’t take 40 minutes to present Detailed enough to allow for criticism, applause, response in general by the audience (fellow software developers/management (that’s me)) Tools: will you be using any? You can use any kind of tool, but don’t BUY anything! Contingency plans: if a tool doesn’t work, if test technique X has problems, what is Plan B?

6. 6 How to Test Test Plans Things to include, unless there is an amazingly good reason not to: SCHEDULE TYPES OF TESTING TO BE DONE, e.g.: Random testing Coverage-based approaches TOOLS METRICS TO EVALUATE TESTING, e.g.: Branch, statement coverage FORMAT OF BUG REPORTS RISKS INITIAL EVAL OF TESTED CODE e.g., LOC, quality, does it compile?

7. 7 BUGS BUGS BUGS BUGS BUGS Due date is in admin slides Send to my email, with subject CS362S11 BUGS [onidid] Include, for both bugs: Relevant code snippet What does it do? How do you trigger it? Why do you think it might be tricky to find? Any cool game consequences?

8. 8 Tools gcov http://gcc.gnu.org/onlinedocs/gcc-3.1.1/gcc/Gcov.html/ Shall I show it to you again? I included flags to use it in the makefile Basic minimal tool use; you probably don’t want to plan testing without including use of some source coverage tool Delta-debugging http://www.st.cs.uni-saarland.de/dd/ Minimization for test cases Code is in python, but very easily adapted even if you don’t know python If you want, we can try it out in class today? Also a good basic point for tools; courtesy to developer, and it will please me as well

9. 9 Tools GinkoTest In the repository

10. 10 Tools: Concolic Testing CREST (Successor to CUTE) http://neutrinographics.com/concolic-testing-and-fault- localization-0 http://neutrinographics.com/concolic-testing-and-fault- localization-0 I haven’t really used it myself; you’re somewhat on your own Might very well automatically generate all the tests you’d care to have Splat Looks like when Ru-Gang graduated, at some point UCLA reclaimed his web pages, which had the download link Probably best not to try for it, given that

11. 11 Tools: Runtime Analysis Daikon http://groups.csail.mit.edu/pag/daikon/ Generally up to date I haven’t used this on C programs in a couple of years; do not know the current method well We can try it out in class, if you wish Valgrind http://valgrind.org/downloads/ Linux only Good for finding memory allocation / access errors

12. 12 Tools: Static Analysis Uno http://spinroot.com/uno/ Static analysis for C code We’ll discuss static analysis a bit later in class, briefly Could give you a hint as to bugs before you test, if you feel like brave exploration

13. 13 Tools: Model Checking Spin http://spinroot.com/spin/whatispin.html Backtracking exploration (“explicit-state model checking”) for C programs Topic of class soon (once we’re done with debugging)! If you want to download/start playing, I suggest grabbing it and then looking in the SPIN directory in the repository CBMC http://www.cprover.org/cbmc/ Bounded model checking: exhaustive up to loop bounds On Windows, needs a Visual Studio install

14. 14 VOLUNTEERS NEEDED Who will join my PROJECT TEST PLAN PRESENTATION ARMY? No draft, though; it wouldn’t help me get re-elected Let’s take hands and start making a schedule now. Will Boshuizen Canaan Hadley-Voth

15. 15 SUBMITTING THE TESTER Requirement: You turn in code such that: > make testdom >./testdom 20 minutes later I can take a look at your test results – failures, success, coverage, etc.

16. 16 THE PROJECT REPORT Contents: Detailed discussion of test approach taken I want to see source code – show, don’t just tell Not a dump of all your code, but representative portions to indicate any of these that apply: Structure of a test case Length of test cases Method for choosing inputs Specification method used How to handle segfaults/crashes Development history of the test system Areas particularly well tested, areas that have not been thoroughly tested – remaining risks of failure Complete set of test statistics gathered: coverages, number of tests performed, number of faults detected, etc.

17. 17 THE PROJECT REPORT Contents: Detailed discussion of the test process Communications with partners Give some representative examples, show me what a bug report and response looked like Revisions to tested code (and to tester) Regression strategy, regression failures discovered Quality of code on first testing Quality of code when finished Lines of code at beginning, when finished Final status of all bugs reported so far (FIXED, WONTFIX, BOGUS…) Details of bugs (possibly of all discovered bugs, if it fits), including the severity, nature, fix, pointer to test case (or full test case if it is small enough)

18. 18 THE PROJECT REPORT Contents: Opinions for future testers What would you have done differently, given time How well did any tools you used work? What infrastructure did you develop or did you need How useful was automated testing vs. hand unit tests vs. code inspection, in your case? I expect a written document rather than a powerpoint here. Ok, it it’s a REALLY GOOD powerpoint, fine. I doubt you can do it in less than 5 pages. That is not a challenge to do it in 4.

19. 19 GENERAL WARNINGS & CAUTIONS Do not just trust that svn is working for you CONFIRM YOUR COMMITS Browse source in the beaversource repository CONFIRM YOUR COMMITS svn add Adds this directory to the repository, but does not check it in Doesn’t actually DO anything by itself svn ci –m “my message” This checks in everything below the current directory Don’t imagine you can do the whole project late in the term, since the early plan/reports aren’t many points. You can’t. There is a strong correlation between “did not turn in the early stuff” and “got a bad grade on the project” “If you start late, you’re going to arrive late” (speeding doesn’t actually get you there that much sooner)