Testing and Playtesting CIS 487/587 Bruce R. Maxim UM-Dearborn

Testing Objectives Testing is the process of executing a program with the intent of finding errors. A good test case is one with a high probability of finding an as-yet undiscovered error. A successful test is one that discovers an as-yet-undiscovered error.

Testing Principles All tests should be traceable to customer requirements. Tests should be planned before testing begins. 80% of all errors are in 20% of the code. Testing should begin in the small and progress to the large. Exhaustive testing is not possible. Testing should be conducted by an independent third party if possible.

Software Defect Causes Specification may be wrong. Specification may be a physical impossibility. Faulty program design. Program may be incorrect.

Good Test Attributes A good test has a high probability of finding an error. A good test is not redundant. A good test should be best of breed. A good test should not be too simple or too complex.

Test Strategies Black-box or behavioral testing knowing the specified function a product is to perform and demonstrating correct operation based solely on its specification without regard for its internal logic White-box or glass-box testing knowing the internal workings of a product, tests are performed to check the workings of all possible logic paths

Strategic Approach to Testing - 1 Testing begins at the component level and works outward toward the integration of the entire computer-based system. Different testing techniques are appropriate at different points in time. The developer of the software conducts testing and may be assisted by independent test groups for large projects. The role of the independent tester is to remove the conflict of interest inherent when the builder is testing his or her own product.

Strategic Approach to Testing - 2 Testing and debugging are different activities. Debugging must be accommodated in any testing strategy. Need to consider verification issues are we building the product right? Need to Consider validation issues are we building the right product?

Strategic Testing Issues - 1 Specify product requirements in a quantifiable manner before testing starts. Specify testing objectives explicitly. Identify the user classes of the software and develop a profile for each. Develop a test plan that emphasizes rapid cycle testing.

Strategic Testing Issues - 2 Build robust software that is designed to test itself (e.g. use anti-bugging). Use effective formal reviews as a filter prior to testing. Conduct formal technical reviews to assess the test strategy and test cases.

Stages of Testing Module or unit testing. Integration testing, Function testing. Performance testing. Acceptance testing. Installation testing.

Regression Testing Check for defects propagated to other modules by changes made to existing program Representative sample of existing test cases is used to exercise all software functions. Additional test cases focusing software functions likely to be affected by the change. Tests cases that focus on the changed software components.

Testing Stages of game testing are similar to those of all software testing: Unit testing Integration testing Validation testing Usability testing System or Performance testing

Test Plans Goals and objectives Test Strategy Components to be tested Resources Schedule Work products Testing procedures (tactics) For completing including test cases for all phases of testing

The next 8 slides come from the Rabin text

The Five Step Debugging Process 1. Reproduce the problem consistently 2. Collect clues 3. Pinpoint the error 4. Repair the problem 5. Test the solution

Step 2: Collect Clues Each clue a chance to rule out a cause Each clue a chance to narrow down the list of suspects Realize that some clues can be misleading and should be ignored

Step 3: Pinpoint the Error Two main methods: 1. Propose a Hypothesis You have an idea what is causing the bug Design tests to prove or disprove your hypothesis 2. Divide and Conquer Narrow down what could be causing the bug Eliminate possibilities from the top down or Backtrack from the point of failure upward

Step 4: Repair the Problem Propose solution Consider implications at point in project Programmer who wrote the code should ideally fix the problem (or at least be consulted) Explore other ways the bug could occur Ensure underlying problem fixed and not just a symptom of the problem

Expert Debugging Tips Question assumptions Minimize interactions and interference Minimize randomness Break complex calculations into steps Check boundary conditions Disrupt parallel computations Exploit tools in the debugger Check code that has recently changed Explain the bug to someone else Debug with a partner Take a break from the problem Get outside help

Tough Debugging Scenarios Bug exists in Release but not Debug Uninitialized data or optimization issue Bug exists on final hardware, not dev-kit Find out how they differ – usually memory size or disc emulation Bug disappears when changing something innocuous Timing or memory overwrite problem Intermittent problems Record as much info when it does happen Unexplainable behavior Retry, Rebuild, Reboot, Reinstall Internal compiler errors Full rebuild, divide and conquer, try other machines Suspect it’s not your code Check for patches, updates, or reported bugs Contact console maker, library maker, or compiler maker

Adding Infrastructure to Assist in Debugging Alter game variables during gameplay Visual AI diagnostics Logging capability Recording and playback capability Track memory allocation Print as much information as possible on a crash Educate your entire team testers, artists, designers, producers

Prevention of Bugs Set compiler to highest warning level Set compiler warnings to be errors Compiler on multiple compilers Write your own memory manager Use asserts to verify assumptions Initialize variables when they are declared Bracket loops and if statements Use cognitively different variable names Avoid identical code in multiple places Avoid magic (hardcoded) numbers Verify code coverage when testing

Game Test Cycles - 1 Alpha test cycles Can be applied when game product has a least one playable logic path Basic user interface should be complete Game should run on minimum hardware configuration Test multiplayer functions if called for Game installer should work and draft manual should exist

Game Test Cycles - 1 Alpha test cycle objectives Test all modules for current product version Create a defect database and test plan Record known defects and performance test results

Game Test Cycles - 2 Beta test cycles Applied after all game features and options have been implemented Game allows testers to navigate all logic paths to allow removal of most game termination defects Final game interface, artwork, and audio should be present Final game installer and manual exist

Game Test Cycles - 2 Beta test cycle objectives Isolate significant defects and performance problems Complete testing, defect removal, and performance testing Complete compatibility testing

Game Test Cycles - 3 Gold release candidates Applied after senior management has reviewed product and defect database All program features will be present Performance is appropriate for release Online documentation is complete All major defects removed Game runs on all supported platforms

Game Test Cycles - 2 Gold release candidate objectives Successful playtesting will occur Similar to acceptance testing, except that publisher determines criteria not end users End product will be potentially shippable Gold release candidate will be classified as a baseline work product (therefore changes are quite risky and must be approved by all stakeholders)

Acceptance Testing Purpose of acceptance testing is to have customers work with software in their own environment Determine that software that requirements have been met Largely a highly scripted affair, since it determines whether developers will get paid or not

Playtesting Game testers play the game and try to user game features in the same way as end users Use game manual and see if game installs according to written directions See if game plays according to game manual instructions Playtesting needs to take place without coaching

Playtesting Goals Playtesting follows software testing Is the game fun? Are there problems with game mechanics? Software Testing + More Tuning game play Tuning game flow experience Does the player the point of the game?

Pragmatics Playtesting is not the same as debugging (which is very code oriented) Better to have a game that plays well and crashes occasionally, than a game that runs great and has a boring ending Testers must feel that developers are reacting to their feedback and criticism in an egoless manner

Who is the right playtester? Need a person who can explain the reasons behind his or her likes and dislikes General impressions (e.g it was fun) are not very helpful It is helpful to involve first time users since they can often spot usability problems and poor game play quickly

Possible Testers Development team members and test team members may not be the best play testers (if they don’t match intended game audience profile) They are motivated to improve the product and know what needs to be tested

Team Structure Fundamental issue create willful blindness Have testing team members be different from your coders is vital Separate software testing team Separate playtesting team Separate SQA group(?)

Playtesting Team Internal team External team Manages playtesting people Does own playtests Recruits outside testers External team Representative sample target players Varying skills Unfamiliar with product Not in love with your product! Not yet bored with the product

External Testers Do they like the game? They may lie to you Politeness effect If they can’t say why they like a game feature, you have a problem When/where do they get frustrated? What common areas? Often skill-based Internal team judges player’s skill, also

Focus Groups Representative sample of target audience for game product Need to determine what feedback is needed back from target audience Can provide feedback on key screen displays, storyboards, interface prototypes early in the design process Early feedback on gameplay can shape direction of final product with less rework

Guided or Unguided Unguided Guided Testers play game without restrictions Better used later in the development process Guided Testers given specific tasks to accomplish Used early in development or during regression testing

Outcomes When play testers can not progress in game the designer should try to figure out the reason If the game is too hard, it needs to be simplified If play testers are doing unexpected things and the results are entertaining the new features should be added to the game

Case study from C.D. Shaw

External Testers: Half-Life People near Valve’s offices who sent in registration cards Designers sit quietly while player struggles Designer takes notes Typical 2-hour session 100 action items First 20-30 sessions absolutely vital Learn what was fun 200 sessions total

Half-Life: Fine tuning Add instrumentation Player position, time, health, weapons Activities: Game save, dying, being hurt, solving a puzzle, fighting a monster… Graph series of sessions together Spot too long with no encounters Spot long periods of too much health Spot long periods of too little health

Half-Life Most important playtesting outcome: Clearly identified good and bad ideas Playtesting provides the evidence needed to abandon bad ideas

Advice Don’t get defensive! Testers: Stick to your findings! The tester opinion is important Testers: Stick to your findings! Respectfully point out problems Mix up the hardware Be honest about the specs on the box Don’t let bad decisions live forever!