1. California Educational Research Association Annual Meeting San Francisco, CA – November 19, 2009 Richard Wainess, Rebecca Buschang, & Deirdre Kerr Using Human Help to Inform Automated Help in Games-Based Learning Environments

2. 2 / 23 Our Goal Create a self-contained pre-algebra instructional game as platform to examine what works in games for learning

3. 3 / 23 Puppetman (a pre-algebra game)

4. 4 / 23 Current Situation CONDUCTED DESIGN STUDY To examine [prototype] game/instructional features and outcomes [learning/attitude ] Two Conditions: Mathy Version and Gamey Version

5. 5 / 23 Issue Players needed help not provided by game

6. 6 / 23 1. Captured help needed 2. Coded the help (3 ways) 3. Analyzed game/test/help data 4. Report Results 5. Discuss Next steps Process

7. 7 / 23 Phase 1: Capture & Code Help IDGame HelpMath HelpComments  - game mechanic  - other (pls note) - basic (e.g., job aide) - instruction  - solution  - other (pls note) Had her look up "unit" and green dots. Helped her with counting number of fraction (numerator). And that 1 1/2 is same as what else? (i.e., 3/2)  - game mechanic  - other (pls note) - basic (e.g., job aide) - instruction  - solution  - other (pls note) 1. Basics of "unit" and green dots. 2. Counting fractions (numerator). 3. How many 1/6 is 1/3?  - game mechanic  - other (pls note)  - basic (e.g., job aide) - instruction  - solution  - other (pls note) 1. Recognizing 1/2 rather than thirds. 2. Counting 1/2s. Plus game level and helper’s initials

8. 8 / 23 Phase 1 Coding Results Help TypeCount Game Help: Game Mechanic 12 Game Help: Game Other8 Math Help: Math Basic 20 Math Help: Math Instruction 30 Math Help: Math Solution8 Math Help: Math Other1 46 participants need help (out of approx 120 participants) 30 needed help 1 time 11 needed help 2 times 4 needed help 3 times 1 needed help 4 times

9. 9 / 23 Phase 2: Examined Help Elements CommentsCoded As Had her look up "unit" and green dots. Helped her with counting number of fraction (numerator). And that 1 1/2 is same as what else? (i.e., 3/2) 1.Asked guiding Questions 2.Prompted help seeking 1. Basics of "unit" and green dots. 2. Counting fractions (numerator). 3. How many 1/6 is 1/3? 1. Asked guiding Questions 1. Recognizing 1/2 rather than thirds. 2. Counting 1/2s. 1. Can’t code 1.One rater examined 120+ help elements and created rubric 2.3 rounds with 2 raters (1/3 of data each round) 3.Modified/clarified/reduced rubric after round 1 4.Inter-rater reliabilities of 71%, 64%, 63% (100% w/discussion)

10. 10 / 23 Phase 1 coding results Help TypeCount Can’t code31 Asked guiding questions17 Explained game control10 Stated math concept in terms of game feature/mechanic6 Gave solution step6 Prompted help seeking5 Explained game mechanic4 Asked for explanation2 Explicit math instruction0 Stated math concept as math0 Gave solution0 Gave solution strategy0

11. 11 / 23 Phase 3: Recoded Help Elements CommentsCoded As Had her look up "unit" and green dots. Helped her with counting number of fraction (numerator). And that 1 1/2 is same as what else? (i.e., 3/2) Help w/ unit Help converting fractions Help w/ green dots Help w/ coil size (denom) Help w/ coil num (numer) 1. Basics of "unit" and green dots. 2. Counting fractions (numerator). 3. How many 1/6 is 1/3? Help w/ unit Help converting fractions Help w/ green dots Help w/ coil num (numer) 1. Recognizing 1/2 rather than thirds. 2. Counting 1/2s. Help w/ coil size (denom) Help w/ coil num (numer) 1.One rater examined 120+ help elements and created codes 2.Second rater coded using same codes (matched 49%) 3.Reviewed, clarified rubric, and reached consensus

12. 12 / 23 Phase 3 coding results Help TypeCount Clarify what one “unit” is25 Determine appropriate coil size (denominator help)19 Convert fractions16 Determine number of coil pieces to use (numerator help)11 Help with moving trampolines/coils8 General help with fractions7 Help with what green dots meant5 Help with placing trampolines in correct grid position5 Help on Level 14 Help combining fractions4 Help with scrolling4 Other help4 Determine some aspect of distance2 Help with understanding arrows2 Help with undo/reset button2 Help breaking coils into smaller parts1

13. 13 / 23 ANALYSIS & RESULTS

14. 14 / 23 Differences in Gameplay One-way ANOVA on Level Reached Only 27% of those reaching or exceeding level 17 needed help Only 35% of those reaching levels 12 to 16 needed help 78% of those who did not reach level 12 needed help

15. 15 / 23 Differences in Learning Regression predicting post-scores (results of ANOVA) (Note: Four models were run. Values below are from fourth model.) No difference in post scores between gamey/mathy, B =.023, p =.879 Difference in post scores between help/no help, B = -.587, p =.001 Kids who got help also scored lower on post-test, when controlling for gender, gameplay, and pre-test Model r2 =.490, p =.000 is a significant predictor of post- test. Adding interaction (help & condition) was a significant predictor of post-test scores ( when added gender, pre-test, gameplay, condition, help, and measures of in-game learning ), p =.033

16. 16 / 23 Differences in Learning (cont’d) Interaction between help/no help and condition Those who received help did worse those who didn’t receive help (p =.001)  B = -.587 for gamey  B = -.564 for mathy Mathy with help did better than everyone (p =.033)  B = 1.095 better than gamey with help  B=.508 better than gamey no help  B =.485 better than mathy no help

17. 17 / 23 Help by Condition Help TypeGamey (N)Mathy (N) Game Help: Other(3)Game Help (5) Math Help: InstructionMath Help (15)(10) PHASE 1 Help TypeGamey (N)Mathy (N) Explained Game Controls(3)Game Help (8) Prompted for ExplanationMath Help (2)(0) Game Solution StepMath Help (4)(2) PHASE 2

18. 18 / 23 Help by Condition (cont’d) Help TypeGamey (N)Mathy (N) Moving Trampoline or CoilGame Help (5)(3) Level 1 (getting started)(1)Game Help (3) How to Scroll(1)Game Help (3) Defining a UnitMath Help (14)(10) How to Reset or Undo(0)Game Help (2) Other Help with Distance(0)Game Help? (2) PHASE 3

19. 19 / 23 Help by Condition Trends Help TypeGamey (N)Mathy (N) Math Help: InstructionMath Help (15)(10) Defining a UnitMath Help (14)(10) Explained Game Controls(3)Game Help (8) Largest Ns ConditionGame HelpMath Help Gamey14 Mathy60 Most help by condition

20. 20 / 23 Next Steps Created the following help topics and instruction to address all areas needing help 1. Determining the Level’s Denominator 2. Determining the Distance Between Jump Points 3. Changing the Denominator of a Coil 4. Common Denominators 5. Reducing the Denominator 6. Least Common Denominator of Two Fractions 7. Matching the Level’s Denominator

21. 21 / 23 Next Steps (cont’d) Next Steps Validate Assumptions through Think-Alouds ( beginning December, 2009 ) Embed more instruction into game  As direct instruction ( January, 2009 )  As contextualized help/feedback  Through an in-game help system

22. 22 / 23 Review Collected “help given data” during design study Coded data three ways Analyzed codes, gameplay, & test scores for themes Used analysis results to define instructional needs Created and are about to test instructional parcels Will begin testing embedded instruction January 2009

23. 23 / 23 BACKUP

24. 24 / 23 In-game Help Comparison: Level 6 StartHelpMake Bucket Place BucketWrong UnitEndDeath Mathy We need to bounce PuppetMan 3 spaces to reach safety! Click on the big coil and scroll up on the mouse wheel to break the WHOLE into more but smaller EQUAL pieces. The WHOLE unit (the space between the red lines) is broken into TWO equal pieces but it will take THREE of THOSE EQUAL pieces to get to the goal. Recall - two HALVES are ONE WHOLE When you are finished loading the trampoline, place it on the blocks to help save PuppetMan! Have you counted the number of EQUAL pieces the WHOLE was broken into? Did you count how many of THOSE pieces you need to reach the goal? Remember ONE WHOLE UNIT is the distance between the red lines. Remember, the coils must all be exactly the SAME size when ADDING them to the trampoline. If you cannot drag a piece in, break the coils into pieces of the SAME size (same denominator). You are mastering the art of trampoline making! Just remember, addition requires that all the coils have to be the SAME size to be put into the same trampoline! Make sure you see how many pieces the WHOLE (between the red lines) is broken into, THEN count how many of THOSE pieces you need to get PuppetMan to the goal. Gamey We need to bounce PuppetMan three half-units to reach safety! Click on the big coil and scroll up on the mouse wheel to break it in half, and then drag the 3 half pieces over! Sorry, no help on this level. When the trampoline says 3/2, place it on the blocks to help save PuppetMan! This is the text that appears the player places some number of trampolines onto the grid If you cant drag a piece in, click on the trampoline and scroll the wheel to change the units! You can also use the scroll wheel to break up the coils into the right units! You are mastering the art of trampoline making! Always remember, all the coils have to be the same size to be put into the same trampoline! DISABLED

25. 25 / 23 In-game Help Comparison: Level 11 StartHelpMake BucketPlace BucketWrong UnitEndDeath MathyYou know the goal, get PuppetMan safely across the spears. But now the WHOLE unit has been broken down differently. Check it out and good luck. The WHOLE UNIT has been broken into THREE pieces and it looks like you need FIVE of them in total. Since you only have HALVES break them into a piece that can also be made into THIRDS. When you are finished loading the trampolines, place them on the right blocks! Have you counted the number of EQUAL pieces the WHOLE was broken into? Did you change HALVES into a unit you can also use to count THIRDS? If you can't drag a piece in, click on the trampoline or coil and scroll the wheel to change the units into the units you need. You can only ADD IDENTICAL pieces. Genius! Sometimes, you'll have some coils leftover! Don't worry at all about that, because PuppetMan is alive and well. Break everything into SIXTHS to make it easier! Two SIXTHS is one THIRD and Three SIXTHS is one HALF. Then you can count both coils and how far you need to jump. GameyYou know the goal, get PuppetMan safely across the spears. But now the WHOLE unit has been broken down differently. Check it out and good luck. The WHOLE UNIT has been broken into three pieces and it looks like you need five of them to get to the goal (two to the first block and three from there to the goal). When you are finished loading the trampolines, place them on the right blocks! Have you counted the number of EQUAL pieces the WHOLE was broken into? Did you count how many of THOSE pieces you need to reach the first block? How many more do you need to reach the goal? If you can't drag a piece in, click on the trampoline or coil and scroll the wheel to change the units into the units you need. Genius! Sometimes, you'll have some coils leftover! Don't worry at all about that, because PuppetMan is alive and well. Break everything into thirds to make it easier! Then make sure you have enough thirds to reach the first block and enough to reach from there to the goal.

26. baker@cse.ucla.edu