1. California Educational Research Association (CERA) San Francisco, CA – November 19, 2009 Gregory K.W.K. Chung Research Issues in Developing Games for Learning and Assessment

2. 2 / ∞ Overview Project overview Why study games for learning? Tensions along the way Some design variables Study results Conclusion and next steps 2

3. 3 / ∞ Project Overview Center for Advanced Technology in Schools (CATS) USC Game Innovation Lab R&D focused on games and simulations for learning and assessment Content focus is pre-algebra (rational numbers, solving equations, functions) Target population is underprepared students Systematic testing of features (instructional variations, game-based) before full-scale implementation

4. 4 / ∞ Why Study Games for Learning? If you build it, they will play (and learn)... Given: Students choose to spend hours playing games Idea: Let’s put academic content in games Magic: Students will play the game, be engaged in the game, and will learn the stuff fait accompli Recall scantron (1950s), word processors (1980s), calculators (1980s), OPAC (1980s), Web (1990s)... It’s going to happen with or without R&D, so let’s figure out ways to shape the process

5. 5 / ∞ Why Study Games for Learning? Help determine the relationship among: Different instructional design variables AND Different game design variables AND Different types of learning outcomes AND Different types of students AND Different types of game outcomes

6. 6 / ∞ Tensions: Games for Learning Math game learning fun math play time efficiency choose to play have to play “pure” math “applied” math basic skills 21st century skills simple tasks complex tasks unobtrusive measures (embedded) obtrusive measures (external) 6

7. 7 / ∞ The R&D Challenge 7 Math outcomes Skills Conceptual understanding Game outcomes Game level Gaminess Instruction Tutorial Feedback Core mechanics Must use math Motivational elements Bling ?

8. 8 / ∞ Game Design Variables Feedback Type Timing Precision Impasse-driven In-game Assessment Scoring Performance sensing 8 Instruction Game mechanics Conceptual Procedural Core mechanics Part of game Motivation Bling

9. 9 / ∞ Outcome Variables Math outcomes Skills Conceptual understanding Game outcomes Student perception of “gaminess” Flow Game level 9

10. Prototype Gamelet

11. 11 / ∞ Game Design Requirements The Outcome Conceptual and computational fluency with rational numbers (fractions) The Math Idea of “unit” and fractional parts Additive operations Denominator  no. of pieces in 1 unit Numerator  no. of pieces Equivalence The Challenge: How to do math without killing the game

12. 12 / ∞ Prototype Game Design Genre Puzzle—need to figure out how to navigate from start to end points Game and Learning Mechanics Jumping/bouncing from point to point Adding coils to go from point to point Only allowed to add pieces of the same fractional size (i.e., common denominator) Need to convert among equivalent units (2/2 = 3/3 = 4/4)

13. 13 / ∞

14. 14 / ∞

15. 15 / ∞

16. Study

17. 17 / ∞ Research Study Research Question To what extent do different kinds of feedback affect understanding of fractions (i.e., unit), game performance, and perception of game play? Design 2 conditions that varied feedback Gamey: Minimal math instruction Mathy: Emphasized math concepts related to unit

18. 18 / ∞ Sample N = 137 9th (30%); 10th (18%), 11th (31%), 12th (15%) Amount of weekly game play 0hr (21%); 1-2hr (40%); 3-6hr (19%); > 6hr (23%) Math achievement Self-reported grades: A’s and B’s (55%), C’s (31%), D’s and F’s (13%) Math pretest: M = 6.34, SD = 3.39, Min. = 0, Max. = 11

19. 19 / ∞ Measures Math outcome Pretest, posttest Game outcome Last level reached, perception of game Game process measures Time, correct fraction additions, incorrect fraction additions Background

20. 20 / ∞ Results Did we build a game? Did students learn from the game? Was there an effect of type of feedback on: Learning? Game performance? Game perception?

21. Did we build a game?

22. Yes

23. 23 / ∞ Results

24. 24 / ∞ Results

25. 25 / ∞ Results

26. Did students learn from the game?

27. It depends

28. 28 / ∞ Did students learn from the game? No overall effects of game play on math posttest scores Not surprising—sample was composed of high and low performers However, our target group—low math performers—appeared to profit from game play Low performers’ posttest scores (M = 3.08, SD = 2.04) were significantly higher than their pretest scores (M = 2.55, SD = 1.22). t (48) = 2.0, p =.05, d = 0.32.

29. Was there an effect of type of feedback on learning?

30. No

31. Was there an effect of type of feedback on game performance?

32. Yes

33. 33 / ∞ Was there an effect of type of feedback on game performance? Students in the mathy condition (vs. the gamey condition): Appear to have gone further in the game (p =.08, d = 0.31) Committed more correct additions (p =.003, d = 0.49) Committed fewer incorrect additions (p =.007, d = 0.48)

34. Was there an effect of type of feedback on game perception?

35. Probably

36. 36 / ∞ Was there an effect of type of feedback on game performance? Students in the mathy condition (vs. the gamey condition): Perceived the game as more game-like (p =.08) Were more willing to use the game as part of school work (p =.06) Agreed more with the statement that the game helped them understand math (p =.003, d = 0.54)

37. 37 / ∞ Summary Did we build a game? (YES) Did students learn from the game? (ONLY LOW PERFORMERS) Was there an effect of type of feedback on: Learning? (NO) Game performance? (YES) Game perception? (PROBABLY)

38. 38 / ∞ Conclusion and Next Steps Beginning to understand conditions under which “mathification” may not hurt game play Speculate that math instruction helped students progress in game Impasse-driven instruction Results promising for the development of a game that includes math content while preserving game aspect Need stronger instructional intervention Building tutorial, just-in-time feedback

39. Backup