1. 1 Instructional Strategies to Improve Learning in Computer Games Harold F. O’Neil and Hsin-Hui Chen, University of Southern California/CRESST AERA v.5 Chicago, Illinois April 10, 2007

2. 2 What Is a Game? A computer game consists of four key components –Settings that are real or imaginary –Roles or agendas for the participants –Rules (real life vs. imaginative) –Scoring, recording, monitoring, or other kinds of systematic measurement Motivation comes from challenge, complexity, fantasy

3. 3 CRESST Model of Learning Content Understanding Learning Communication Collaboration/ Teamwork Problem Solving Self-Regulation

4. 4 Content Understanding Assessing Problem Solving Via Games Domain-Dependent Problem-Solving Strategies Self-Regulation Metacognition Self- Monitoring Planning Motivation Effort Self- Efficacy

5. 5 Research Questions Will games increase players’ problem solving? Will adding effective instructional strategies to commercial off the shelf games improve problem solving? Trade-off between development and selection

6. 6 The Specification of What We Are Teaching Is Essential From goal/objective of teaching leadership, situational awareness, decision making, tactical problem solving –The instructional strategies follow Nature of feedback, timing of feedback, take- home packages, instructor training, homework assignments, etc. –The type of assessment follows Different assessment measures, after-action reviews

7. 7 Do Games Train? — Literature The research indicates that computer games are potentially useful for instructional purposes and are hypothesized to provide multiple benefits –Promotion of motivation; improvement of knowledge and skills; facilitation of metacognition Limited empirical research in journals conducted on games topic (19 studies, 1990–2005) –Adults, empirical (qualitative/quantitative) –PsycINFO, Education Abs, SocSciAbs In 2006, DOD technical report literature added 4 additional reports –Only one relevant empirical study on massive multiplayer games

8. 8 Different Mental Models 1 Content understanding, problem-solving, self-regulation, communication, team skills. 2 Action, role planning, adventure, strategy games, goal games, team sports, individual sports (Laird & VanLent, 2001). 3 Implicit vs. Explicit: During or after (AAR).

9. 9 Check Validity of Instructional Strategy Embedded in game –Usually inductive discovery approach —Usually doesn’t result in learning (Kirschner, P. A., Sweller, J., & Clark, R. E. 2006. Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based learning. Educational Psychologist, 41, 75-86.) What Works in Distance Learning –Good instructional practices that can be applied to games

10. 10 Selection of Game for Research Off-the shelf games lacking learning objectives and assessment of learning Use wrap around instructional & assessment strategies as no access to source code

11. 11 SafeCracker Puzzle-solving game –Example of problem solving No special background knowledge, motor skills, or extraordinary visual-spatial ability required Adult-oriented Single-player game Pacing controlled by players Not popular

12. 12 Common Methodology Participants: –Young adults selected to have no experience of playing SafeCracker but game players Measures: –Knowledge mapper –Retention and transfer questions analogous to Mayers’ –Trait self-regulation questionnaire

13. 13 Knowledge Mapper

14. 14 An Example of Scoring Map Concept 1LinksConcept 2Expert1Expert2Expert3 Keyis used forSafe111 RequiresKey110 CatalogcontainsClue101 SafeContainsClue001 Final score= total score÷ number of experts =8÷3= 2.67

15. 15 Problem Solving Strategy Measure Domain-specific problem-solving strategies measured by open-ended questions Modifications of previous researchers (Mayer, 2001; Mayer & Moreno, 1998; Moreno & Mayer, 2004)

16. 16 Problem-Solving Strategy Measure Retention Question List how you solved the puzzles in the rooms. Transfer Question List some ways to improve the fun or challenge of the game.

17. 17 Scoring of Retention Question and Transfer Question Based on the number of predefined major idea units correctly stated by a participant regardless of the wording.

18. 18 Measurement of Self-Regulation Trait self-regulation questionnaire (O’Neil & Herl, 1998). –planning –self-checking –self-efficacy –effort

19. 19 Study I, II & III Study I– Without effective instructional strategies. Study II– With worked examples. Study III– With navigational aids.

20. 20 Purpose of the Study I To evaluate a computer game (SafeCracker) with regard to its effectiveness for improving problem solving

21. 21 Formative Evaluation Measures design and tryout Checking of validity of instructional strategies embedded in game against research literature Feasibility review Revisions implemented O’Neil’s Framework (2002)

22. 22 Data Analysis Knowledge Map Retention Test Transfer Test MMM Pretest 2%8%7% Posttest4%15% 12% t(29) = 4.32, p <.01 t(29) = 12.66, p <.01 t(29) = 7.05, p <.01

23. 23 Discussion/Implications There was an increase in problem-solving. But it was small. Existing instructional strategies (discovery learning) in the game were not effective. More research on a game designed with effective research-based instructional strategies –Worked examples (Danny Shen) –Pictorial aids (Richard Wainess) –Just-in-Time Worked Examples (Joan Lang) –After-Action Review

24. 24 This study provides a research environment with reliable and valid measures of problem solving: –knowledge maps –retention and transfer questions –trait self-regulation questionnaire Used in RSOE/USC game research Discussion/Implications (cont.)

25. 25 Study II Wrap-Around Instructional Strategy (Shen & O’Neil, In-Press) Will participants in the worked example group increase their problem solving in a game-based task (i.e., SafeCracker) after studying worked examples compared to the control group?

26. 26 Worked Examples Worked examples are procedures that focus attention on problem states and associated operators (i.e., solution steps), enabling students to induce generalized solutions or schemas (Sweller, 1998). Many researchers investigated the efficacy of using worked examples in classroom and computer-based instruction and provided evidence of the effectiveness of worked examples instruction (Cooper & Sweller, 1987; Mayer & Mautone, 2002; Ward & Sweller, 1990). No research used worked examples in a game-based environment.

27. 27 A Sample of a Worked Example Room 5: Constructor Office Goal: Open the Liberty Safe Step 1: Recognize the Buttons, the Lights, and the Handle.

28. 28 Results Worked example instruction produced a significant increase in content understanding compared to the control group.

29. 29 Worked example instruction produced a significant increase in problem-solving strategy question of retention compared to the control group.

30. 30 Worked example instruction produced a significant increase in problem-solving strategy question of transfer compared to the control group.

31. 31 Results (cont.) Alternative Problem Solving Measure

32. 32 Discussion/Implications The worked example group significantly improved more than the control group in content understanding and problem-solving strategies. However, the improvement was small. This study provided evidence that using worked examples could be one of the good instructional methods to facilitate adults ’ problem solving with a commercial off-the-shelf computer game. In order to obtain greater improvement, in future studies the worked example instruction could add: –Just-in-Time –Fading procedure –Self-monitoring

33. 33 General Research Results Study I– Problem solving increased somewhat after game playing. Study II– Problem solving increased significantly more with worked examples. Study III– Navigation maps did not affect problem solving.

34. 34 What Are Continuing R&D Issues? Can we leverage game technology for training? –Embedded instructional and assessment strategies –Wrap-around instructional and assessment strategies

35. 35 Walk-Away Issues How are games currently used effectively for adults? –Limited evaluation data (qualitative or quantitative) to answer this question –There is little empirical work in the literature on effectiveness of games for training of adults Analytically, would you predict that commercial off-the-shelf games should teach? –No

36. 36 Walk-Away Issues (cont.) What support and guidance would help training game developers to do a better job? -Alignment with What Works in Distance Learning Instructional strategies that could work - Wrap-around or embedded instructional and assessment strategies

37. 37 CRESST Web Site http://www.cresst.org or any search engine: type CRESST honeil@usc.edu

38. 38 Back-Up

39. 39 code safe clue tool room trial-and -error combination desk searching key direction floor plan compass book catalog contains requires contains Safecracker Expert Map III used for contains used for leads to contains part of contains used for contains part of leads to part of leads to used for contains part of used for leads to contains part of leads to requires part of used for leads to contains used for prior to contains part of requires prior to requires contains used for contains leads to used for contains used for part of requires contains leads to