1. Researching the Effectiveness of Serious Games: Asking the Right Questions Dr. Jennifer Solberg US Army Research Institute for the Behavioral and Social Sciences

2. Serious games are being increasingly incorporated into learning environments. Examples: Classroom Education Games for Health Military Training What sorts of questions should we be asking? Really, there’s only one. The Rise of Serious Games

3. The One Big Question Do Serious Games Work? Are serious games an effective means of training? Are they effective above and beyond current methods? It’s that simple! If not, nothing else matters It’s not that simple! Many factors are undoubtedly involved in the effectiveness of serious games.

4. Overview Current thought on the One Big Question (OBQ) Answers so far Are we asking the right questions? What questions we should be asking The Player The Game The Process Where we can find the answers Performance within and outside the game Individual difference measures Preference measures Where do we go from here?

5. Current Thought on the OBQ Is there evidence that serious games are effective for training? Sometimes. Is there evidence that serious games in general are effective for training above and beyond traditional methods of instruction ? Not really. (Hays, 2005) What Does this Mean?

6. Current Thought on the OBQ “The empirical research of the instructional effectiveness of games is fragmented, filled with ill defined terms, and plagued with methodological flaws.” (Hays, 2005) “There appears to be a consensus among a large number of researchers with regard to the negative, mixed, or null findings of game research, suggesting that the cause might be a lack of sound instructional design embedded in the games” (O’Neil et al, 2005) “Successful learning in serious games involves a complex interaction between the player and the game. In order to develop serious games that work, we must ask the right questions about this interaction.” (Solberg, this week)

7. Asking the Right Questions Any type of learning involves three components: the player, the instructional material, and the learning process itself. We need to ask questions about all three. The Player: What characteristics of the player make them more likely to learn through serious games? The Game: What aspects of the game itself can we manipulate to maximize effectiveness? The Process: In what context should the game be delivered to the player to maximize the likelihood of learning?

8. Is a game-based environment conducive to our normal learning and memory processes? Kirschener et al (2006). The trend of relying on unstructured learning is contrary to cognitive architecture. Clark’s talk yesterday Are serious games conducive to normal interactions in a multi- player setting? Method of communication Restriction of actions The Player

9. Do some people have characteristics that lend themselves to success in game-based learning? You may be surprised! Examples: Sex Carr (2005). Female players avoid female avatars in game environments. Age/Generation Cognitive ability Personality Learning style preference Propensity to play games Orvis et al. (2005, 2006). Cadets’ gaming experience Prior experience with training material

10. The Game How can we design serious games to maximize the likelihood of learning? Current thought: –Serious games should be tied to structured learning objectives. –Serious games should be motivational. What aspects of serious games can we manipulate to assess effectiveness? Modality (of instructions, feedback)Type of instructional material Changes in difficulty levelComplexity (part v. whole task) Transparency of instructional materialDuration (effects on motivation) In game feedback/scoreHelp (scaffolding, availability)

11. The Process Does the way in which serious games are administered affect the quality of learning? Possible factors: Pre-game instruction (game play and training domain) Presence of instructor Feedback during/after game (modality and timing) Effects of retesting We have lots of questions. Where can we find the answers?

12. Finding the Answers Where can we find the answers? Performance measures within and outside the game Individual difference measures Preference measures These measures make up the types of data used in serious games research. –It’s important to consider including measures in the game design so that research is possible later on. Validation Improvement Trouble-shooting

13. Performance Measures Within-game: How well do players play the game? Score Time to completion Qualitative decisions/actions Relative position to others? Communication between players Outside the game: How well do players transfer skills to other tasks? “Real world” task Nolan & Jones (2005) squad exercise performance after training with COTS game Tests Retention, recognition –Training domain knowledge –KSAs

14. Individual Difference Tests In order to determine the extent to which player characteristics can affect learning in serious games, measures of these characteristics should be considered. Perceptual/attentional tests Cognitive factors (e.g. decision-making ability, memory span) Personality factors (e.g. extraversion, conscientiousness, learning style preference)

15. Preference Measures An important aspect of serious games is their ability to motivate. Gaming should be fun. Preference measures provide data on the enjoyability of the game. Also, Preference for game-based instruction Preference for modality of instruction (e.g. visual v. audio)

16. Too Much Information? How much performance information is enough? What kind is most informative? This is a topic of debate (even in my lab), and largely depends on constraints of project Subjectivity v. objectivity Need for rapid deployment (especially in military) Need for troubleshooting Need to answer theoretical questions Should there be a “standard” methodology?

17. Where Do We Go Now? Assessing the answer to the OBQ will require asking questions about the interaction between The Player The Game The Process Some answers can be found in: Performance measures within and outside the game Individual difference measures Preference measures

18. Where Do We Go Now? These are all empirical questions that need to be answered! Serious games are being implemented whether or not we answer the OBQ! Non-trivial training domains Military training Medical/health instruction

19. An Afterthought The Author How easily does the author of the game interact with the authoring interface? –May affect ability to incorporate training objectives into game environment. How much control does the author need? –Sophistication of scenarios v. accessibility –Decision based on finances, access to facilities, need for rapid deployment