1. 8/15/2015 Center for creative technologies Evaluating the Learning and Motivation Effects of Serious Games Richard E. Clark Rossier school of Education Center for Creative Technologies August 25, 2006

2. 8/15/2015 Topics Evidence from games research on learning and motivation benefits 6 problems identified by research Three evaluation strategies: 1.Testing learning and motivation 2.Blending robust learning and motivation pedagogies into game environments 3.Game and non-game comparisons

3. 8/15/2015 Current Research Evidence In 4,000+ articles only 19 studies in peer- reviewed journals (O’Neil et al, 2005) None provide evidence for learning due to “serious games” Small learning gains attributed to pedagogy that did not require game Gredler (1996) reported similar finding

4. 8/15/2015 Current Research Evidence Problem 1: Games were not clearly defined – Suggested definition is: Games “consist of (linear goals and) rules that describe allowable player moves, game constraints and privileges (e.g. extra turns) and illegal actions … they need not relate to real world actions” (Gredler, 1996, p. 523)

5. 8/15/2015 Current Research Evidence Problem 2: Games confused with simulations – Simulations defined as: Depicting “a dynamic set of (not necessarily linear) relationships among several variables that (1) change over time, and (2) reflect authentic causal processes” (Gredler, 1996, p. 523) Serious games and simulations are often blended

6. 8/15/2015 Current Research Evidence Problem 3: Pedagogy in games most often based on “unguided discovery” Minimal guidance only works with top experts Overwhelming evidence discovery does not help novices learn (Mayer, 2004; Kirschner et al, 2006) Discovery and some game design techniques overload working memory and decrease learning (Clark and Choi, 2005)

7. 8/15/2015 Current Research Evidence Problem 4: Learning “with” or “from” a game? Is the pedagogy transferable? Example: Mayer et al (2002) found that visual modeling guidance aided learning in a geology simulation-game (.45 effect size) Visual modeling is an example of learning “with” a game – could be used effectively with many delivery platforms “With” game learning can provide varied practice

8. 8/15/2015 Current Research Evidence Problem 5: Tests of learning are most often unreliable and invalid Learning cannot be measured by self report Inverse relationship in many instances Two types of knowledge Declarative – can memorize and recall Procedural – can apply/do something

9. 8/15/2015 Current Research Evidence Problem 6: Tests of motivation are most often unreliable and invalid Self reported enjoyment does not aid learning Inverse relationship in many instances Motivation is assessed with three outcomes 1.Active choice (Do they choose to start something?) 2.Persistence (Do they persist?) 3.Mental effort (Do they invest adequate mental effort?)

10. 8/15/2015 Impact on learning Ability contributes 30 to 45% of variance in learning Motivation contributes 18 to 26% of variance Three “causal” variables account for most variance VALUES (self reported interest and utility) SELF EFFICACY (self report of specific ability) EMOTION (self report of anger or depression) All variables are presumed to influence “perception of self-control of outcomes” (Ackerman et al, 1995; Colquitt et al, 2000)

11. 8/15/2015 Motivational Strategies Environmental variables that influence value, efficacy and emotion? Goals (C 3, Mastery/Performance) Incentives (praise, money, winning) Attributions ( Control causes for novel, negative events ) Mood or Emotion (positivity and optimism) Model ( Competence, similarity, credibility, enthusiasm ) Counter Intuitive Messages (for cognitive conflict)

12. 8/15/2015 Goals Concrete Current Challenging Mastery/Performance External Attributions Suggested control & stability causes about novel or negative events Modeling Model Competence Model Similarity Model Credibility Model Enthusiasm Values Interest Importance Utility Self and Group Efficacy Expectation of success in specific situation Confidence in group skills and collaboration Internal Attributions Belief about control & stability of novel or negative events Mood Affect Emotion Active Goal Pursuit (Has intention turned to action?) Persistence (Have they stopped working on the task?) Mental Effort (Are they working to develop new knowledge?) Expertise and Meta-cognitive Strategies Planning Tuning Selecting Connecting Monitoring Goal Directed Achievement Environmental Factors Psychological Factors Motivated Behavior Knowledge and Learning Strategies Performance Motivation For Learning     Incentives Reinforcement Praise Money © 2005 Richard Clark

13. 8/15/2015 Team Motivation Individual motivators: Value, Self-Efficacy, Positive Emotion Team Motivators = individual motivators + 1.Belief in Team Collaborative Ability 2.Belief that team members have required skills Team Motivation Problems: Social Loafing (Eliminated by individual evaluation) Negative perceptions of weak members Perception that Prima Donna’s refuse collaboration

14. 8/15/2015 Mental effort has an inverted U relationship with task self efficacy. High Mental Effort Low Low Task Self Efficacy High DeclarativeProcedural Novelty Familiarity

15. 8/15/2015 Three Evaluation Strategies I.Test Learning: Use Kirkpatrick’s 4 Level Model (Kirkpatrick, 1994) 1.Reactions (Value, Emotion, Efficacy) 2.Learning & Motivation Learning: Recall and Application Motivation: starting, persisting, mental effort 3.Transfer (Can they do it in life after the game?) 4.Impact (Did “doing it” help achieve a larger goal?)

16. 8/15/2015 Three Evaluation Strategies II. Test Pedagogy: Build guidance into game Effective Guidance has four elements: 1.Connecting new learning to prior experience 2.Authentic problems that reflect objectives 3.Demonstrations of how to solve problems 4.Part task and whole task practice with feedback that focuses on player strategy

17. 8/15/2015 Three Evaluation Strategies III. Test Game and Non Game Platforms Compare game pedagogy with the same pedagogy delivered on non-game platforms Assess reactions, learning and motivation Time to learn Measure transfer to non game environment* Collect cost/benefit data

18. 8/15/2015 Conclusion The most likely benefits of serious games are: An opportunity to present controlled but increasingly varied, challenging and authentic problem solving. A platform that allows us to study the evolution of learning and motivation for different pedagogies, learners and learning tasks. A realistic and engaging vehicle to study long-term practice and automation of complex expertise. An opportunity to study the interaction between emotion and problem solving.

19. 8/15/2015 References: Games, Learning Clark, R. E. and Choi, S. (2005). Five design principles for experiments on the effects of animated pedagogical agents. Journal of Educational Computing Research. Clark, R. E. (January 2005) Five Research-Tested Group Motivation Strategies. Performance Improvement Journal. 13-17. Kirschner, P., Sweller, J., & Clark, R. E. (2006). Why minimally guided learning does not work: An analysis of the failure of discovery learning, problem-based learning, experiential learning and inquiry-based learning. Educational Psychologist, 41(2). 75-86. Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels. San Francisco, CA: Berrett- Koehler Publishers, Inc. Mayer, R. E. (2004). Should there be a three-strikes rule against pure discovery learning? American Psychologist, 59(1), 14-19. Mayer, R. E., Mautone, P., and Prothero, W. (2002) Pictorial aids for learning by doing in a multimedia geology simulation game. Journal of Educational Psychology, 94(1), 171-185 O’Neil, H. F., Wainess, R., and Baker, E. (December 2005) Classification of learning outcomes; Evidence from the games literature. The Curriculum Journal. 16(4), 455-474. Yildir, I. (2004). Group motivation and performance indicators in an online, team role-playing game. Unpublished Ph.D. dissertation presented to the faculty of the Rossier School of Education, University of Southern California, Los Angeles.

20. 8/15/2015 References: Motivation Ackerman, P, Kanfer, R. and Goff, M. (1995) Cognitive and Non-Cognitive Determinants and Consequences of Skill Acquisition. Journal of Experimental 270-304 Psychology: Applied. 1(4), Bandura, A. (1997) Self-Efficacy: The exercise of control. NY: W. H. Freeman Clark, R. E. (1999) Yin and Yang Cognitive Motivational Processes Operating in Multimedia Learning Environments. In van Merrienböer, J. (Ed.) Cognition and Multimedia Design. Herleen, Netherlands: Open University Press. 73 – 107 Clark, R. E. (January 2005) Five Research-Tested Group Motivation Strategies. Performance Improvement Journal. 13-17. Clark, R. E. (2003) Fostering the work motivation of individuals and teams. Performance Improvement, 42(3), 21-29. Colquitt, J., LePine, J., and Noe, R. (2000) Toward an Integrative Theory of Training Motivation: A meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 83(5), 678-707 Pintrich, P. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology. 95(4), 667-686 Pintrich, P, and Schunk, D. (Eds) (2002) Motivation in Education. Second Edition. Englewood Cliffs, NJ: Prentice-Hall.