1. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 GaLA Game and Learning Alliance The European Network of Excellence on Serious Games WP3 – SG Application Fields Michela Mortara, Chiara Eva Catalano, CNR – IMATI

2. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 The mission of WP3

3. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Synthesis of previous years Focus: diffusion Year 1 Collection of SGs in the application domains Best practices and challenges in adoption/development Recommendations for maximizing SG diffusion Focus: learning impact Year 2 Current practices and challenges in evaluating the learning impact Plan for conducting SIG user studies Recommendations for maximizing and evaluating the learning impact Focus: user studies on SG effectiveness Year 3 Empirical evidence of effectiveness of representative SGs Validation of previous recommendations

4. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Focus and achievements of Year 4 1.SG evaluation GALA SG Evaluation Framework (SG-EF) GamEval toolkit 2.Roadmapping Research challenges from the application perspective 3.Dissemination Thematic Workshops Scientific papers SG descriptions Blog news

5. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Evaluation Framework From the fields: general evidence of effectiveness as the main barrier for adoption –need for a formal way to evaluate SG effectiveness Focuses on the evaluation of the learning impact –Design of the experiment Formulation of research hypotheses The GALA SG EF is addressed to: –developers –teachers, trainers, financers and other stakeholders –researchers

6. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 A blended approach SURVEY OF THE DOMAINS Y1,2 USER STUDIES Y3 GUIDELINES, RECOMMENDATIONS Y1,2,3 STATE OF THE ART MODELS FOR EVALUATION Y2,4 GALA SG DESCRIPTION TEMPLATE WP1 GALA SG EVALUATION FRAMEWORK GAMEVAL TOOLKIT Y4

7. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 SOA Models Dominance of single-case studies, mainly games for children in formal education –Lack of general frameworks which cover lower and higher education, professional training or personal, informal learning, or generic context of application. Few high-quality evaluation frameworks address the learning impact Limited number of generic concepts that should be taken into consideration when evaluating SGs. –Do not specify what factors to be measured and how to do so –not easy to use for hypothesis generation and testing

8. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Amory and Seagram 2003 (GOM) Amory 2007 (Game object model version II) The Game Object Model (GOM) explores the concept that educational games should be: relevant, explorative, emotive, engaging, and include complex challenges. Game Space - Play, Exploration, Challenges, Engagement These are the core elements of a game to keep the player moving through the game. These components are important to maintain the interest of a player –Visualization Space - Critical thinking, discovery, goal formation, goal completion, competition, Practice, story-line –The visualization of the game addresses the physical and emotion experience the player goes through during the game. Elements Space - Fun, graphics, sounds, technology Here are the gameplay elements within the GOM. How does the game play? Is it fun? Do the graphics engage the user? Do the graphics represent realism or are they stylized? Do the sounds help create an engaging experience or do they pull the player out of the game. Is the technology a help or hindrance for the player? –Actors Space - Drama, interaction, gestures –The physical actions and reactions of the players and students involved. How are students interacting with each other during gameplay? –Problem Space Communications - Reading, writing, speaking Communication, successful communication, happens often in a game without the player thinking about it. This communication is not simply between player and game, but student and teacher and student to student peer relationships as well. Literacy - Visual, Logical, mathematical, computational Here is where many traditional educational skills & standards (libraries, education, subject discipline) would be inserted into the GOM. Memory - Short-term, Long-term Some skill sets are intended and reinforced for success for game play and studying in the short team. There may be specific skills in the video game that are intended only for the short term (a new attack or item to reach a new location / enemy). Long-term memory needs prepare the player not only for a specific action, they the player to understand the game. Motor - Manipulation, Reflex Here are the physical movements required by the game and lesson. What actions and movements are required? A consideration to those with physical disabilities can be considered here.

9. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Amory et al 1999 and 2001 conceptualization of design principles of educational games (guideline for design) Describes relationships btw pedagogical dimensions and game elements like OO

10. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Amory 2007 – GOM II Provides a theoretical basis for the design of educational games Evaluation: create a check list of all necessary criteria (concrete interfaces) and to evaluate game design specification against such a check list Twofold evaluation: 1.pedagogical fit of a game 2. how well a game achieves the educational objectives “theorical” evaluation! Social Space object added

11. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 RETAIN Model – Gunter et al. 2007 Evaluates DESIGN to PREDICT overall effectiveness Only qualitative high level factors to evaluate fidelity to pedagogical principles Only for educational games in formal context Refers to Bloom’s levels A focus group of experts rates a game wrt: Relevance, Embedding, Transfer, Adaptation, Immersion, Naturalization Can help teacher evaluate COTS game for use in classroom We focus on L.I. = Naturalization (evidence of effectiveness) and address the other factors as causal / mediating- going in much more detail and adding quantitative measure. We also evaluate the evidence of impact on real users rather than predict it looking at the design They assume that if it is pedagogically sound, then it is effective, disregarding the player (the context is fixed).

12. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 4DF provide a conceptual framework for exploring immersive learning High level, abstract model Provides factors to be taken into account when selecting or using a game in a learning practice

13. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Need to include more factors in the context variables, e.g. the technological setup available Need to specify detailed levels in the learning goals and outcomes to measure precise achievements Need to specify factors related to game design choices and gameplay features Particularly focussed on corporate training Need to include cultural factors Conceptual Model of GBL Intervention by [Mayer et al. 2014].

14. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 The framework IMPACT QUALITY GAMEPLAY PARTICIPANT CONTEXT

15. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Close-up Satisfaction P2Player Satisfaction P2Facilitator Satisfaction Artefacts P2Design Satisfaction Learning Learning Curve Effective Learning Time Time on task, Time in game, Duration of intervention, Timing of intervention Pedagogical Approach QUALITY

16. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Properties of SG-EF broad scope: no restriction about –topic (e.g., health, security, engineering, soft skills) –aim (e.g., education, training, awareness raising, behavioural change) –expected context of use (e.g., with or without facilitation, single- multi-player or team play). technical aspects + pedagogical principles fine level of granularity for the educational objectives a number of variables is suggested for each factor –e.g., Personality Traits  Openness, Conscientiousness, Tolerance, Extraversion, Emotionally, Integrity, Agreeableness, … –both qualitative and quantitative variables

17. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 The GamEval toolkit (leader: PLAYGEN) PARTICIPANTS Personality Measure the individuals' social predispositions along : Networking ability, Apparent sincerity, Social astuteness, Interpersonal influences Social Dispositions PARTICIPANTS Personality Determine personality aspects of the individuals examined using the BIG FIVE, Myers Briggs, HEXACO or similar method along: Openness, Conscientiousness, Tolerance, Extraversion, Emotionality, Integrity, Agreeableness, Neuroticism, etc. Personality Traits CONTEXT Game Domain Identify the technologies utilised in the game : PC, MAC, Console, Smartphone, Android, iPhone, Portable console, Tablet, other Technology Platform IMPACT Learning Goals Measurements of change in skills related to physical movement and coordination covering; Perception, Set, Guided response, Mechanism, Complex overt response, Adaptation, Origination Psycho-motor skills QUALITY Artefacts Measure the satisfaction of the player toward chosen Game Element cards along one or more of : Clarity, Realism, Complexity, Usability, Ease of use, Attractiveness, Malfunctions, Storytelling, Control, Graphics, Audio, Metaphors P2Design Satisfaction

18. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 The GamEval Toolkit (leader:PLAYGEN) The ‘game’ is played using a deck of cards on two boards –Part 1 is about identifying the most relevant elements that are of interest to the developer, researcher, client, end-user - or indeed any stakeholder –Part 2 is concerned with defining the research hypotheses. The cards are in 5 suites: –Context : The Team, Organisational Environment, Game Domain –Participant : Personality, Involvement, Background, As a Gamer, As a Learner –Quality : Satisfaction, Artefacts, Learning –Gameplay : Emotional Response, Player Behaviour, Game Elements –Impact : Learning goals, Learning Outcomes, Sustainability

19. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Part 1 Identifying most important / relevant variables to measure Example – ICURA: PARTICIPANTS Background Determine the individuals background along : Age, Sex, Nationality, Residency, Socioeconomic, Diversity, Adaptation, Traditions, Humour Personal & Cultural Background PARTICIPANTS As a Gamer Determine the individuals' frequency of engagement with one or more of : Traditional Games, Entertainment Games, Learning / Training / SGs Frequency of Play PARTICIPANTS As a Learner Determine motivations towards learning along Intrinsic and Extrinsic : Interest/Enjoyment, Perceived Competence, Effort/Importance, Pressure/Tension, Perceived Choice, Value/Usefulness, Relatedness Motivation for learning QUALITY Artefacts Measure the satisfaction of the player toward chosen Game Element cards along one or more of : Clarity, Realism, Complexity, Usability, Ease of use, Attractiveness, Malfunctions, Storytelling, Control, Graphics, Audio, Metaphors P2Design Satisfaction GAMEPLAY Game Elements Determine the UI elements used in the game ; scoreboard, screen, keyboard, charts, physiological sensors, time indicator, performance meter, odometer, dashboard, cockpit, tactile output, touch-screen, stylus pen, point of view, point & click interface, cameras, bio-feedback, aural output, warning, alarm, notification, 3D, 2D, etc. User Interface GAMEPLAY Game Elements Measure impact of the rules used in the game; Bonus, checkpoint, difficulty levels, dynamic difficulty adjustment, mission, tasks, dialogues, segmentation, game ends, lives, level, narrative, spatiality, matching, social interaction, social networking, time, etc. Rules GAMEPLAY Game Elements Determine the type of entity manipulation used in the game; collect, encounter, acquire, gain, etc. Entity Manipulation IMPACT Learning Goals Measurements of change in cognitive skill acquisition using Bloom's revised taxonomy covering; Remembering, Understanding, Applying, Analysing, Evaluating, Creating Cognitive skills IMPACT Learning Goals Measurements of behaviour or attitudes change on : Emotional, Moral, Relational dimensions, and/or receiving phenomena, Responding to phenomena, Valuing, Organization, Internalising values. Behaviour a Attitude & Affective Change IMPACT Learning Sustainability This requires the reassessment of learning goals at a later point in time, preferably, in the learner’s work context and evaluated by line managers. Sustainability -5 +2 +3 +5 +4

20. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Part 2 IMPACT Learning Goals Measurements of change in cognitive skill acquisition using Bloom's revised taxonomy covering; Remembering, Understanding, Applying, Analysing, Evaluating, Creating Cognitive skills Research Question: How much effective the following game elements are in determining cognitive outcomes in Icura: narrative, dialogues, tasks ? GAMEPLAY Game Elements Measure impact of the rules used in the game; Bonus, checkpoint, difficulty levels, dynamic difficulty adjustment, mission, tasks, dialogues, segmentation, game ends, lives, level, narrative, spatiality, matching, social interaction, social networking, time, etc. Rules

21. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Part 2 IMPACT Learning Goals Measurements of change in cognitive skill acquisition using Bloom's revised taxonomy covering; Remembering, Understanding, Applying, Analysing, Evaluating, Creating Cognitive skills Research Question: which element does determine longer sustainability of the cognitive outcomes in Icura: narrative, dialogues, tasks ? GAMEPLAY Game Elements Measure impact of the rules used in the game; Bonus, checkpoint, difficulty levels, dynamic difficulty adjustment, mission, tasks, dialogues, segmentation, game ends, lives, level, narrative, spatiality, matching, social interaction, social networking, time, etc. Rules

22. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Part 2 GAMEPLAY Game Elements Determine the type of entity manipulation used in the game; collect, encounter, acquire, gain, etc. Entity Manipulation PARTICIPANTS As a Gamer Determine the individuals' frequency of engagement with one or more of : Traditional Games, Entertainment Games, Learning / Training / SGs Frequency of Play IMPACT Learning Goals Measurements of change in cognitive skill acquisition using Bloom's revised taxonomy covering; Remembering, Understanding, Applying, Analysing, Evaluating, Creating Cognitive skills Research Question: How much the players’ ability influences the acquisition of cognitive skills in Icura?

23. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Summary a comprehensive and systematic collection of concrete factors worth being investigated and variables worth being measured in an experimental evaluation of SGs. can support stakeholders in testing the efficacy of games and researchers in finding new correlations involving the Learning Impact The SG-EG can be further extended and enriched with: –New factors and variables –Accepted, standard tools to measure them

24. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 “impact” of WP3 Evaluation Framework Contribution to Roadmap Final Thematic WS [qnt: target: 6] # participants [qnt] / #external to GALA [qnt] Quality of the host event [qlt] Degree of interest and usefulness for attendees [qlt] Prestigious hosting events (DigitalHeritage2013, Int. Multiconference on Modeling & Simulations, World Manufacturing Forum, ASME, Serious Play, …) 200+ participants overall New SG descriptions # of SG studies online [qnt, target: 6] Publications # of publications [qnt, target: 12] Quality of publications [qlt] Blog posts # of thematic blog posts [qnt, target: 60] New associate partners / SGS members 99 99 3131 6363 [All activities done, all targets reached!!!]

25. Final Review Meeting European Commission Information Society and Media Luxembourg, 2-3/12/2014 Thank you for your attention!