Evaluation of the effectiveness of games from a psychoeducational perspective Ana Cristina Almeida 5th March 2014

human sciences // technology // Potential of serious games to increase the effectiveness of learning (literature evidence in thousands of references) But many studies have not produced firm conclusions; lack of high quality empirical evidence (Hays, 2005; de Freitas & Jarvis, 2007; Girard, Ecalle & Magnan, 2013) affordances of digital environments engagement, achievement (how? Which processes) All games are serious (Ben Sawyer & Peter Smith, 2008) Feedback is a critical aspect making learning meaningful (deeper and sustained) Adequate / immediate feedback while playing enable self-regulation

Theoretical foundations Theories of learning /cognition Theory of flow (Csikszentmihalyi) Cognitive load theory (Sweller) Theory of affordances (Gibson) Experiential learning (Kolb) Situated learning (Lave; Wenger) Multimedia Learning (Mayer) … and

Cognitive theory of multimedia learning cognitive processes (Mayer, 2011)Mayer, 2011 – Selecting relevant information (words or imago) – Organizing the selected information into a coherent representation – Integrating the representation with previous knowledge

flow experiences: a measure of student engagement Gaming models Learning flow (game boost player interest in the subject; increase motivation) Invitation to think, understand and build up meaningfulness Problem-solving Games as problem spaces where gamers create immersive experiences and players use them situating their knowledge Player validates the game and extend it through its use  Metagaming (Landstrom, 2010) Following hints or the modeled path / instructions of the game design is monitoring problem solving by experts The schemata / mental model maps problem-solving Helps transfer of learning concepts and procedures Playing a game with a tutorial can activate cognitive skills that enable its reuse into new environments (represented knowledge)

Flow: from gaming models to modeling play Game-based assessment Player experience Observational criteria Problem-solving scenarios / Game context Gaming Model Flow antecendents challenges matched to player’s skill level, clear goals, unambiguous feedback, sense of control, Playability Kiili (2006) Flow experience

GBL = opportunity to foster and assess learner processes High quality assessments generally across many venues Applied in gaming Assessment strategies according to a framework (e.g., Berkeley Evaluation and Assessment Research System - Wilson & Sloane, 2000; Wilson, 2005; Scalise & Wilson, 2012): – 4 principles = building blocks to engage in formal measurement Embedded assessment becoming part of the learning activities  task design and formal callibration Interacting components required: – A model of student cognition and learning in the field of study – Tested assessment questions and tasks to observe responses – Inferences about student competence mapping back to the model

Effects of SG It is assumed that serious games influences learning in 2 ways: by changing cognitive processes and by affecting motivation. While Girard, Ecalle & Magnan (2013) in Serious games as new educational tools: how effective are they? A meta- analysis of recent studies concludes about the «impossibility of reaching any reliable conclusion concerning the effectiveness of VGs and SGs in learning»Serious games as new educational tools: how effective are they? A meta- analysis of recent studies A Meta-Analysis of the Cognitive and Motivational Effects of Serious Games (Wouters, van Nimwegen, van Oostendorp & van der Spek, 2013), found that serious games were more effective in terms of learning, and were not more motivating than conventional instruction methods. A Meta-Analysis of the Cognitive and Motivational Effects of Serious Games

Principles addressed by assessment system provide diagnostic information Assessment based on a development perspective of learning Clearly aligned with the goals of instruction Must produce valid and reliable evidence of what learners know and can do Assessment data should provide useful information to teachers and students to improve learning outcomes Self-regulated learning Boekaerts’ dual processing model describes how learning goals interact with well-being goals

basic requirements of learning environments design of the learning process with the support of interactive and immersive environments (structured as games). Games can offer an incredibly engaging environment where users ‘learn by doing’. Users make and learn from their own mistakes in a controlled environment. This trial and error based approach supports well learning and is able also to improve teamwork, social skills, leadership and collaboration (According to SEGAN Progress Report, 2013). does-it-work/

effectiveness of serious games Wouters, van der Spek & Oostendorp (2011) state that effectiveness of games the is often measured by verbal means the alternative: structural assessment (e.g., Pathfinder) measuring the learners’ knowledge organization and compare this with a referent structure which comprises three steps: knowledge elicitation knowledge representation knowledge evaluation

The underlying assumption of the structural assessment approach is that we organize our knowledge in knowledge structures containing the important concepts of a domain and the relations among those concepts (Kraiger et al. 1993). often applied in cognitive psychology; to capture underlying organization of knowledge: – conducting a cognitive task analysis – analyzing instructional experts / instructors or – simply considering the agreed core concepts in a domain material, interviewing (Trumpower et al., 2010)

Changes in the knowledge structures due to learning or training have been investigated with techniques such as: – the ordered-tree techniques (Jonassen et al. 1993) – hierarchical cluster analysis (Adelson, 1981) – relationship-judgment tests (Diekhoff 1983) – concept maps (Keppens &Hay, 2008) – multidimensional scaling (Gonzalvo et al. 1994) – network techniques (Goldsmith et al. 1991) – Also efforts have been made to integrate and fully automate the different steps in the assessment of knowledge structures (e.g., HiMatt cf. Pirnay-Dummer et al., 2010; Pathfinder, cf Schvaneveldt et al., 1985)

Properties of SG Multimodality: content knowledge presented through a combination of visual, auditory, haptic and other sensory modalities /manifestations  empirical research testing impact (multiple sensory channels, Sundar, 2008) Interactivity –  …and taken in human dynamics, let’s talk about:

Dynamic assessment Intelligence is changeable Added Theories to understand and modify cognitive status: Proximal Developmental Zone, Lev Vygotsky Structured Cognitive Modifiability, Reuven Feuerstein Interactive approach instead of static traditional way to assess cognition by mapping