1. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly GaLA Game and Learning Alliance The European Network of Excellence on Serious Games SIG3.5 – SGs for Humanities & Heritage user studies Michela Mortara, Chiara Eva Catalano, CNR – IMATI Jeffrey Earp, CNR-ITD

2. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Overview Icura [ongoing, 50%] Playing History – The Plague [started, 20%] Playing History – The Plague [started, 10%]

3. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly USER STUDY #1: ICURA

4. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Game description Educational objective: cultural awareness - Japanese etiquette Genre: adventure Environment: 3D Platform: PC Intended educational context: informal (at home) Developer: TU-Wien Duration: less than 1 h Target: secondary school and higher Previous studies: evaluation done by the developers at the Wien university on a group of 20 students. Reports on papers and PhD. thesis

5. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Previous evaluation Pre-questionnaire (demographic data & game habits) Pre-test with 12 questions ( on language basics, behaviour & etiquette, culture and society) Game session Post-test (same 12 questions) Final questionnaire about user satisfaction Observations: On average from +5 to +10 correct answers btw pre and post –Selective attention bias Learning gain linked to the way the content is communicated (information agent vs quest) Lacking estimate of retention

6. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Our setting Same phases but different questions in the pre-test to avoid the selective bias Higher participant number (CNR, RWTH, ORT, DIBE) Open answers to assess higher level in Bloom’s taxonomy (post questionnaire) e.g.: What have you learnt from the game and how (through what mechanics)? Did the game increase your interest towards Japan/ Japanese culture? Would you read more about Japan? Would you visit it? How would you summarise Japanese culture in 3 adjectives? What can you deduce about Japanese culture from the facts you learnt in the game? How would you compare Japanese and Western culture? Plan to repeat the post test after a few months to evaluate retention

7. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Current status CNR session –10 university students –Colleagues&friends RWTH session –17 university students ORT session –12 people –students DIBE session

8. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Partial outcomes

9. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Partial outcomes

10. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Partial outcomes

11. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly PLAYING HISTORY – THE PLAGUE

12. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly The game Educational objective: history, historical reconstruction Genre: adventure Environment: 3D Platform: browser (Unity Web player) Intended educational context: formal (at school) Developer: Serious Games Interactive Duration: nearly 3 hours Target: 8-13 Previous studies: evaluation done by a master student with two primary school classes in Denmark. Report in her Master thesis (in Danish).

13. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly The experience: a Joint Research Action JRA involving CNR-IMATI (SIG3.5), CNR-ITD (TC2.8, T6.3), SGI and DIS (Deledda International School) in Genova Objectives: –explore the suitability and effectiveness of serious games in the field of humanities and cultural heritage in schools; –evaluate the educational efficacy of the game and the level of student engagement and enjoyment triggered by the learning activities; –investigate critical aspects of SG deployment in formal ed./school contexts, especially teacher’s role –examine cultural/educational issues related to game’s adoption in Italy (game produced in northern Europe and set in medieval Italy)

14. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly The context School: English-speaking school in Genoa following International Baccalaureate & IB Middle Years programmes Classes: 2 classes Year 7 (11-12 year olds). Approx. 35 pupils Deployment settings: computer lab, classroom, home access

15. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Research framework Approach: qualitative, design based research – DBR CollectiveDBR Collective Tools: –question matrix for gathering teachers’ GBL/ICT experience & attitudes, plus their expectations, intentions and goals for the experience –pedagogical planning tool (teachers document intentions/goals of intervention and plan learning activity sequence) –researchers’ observation sheets for monitoring classroom activities –pupils’ pre/post tests (in teacher’s resource pack from SGI) –mind map recording pupils’ collective pre/post knowledge/awareness –teachers’ monitored focus group –teacher questionnaire –reflection sheet for recording researchers’ attitudes, intentions, goals and interventions in the experience

16. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Activity Phases 1 & 2… 1. Teacher preparation –introduction to the experiment, the game and its teaching resource pack –teachers’ group discussion of matrix questions (researcher compiled) –tech. training session (game’s teacher functions; ITD pedagogical planner) –teachers consolidate pedagogical plans 2.Deployment –pupil’s individual pre-test & collective brainstorming of pre-knowledge (making collective mind map) –class briefing: game presentation + play –gameplay sessions in pairs (plus possible home play) –pupil summative test & debriefing (re-examining collective mind map) =total 4/5 school periods with researcher monitoring & support

17. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly … Activity phases 3 - 6 3.Teacher debriefing –focus group (inc. re-examination of matrix questions and pre- deployment responses, and of pedagogical plan/s) –final questionnaire 4.Analysis of data gathered from phases 1-3 5.Follow up –Repeat of post test to evaluate pupil retention 6.Final data analysis and reporting

18. Madrid April 15-17, 2013 European Commission Information Society and Media General Assembly Current status Series of organisational meetings held involving CNR, DIS & SGI Activity phase 1 commenced –introduction to teachers –discussion / compiled of teachers’ of question matrix Issues with running game on DIS network have delayed commencement of deployment sessions with pupils Solution: DIS agreed to move students to CNR (2 sessions in May)