1. A Lifecycle Approach to Evaluating MyArtSpace Giasemi Vavoula with Julia Meek Mike Sharples Peter Lonsdale Paul Rudman

2. 2 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk Outline What is MyArtSpace What is Lifecycle Using Lifecycle to evaluate MyArtSpace Results: Mobile learning for schools in museums Conclusions

3. 3 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk MyArtSpace – Situation of concern School visits to museums Visits often isolated from classroom work Learning is a cumulative process involving connections and reinforcement among the variety of learning experiences people encounter in their lives: at home, during schooling, and out in the community and workplace (Dierking et al. 2003) making the links between school and museum learning explicit, genuine, and continuous affords real opportunities for school students to have enjoyable learning experiences in both settings. (Griffin 2004) Making the visit personal and relevant Static displays, generic labels Structuring enquiry learning. Children need specific support in: planning appropriate investigations managing investigations interpreting results (de Jong et al., 2006) Design linked pre, on, and post- museum visit learning experiences P ersonalise interaction with exhibits Design enquiry-led museum learning

4. 4 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk MyArtSpace – What it is Service on mobile phones for enquiry-led museum learning Aim to make school museum visits more engaging and educational Learning through structured enquiry, exploration, connection Combines physical space (museum, classroom) virtual space (online store and gallery) personal space (mobile phones) Museum test sites Urbis (Manchester) The D-Day Museum (Portsmouth) The Study Gallery of Modern Art (Poole) About 3000 children during 2006

5. 5 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk 2.At the museum: Students are given Nokia 6680 multimedia mobile phones Use phones to collect exhibits by typing a two-letter code Prompted to type their reason for collecting Document their experience (take pictures, record sounds, write comments) All exhibits and recordings are sent automatically to a personal web space MyArtSpace – How it works 1.In the classroom: Discuss big question to explore, by collecting evidence from the museum visit 3.Back at school: Personal website shows their notes, recordings, pictures, exhibits They can view others collections, and items provided by the museum They can create personal online galleries to show teachers and classmates, and to friends and family outside school

6. 6 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk Lifecycle – What it is A methodology for the evaluation of Educational Technologies Basic premises: Start evaluation very early in the design process – and keep evaluating throughout Undertake evaluation activities at key points addressing questions: Who are the stakeholders and the users of this evaluation activity? Why is this activity being undertaken (purpose/goals)? What evaluation methods/resources will be used When should the evaluation be undertaken? How will the results be communicated? Feed evaluation outcomes into following stages, or re-iterate previous stages Lifecycle Evaluation Toolkit

7. 7 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk Lifecycle – What it is (2) Shares concepts and methods with software engineering, educational and learning technology evaluation (User-Centred Systems Design, lifecycle approach to software engineering, etc.) Emphasises the central role of Evaluation throughout learning system design, from system conception to final assessment. Provides a framework for planning evaluation activities and using their outcomes. Meek (2006), Meek & Sharples (2001) Contact: n.meek@ntlworld.comn.meek@ntlworld.com

8. 8 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk Lifecycle for MyArtSpace - Stakeholders The Sea: MyArtSpace design and development team Culture Online (UK Dept. for Culture, Media and Sport): funding body Teachers: users Students: users Local Education Authority reps: expert users / promoters Museum curators / education experts: users / clients / test sites Education/accessibility consultants: experts

9. 9 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk Lifecycle for MyArtSpace – Evaluation activities Project PhaseActivityMethods ConsultationConsultation workshopRound-table discussion ConsultationRequirements analysisWorkshop Design - BuildUsability of designsHeuristic evaluation BuildFirst user trialField study BuildSecond user trialField study DeployUsability of designsHeuristic evaluation DeployThird user trialField study DeployStakeholder consultationStructured interviews DeployTeacher evaluationQuestionnaire/interviews

10. 10 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk Lifecycle for MyArtSpace – Evaluation activities (2) Consultation workshop Development team, funding body, evaluation team/mobile learning experts Improvement of initial concept Requirements workshop Development team, funding body, teachers, LEA reps, evaluation team, museum reps, accessibility expert Educational, user and accessibility requirements Usability evaluations Heuristic evaluations of mobile phone, student/teacher web interface, museum curators web interface; at critical points in interface design Field studies Full scale trial with KS2 students at Urbis; main focus: system performance, usability Full scale trial with KS3 students at D-Day; main focus: issues from previous trial, educational experience, usability Final full scale trial with KS3 students at D-Day; main focus: educational and user experience, system impact Stakeholder interviews After-pilot deployment interviews; focus: great expectations – fulfilled? Teacher surveys Questionnaires and semi-structured interviews with teachers; main focus: what works and what doesnt, why, perceptions and attitudes, space for improvement Jan 05 Dec 06 Jan 06 Aug 05 Jun 05 May 05 Conception Consultation Design Build Deploy Market Project Timeline

11. 11 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk Lifecycle for MyArtSpace – Use of evaluation output MyArtSpace concept MyArtSpac e designs MyArtSpac e system Formulate concept Requirement s analysis Usability evaluation s Field Studies Evaluation activity System development stage Teacher evaluation s MyArtSpac e marketing Stakeholder Consultation

12. 12 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk Lifecycle for MyArtSpace – A focused framework Emergent evaluation framework for mobile learning systems and services ExpectationsRealityGaps Micro Individual activities performed during the learning experience – focus on usability/usefulness How do stakeholders expect the system to work technically? How does the system work technically for stakeholders? What were the gaps between technical expectations and reality? How does the Micro level affect the Meso level and vice versa? Meso Educational aspect of the experience – focus on learning breakthroughs and breakdowns How do stakeholders expect the system to work educationally? How does the system work educationally for stakeholders? What were the gaps between the educational expectations and reality? How does the Meso level affect the Macro level and vice versa? Macro Effect on educational practice – focus on educational transformations What was the stakeholders vision for the system? What impact / transformations does the system bring about? What were the gaps between the expected system impact and the actual impact the system appears to have?

13. 13 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk Lifecycle for MyArtSpace – Preliminary results Mobile learning for schools in museums: Micro level (individual learning activities) Students enthusiastic about creating their own content: taking dozens of pictures and recording lots of audio Text comments not very popular – in recent trial, they override text comments and record audio instead Meso level (the learning experience) Made me look at artwork more Students stop before exhibits and think – should I collect? Is this relevant? Prompted to abstract main ideas of their galleries by e.g. asking to provide a good title Durable visit output to take away Macro level (the educational experience) Connect museum visit to school learning (didnt really use museum visits for specific purposes (before), give us an easier follow up lesson to make the value of the trip more worth while) Increase engagement

14. 14 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk Lifecycle for MyArtSpace – Preliminary results (2) Example inter-level influences (micro-meso): Aggressive collecting: the more open and exploratory the learning task is at the meso level, the more overuse is made of collection activities at the micro level… … which makes harder the post-visit manipulation of collections at the micro level, which in turn hinders the learning experience at the meso level learning task needs to be well designed and appropriately constrained pre, on and post museum experiences need to be allocated enough time No explicit link between pictures and comments at micro level results in difficulty in interpreting pictures post-museum at the meso level Functionality needed: annotate picture with audio/text comment

15. 15 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk Conclusions MyArtSpace: Encourages students to stop and think about each exhibit Engaging, fun Consolidates formal and semi-formal learning experiences MyArtSpace is being re-branded as OOKL for commercial release Lifecycle: Focus on both usability and pedagogy early in the design Helped draw a full picture of the use of new technology to be understood, assessed and improved Involved stakeholders in evaluation activities early on, making it everyones business to provide input to evaluation activities and make use of the outcomes!

16. 16 / 16 WMUTE 06g.vavoula@gmail.com, g.vavoula@open.ac.uk Partners - contacts Culture Online (Department of Culture, Media and Sport) http://www.cultureonline.gov.uk/ The SEA – software development http://the-sea.com University of Birmingham (CETADL) University of Nottingham (LSRI) Urbis, Manchester Study Gallery, Poole D-Day Museum, Portsmouth Lifecycle Ltd Contact: Julia Meek (n.meek@ntlworld.com)n.meek@ntlworld.com Partners: Julia Meek, Peter Lonsdale, Paul Rudman, Mike Sharples, Giasemi Vavoula (MAS evaluation team leader),