1. 21/04/2015 Electronic Management of Assessment

2. Overview » Introductions » Jisc Electronic management of Assessment (EMA) project and study » Headline findings so far » Prioritising challenges - activity » From challenges to solutions » Solution scoping - activity Electronic management of assessment2

3. Electronic management of assessment (EMA) project » Recently completed – a 3 year programme exploring technology-enhanced assessment and feedback » 4 main themes: › Feedback and feed forward › Enhancing student employability › Strategies for change › Electronic management of assessment (EMA) Electronic management of assessment3 Background and context CC BY-NC-SA 2.0 jackhynes http://bit.ly/jisc-assesshttp://bit.ly/jisc-assess and http://bit.ly/jiscdsafhttp://bit.ly/jiscdsaf

4. Electronic management of assessment (EMA) project » EMA - benefits and drivers are clear » Challenges surfacing relating to process, practice and technology » Working in collaboration with the Heads of e-Learning Forum (HeLF) and UCISA to maximise the benefits technology can offer in the management of assessment Electronic management of assessment4 Background and context CC BY-NC-SA 2.0 ©comedynose via Flickr

5. The EMA study » To gather examples of practices, policies & process as well as approaches to implementation » To identify the challenges around EMA, in terms of pedagogy/culture, process or technology and to identify existing solutions where they can be found » To provide interim guidance to institutions on the state of play and effective practice with approaches to implementation » To make recommendations to institutions, systems providers and Jisc on priorities to address Electronic management of assessment5 Aims CC BY-NC-SA 2.0 ©comedynose via Flickr

6. Information sources » HeLF EMA surveys 2011, 2012, 2013 » Jisc assessment and feedback programme (2011 to 2014 involving over 30 institutions) » Other Jisc learning and teaching related projects » Literature review » Online questions (April 2014) – 90 responses from 70 different institutions (65 HE and 5 delivering HE in FE) covering all four UK nations » Think Tank (May 2014) 32 institutions represented » Interviews with learning providers Electronic management of assessment6

7. Headline findings » Nuanced responses but broad trends visible » Localised initiatives but beginning to scale up » System integration is a key problem area » EMA exposes variability in business processes Electronic management of assessment7 “The use of technology is now a fundamental part of the support for assessment and feedback practice across the sector, but there are few examples of fully integrated approaches to supporting the whole assessment and feedback lifecycle ” Jisc, 2014

8. Lots of localised initiatives Electronic management of assessment8

9. Loose system integration Electronic management of assessment9

10. 10 Variation in business processes

11. Pain points Electronic management of assessment11

12. What causes the pain? » Resistance: many stakeholders take narrow view; little joined-up thinking » Scaling up: may expose technology issues & ‘harden’ staff attitudes » Feedback: storage & access issues; good practice can be hard to scale » Practice: may be simple but process workarounds cause frustration & resentment » Technology: limited range of core products how hard can it really be? » Formative approaches: less well developed than summative » Anonymity: both ends of spectrum motivated by student pressure Electronic management of assessment12

13. A Lifecycle Model Electronic management of assessment13

14. Top 10 Challenges 14Electronic management of assessment

15. Top 10 Challenges 15Electronic management of assessment

16. Activity - the challenges 1. Review the challenges as identified in the sheets on your tables (or visit the blog) 2. Individually or in pairs tick the challenges that are causing you pain 3. As a table, discuss and rank which you think are the biggest barriers (1 = most important, 5 = least important) 4. Share on your tables any enablers/solutions to these problems Electronic management of assessment16 How?

17. Where are we now? » Review of the EMA landscape across UK HE and FE completed. » Identified and prioritised the challenges around EMA » Workshops in Dec 2014 and Jan 2015 explored possible solutions to these prioritised challenges Electronic management of assessment17 http://ema.jiscinvolve.org/

18. Where next? Electronic management of assessment18 Phase 1 March – July 2014 Landscape review providing initial identification of challenges and outline of the state of play with regards to EMA Phase 2.2 Oct – Aug 2015 Ongoing development of resources for the ‘toolkit’ based on the assessment and feedback Moving from challenge to solutions – 4 projects going forward Ongoing capture of resources – ‘EMA toolkit’ Landscape review – challenges prioritised

19. Four projects » EMA Requirements Map » Feedback hub » Reliable submissions » EMA systems integration web resource » Assessment & feedback toolkit Electronic management of assessment19

20. EMA requirements map » Project to identify, validate and specify key EMA requirements and workflows, through development of generic models that represent the diversity of practice. » Providing clarity and transparency on workflows (particularly from submission to return of grades). Helps systems suppliers better support good pedagogic practice and helps institutions review their own practice. » Develop a visual way of presenting workflows; map current systems to workflows; and engage with suppliers and developers to fill gaps. Electronic management of assessment20

21. Feedback hub » System independent tool for aggregating, organising and presenting feedback at a programme level, for both staff and students. Also enables interaction around feedback between staff and students. » Students benefit from aggregated view of their feedback to support self-reflection on progress; lecturers see a more holistic view of students’ progress and better understand an individual’s progression to identify where intervention or support is needed. Electronic management of assessment21

22. Systems integration » Resource to address lack of interoperability between marking systems and student records systems and subsequent need for ‘workarounds’ by staff. » Helps institutions find solutions to EMA systems integrations issues via improved workflows and by sharing existing integrations (use cases and code). » Gaps can be surfaced and prioritised for potential development. » A community of practice might support this resource. Electronic management of assessment22

23. EMA toolkit » Web-based toolkit of resources: practices, processes, policies, technical integrations etc based around the assessment and feedback lifecycle. » Examples of solutions, enabled by technology, based on pedagogy and underpinned by research, that can be repurposed for local contexts. » Addresses the question of ‘what does good assessment design look like?’ and enhances assessment literacies of staff and students. » Navigation routes through the toolkit e.g. by identified workflow and stakeholder groups (academic staff, administrative staff, student Electronic management of assessment23

24. Activity Choose one project area to work on The aim is to help to shape the detail of these projects. Discuss: » What would you like to see from this ‘solution’ and how might you use it? » What do you have that you can contribute? » How can we best reflect the needs of HE and FE within this project? » Sign up to get involved: http://ema.jiscinvolve.org/wp/2015/04/05/solution-development- is-under-way-can-you-help/ Electronic management of assessment24

25. Activity The aim is to help to shape the detail of these projects. Discuss: » What would you like to see from this ‘solution’ and how might you use it? » What do you have that you can contribute? » How can we best reflect the needs of HE and FE within this project? › Toolkit: http://bit.ly/EMAyorkhttp://bit.ly/EMAyork › EMA requirements map http://bit.ly/sysyorkhttp://bit.ly/sysyork › Feedback hub scoping activity http://bit.ly/feedyorkhttp://bit.ly/feedyork » Sign up to get involved: http://ema.jiscinvolve.org/wp/2015/04/05/solution-development-is- under-way-can-you-help/ Electronic management of assessment25 Choose one project area to work on:

26. Find out more: » Lisa Gray lisa.gray@jisc.ac.uk lisa.gray@jisc.ac.uk » Landscape report available from: bit.ly/jisc-ema bit.ly/jisc-ema » Join the conversation on the blog: ema.jiscinvolve.org/ ema.jiscinvolve.org/ and on twitter #jiscassess » Join the mailing list: www.jiscmail.ac.uk/tech- enhanced-assessment » Guides, case studies, videos and other resources on assessment and feedback: bit.ly/jisc-assess bit.ly/jisc-assess 26Electronic management of assessment

27. Any questions?