1. Better Benchmarking: A discussion on what works and what could work better Ben Daniel (HEDC) Judy Fisher (Library) Simon Hart (Library) Emerson Pratt (ITS) Sarah Stein (Distance Learning) Quality Forum, April 2016

2. Overview 1.Discussion on recent/current benchmarking activity 2.Definitions of benchmarking 3.Our project – the process we used 4.What we have learned from our benchmarking activity 5.Where to next?

3. Discussion Handout: “Better Benchmarking past/current benchmarking activities”

4. Benchmarking: definitions “…the process of identifying best practices and learning from others. …enables a better understanding of practice, process or performance and provides insights into how improvements might be made.” Jackson, N. (2001) Benchmarking in UK HE: an overview Quality Assurance in Education, 9 (4), 218 – 235. “The self ‐ assessment activity will ultimately facilitate an institution knowing itself just that little bit better, that is, against what has been proposed as ‘good practice’ by the Performance Measures in the Benchmarks. The desired outcome is for each institution to identify their strengths and weaknesses and ways they can facilitate the actions required to make enhancements in these areas where appropriate.” ACODE (2014). Benchmarks for technology enhanced learning. http://www.acode.edu.au/course/view.php?id=5, p. 9. http://www.acode.edu.au/course/view.php?id=5

5. Benchmarking… Focus on the process…. on how results are achieved, rather than only on the outcomes Booth, S. (2015). International peer review benchmarking for quality higher education. Ako Aoteraroa (www.akoaotearoa.ac.nz)

6. Australasian Council on Open, Distance and eLearning (ACODE) (2014). Benchmarks for technology enhanced learning. http://www.acode.edu.au/course/view.php?id=5

9. ACODE Benchmarks…. 1.Institution-wide policy and governance for technology enhanced learning; 2.Planning for institution-wide quality improvement of technology enhanced learning; 3.Information technology systems, services and support for technology enhanced learning; 4.The application of technology enhanced learning services; 5.Staff professional development for the effective use of technology enhanced learning; 6.Staff support for the use of technology enhanced learning; 7.Student training for the effective use of technology enhanced learning; 8.Student support for the use of technology enhanced learning

10. ACODE Benchmark elements….  Scoping statement  Good practice statement  Performance indicators  Performance measures (on a 5 point scale)  A place to provide a rational and evidence to support your assessments, and  An area to note an initial recommendation which may be useful for future improvement.

14. Initial Repeatable Defined Quantitatively managed Continuous improvement 5 4 3 2 1 optimized measured confirmed documented ad hoc Capacity Maturity Model Paulk, M. C., Curtis, B., Chrissis, M. B. & Weber, C. V. (1993). Capability maturity model, version 1.1. Institute for Software Research Paper 7.

15. What we did 1.Followed the suggested ACODE benchmarking process. 2.Benchmarked within our departments. 3.Benchmarked as a group. 4.Agreed overall ratings across the group. 5.Discussed use of each benchmark at academic department level. 6.Summarised all ratings. 7.Summarised how the benchmarking activity has informed our own departmental reflections/planning. 8.Identified benchmarks for a smaller, more manageable set. 9.Planning for “roll out” to academic departments is now underway. 10.Benchmarking with other institutions?

16. What we have learned from the process… Ben Daniel Coordinator, Educational Technology, HEDC

17. HEDC Educational Technology: Benchmarking Logic Model

19. What we have learned from the process… Judy Fisher Learning Services Librarian, University Library

20. The Library in Context Broad definition: Technology for delivery of resources Specific definition: TEL instructional programmes deliver research skill support

21. Benchmarks: the supporting evidence Scored highly in strategy, policy and operational planning Reliant on other internal and external providers for technology support

22. Library TEL instructional programmes Reliant on the LMS Evaluation and assessment Marketing and communication Digital Strategy

23. What we have learned from the process… Emerson Pratt Teaching & Learning Facilities Manager, ITS

24. One university, many tales

25. The ACODE benchmarking tool asked us to rate ‘the university’ but in reality there is no ‘one’ answer

26. General Conclusions Difference in maturity of processes and systems through-out the university Some departments have excellent availability, technical support, & processes to support ICT integration into teaching/research. Others do not….

27. General Conclusions Strong processes for managing central IT systems such as Blackboard, Lecture Capture, Zoom and other institutional systems. While there is a lot of room for improvement… we are doing ok (no one has the ‘right’ answers)

28. What ITS is doing Creating a community of practice to mentor departmental IT staff Developing a consultation service for academics Finding ways to be involved earlier in the process of developing new programmes

29. What we have learned from the process… Sarah Stein Director, Distance Learning

30. Distance Learning many dependencies & interdependencies highlighted intersections, cross-overs, overlaps better understanding of how things happen  insights into current and future developments, operations and strategic action

31. What we have learned from the process… Simon Hart Policy, Planning and Evaluation Librarian, University Library (project manager)

32. Open Simple Flexible Real What we have learned from the process…

33. Discussion Handout: reduced list of ACODE Benchmarks to consider Questions for discussion: 1.Thinking about what you have heard today, what would you apply to a TEL or other benchmarking process within your department/unit/group. 2.What do we need to do as an institution to support better (process) benchmarking at department, institution and inter-institutional levels?

34. Contact us anytime to chat to us about our Better Benchmarking process using the ACODE Benchmarks for Technology Enhanced Learning: Ben Daniel, HEDC, ben.daniel@otago.ac.nzben.daniel@otago.ac.nz Judy Fisher, Library, judy.fisher@otago.ac.nzjudy.fisher@otago.ac.nz Simon Hart, Library, simon.hart@otago.ac.nzsimon.hart@otago.ac.nz Emerson Pratt, ITS, emerson.pratt@otago.ac.nzemerson.pratt@otago.ac.nz Sarah Stein, Distance Learning, sarah.stein@otago.ac.nzsarah.stein@otago.ac.nz Contact Quality Advancement for further information: quality-advancement@otago.ac.nz