1. Effective application of quality assurance & enhancement procedures to e-learning courses Workshop at University of Reading 7 July 2008

2. Welcome

3. Project team Institute of Education Dr Harvey Mellar Dr Magdalena Jara University of Reading Maria-Christiana Papaefthimiou University of Derby Dr Dave O'Hare UoL External System Brian Sayer University of Teesside Eileen Webb Quality Assurance Agency Dr David Cairns

4. Workshop background  IOE Pathfinder Pilot – PREEL Included revision of QA procedures for e-learning courses  IOE research study on QA/QE in e-learning Examined effectiveness of internal quality assurance procedures for online courses  Reading Pathfinder DIRECT Driving Institutional Reform: Exploring Change with Technology  Quality Assurance Agency In 2006 the QAA began a new cycle of audits with a more enhancement- focused approach

5. Purposes of the workshop  Identify any issues in quality assurance/enhancement procedures that arise from the use of technology in teaching and learning  Identify how these issues are reflected in your own current institutional procedures  Identify how the internal QA/QE procedures of other institutions deal with these issues  Identify ways to make your institutional QA/QE procedures more effective in assuring and enhancing the quality of TEL.

6. Day overview 10.00Introduction 10.15Part 1: Making terms explicit 10.55Part 2: Internal QA/QE procedures for TEL 11.35 Coffee 11.45Part 3: Issues affecting effective implementation of internal QA/QE procedures for TEL 12.45Lunch 13.30Part 4: Planning actions to improve the effective application of QA/QE procedures to TEL 14.40QA/QE in e-learning SIG 15.00Close

7. Part 1 Making terms explicit

8. Technology  Technology enhanced learning  E-learning  On-line teaching

9. QAA’s framework - IQAPs  Internal quality assurance procedures Each [HEI] is responsible for the standards and quality of its academic awards and programmes. Each has its own internal procedures for attaining appropriate standards and assuring and enhancing the quality of its provision. In particular, institutions address their responsibilities for standards and quality through: the assessment of students; their procedures for the design, approval, and the monitoring and review of programmes. QAA (2003) A brief guide to quality assurance in UK higher education. www.qaa.ac.uk

10. Quality enhancement.. the process of taking deliberate steps at institutional level to improve the quality of learning opportunities....[…] … an aspect of institutional quality management that is designed to secure, in the context of the constraints within which individual institutions operate, steady, reliable and demonstrable improvements in the quality of learning opportunities. QAA (2006) Handbook for institutional audit: England and Northern Ireland The definition of 'enhancement' QAA has adopted …. leaves room for institutions to follow their own definitions of 'enhancement'. Some institutions may define enhancement as 'continuous improvement', others as 'innovation' and there may be other definitions. QAA (2007) Audit and enhancement: further guidance for institutions and QAA audit teams

11. QE and TEL  Enhancement improvement or innovation  TEL Is an innovation, hence enhancement? Need to improve TEL?

12. Task 1 (15 minutes) In each group…  What do you mean by QE?  How can you use the internal quality assurance procedures to achieve QE? Report back – one point per group

13. Swinglehurst, D. (2006) Peer Observation of Teaching in the Online Environment: an action research approach. Final Report. Centre for Distance Education

14. QA/QE – the difference Quality assurance and quality enhancement are not the same as the first is concerned with determining that objectives and aims have been achieved, while quality enhancement is concerned with making improvements.  Quality enhancement is part of a wider framework in which quality control, quality assurance, quality enhancement and transformation are stages in the management of quality  Common belief that quality assurance leads naturally to quality enhancement is not correct, as most quality assurance efforts are by and large concentrated in accountability Middlehurst, R. (1997), 'Enhancing Quality'. In F. Coffield and B. Williamson (eds), Repositioning Higher Education. Buckingham: SRHE and Open University Press.

15. Retrospective vs prospective Quality assurance may be either ‘retrospective’ or ‘prospective’ depending on the type of quality it is aiming to assure.  Retrospective QA looks into the past to make a judgement with a focus on accountability  Prospective QA is concerned with the present and future, focusing on quality as fit for purpose, and encouraging improvement  Defines QE as the internal mechanisms that an institution puts in place to continually review and improve practice Biggs, J. (2001), 'The reflective institution: assuring and enhancing the quality of teaching and learning'. Higher Education, 41, 221 - 238

16. Accountability vs improvement … external evaluations have accountability and compliance focuses rather than the encouragement of continuous quality improvement of the student experience. In most institutions where it occurs, improvement of the student experience is a function of internal review and monitoring processes, usually heavily reliant, nowadays, on student feedback, examiners reports, internal improvement audits, periodic revalidation of programmes of study and staff teams critically self-reflecting on their everyday practice.  All internal processes of quality monitoring have a greater effect on the quality of the provision than the external monitoring processes Harvey, L. (2005), 'A history and critique of quality evaluation in the UK'. Quality Assurance in Education, 13 (4), 263 – 276.

17. Part 2 Internal QA/QE Procedures for TEL

18. Internal QA procedures 1. Validation 2. Module evaluation 3. Annual course review 4. Periodic course review 5. Student representation 6. External examiner 7. Peer observation 8. Other…

19. Task 2 (15 minutes) In each group… Considering the internal QA procedures in place in your own institution, discuss:  Does TEL require new/changed internal QA procedures?  Does TEL enable new/changed internal QA procedures? Report back

20. National overview (QAA audits 2003-2006)

22. HEIs reporting variations in the application of the procedures to e-learning Procedure Total (n=129) External Examiners Student feedback1411% Student representatives Staff feedback Team meetings/staff development Annual Review119% Periodic Review108% Validation/Approval3527% Peer Review Other QA procedures

23. Variations Student feedback  Specially tailored forms  Implementation of online surveys and discussion forums to collect feedback (to improve low response rates) Annual Review  Additional issues to be considered by review team  Additional scrutiny  Additional section to be discussed Periodic Review  External assessor with experience in e-learning  To include an evaluation of the development and effectiveness of teaching, learning and assessment strategy and method

24. Validation Additional consultations/approval steps:  Consultation/approval by specialised units  Proposals seen by additional committees Sustainability:  Additional scrutiny of economic viability and availability of resources (risk assessment, contingency plans) E-learning mode of delivery:  Demonstrate appropriateness of delivery method, explain how student discussions will be facilitated Materials:  Provide sample materials for review  Internal and external peer review Validation Panel members:  At least one member of the panel needs to have expertise in the area Academic staff:  Extra checks on staff development needs

25. Coffee break

26. Part 3 Issues affecting effective implementation of internal QA/QE procedures for TEL

27. QA/QE for TEL The literature identifies differences between TEL and campus-based learning:  Disaggregated processes  Distributed teams  Distant location of students  Openness to review

28. Research study  Four case studies Postgraduate courses in a range of universities Explore how dual-mode HEIs approach the application of their internal QA procedures to their e-learning courses  For each case study QA documentation Interviews with stakeholders  Comparative examination of data: Map of issues not captured by the QA procedures Identification of those aspects of the courses which were impacting on the implementation of the procedures

29. Research results The application of the QA procedures was affected by:  Organisational context in which courses are located  Disaggregated processes  Distributed teams  Distant location of students  Openness to review

30. 1. Organizational context - detachment Online courses were often in a ‘detached’ position in their institutions, which created both a sense of autonomy and isolation  On-line courses off senior management’s agendas  Courses without central oversight  This led to failure to collect relevant information

31. 2. Disaggregated processes Distributed organisation of teams in terms of roles affected the levels of coordination and communication among team members  Unclear distribution of responsibilities  Collaboration with external and specialised units added further complexity

32. 3. Distributed teams - coordination Online courses were often taught by a mixture of full time tutors, tutors with fee-based contracts and tutors working from home or elsewhere. Course teams were scattered and course leaders were often not aware of the coordination requirements of a distributed team  On campus team with face-to-face meetings, tutors fully integrated  Course leaders tend to use the same coordination and feedback mechanisms used for on-campus staff

33. 4. Distant location of students Distant location of students also affected the quality assurance mechanisms, as it obstructed the implementation of some of the procedures in their current form (e.g. the use of student representatives)  Compensated for by a strong and trusting relationship between students and tutors  Opportunity was only occasionally taken up by teams

34. Effect on internal procedures  Annual Reviews  Module evaluations  Team meetings

35. Annual Reviews  Approached as an administrative burden, which had to be written up just for accountability purposes Senior management is perceived as not prepared/not able to understand the relevance of TEL issues  Annual review often detached from its enhancement function Perceived as an event rather than a process Teams render two different accounts, one for external consumption and another to be used internally

36. Module evaluations  Distant location of students…  low response rates  teams discarding the results as invalid, regardless of their content  Distributed organisation of teams…  unclear allocation of responsibilities  led to responses being left untouched or only superficially analysed

37. Team meetings  Distributed teams  absence of a structure of formal meetings  put at risk the team’s capacity to deal with the issues identified and the monitoring of their resolution Evidence:  consistency in the information managed by team members  mechanisms for coordination and feedback relying on informal encounters  Adjustments to overcome the limitations usually took the form of increasing the formalisation of the communication and coordination channels either online or face-to-face

38. Task 3 (35 minutes) Select one internal QA procedure In each group: Quickly review the procedure’s purpose and how should it work Quickly review how the procedure is intended to aid quality enhancement Describe any issues impacting on the use of this procedure for QA/QE arising from the use of TEL Suggest solutions Produce a one page written report: List issues identified affecting the procedure selected List suggestions for changes/improvement

39. Lunch break!

40. Part 4 Planning actions to improve the effective application of QA/QE procedures to e-learning courses

41. Techniques - research & practice  Validation criteria  Evaluation of e-learning courses  Embedded evaluation  PROPP Peer Observation  Periodic review framework

42. 1. Additional validation criteria (IoE)  Staff experience of, or training in, use of on-line teaching methods  Evaluation plan  Arrangements for the support of students with disabilities  Explicit course management procedures (including the role of administrators, and a named person responsible for quality assurance)  Arrangements for obtaining feedback from students  Explicit arrangements for tutor peer observation

43. 2. Recommendations for evaluation  Obtain feedback from all stakeholders: students, tutors, administrators, technical support  Organise frequent formal staff meetings, face-to-face and/or online, with agendas defined by the staff, and covering all key issues  Carry out evaluation as an integral part of the activities of the course, both during and after the course  Ensure that responsibility for collection and analysis of results is clearly assigned  Take advantage of the technology in use in the course to collect feedback Evaluation of e-learning courses WLE Occasional Paper 4

44. 3. Embedded evaluation  Project studying e-learners’ experiences in a mixed- mode course  Student learning is enhanced by evaluation which is concurrent with teaching Embedding evaluation tasks as part of the activities of the course encouraging students to think about their own learning and how the course design, materials and/or activities have supported them (or not) in this pr Daly, C. Pachler, N., Pickering, J. & Bezemer, J. (2006) A study of e-learners’ experiences in the mixed-mode professional degree programme, the Master of Teaching.

45. Examples of evaluation questions Writing online or in face to face seminar MASTER OF TEACHING Only two of you had prior experience of learning online before joining the MTeach, so this has been a new way of communicating to learn for everybody else in the group. These discussions form a significant alternative to talking about issues at the face-to-face days or in traditional seminars. What has it been like to ‘discuss’ by using writing to communicate with each other like this? RESEARCH SYNTHESIS FOR POLICY AND PRACTICE Conducting a systematic review is a collaborative process, involving both face-to-face meetings and online discussions/activities. The course has been designed to mimic this process - carrying out activities both in the workshops and online - how has it been to learn like this? Please offer your thoughts…

46. 4. PROPP Peer Observation  Project studying the implementation of peer observation of teaching for tutors of an online course  Development & piloting of approach to peer observation regular meetings course tutors are held at each meeting, individual tutors bring an example of a problem, issue or a project (e.g. feedback prepared for a student, transcript of a virtual seminar, marks awarded for student’s work) this material is used as the basis for discussion with other tutors supporting reflection, and challenging assumptions Swinglehurst (2008) Peer Observation of Teaching in the Online Environment: an action research approach

47. Example of a PROPP meeting Topic  How do we assess our students? Evidence:  Two student tutor marked assignments with the marks and tutor feedback Main outputs  The instructions in the assignment task were ambiguous, and so needed to be altered  The task needed to be broken down into smaller components  Written feedback should include specific examples rather than relying on general comments which may not be well understood by students  Feedback should refer back to previous feedback given to students, it is worth exploring the possibilities for enabling this more easily within the VLE

48. 5. Periodic review framework University of Reading Pathfinder DIRECT - Driving Institutional Reform: Exploring Change with Technology

49. Plenary discussion Summaries of discussions from Task 3 Discussion: How can your institutional procedures be made more effective in assuring and enhancing the quality of TEL? What obstacles stand in the way of improving these institutional procedures ?

50. QA-QE SIG  JISCmail list: ‘QA-QE SIG’ to join go to http://jiscmail.ac.uk and sign uphttp://jiscmail.ac.uk  More information: www.lkl.ac.uk/research/qa-elearning

51. Evaluation of workshop  External evaluator (Charlotte Creed)  By email: link to online evaluation (10 min max)