Evaluating E-Learning Efficacy University of York, UK Wayne Britcliffe and Simon Davis Edinburgh Napier Learning and Teaching conference 14 th June 2012 What’s going on?

Drivers  Initially approached by our Department of Health Sciences: –Health Sciences have extensive and embedded online support for teaching activity –Large student numbers and a number of CPD courses as well as UG and PG courses –Devolved VLE administration/acknowledged VLE Coordinator  Approach backed by Health Sciences senior management –Their primary goal was to establish the level of online engagement from staff and students and also try and get an idea of its impact  Language and Linguistics, Law, Economics, Environment, Management –Improve dialogue and inform departmental strategy, baseline development etc

Evaluation challenges  Students range from Undergraduates on campus through to CPD distance learners  Large number of modules to ‘audit’ (e.g. HS = 217)  How to broadly classify and indicate activity without being judgemental about the effectiveness of that activity?  Establishing actual impact on learning or quantifying where value has been added can be very difficult  E-Learning evaluation not built into course design and the focus of evaluation endeavours quite wide  Difficulty in getting good survey (survey fatigue) and focus group coverage across the whole offering

Process – Step 1  Consultation –Departmental VLE coordinator liaised with departmental management  Establish reporting requirements  Get senior backing for survey/report activity –E-Learning Team liaised with the departmental VLE co’ordinator to develop evaluation plan  Scoping of evaluation aims  Identify useful review and performance indicators  Establish data collection methods

Evaluation plan Focus Establish level of on-line activity per course focussed around usage of each course’s discrete VLE module Try and establish the impact of the on-line support/activity on learning Key Questions How has each VLE module been used by teaching staff? How has each VLE module been used and received by the students? Stakeholders For feedback : The department’s teaching staff, the department’s students For evaluation : ELDT, dept’s support staff and management Time Scale/Dependencies Need established late 2011 HS draft report required by mid-March 2012 Instruments and methods Statistical querying of the VLE database Survey of staff users Survey of student users Focus groups with students Systematic audit of current VLE sites

Process – Step 2  Data Capture –Statistical querying of the VLE database  VLE administrator written queries based on stat data that is deemed useful –Survey of staff users  Develop survey questions –Survey of student users  Develop survey questions –Focus groups with students/staff  Develop focus group questions –Systematic audit of all current academic year on-line sites  Decide on ‘positioning’ criteria and also on what to qualitatively look at

Data Capture – Closer look at module audit – 3E  Systematic audit of all current academic year on-line sites

Enhance – Health Sciences Example

Extend – Health Sciences Example

Empower – Health Sciences Example 1

Empower – Health Sciences Example 2

Process – Step 3 – Analysis and recommendations EARL (reading lists)

Process – Step 3 – Analysis and recommendations  Some survey feedback from students: “Better training for teaching staff, so they are able to offer the different opportunities for as example group files/file exchange/place to save articles and other documents to make them accessible to the whole group. This would decrease the amount of s sent between groups and make communication easier.” “The Wiki section of the VLE could use clear directions on conventions and getting started.” “I don't consider myself computer illiterate, however The VLE in my opinion is extremely difficult to negotiate.”

Process – Step 3 – Analysis and recommendations  Some survey feedback from staff: “Departmental strategy for implementation of VLE in teaching should be developed (and adhered to) - based on needs/requirements of both staff and students - this could (should) be tied into staff development so that awareness of what the VLE can do and how it can support their pedagogical framework and desired course outcomes is increased.” “VLE sites are generally set up by someone else, and I'm not sure if they consult about structure, nomenclature or branding.” “I find it difficult to invest in developing and maintaining the sites due to workload and technical skills.”

Process – Step 3 – Analysis and recommendations  In Brief: Early draft recommendations –Staff support and training/student support  Work with staff (targeting programmes rather than individuals) to help improve understanding and integration of E-learning delivery. Improve student induction to on-line tools/content and support conduits –Template development  Develop templates that suit specific programmes (not one size fits all) and make sure staff understand how to use them

Moving forward  Refine evaluation processes –Bespoke survey questions/focus group questions –Create a transferable framework  Staff support and training –PGCAP: 3 E forms basis of “Intro to E-Learning session” –Bespoke dept training: “Going further with VLE” –Position local exemplars within the framework –Build evaluation into module design so it’s easier to establish actual impact

Questions? Wayne Britcliffe and Simon Davis University of York, UK

References Smyth et al., 2011, Benchmark for the use of technology in modules, Edinburgh Napier University Higher Education Academy (2008), Challenges and Realisations from the Higher Education Academy/JISC Benchmarking and Pathfinder Programme: An End of Programme Review by the Higher Education Academy, Evaluation and Dissemination Support Team, September 2008, available at content/uploads/2008/09/Bench_and_PathFinalReview pdf content/uploads/2008/09/Bench_and_PathFinalReview pdf ACODE (2007), ACODE benchmarks for e-learning in universities and guidelines for use, June 2007, available at