Student Surveys - Best Practice Eric Bohms “If you can't measure it, you can't manage it” Peter Drucker – Drucker Institute

Student Surveys - Best Practice Session Agenda: Best Practice Survey Paper vs. Online: the ongoing debate Best practices for HE survey administration Ideas for driving cost and efficiency saving Why surveys? The ‘Audit’ culture vs. the ‘Feedback’ culture

Student Surveys - Best Practice Population: UK Quality Managers and University Administration staff at 58 institutions 2010

Paper vs. Online Advantage Disadvantage Instant feedback High response rate Easy for the student Low deployment cost Easy to administer Location independent Survey Administration Printing costs Hand writing Low completion rates (statistically significant) Strong opinions likely Amend green thing Amend arguments 5

Online Surveys Paper Data Capture Hybrid Add-on Survey Outsourcing The Debate Continues... Online Surveys Paper Data Capture Hybrid Add-on Survey Outsourcing Amend green thing Amend arguments 6

Things to Consider When Managing Surveys in Higher Education: Survey Best Practice Amend green thing Amend arguments 7

Survey Best Practice Use in-class time Treat survey like an assignment Give proper time to complete Amend green thing Amend arguments 8

Most likely to complete Survey Best Practice Paper: In class surveys: ‘herd Instinct’ Most likely to complete Satisfied Dissatisfied Amend green thing Amend arguments +30% higher response with in-class paper surveys over email Online surveys 9

Most likely to complete Survey Best Practice Online: In class surveys: ‘herd Instinct’ Most likely to complete Satisfied Dissatisfied Amend green thing Amend arguments Same principle: higher response with in-class Online surveys over email to participants 10

Survey Best Practice Amend green thing Amend arguments 11

Why is obtaining high response rate important? Survey Best Practice Why is obtaining high response rate important? Institutional Research / Overall Student Satisfaction Survey Universe The University Amend green thing Amend arguments Possible to draw some conclusions from a low response sample 12

Why is obtaining high response rate important? Survey Best Practice Why is obtaining high response rate important? Course & Module Evaluation Survey Universe A 25% response rate from a class of 20 is only 5 students who’s opinions may differ greatly The Course Amend green thing Amend arguments The challenge is gaining representative data from each and every course or module 13

Course & Module evaluation is not easy Survey Best Practice Course & Module evaluation is not easy Its best in evaluating a course and the teaching to gain a majority opinion 50%+ response It is important to be able to compare to show improvement (against dept, university average, over time) Online survey systems fall short - average 15-30% response rate Amend green thing Amend arguments 14

Survey Best Practice Other considerations Gain stakeholder buy-in as the key to participate – The ‘myth of survey fatigue’ Create surveys that make sense and report on only what you need to know Reinforce the concept of anonymity to get honest feedback Amend green thing Amend arguments 15

Efficiency and cost savings Ways to drive efficiency and cost savings for surveys in Higher Education Amend green thing Amend arguments 16

#1 Centralise survey administration Efficiency and cost savings #1 Centralise survey administration Avoid ‘academic run’ quality enhancement Run research projects on a common platform Know what surveys are being run at all times Amend green thing Amend arguments 17

Efficiency and cost savings Amend green thing Amend arguments 18

Efficiency and cost savings Amend green thing Amend arguments 19

Individual Academic with 5 courses = 1 day per semester Efficiency and cost savings Academic Survey Design Data Capture Analysis Deploy Survey Academic run survey admin: Amend green thing Amend arguments Individual Academic with 5 courses = 1 day per semester 20

Department with 50 academics Efficiency and cost savings Dept 1 Academic run survey admin: Amend green thing Amend arguments Department with 50 academics 21

20 departments / 1000 instructors = 1000 teaching days per semester Efficiency and cost savings Arts Business School Engineering 1 1 1 Law School Languages Sciences 1 1 1 Amend green thing Amend arguments 20 departments / 1000 instructors = 1000 teaching days per semester 22

Efficiency and cost savings Main point: Funding cuts to teaching means less time and more work for academics 1000+ teaching days per semester is better spent teaching rather than on survey admin Amend green thing Amend arguments 23

Efficiency and cost savings 3 weeks .5 Centralised survey admin: Amend green thing Amend arguments Best practice example: London School of Economics 23 Departments /1000+ staff / 9000 students Complete survey administration 15 days per semester (Deployment: 4 days / Data Capture: 10 days / Reporting: 1 day) 24

#2 Create standardised questionnaires Efficiency and cost savings #2 Create standardised questionnaires Develop compulsory question sets for lectures and modules Focus on teaching & learning and the student experience Biology English Teaching Quality & the Student Experience Amend green thing Amend arguments 25

#3 Standardise core reporting functions Efficiency and cost savings #3 Standardise core reporting functions Develop and implement a simplified reporting strategy Supply stakeholders only with what they need Standardised reporting allows for comparison to show trending over time Amend green thing Amend arguments 26

#4 Utilise advanced technology Efficiency and cost savings #4 Utilise advanced technology End of the day: Surveys can be a powerful tool for making decisions and transforming an organisation If you are invested in a survey system that produces low response rates it is not going to meet the diverse requirements of Higher Education institutions Survey administration should be a second job not a day job Amend green thing Amend arguments 27

Quality Assurance (Audit) Why Surveys? Quality Assurance (Audit Culture) Quality Enhancement (Feedback Culture) Quality Enhancement Quality Assurance (Audit) Inhibits learning from outcomes Promotes incremental improvement of academic practice Discourages risk taking (low risk appetite) Encourages and requires frank reporting Facilitates transformational change to surface & deep conditions for learning Supports and manages risk taking QAA EESC OFFICER Amend green thing Amend arguments External Examination Sub Committee Officer “(There is a need for) a more holistic view of how a University’s administration and procedures affect a department’s teaching.” Sue Wright, Enhancing the Quality of Teaching in Universities 28

Quality Assurance vs. Enhancement EvaSys is designed to provide a ‘holistic’ platform to drive quality enhancement efforts across the organisation facilitating continuous improvement Develop Process Implement Process Evaluate Process Plan for Outcomes Amend green thing Amend arguments 29

Student Surveys - Best Practice Eric Bohms Presented by: Philip Wyse Conclusion Student Surveys - Best Practice Eric Bohms Presented by: Philip Wyse Questions? Thank you for your time!! EvaSys Best Practice Blog: www.ericbohms.wordpress.com