1. Colour coding, KPIs and the Road to Damascus Shedding new light on reporting the student experience at the University of Sydney Rachel Y. Symons Quality Assurance Officer (Learning and Teaching)

2. “We can learn from experience if we are ready to adapt that experience to changed conditions” JC Masterton (1972): The Double Cross System in the war of 1939-1945

3. Scene setting Reports Marks 1 & 2 Colour coding Taxonomy upgrade NVivo7 Reports Mark 3 What next?

4. Reports Mark One (2003 – 2006) Areas of best practice Suggested improvements Ordered by percentage of comments received Not linked to SCEQ scales Redundant with introduction of KPIs

5. Reports Mark Two (2007) Aligned with KPIs –Generic Skills –Good Teaching –Overall Satisfaction List of key issues Minimal sample comments Results as raw numbers Undergraduate and postgraduate coursework combined on one page

6. Feedback and A new broom Dense Not user friendly Difficult to interpret Percentages not raw numbers Confusing to novice Need for succession planning

7. Colour coding and KPIs Easy identification of related data across all documentation –Taxonomy –Recording analysis –Communicating information Linked to KPIs / SCEQ scales Generic Skills Good Teaching Overall Satisfaction Learning Community Curriculum Assessment Workload Clear Goals and Standards

8. Taxonomy enrichment Describes the concepts Coverage of each component Answers the question Illustrates with student comments

9. The Road to Damascus NVivo7 + Boolean logic = technological conversion Combines manual and automated analysis Use of pre-set base queries Faculty customisation

10. Benefits of NVivo7 methodology No need to print source documents Simultaneous coding Time saving Direct transfer of comments into reports Reduces impact on environment

11. Revitalising the reports Consultation with stakeholders and communications experts Developed quality product Beneficial to all stakeholders More actionable than previous reports

12. Reports Mark Three (2008) KPI alignment SCEQ and CEQ data Executive summary Trends Comparative quantitative and qualitative data (2001 – 2007) Key issues (2007) Customisation by faculty Separate reports undergraduate and postgraduate coursework

13. Reports Mark Three (2008) Key results Faculty scores for Good Teaching (KPI 15; Section 2) –have increased by 8% since 2005 for current students (SCEQ) –have been consistently above the University average for current students (SCEQ) since 2000 –are lower than previous years for graduates (CEQ) –are lower than the University average for graduates (CEQ) –domestic and international students are recording similar experiences Analysis of the qualitative data from current students (SCEQ) indicates that experiences of teaching within the Faculty are evenly divided between students who consider this to be one of the best areas of their experience and those who suggested improvements. For graduates (CEQ), a higher percentage of comments received from respondents focused on areas of improvement rather than areas of best practice.

14. Reports Mark Three (2008)

15. SCEQ 2005 SCEQ 2007 CEQ 2006 CEQ 2007 Areas of best practice Domestic46% 43% 25% 30% International0% 50% 46% 33% All32% 46% 28% 30% Suggested improvements Domestic49% 52% 25% 40% International38% 36% 45% 20% All46% 28% 38%

16. Reports Mark Three (2008) Key issues for coursework students (SCEQ 2007; CEQ 2007) Areas of best practice Current students (SCEQ 2007) Domestic (n=44) International (n=32) All (n=76) Good teaching experiences Use of innovative and up to date teaching methods Ability to keep students interested in the subject Learning resources are up to date 9%19%13% Sample comments There are many dedicated staff who put a lot of effort into their teaching Lectures stimulate further thinking and understanding rather than just presenting facts and memory work

17. Have we succeeded? Improved arrangement Easy identification of areas –colour coding –collation by KPI Used to inform –curriculum review –marketing Aids in prioritising areas for improvement Incorporation of qualitative and quantitative data plus executive summary particularly helpful

18. What next? Investigate strategies for “closing the loop” Report on faculty/ student administration Cross university reports –eLearning –Research-enhanced learning and teaching –Library –International students Alignment with AUSSE Scales and survey items

19. What next? Commitment to quality improvement –Reports continue to be actionable and accessible –Taxonomy continuous w-i-p Flexibility of NVivo7 will accommodate range of analytical reports

20. “We can learn from experience if we are ready to adapt that experience to changed conditions”

21. Useful information Taxonomy; methodology; reports http://www.usyd.edu.au/learning/evaluating/open_ended_responses.shtml http://www.usyd.edu.au/learning/evaluating/open_ended_responses.shtml Contact information r.symons@usyd.edu.au 02 9351 6560