College of Science and Engineering Evaluation of the Learning and Teaching Strategy: The Way Forward? Velda McCune Centre for Teaching, Learning and Assessment

The current evaluation strategy Questionnaire 1 - given at the start of the semester responsible learning possible influences on responsible learning from the students Questionnaire 2 responsible learning perceptions of the course Qualitative data group interviews in selected courses

Advantages of the present strategy Allowed us to demonstrate correlations between - responsible learning and student influences (e.g. intrinsic motivation positively related to RL) responsible learning and students perceptions’ of courses (e.g. informative feedback positively related to RL) Qualitative data offered richer insights into the students’ experiences e.g. the value of opportunities to test understanding (and there is more to come from these data)

Advantages of the present strategy (cont’d) Provided an indication that the courses implementing the strategy were typically well received Provided a positive view of students’ motivation Allowed some specific feedback to course teams (But there is more to do here) Could be integrated with a College-wide evaluation process

Disadvantanges of the present strategy There are often a lot of missing data from the questionnaires It is difficult to infer causality because - courses are typically only modified in modest ways we typically don’t have access to pre and post intervention data courses not officially involved in the strategy are often quite in line with it missing data makes comparing the same student across courses problematic Struggling or failing students are the least likely to complete questionnaires Qualitative data is really needed to make sense of the questionnaire findings but it is time consuming and costly to collect and analyse

Disadvantanges of the present evaluation strategy (cont’d) The Learning and Teaching Strategy covers a wide range of possible changes to courses which makes it difficult to have a clear cut picture of its impact It is rare to show clear cut findings with such measurement instruments in large first year courses The current questionnaire does not cover all of the areas in the National Student Survey We do not have the staffing levels to scale up this evaluation much more and our turn around time on the data could be better There is a question as to whether more data collection of the same kind will really add to what we know

Alternative 1 - The same overall approach focused on fewer courses This would allow more complete quantitative data to be collected (e.g. turning up in person to collect it, following up non-response) This would allow richer qualitative data to be collected (e.g. more interviews with students, interviews with staff) This would allow a more reasonable turn around time on the data This might allow us to do additional types of analysis (e.g. looking at grades or prior qualifications)

Alternative 2 - In depth development and evaluation case work in very few courses This would allow a very detailed picture of a small number of courses, which could be written up so as to show their potential relevance to other courses This would allow a more integrated approach to educational development and evaluation work, which would make successful substantive changes more likely The process can be tailored and iterative (e.g. questionnaires can be designed to tap into particular innovations) This could only be done in very few courses e.g. one per semester

Alternative 3 - Focus on the experiences of particular groups of students A focus on the most successful students would help us to understand the roots of responsible learning (e.g. in their prior experiences or attitudes) Data from successful students could be worked up into guidelines for all students A focus on struggling or failing students would help us to understand what might get in the way of responsible learning It would be very important to look at the effects of courses through the eyes of these students (and not to assume a ‘good’ student ‘bad’ student model)

Alternative 4 - Focus on the experiences of academic staff It may be that the most measurable impacts of the strategy will be on staff attitudes and expectations Academics views of themselves as teachers are fundamental to how they approach their teaching We could collect data about what supports or hinders educational change

Alternative 5 - Focus on a specific topic for a semester (e.g. feedback) Would allow an in-depth approach to a topic which might support faster progress Data collection could focus on courses which were of particular interest on the topic Would allow time for wider literature search Would allow time to make more of examples of good practice from other HEIs