Presentation on theme: "Experiences of audio assessment feedback: Staff and Student experiences Carol Ekinsmyth Department of Geography University of Portsmouth"— Presentation transcript:
Experiences of audio assessment feedback: Staff and Student experiences Carol Ekinsmyth Department of Geography University of Portsmouth (firstname.lastname@example.org)
Why audio feedback might be important Teaching and learning in higher education is often confounded by dominant cultures and quality audit frameworks that prevent the achievement of these desirable goals: The sharing between teachers and learners of integrated knowledge structures (Kinchin, Lygo-Baker and Hay (2008) The breaking down of pedagogic resonance (Trigwell and Shale (2004), Kinchin et al (2008) Helping students arrive at and move beyond liminal space and develop understanding of threshold concepts (Meyer and Land, 2005)
Assessment and feedback Researchers agree that the majority of student effort (learning?) is geared towards assessment (Carless et al, 2006; Brown, 2005; Gibbs, 2006) Carless et al (2006) encourage us to engage in learning orientated assessment Smith (2007, 11) argues: If assessment is for learning, then feedback needs to focus on what the student could do differently and how to apply that learning to future work. Price (2007) and Baumard (1999) argue that the trend towards over-reliance on written feedback in HE crushes the ability to effectively encourage tacit knowledge formation and concentrates instead only on explicit knowledge
Assessment and feedback Good feedback comprises not just commentary about what has been done, but suggestions for what can be done next (Brown, 2007) And she argues; I believe that concentrating on giving students detailed and developmental formative feedback is the single most useful thing we can do for our students (Brown, 2007) Race (1999) argues...feedback quality and quantity are probably the most important factors in enhancing student learning And yet, my experience suggests that Staff are reluctant to spend a lot of time on feedback......
Why the reluctance over feedback? Time pressures Uncertainty about what feedback is for: Kinchin et al (2008) talk about more fundamental barriers – complicity between teachers and students in non-learning outcomes Correcting errors, providing guidance for future work, justifying the mark, satisfying quality assurance processes, diagnosing difficulties, the list goes on. – Price (2007)
How can audio feedback help? It cant - if teachers are unwilling to use it We perhaps need to concentrate on overcoming the barriers to teacher-engagement Audio feedback seems especially suited to feed- forward as a supplement to assessment proformas that often concentrate on feedback Used in carefully identified places in the curriculum, it may be a powerful tool in breaking down pedagogic resonance
Why? Because it enables the students to get closer to the thought processes (and thus community of practice) of the academic than written feedback does Face-to-face contact might not be better as it can be intimidating and the details often cannot be recalled by students (my research findings)
The Project GEES/LTSN (JISC Sounds Good project) Aims: To evaluate the utility, efficiency, popularity and effectiveness (amongst students and staff) of audio feedback on assignments (range of types) for undergraduate students within the discipline of Geography.
Other reasons to bother UK National Student Survey (2005-2008) shows that the area that students are least satisfied with is assessment and feedback (Race and Pickford 2007) The centrality of assessment to the student experience is widely acknowledged (Ribchester et al (2008), Brown 2005) Research has shown that on average, 90% of all student study activity is directly related to assessment (Ribchester et al (2008)) Feedback that academics provide may not be effectively used by all students (Higgins et al (2002)) Academics and students have been shown to differ on their perception of what makes good feedback (Carless (2006)) …the literature on student experiences of feedback tells a sorry tale (Handley et al (2007, p1) quoted by Ribchester et al (2008)) Shriver (1992) has shown that hearing feedback helps writers appreciate the effects of their writing on the reader (Merry and Orsmond (2007)) The technologies exist so why not experiment!
An example My feedback – A, B Probably easier to leave feedback when the work is good rather than poor.
Reactions: Student experiences Student audio files - Response D Response A Response B Response D
Student Reactions: Positive thoughts Key aspects of audio feedback that students liked were; Its more detailed nature Tone of voice (enriches experience) Clarity – You can take it in easier Not reliant on handwriting Its more personal nature – I listened carefully because it is more personal Its potential for feed-forward
Students taking more notice? I think it is more articulate than written text. I think it allows you as a marker to articulate your points a lot better than a written line of text does - and it allows you as a student to take the feedback on board more. I mean, I normally dismiss the written feedback form if Ive done really badly And you cant read ahead. You have to listen to it all. You dont know whats coming (general laughs). When youve got a sheet you can skim-read it ….
Student Reactions: Positive thoughts It is definitely more memorabl e because it is a different way Written feedback is often too brief and difficult to read I take information in better when it is in a spoken form I find it interesting to hear how you are criticising my work (tone) Overall I think it is extremely useful I really liked it Very easy to use It was the most feedback I have ever had on a single piece of work
Student Reactions: Negative thoughts It doesnt replace the written feedback Nothing visual to look at Can be too personal for comfort You cant answer back/ engage in dialogue
Student Reactions: Negative thoughts Strange at first – felt like my lecturer was in the room with me..perhaps made you more nervous.. Builds a bit of tension I found that some of the feedback was very critical and it upset me more as an audio feedback than a written feedback Personally I do prefer hard copy – Im a bit old fashioned like that … so I do think you shouldnt loose hard copy of feedback – you can keep that for life then My main dislike was that I had nothing visual to look at It was a bit more personal, therefore I feel harder to take the criticism It would wear off. It cant be used all the time – or it wont go in it wont be registered at all well
Staff Reactions: Focus group discussion There was a reluctance to like this method – why? Reactions focused on issues of: Exposure and feedback quality Feasibility/practicality Appropriateness Best applications Context limitations and litigation Language Skill acquisition
Staff Reactions: Exposure and feedback quality Do we hide behind written feedback conventions?
Staff Reactions: Exposure and feedback quality Do we hide behind written feedback conventions? When I was doing audio, I couldnt hide anything… it made me think more about what good feedback constitutes You can hide more in written feedback – you are much more explicit when speaking something so you need to think much more carefully This isnt to suggest that teachers are intellectually lazy, but under inevitable time pressures during summative marking processes, it was felt that feedback short-hand comments require less time and effort than comprehensive spoken feedback. Do feedback proformas ease or exacerbate this?
Staff Reactions: Exposure and feedback quality Written feedback, especially that coded onto generic feedback sheets perhaps leads us to write feedback in a form of short-hand that needs to be deciphered by the students. Do we teach students to decipher this short-hand well enough? Staff can hide behind this short-hand which, for reasons of lack of time or the repetition of the process, reduces the need for great qualification of opinions or development of comments. In audio feedback, you are able to develop this feedback with for example............. You do not develop your arguments thoroughly enough You need to contextualise these findings by referring to debates in the academic literature Your discussion is too anecdotal
Staff reactions: A consensus Feed-forward rather than feed-back is where the greatest strength of this method lies Not everyone however is convinced of the need for thorough feed-forward – why? Unitised curricula structures Goal-oriented teaching and learning structures Economies of practice for teachers and students (Kinchin and Hay 2007, 98) Teachers and students complicit in a cycle of non-learning (Kinchin and Hay 2007, 98)
Staff reactions: Feasibility and practicality It takes longer if you are going to do it properly (i.e. Use the method to its greatest potential) I think that if had undertaken better preparation for my 5 minutes of audio feedback, by preparing the comments in advance, then the work necessary (and time taken) would have been much greater than what I do when Im giving written feedback, and we do need to keep coming back to that My gut feeling will still be … phew…. its a lot of work I dont think it will ever be widely applicable At the moment based on my own experiences, I think the same, its beyond us but it may be because it is new, its learning a new language and new skills
Staff reactions: Practicality You do need a quiet room to sit down … we work in a fragmented way and slot in marking … to make it efficient. Sitting down to record a narrative was problematic. To have a continuous narrative for 5 minutes was difficult.
Staff reactions: Practicality and disembodiment The practicality of it, to me, is my main barrier – I think ideologically and pedagogically there are some benefits but the shear practicality, the fact that we almost disembody the feedback from the written text and then to reconnect – Im not sure there is anything lost but I expect there is… but simply the recording of it, thinking of what you are going to record, it is a two-stage process whereas when writing, it is all in one
Staff reactions: Language and Litigation It is much more difficult if you are being natural to control the language that you are using. Now in our feedback we have to work to language criteria – its a very controlled environment – excellent is a first, very good is a 2.1 – so audio feedback, its almost impossible to work within those limitations - and if you do work to them, it is pointless. The audio feedback has to be different from the written in that it can display some emotion or passion Do we need to accompany audio feedback with written guidance as a disclaimer?
Staff reactions: Applications Id be happier giving feedback in a formative sense released from the constraints of grade point criteria and the associated language We have sanitised our written feedback so it is probably less memorable. Theres clearly a role for it, you can motivate and stimulate much more easily it is much more memorable. Formative feedback Feed-forward But what of the novelty value? Id be a little wary of replacing written comments with audio feedback – not least from a quality perspective, records for External Examiners and so on
Staff reactions: Re-accessibility The re-accessibility aspect is important. They can go back to parts of the written feedback but would they listen to a 5 minute file again if they only needed a bit? Also, they have to listen to the negative criticism again if there is some, and they might not want to do this and thus may not be able to bear to listen again Will students keep it and listen to it again? How do they keep it with the work? Do they need instructions?
Conclusions/ Recommendations I Excellent for formative feedback/ feed-forward – perhaps not so useful (or necessary) for summative feedback. Staff were more cautious than students, remaining unconvinced overall of the cost-benefits but recognising great strengths and potentials if used judiciously. For summative feedback, neither staff nor students recognise audio feedback as a replacement for the rule- governed written feedback. Students in particular felt that audio feedback provided a much more detailed and richer account of the strengths and weaknesses of the work.
Conclusions/ Recommendations II Students felt that hearing feedback was more effective and memorable than reading it but both staff and students felt that the sporadic use of audio feedback was important in this respect. Thus the judicious use of use of the audio medium at the most appropriate moments would be the best approach. Staff had concerns about the need to be very careful about language and tone in audio feedback, and felt that a separation of intent between written feedback and audio feedback would be useful.
In a nutshell Excellent for feed-forward Saves time in delivering detailed feed-forward Not time-efficient for vast piles of scripts Best times to use method in the curriculum/degree programme need identifying A key problem is breaking through established economies of practice of teachers with regard to feedback Central to this is a re-evaluation amongst practitioners of the goals, possibilities and importance of feedback
Thank you Carol Ekinsmyth Principal Lecturer Department of Geography University of Portsmouth
References Baumard, P (1999) Tacit Knowledge in Organizations, London, Sage Brown, S (2007) Feedback and feed-forward, Centre for Biosciences Bulletin, 22, Autumn 2007 Brown, S. (2005) Assessment for learning. Learning and Teaching in Higher Education 1, 81- 89. Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher Education, 31, 219-233. Carless D, Joughin G and Mok M (2006) Learning-orientated assessment: principles and practice. Assessment and Evaluation in Higher Education 31, 4, 396 Gibbs G (2006) How assessment frames student learning. In C Bryan and K Clegg (Eds.), Innovative Assessment in Higher Education, Routledge, London. Handley, K., Szwelnik, A., Ujma, D., Lawrence, L., Millar, J., Price, M. (2007). When less is more: Students experiences of assessment feedback. Paper presented at the Higher Education Academy Conference July 2007. http://www.heacademy.ac.uk/assets/York/documents/events/conference/E5.doc http://www.heacademy.ac.uk/assets/York/documents/events/conference/E5.doc Higgins, R., Hartley, P., Skelton, A. (2002).The conscientious consumer: Reconsidering the role of assessment feedback in student learning. Studies in Higher Education, 27(1), 53-64 Handley, K., Szwelnik, A., Ujma, D., Lawrence, L., Millar, J., Price, M. (2007). When less is more: Students experiences of assessment feedback. Paper presented at the Higher Education Academy Conference July 2007. http://www.heacademy.ac.uk/assets/York/documents/events/conference/E5.doc http://www.heacademy.ac.uk/assets/York/documents/events/conference/E5.doc
References Kinchin I, lygo-Baker S and Hay D (2008) Universities as centres of non-learning, Studies in Higher Education 33, 1, 89-103 Merry, S., Orsmond, P. (2007) Feedback via MP3 audio files. Bioscience HEA Academy Centre for Excellence Bulletin, Autumn 2007 Meyer J and Land R (2005) Threshold concepts and troublesome knowledge (2) – epistemological considerations and a conceptual framework for teaching and learning, Higher Education, May Price M (2007) Should we be giving less written feedback?, Centre for Biosciences Bulletin, 22, Autumn, 9 Race, P., Pickford, L. (2007). Making Teaching Work: Teaching Smarter in Post-compulsory Education. Sage, London. Race, P (1999) Enhancing Student Learning, Birmingham: SEDA Ribchester, C., France, D., Wakefield, K. (2008) It was just like a personal tutorial: Using podcasts to provide assessment feedback, paper presented at the Higher Education Academy Conference July 2008 Shriver, K. (1992) Teaching writers to anticipate readers needs. Written Communication 9(2) 179-208 Smith B (2007) Is assessment really for learning?, Centre for Biosciences Bulletin, 22, Autumn, 11 Trigwell K and Shale S (2004) Student learning and the scholarship of University teaching,, Studies in Higher Education 29, 4, 523-36
Phases and progress Phase one – Experimentation 1.Using audio feedback in a range of ways (individual feedback, group feedback, generic feedback, bite-sized feedback 2.Trialling WIMBA as the enabling platform
Phases and progress Phase 2 1.Staff experiences of using audio feedback recorded in a videoed focus group. Video to be podcasted for use in staff-training workshops. 2.Student experiences and evaluations of the method elicited through Wimba voice email, focus groups and questionnaires.
Phases and progress: Future plans Phase 3 Investigation into where in the curriculum audio feedback can be most effective Phase 4 Evaluation and dissemination (including production of staff-training materials)
Constraints With limited time and large student numbers, any method will need to be measured in terms of a cost-benefit analysis Giving more detailed feedback takes time – so what methods are available that might help in this respect?
Phil Race (2008) – on feedback (www.phil-race.co.uk retrieved 12.12.08)www.phil-race.co.uk Two of his key recommendations are: Making sure they get feedback quickly enough so that they still care about it Ensuring that they get plenty of feed- forward, so they can make their next piece of work better
Audio feedback – my method Read work, make notes against the assessment criteria. Also make notes on each section of the work (perhaps paragraph by paragraph for a traditional essay). This might be in the margins of the work. Take a note of overall feedback. Make a voice recording of this feedback. Send to student. Provide summary notes on a conventional marking sheet for return to the student.
Staff Reactions: My experiences Can be daunting at first leaving the recording but you quickly gain confidence and do not feel the need to listen to the message Despite worries about tone of voice or in authenticity of sound, if you are natural and speak as you normally would, the recordings are well-received You get quicker at the whole process – at first you think it will never work. Having said this, it is more time- consuming than methods I am used to (but the quality of feedback is much greater) In spoken form, I could go into depth about issues and make suggestions that I simply wouldnt have time (or inclination) to do in written-form. Minute for minute therefore, it is much quicker to speak than to write. You need to record the feedback after you have read an
Audio feedback- Colleague As method Read work and record feedback instead of writing it on marking sheet – trying to take no longer than he would normally take in conventional marking/ feedback production
Audio feedback – Colleague Bs method Provide feedback in a PowerPoint presentation – scanning in sections of the student work and using narration record to provide explanation of points.
Hardware & Software The recordings could be made using any MP3 or digital recording devise but sending files then takes time. VLE/Wimba Voicemail can be used and for this, only a headset is needed. PowerPoint can be used for audio-visual feedback