1. Directorate of Human Resources The use of the Nominal Group Technique in evaluating student experience Diana Williams OCSLD

2. Oxford Centre for Staff and Learning Development Background National Student Survey provides quantitative information about the student experience Concerns that the lowest scoring response (3.1) " /Feedback on my work is prompt/" had not changed

3. Oxford Centre for Staff and Learning Development Aims of the study: To use a systematic method of evaluation to gather qualitative data on the student experience of feedback on their assessed work To improve the assessment process for students. To explore the use of an alternative approach to the collection of student evaluation data

4. Oxford Centre for Staff and Learning Development Nominal Group Technique Originally developed as an organisational planning tool by Delbecq, Van de Ven and Gustafson in 1971 Also been used in higher education for various purposes

5. Oxford Centre for Staff and Learning Development Curriculum innovation: reactions of students to new courses in higher education (Chapple & Murphy, 1996) Assessment: to incorporate college students’ perspectives into assessment, (Farone et al., 1998), Course evaluation: investigated the strengths and weaknesses of BEd programmes (O’Neil, 1981)

6. Oxford Centre for Staff and Learning Development Nominal group technique (NGT) Structured group interview technique Takes a qualitative approach Draws quantitative estimates The

7. Oxford Centre for Staff and Learning Development Why? It generates issues that are important to the students themselves, rather than generating answers to questions posed by the evaluator

8. Oxford Centre for Staff and Learning Development Facilitates a balance of participation across members Allows equal weighting of all contributions in the final decision making stage

9. Oxford Centre for Staff and Learning Development More structure than the focus group, but still takes advantage of the synergy created by group participants

10. Oxford Centre for Staff and Learning Development Who? Second year undergraduates or first year diploma students 2/3 groups with 7-8 participants each Over lunch – 45 minutes

11. Oxford Centre for Staff and Learning Development 'How can feedback on my assessments be improved?'

12. Oxford Centre for Staff and Learning Development Data Collection 1)Generation of ideas by individuals 2)Round-robin 3) Clarification and elaboration of each idea 4) Anonymous ranking

13. Oxford Centre for Staff and Learning Development Data Analysis 1)The facilitator collects and calculates the ratings and records the cumulative rating for each idea 2) Discussion of ideas can be recorded and analysed for further qualitative data

14. Oxford Centre for Staff and Learning Development First comment recorded: /I think some of the feedback could be more prompt/

15. Oxford Centre for Staff and Learning Development A More prompt B Contact between tutors on advice they are giving C Feedback sheet – strengths and weaknesses D Individual, personal feedback – how you are doing? Where you are, what you need to do E Follow up chat after feedback F More time – technical tutorials G Group tutorial to discuss with three tutors) H 2 / 3 minutes 1:1 tutorial before main tutorial I Clarify system of marking

16. Oxford Centre for Staff and Learning Development J Relate to our personal aspirations (learn new things, learn about design) K During the year – where I stand L Peer assessment from year above M Peer assessment

17. Oxford Centre for Staff and Learning Development Rankings Participant RParticipant SParticipant M Prompt feedback 122

18. Oxford Centre for Staff and Learning Development /I know personally then I find it hard to continue with what I was doing coz I didn’t know how I had done in the previous subject so sort of holding me back/ /And also everything is related/ /unless you know if you have done well you cant really move on/ /Because in a way was there any point if I had gone down the wrong tangent is there any point in me going onto the next bit/

19. Oxford Centre for Staff and Learning Development /I know like from **** we get my unit get pretty prompt emails in the next couple of days which is good /but ummm/ / But generally with **** you know how it goes/

20. Oxford Centre for Staff and Learning Development /A feedback sheet with strengths and weaknesses that sends out something more individual / /Every individual is different its so hard to categorise somebody on a sheet like that/ /I think what we feedback could relate more to what we want to get out of the course what are our aspirations what do we want to get out of the course/ /Some are here to improve their design, to try new things, the thought process tie it in more tightly/

21. Oxford Centre for Staff and Learning Development Discuss in threes Who you might use NGT with if at all? For what purpose? How would you use it? If not discuss some of the reservations or hindrances to its use