Presentation on theme: "Transforming assessment through the tenets"— Presentation transcript:
1 Transforming assessment through the tenets Sally Brown and Brenda Smith23 April 2013
2 This session aims to offer: Some thoughts on the nature of assessment as a transformative process to enhance student learning;An overview of the six tenets;A review of how tenets one and two integrate holistically with the other tenets;Some practical suggestions to support your work based on tenets 1 and 2;Opportunities to develop your work and share ideas and issues so you can start to make some meaningful changes at your institutions.
3 Background to ‘A marked improvement’ Stems from firm evidence that assessment is not‘fit for purpose’ (Race 2010, Brown, 2010);Aims to take a radical approach, recognising that it is time for significant reappraisal of assessment policy and practice (ASKe, Weston Manor Manifesto);Builds on expertise, perspectives, and previous work;Takes an evidence-informed approach;Encourages for assessment to be seen as an integral part of the learning experience.
4 Why change is needed QAA subject reviews Burgess Report - “system no longer fit for purpose” (p5)QAA - “…it cannot be assumed students graduating …. will have achieved similar standards” (2007)Media accusations of dumbing down and grade inflationNational Student Satisfaction Survey“There is considerable scope for professional development in the area of assessment” (Yorke et al, 2000, p7)Students become more interested in the mark and less interested in the subject over the course of their studies (Newstead 2002, p2).
5 Tenet 1“The debate on standards needs to focus on how high standards of learning can be achieved through assessment. This requires a greater emphasis on assessment for learning rather than assessment of learning”
6 Why change is neededOur current systems focused on marks and grades are not working:Belief that it is possible to distinguish the quality of work to a precision of one percentage point (Elander & Hardman, 2002);Belief that double-marking will ensure fairness and reliability (Polani, 1998);Belief that consistency can be achieved through conformity, and simple numerical rules (e.g. level 1 essay 3,000 words, level 3 essay 5,000; or no more than two pieces of assessment per module);The combination of scores, which obscures the different types of learning outcome represented by the separate scores;The distortion of resulting degree classifications by the application of idiosyncratic institutional rules (e.g. Armstrong et al, 1998).
7 Assessment for learning Students developing as learnersInformal feedbackFormal feedbackPractice, rehearsalFormative and summativeAuthentic assessmentSambell, K., McDowell, L. and Montgomery, C. (2012) Assessment for Learning in Higher Education. Abingdon, Oxon: Routledge.Holistic model, an overall approach (rather than particular strategies or techniques that might be applied)Term ‘learning-oriented assessment’ might be used (see Carless  Learning-oriented assessment: Principles, practice and a project. In L. H. Meyer et al (Eds.), Tertiary Assessment & Higher Education Student Outcomes: Policy, Practice & Research (pp79-90). Wellington, New Zealand: Ako Aotearoa)Six principlesAuthentic assessment – engaging, meaningful assessment tasksStudents developing as learners – effective attributes and skills to self-assess and evaluate their own learningInformal feedback – Encouraged, used throughout a programme (e.g. in-class group discussions, peer-review)Formal feedback – Range of forms of feedback, used at a number of stagesPractice, rehearsal – Students need opportunities to learn and practice (e.g. using linked assignments)Formative and summative – Providing an appropriate mix of these two key types of assessments(Sambell, et al, 2012)Adapted from Sambell, McDowell and Montgomery (2012, p5)
8 We need to focus on high standards of achievement and to clarify what we seek “Even when lecturers say that they want their students to be creative and thoughtful, students often recognise that what is really necessary, or at least what is sufficient, is to memorise” (Gibbs, 1992, p10)We need to emphasise for students the importance of spending rich and purposeful time on task, not just going through the motions
9 Key questions: To what extent and how do you evidence this? Are you confident that assessment tasks demand high standards of learning?Is assessment for learning given emphasis in relation to assessment of learning?Do you ensure an appropriate balance between formative and summative assessment?Is assessment and feedback planned within and across a programme to ensure appropriate student preparation and practice before summative assessment takes place?(From ‘A marked improvement’)
10 Tenet 2“When it comes to the assessment of learning, we need to move beyond systems focused on marks and grades towards the valid assessment of the achievement of intended programme outcomes”
11 Why change is neededMeaningful understanding of standards requires both tacit and explicit knowledge (O’Donovan et al. 2004)“We can know more than we can tell” (Polanyi, reprinted 1998, p.136)Verbal level descriptors are inevitably ‘fuzzy’ (Sadler 1987)There is a cost (in terms of time and resources) to codifying knowledge which increases the more diverse an audience’s experience and language (Snowdon, 2002)Tacit knowledge is experience-based and can only be revealed through the sharing of experience – socialisation processes involving observation, imitation and practice (Nonaka, 1991).
12 Key questions: To what extent and how do you evidence this? Is their an emphasis on assessment for learning over systems focused on marks, grades and reliability?Does the assessment design process ensure valid assessment of the intended learning outcomes?Is there a trade off between reliability and validity of assessment?Are assessment decisions in relation to design, development and variety made within a programme context and focused on learning outcomes?(From ‘A marked improvement’)
13 Tenet 3“ Limits to the extent that standards can be articulated explicitly must be recognised since ever more detailed specificity and striving for reliability, all too frequently, diminish the learning experience and threaten its validity. There are important benefits of higher education which are not amenable either to the precise specification of standards or to objective assessment”(for further discussion at later targeted forums, but here as part of a holistic discussion of the tenets as a whole)
14 Why change is neededThe most significant factor in student academic success is student involvement fostered by student/staff interactions and student/student interactions (Astin, 1997)The only common factor in a study of departments deemed excellent in both research and learning and teaching is high levels of student involvement (Gibbs, 2007)“Participation, as a way of learning, enables the student to both absorb, and be absorbed in the culture of practice” (Elwood & Klenowski, 2002, p. 246)
15 Tenet 4“Assessment standards are socially constructed so there must be a greater emphasis on assessment and feedback processes that actively engage both staff and students in dialogue about standards. It is when learners share an understanding of academic and professional standards in an atmosphere of mutual trust that learning works best”(for further discussion at later targeted forums, but here as part of a holistic discussion of the tenets as a whole)
16 Why change is neededImportant aspects of complex, high-level learning outcomes can only be achieved when students are allowed time to ‘come to know’ the standards in use by the community:Slowly learnt academic literacies require rehearsal and practice throughout a programme (Knight and Yorke 2004)The achievement of high-level learning requires integrated and coherent progression based on programme outcomesWhere there is a greater sense of the holistic programme students are likely to achieve higher standards than on more fragmented programmes (Havnes, p. 2007)Students need to engage as interactive partners in a learning community, relinquishing the passive role of ‘the instructed’ within processes controlled by academic experts (Gibbs et al, 2004)
17 Tenet 5“Active engagement with assessment standards needs to be an integral and seamless part of course design and the learning process in order to allow students to develop their own, internalised, conceptions of standards and monitor and supervise their own learning”(for further discussion at later targeted forums, but here as part of a holistic discussion of the tenets as a whole)
18 Why change is neededChanges in higher education (e.g. massification, reduced unit of resource, expectations of increased productivity in staff) threaten the ‘health’ of disciplinary communities and their ability to share and exemplify professional judgement.There has been slow progress in the professionalisation of university teachers.There has been limited attention paid to professional assessment practice.Reliance on the external examiner system to mediate standards within the system is misplaced (Newstead and Dennis,1994).If some aspects of high-level learning can only be assessed using professional judgement then we need to ensure that judgement is indeed – professional!!
19 Tenet 6“Assessment is largely dependent upon professional judgement and confidence in such judgement requires the establishment of appropriate forums for the development and sharing of standards within and between disciplinary and professional communities”(for further discussion at later targeted forums, but here as part of a holistic discussion of the tenets as a whole)
20 Why change is neededThe current approach to assessment standards is inadequate;The quest for reliability is getting in the way of learning;We should be enabling students to achieve high level, complex learning;Active involvement by staff and students in the learning community is essential to reach common understandings of assessment standards;To achieve change there may need to be a review and evaluation of the allocation of time and resources within HE.
21 Task: working in institutional teams Using the framework on the next slide, focus on tenet one to interrogate your own practices to date and where you can make improvements;Identify between three and five ways in which you would like to make changes in assessment practice to make your assessment focus more on assessment for learning.Prepare some summary bullet points on a flip chart sheet for sharing with other HEI teams.
22 Tenet 1 – Key Points Assessment for learning Learning and assessment should be integrated, fully aligned, challenging & on-goingAssessment should build on previous knowledgeThere should be a balance between formative and summative assessmentStudents need time for preparation & practiceA range of approaches to feedback are neededStudents need to give and receive feedback and feed-forward and engage in a dialogue re their learningTasks should involve the active engagement of students developing the capacity to find things out for themselves & learn independentlyAre your students enabled to broaden their skills, knowledge and experience?Are a range of assessment methods used and constructively aligned?Is your assessment linked, cumulative and challenging?Are your students given opportunities to prepare and practice?Do your students engage in self and peer assessment & feedback?Do you help students and staff to understand the role of assessment and feedback in learning?Do your assessment tasks encourage independent thinking?Is your assessment spread throughout the academic year?
23 Task: working in institutional teams Using the framework in the next slide, discuss with colleagues what you consider your top priorities to work on from the questions and suggestions provided there;Identify ways that you would measure and evaluate changes as a result of the actions you plan to take;Ask a spokesperson to be prepared to report back on this in plenary.
24 Tenet 2- Key Points Ensuring assessment is fit for purpose Systems that focus on marks and grades should be reviewedEmphasis should be on assessment validityStudents should be offered a choice of assessments suited to their needsAssessments should demonstrate students have achieved the programme learning outcomesCourse/programme teams needs to collaborateAssessment methods should be chosen for more than reliabilityDo your students summative marks reflect the achievement of the programme learning outcomes?Is formative assessment built into your assessment cycle and does it feed forward?Do your colleagues work collaboratively across a programme to ensure coherence and validity?Do your staff ensure assessments are linked to the learning outcomes?Are you confident that students being marked by different people or the same people at different times (inter & intra-tutor reliability) will achieve equivalent marks?Does your University have a strategy for staff engagement and development in assessment and feedback to ensure up-to-date knowledge & experience?
25 Programme learning outcomes should reflect what students should achieve Making it clear to students what is expected of them;Making it clear to teachers what students are expected to learn in their own and other modules; Helping teachers to select the most appropriate teaching strategy for the intended learning outcomes e.g. lecture, seminar, tutorial, group work, discussion, student presentation, laboratory work.Helping teachers to select the most appropriate assessment style to assess the achievement of the learning outcomes, e.g. project, essay, performance assessment, multiple‐choice questions, exam;Having a focus on programme learning outcomes – staff therefore need time to collaborate;Are you confident that students being marked by different people or the same people at different times (inter and intra-tutor reliability) will achieve equivalent marks?
26 Methods of Assessment Assignments and the learning opportunity they offer Communicate information and ideas in other than the written wordBecome more adept at writing in different forms, formats and ‘genres’To understand the benefits and challenges of working collaboratively
27 Methods of Assessment Assignments and the learning opportunity they offer Communicate information and ideas in other than the written wordSeminar, poster, video, video conferencing or other multimediaBecome more adept at writing in different forms, formats and ‘genres’Design or proposals; book reviews; case report/study; web pages; journal articlesTo understand the benefits and challenges of working collaborativelyGroup problem solving; joint book reviews; team presentations; role plays; group exhibition
28 Balancing and timing formative and summative assessments Is the schedule of assignments and the feedback on these assignments made clear to students across the year?Do you provide multiple opportunities for small low stake assessments with feedback?Are your students given opportunities to prepare and practice?Is there an appropriate balance between summative and formative assessment?Is assessment and feedback planned within and across programmes and are staff given time to ensure this happens?University ofPoppleton2013
29 Total study effort = 68 hours Figure 1: A student’s estimate of his weekly study effort (in hours) on a programme with no assignments and an exam in week 12Total study effort = 68 hoursGibbs (2010)
30 Total study effort = 78 hours Figure 2: A student’s estimate of her weekly study effort (in hours) on a program with three assignments due in weeks 4, 7 and 10 and an exam in week 12Total study effort = 78 hours(Gibbs 2010)
31 Assessment for Learning Do your students always know what the criteria are for all work they are given? How do you help students to understand the criteria? Could the students re-write the criteria in their own words?
32 Key issues in feedback – develop the capacity to use feedback effectively Students can’t read our writingThere is too much emphasis on grades and marks at the expense of learningThe feedback given is often not very useful and comes too lateStudents are not actively encouraged to self reflectLittle or no use is made of peer/self assessment and feedbackLittle dialogue takes place around feedbackStudents have little opportunity to collate feedback over time and act upon itMost feedback does not feed forward, it only tell students what they have done that is incorrectVery little use is made of feedback as a normal part of the learning and teaching processStaff vary in their approach to feedback
33 Some thoughts on promoting the engagement of students by helping them to get confident with assessingGive students in groups a selection of essay or other relevant piece of workExcellent, Good, Pass, Near pass, FailLet the students decide individually which is which and then compare notesIn groups get them to tease out the differences between them and express them as criteriaProduce suggestions for writing an effective essay or relevant piece of workGet them to write appropriate feedback commentsExchange the comments with a different group and ask them to comment on the feedback. Is it helpful, would they know what to do differently, does it feed forward etc?What suggestions would they give to the group to enhance the feedback comments?
34 How you might help colleagues develop the capacity to use feedback effectively In small groups exchange your assignments and compare the comments receivedDo you understand the comments, if not discuss what you think they might mean?How would you respond to them?What would you do differently next time if you redid the same assignment and applied the learning to a new and different assignment?Return marked work back to the individual but with no grade or feedbackGet them to mark their own workCollect in their grades (Pass a sheet round)9 out of 10 students will be within 5%Arrange to talk to those where the difference is more than 5% or one grade
35 Saving time – a range of approaches to feedback need to be in place Giving generic feedback within 24 hours on the VLEOffering face- to-face feedback with whole group/small groupsUsing peer feedback in groupsUsing technologies including:A sound file – AudacityDragon softwareStatement Banks“Sounds good project”Encouraging students to use self feedbackHelping students to recognise that feedback can come from many different sources (including themselves) and given in different ways
36 Involve the students- Feedback as a dialogue For Student CompletionFor Staff CompletionThese are areas of my work that I think are good for the following reasonsPlease comment on the following areas of workWhat I want to improve or do differently next timeThe mark I think this piece of work deserves is“When I get to sit down with my tutor and discuss my work – that’s probably the best form of feedback I can get”
37 Promoting the engagement of students across modules and programmes LearningMost significant feedback commentsWhat did you understand these to mean?AnalysingThings that I did that attracted positive commentsThings that I did that attracted critical commentsAny recurring trends or comments?Planning and feeding forwardThings I can do to build on the positive feedbackThings I can do to address the critical feedback(after Race, 2007).
38 Bringing this thinking together All the tenets together provide a coherent agenda to bring about a marked improvement in assessment in your HEI;The focus of the rest of the morning is to present your initial plans of action for enhancement with another pilot institution so that ideas and practices can be shared, developed and critiqued.
39 Paired activity in your institutional task group Using some ideas from the presentation and the publication “A Marked Improvement”, review how you might work with your colleagues back in your own institution to embed issues arising from Tenet 1 (Assessment for Learning) and Tenet 2 (Ensuring assessment is fit for purpose).Working with another institutional task group, please present your plans to date and share your ideas and any issues that have arisen.Now consider what are the opportunities and challenges that these two tenets offer and how might your plans change as a result of your conversations with colleagues here?
40 References and further reading 1 Armstrong, M., Clarkson, P. and Noble, M. (1998), Modularity and credit frameworks: the NUCCAT survey and 1998 conference report, Newcastle-upon-Tyne, Northern Universities Consortiuum for Credit Accumulation and Transfer. ASKe Weston Manor manifesto (accessed April 2012) Assessment Reform Group (1999) Assessment for Learning : Beyond the black box, Cambridge UK, University of Cambridge School of Education. Astin, A. (1997), What matters in college? Four critical years revisited. San Francisco: Jossey Bass. Bloxham, Sue. "Marking and moderation in the UK: false assumptions and wasted resources." Assessment & Evaluation in Higher Education 34.2 (2009): Brown, G. with Bull, J. and Pendlebury, M. (1997) Assessing Student Learning in Higher Education, London: Routledge. Brown, S. and Knight, P. (1994) Assessing Learners in Higher Education, London: Kogan Page. Brown, S. Rust, C. & Gibbs, G. (1994) Strategies for Diversifying Assessment, Oxford: Oxford Centre for Staff Development. Brown, S. and Glasner, A. (eds.) (1999) Assessment Matters in Higher Education, Choosing and Using Diverse Approaches, Maidenhead: Open University Press. Brown, S. and Race, P. (2012) Using effective assessment to promote learning in Hunt, L. and Chambers, D. (2012) University Teaching in Focus, Melbourne: Australian Council for Educational Research( Acer Press). P74-91
41 References and further reading 2 Brown, S. (2011) First class: how assessment can enhance student learning in Blue Skies: new thinking about the future of higher education, London: Pearson. Boud, D. (1995) Enhancing learning through self-assessment, London: Routledge. Burgess, Robert. Beyond the honours degree classification: Burgess Group final report. Universities UK, Elander, J. and Hardman, D. (2002), ‘An application of judgement analysis to examination marking in psychology’, British Journal of Psychology 93, pp Elwood, J. and Klenowski, V. (2002), ‘Creating communities of shared practice: the challenges of assessment use in learning and teaching’, Assessment and Evaluation in Higher Education, 27, pp Gibbs, G. (1992), Improving the quality of student learning. Bristol: TES Gibbs, G (2007) ‘Departmental leadership of teaching’ Presented at The Oxford Learning Institute research seminar series, University of Oxford. Hilary term Gibbs, G. 2010, Using assessment to support student learning, Leeds Met Press, Leeds. Gibbs, P., Angelides, P. and Michaelides, P. (2004), ‘Preliminary thoughts on a praxis of higher education teaching’, Teaching in Higher Education, 9, pp Havnes, A. (2007), ‘What can feedback practices tell us about variation in grading across fields?’ Presented at the ASKe Seminar Series, Oxford Brookes University, 19th September. Laming, D. (1990) The reliability of a certain university examination compared with the precision of absolute judgements, Quarterly Journal of Experimental Psychology Section A—Human ExperimentalPsychology, 42 (2), pp. 239–254. Knight, P.and Yorke, M (2004),Learning, Curriculum and Employability in Higher Education. London: Routledge. McDowell, L. and Brown, S. (1998) Assessing students: cheating and plagiarism, Newcastle: Red Guide 10/11 University of Northumbria.
42 References and further reading 3 Nicol, D. J. and Macfarlane-Dick, D. (2006) Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education Vol 31(2),Newstead, S. E. and Dennis, I. (1994), ‘Examiners examined: the reality of exam marking in psychology’, The Psychologist, 7, ppNonaka, I. (1991), ‘The knowledge-creating company’, The Harvard Business Review, November-December, ppO’Donovan, B., Price, M. and Rust, C. (2004), ‘Know what I mean? Enhancing student understanding of assessment standards and criteria’, Teaching in Higher Education, 9, ppPickford, R. and Brown, S. (2006) Assessing skills and practice, London: Routledge. Race,P., Brown, S. and Smith, B. (2005) 500 Tips on Assessment: 2nd edition, London: RoutledgePolanyi, M. (1998) The tacit dimension, in: L. PRUSAK (Ed.) Knowledge in Organization (Boston, MA,Butterworth Heineman).Race, P. (2010) Making learning happen (chapter 4) Sage, LondonRace, P. (2001) A Briefing on Self, Peer & Group Assessment, in LTSN Generic Centre Assessment Series No 9 LTSN York.Race P. (2006) The lecturer’s toolkit (3rd edition), London: Routledge.Race (2007) How to get a good degree , London, Routledge.Rust, C., Price, M. and O’Donovan, B. (2003) Improving students’ learning by developing their understanding of assessment criteria and processes, Assessment and Evaluation in Higher Education. 28 (2),
43 References and further reading 4 Sambell, K McDowell, L and Brown, S 1997, ‘“But is it fair?” An exploratory study of student perceptions of the consequent validity o assessment’, Studies in Educational Evaluation, vol. 23 (4), pp Knight, P. and Yorke, M. (2003) Assessment, learning and employability Maidenhead, UK: SRHE/Open University Press.Sadler, D. R. (1987), ‘Specifying and Promulgating Achievement Standards’, Oxford Review of Education, 13, pp. 191–209.Sadler, DR 1989, ‘Formative assessment and the design of instructional systems’, Instructional Science, vol. 18, ppSadler, R. (2008) Assessment of Higher Education, in International Encyclopaedia of EducationYorke, M. (1999) Leaving Early: Undergraduate Non-completion in Higher Education, London: Routledge.Yorke, Mantz, and Bernard Longden. (2007)"The first-year experience in higher education in the UK: Report on Phase 1 of a project funded by the Higher Education Academy."Yorke, M., Bridges, P and Woolf, H. (2000), ‘Mark distributions and marking practices in UK higher education; some challenging issues’, Active Learning in Higher Education, vol. 1, no. 1, pp