Presentation on theme: "The role of teachers in the assessment of learning"— Presentation transcript:
1 The role of teachers in the assessment of learning Outcomes of the Assessment Systems for the Future project of the Assessment Reform GroupFunded by the Nuffield Foundation
2 Assessment Systems of the Future Project Funding: Nuffield FoundationFocus: summative assessment of school pupils and the role that assessment by teachers can take in thisDuration: Sept 2003 to June 2006Method: 5 expert seminars and 2 wider sets of consultation/dissemination conferences
3 Causes for concern‘High stakes’ testing causes frequent testing and ‘teaching to the test’Frequent testing affects children’s motivation for learningTeaching to the test restricts the curriculum and teaching methodsExternal testing encourages more internal testingReduces opportunities for formative assessment
4 Result Too much testing Validity is low (we do not get good information about learning, only about how good the children are at taking tests)High costNegative impact on learning and teachingNo evidence that testing ‘drives up standards’
5 Costs165 = hours per year spent by Year 6 teachers on tests in England222 = hours per year on other assessment and reporting activities84 = hours per year spent by Year 6 children on taking tests£240m = cost of time spent on tests and examinations (primary + sec)£610m = total direct and indirect cost of tests and examinations (primary + sec)(2003 estimates based on surveys)
6 Properties of summative assessment What learning outcomes are assessed (validity)How accurately and consistently they are assessed (reliability)Impact of the assessment (on teachers, learners, the curriculum and teaching methods)Cost (time for teaching and learning and other resources)
7 Properties depend upon… How the assessment is carried outBy testingBy teachersCombinationWhat the results are used forInformation on individual studentsEvaluation/ accountability of teaching and teachers (groups)Monitoring the system (groups)
8 Uses of summative assessment in a national assessment system Individual results:Internal to the school/college (records, reports, guidance)External to the school (certification, selection, meeting statutory requirements)Group results:Evaluation (teachers, schools and local authorities)Monitoring (year on year comparisons of averages at regional or national level)
9 What ought to be assessed? The full range of understanding, skills, competencies and attitudes that are the goals of a modern educationIn particularLearning with understanding, shown in ability to apply knowledge rather than recall factsUnderstanding learning, being able to reflect on the learning process and aware of how to learnLearning with understandingWhy we value understanding seems obvious at first glance and it frequently occurs in the expression of learning goals, but the reasons are, of course, dependent upon what we mean by it. It is quite different from knowing facts, although it requires factual knowledge. Understanding shows in the ability to organise knowledge, to relate it actively to new and to past experience, forming ‘big’ ideas, much in the way that distinguishes ‘experts’ from ‘novices’ (Bransford et al, 1999). Big ideas are ones that can be applied in different contexts; they enable learners to understand a wide range of phenomena by identifying the essential links between different situations without being diverted by superficial features. Merely memorising facts or a fixed set of procedures does not support this ability to apply learning to contexts beyond the ones in which it was learned. Knowledge that is understood is thus useful knowledge, something that pupils appreciate when they constantly ask for their learning to be relevant to their lives.Learning how to learnWe value the development of students’ awareness and understanding of the process of learning – the development of meta-cognition - because we recognise the need to prepare young people for life and work in the rapidly changing society of today and tomorrow. Pupils need to be able to think about and reflect on their learning as part of the learning process and acquire a set of effective learning practices.
10 About reliabilityTests and examinations are assumed to be reliable, butStrong research evidence that up to 30% pupils are given incorrect grades or levelsTeachers’ own tests may be even less reliable and result in wrong decisions being madePublished data on the reliability of tests and examinations is limited, but what there is indicates that an unacceptable proportion of students are placed in the wrong levels or grades eg up to 30% according the Wiliam (2001) (Reliability, validity, and all that jazz. Education 3-13, 29 (3) 17-21)Study of some tests created by teachers suggests that these are of considerably lower reliability, which could have serious consequences for pupils if they are used to decide future course opportunities.
11 Dominant impact of ‘external’ assessment Reasons:Often ‘high stakes’ for individual students‘High stakes’ for teachers when results used for accountabilitySerious impact because of:Influence on teachers’ own summative assessmentsInfluence on the use of formative assessmentPreference for using tests & examinations
12 The need for changeIf current ways of assessing outcomes are not including important learning outcomes, are not as reliable as assumed and have negative impacts, then we must look for alternativesMost users of assessment want evidence of both academic and non-academic achievementsBecause of the strong influence of what is assessed on what is taught, we must find ways of assessing important outcomes such as learning with understanding and understanding learning.In addition, most users of assessment information want to see evidence of both academic and non-academic achievement. Pupils and parents want all students’ achievements, including what is learned outside the classroom, to be reported and credited. Both employers and higher education admission tutors want to be able to identify students who are independent learners, who show initiative and perseverance and have learned how to learn. Reporting such outcomes when students leave school is not enough; the progress towards them needs to be monitored throughout school. Consequently such outcomes must be included in valid summative assessment.
13 Some advantages of using assessment by teachers Potential for the full range of goals to be included when teachers collect evidence as part of normal work with studentsCan relieve the pressure of terminal tests and examinationsTeachers can use information about students formatively as well as summativelyCan release time and other costs for alternative useHow well the assessment reflects the range of achievements that it is intended should be included will depend not only on what evidence is collected but on how it is collected. Although some outcomes can be assessed by written tests most require evidence of different kinds, some from open-ended tasks and others from evidence gathered over time. When developing tests, concern for reliability of marking tends to restrict the range of tasks. Examples are the science practical examinations taking the form of routine procedures making little cognitive demand on students, but easily scored. This threat to validity can be minimised by using a wider range of evidence including teachers’ judgements across a range of activities.Use of evidence gathered and judged by teachers can improve the match between the range of intended learning and the information provided in the assessment since:as part of their regular work, teachers can build up a picture of students’ attainment across the full ranges of activities and goals. This gives a broader and fuller account of achievement than can be obtained through tests, which can only include a restricted range of items.Freedom from test anxiety means that the assessment is a more valid indication of students’ achievement.Evidence of learning can be used formatively as well as summarised when required to summative purposesTime and money can be saved, as we will see later.
14 Key points: from experience in countries in & outside the UK Where teachers have become dependent upon external tests - at least two years of trial and evaluation needed for the value of new practices to be properly judged Top-down approaches are not as effective as ones involving teachers in building up necessary procedures Effective moderation and professional development are key factors in establishing confidenceThe project gathered information about the ways in which assessment by teachers is featuring in current practice in, and future plans for, assessment systems in the four countries of the UK, including the assessment of vocational skills, and in countries outside the UK. The experience in Queensland, over 32 years, of using assessment by teachers as the basis for the Senior Certificate was described by Graham Maxwell. The Berkeley Evaluation and Assessment Research (BEAR) system was presented by Mark Wilson. These main points from the discussions helped in identifying basic principles that the project considers should guide the development of an assessment system in which teachers’ assessment has a substantial role, listed later.
15 What about the disadvantages? Teachers’ judgements often perceived as being unreliableIncrease in work load for teachersCan lead to the same distortion of teaching as testing if used for high stakes accountabilityEach of these points will be considered in more detail. Starting with reliability.
16 What can be done - about reliability ? Evidence of unreliability of teachers’ judgements comes from studies where no guidance or training was givenResearch shows that, with appropriate pd and moderation procedures, teachers’ assessment can be highly reliableThe training and moderation required have benefits beyond reliability of results; they enhance the quality of teaching and learningAccess to a bank of well designed tasksAs noted, experience of implementing summative assessment based on teachers’ judgments in other countries shows that it is possible to reach a satisfactory degree of reliability. Moderation and training are the key factors.The research evidence of low reliability comes from studies of situations where no attempt was made to prepare teachers for taking a major role in summative assessment. However, the research also shows that when criteria are well specified (and understood) the teachers are able to make judgments of acceptable reliability. The moderation procedures that are required for quality assurance can be conducted in a way that provides quality enhancement of teaching and learning. There is no doubt, however, that robust and permanent procedures for quality assurance and quality control of teachers’ judgments are needed.Further, teachers can be helped by access to a bank of well-designed tasks, with marking criteria, which exemplify activities through which pupils can work towards important goals, such as critical reasoning and the application of knowledge in new situations. Good tasks can exemplify objectives in operationCan supplement teachers’ judgements for individuals where existing evidence is insufficientCan complement teachers’ judgements by providing uniform evidence for all pupilsMust be seen as helping teachers to come to their judgements not supplanting them.
17 What can be done – about workload ? Teachers already spend a large proportion of their time on assessmentIn England, this amounts tonearly 400 hours/year in Y6, of which almost half is on internal or external tests or test preparation; for pupils about 84 hours/year or almost four weeksin Y7, 8 & 9 the time is about 100 hours per class for subject teachers; for pupils 20 hours per subject per yearSaving half of this would more than compensate for extra time on moderationEstimating costs of assessment is notoriously difficult; the uncertainly of concepts and the complexity of variables mean that all reports have to be treated with great caution. The evidence available from three reports and some examples from individual schools provides some very tentative estimates of the time and direct costs of summative assessment currently in England.(Sources: Financial Modelling of the English Exams System , report from PriceWaterhouseCoopers (PWC) for the QCAThe Cost of Assessment. Centre for Science Education, Sheffield Hallam University 2003Survey of secondary schools by the Secondary Heads Association (SHA))At KS 1& 2 Teachers’ time spent specifically on testing (both internal and external) varies from zero in Y1 to 20 hours per year in Y6. The actual time spent on national testing is much smaller than the time spent on regular tests given by the teacher. Other evidence would suggest that this is the result of teachers giving students practice tests and of assuming that tests are necessary, in preference to their own judgements. Teachers already spend about hours per year (upto an hour a week on moderation).At KS 3 national tests and other tests such as end of module tests take about 30 to 35 hours per subject and other assessments about 54 hours (with some further time being taken in report writing and parents’ evenings). Currently little time is spent on moderation in KS3. Fewer tests and more assessment by teachers would increase the need for moderation. However, the time saved in testing more than compensates for the 3% of time necessary to provide half a day every three weeks for moderation activities.
18 What can be done – about the ‘high stakes’ ? Judging schools based on test resultsdoes not reflect all that a school strives forresults in disproportional attention to ‘borderline’ pupilsencourages teaching to the testMore valid and reliable methods for school evaluation taking context into account should be usedHigh stakes should be transferred to how well school meets a range of goalsThe high stakes for teachers and schools of test results comes from using them to set targets and to judge schools by how many pupils reach certain levels. But this is a flawed approach, since it is not providing the information required for accountability. Further, the ‘naming and shaming’ of schools for not meeting targets has a severely detrimental impact rather than improving standards (Hattersley, Guardian, 21 Feb 2006).Not only are the results unlikely to reflect the full range of educational outcomes which a school strives for and for which it should be held accountable, but to reach the target, attention is focused on those students who are close to the required level, with less attention to those who are either too far below or are already above the target level. A third point is the focus on the narrow requirements of passing the test or examination.For a more positive impact, accountability is best based on information about a range of student achievements and learning activities, judged by reference to the context and circumstances of the school and used positively to improve students’ opportunities for learning. The information used in accountability should include information about the curriculum and teaching methods and relevant aspects of students’ backgrounds and of their learning histories. Some good examples exist in various school self-evaluation guidelines.( For example: How Good is Our School in Scotland, in England the emphasis on schools self-evaluation in ‘A New Relationship with Schools’ (DfES and Ofsted, 2004) and in Wales Guidance on the Inspection of Primary and Nursery Schools (September 2004) and Guidance on the Inspection of Secondary Schools (September 2004) (Estyn).)
19 ConclusionsWe want a system capable of providing reliable information about a wide range of pupil competencesSystems depending primarily on test results do not provide thisNo approach to summative assessment is without problems and some negative impacts on pupils and teachersResearch evidence suggests that a system making appropriate use of assessment by teachers has far fewer negative consequences than one based on tests…contd
20 …continuedProcedures are needed to help teacher understand and use criteria consistently also benefit teachingAssessment procedures need to be transparent to gain the confidence of usersSummative assessment should be carried out only when needed to report progress, at other time assessment should have a formative functionProcedures should enable evidence used for formative assessment to be reviewed against summative criteria …Contd
21 …continuedTo reduce the ‘high stakes’ for schools that lead to distortion of the curriculum and teachingSystems for school accountability should not depend solely on pupils achievement resultsTo provide more valid and useful information about national and regional standardsAchievement should be monitored by assessing a sample of pupils using a wide evidence base
22 Implications for action: policy Recognise the short-coming of current assessment policies in relation to validity, reliability, cost and impact on the curriculum, and teachingConsider replacing national testing, where it exists, by moderated teachers’ judgmentsDivert resources from tests into quality assurance and enhancementReview the role of teachers’ assessment in examinations for year oldsPromote openness in assessment procedures
23 Implications for action: school management School assessment policy should require summative assessment only when necessary for reporting progress, not more frequentlyEstablish, maintain and protect time for quality assurance procedures for internal summative assessment,Ensure parents understand the formative and summative use of assessmentResist pressure for test data and encourage positive discourse about assessment
24 Implications for action: teachers Ensure that assessment is always used to help learningOnly when a summative assessment report is needed, ensure that best evidence is reliably judged against relevant criteriaInvolve pupils in self-assessment and help them to understand the assessment criteriaTake part in moderation of judgments and other quality assurance procedures.Use tests only when most appropriate, not as routine.
25 Implications: inspectors and advisers Help schools to establish assessment policies that encourage formative use of assessment and moderation procedures for summative assessmentFor summative assessment encourage use of a range of pupils’ achievementsEnsure that appropriate professional development in assessment is availableHelp schools set targets based on self-evaluation across a range of data not only on levels achieved by pupils
26 Implications: teacher educators Ensure that courses allow adequate time fordiscussion of the different purposes of assessment and the uses made of assessment datatrainees and participants to identify, sample and evaluate different ways of gathering evidence of pupils’ performancegiving experience of generating assessment criteria linked to specific learning goalsconsidering evidence of bias and other sources of error in assessment and how they can be minimised
27 Some referencesSee the ARG website for information and reports from the ASF project:Recent relevant ARG publicationsThe Role of Teachers in the Assessment of Learning (2006) available on the ARG website and from the CPA Office, Institute of Education, London WC1H 0ALGardner, J (Ed) Assessment and Learning (2006) London: Sage
28 Formative and summative assessment working togetherLesson goalsLearningactivitiesCriteria forreporting levelsabcdetcModeration