1. The role of teachers in the assessment of learning
Outcomes of the Assessment Systems for the Future project of the Assessment Reform Group
Funded by the Nuffield Foundation

2. Assessment Systems of the Future Project
Funding: Nuffield Foundation
Focus: summative assessment of school pupils and the role that assessment by teachers can take in this
Duration: Sept 2003 to June 2006
Method: 5 expert seminars and 2 wider sets of consultation/dissemination conferences

3. Causes for concern
‘High stakes’ testing causes frequent testing and ‘teaching to the test’
Frequent testing affects children’s motivation for learning
Teaching to the test restricts the curriculum and teaching methods
External testing encourages more internal testing
Reduces opportunities for formative assessment

4. Result Too much testing
Validity is low (we do not get good information about learning, only about how good the children are at taking tests)
High cost
Negative impact on learning and teaching
No evidence that testing ‘drives up standards’

5. Costs
165 = hours per year spent by Year 6 teachers on tests in England
222 = hours per year on other assessment and reporting activities
84 = hours per year spent by Year 6 children on taking tests
£240m = cost of time spent on tests and examinations (primary + sec)
£610m = total direct and indirect cost of tests and examinations (primary + sec)
(2003 estimates based on surveys)

6. Properties of summative assessment
What learning outcomes are assessed (validity)
How accurately and consistently they are assessed (reliability)
Impact of the assessment (on teachers, learners, the curriculum and teaching methods)
Cost (time for teaching and learning and other resources)

7. Properties depend upon…
How the assessment is carried out
By testing
By teachers
Combination
What the results are used for
Information on individual students
Evaluation/ accountability of teaching and teachers (groups)
Monitoring the system (groups)

8. Uses of summative assessment in a national assessment system
Individual results:
Internal to the school/college (records, reports, guidance)
External to the school (certification, selection, meeting statutory requirements)
Group results:
Evaluation (teachers, schools and local authorities)
Monitoring (year on year comparisons of averages at regional or national level)

9. What ought to be assessed?
The full range of understanding, skills, competencies and attitudes that are the goals of a modern education
In particular
Learning with understanding, shown in ability to apply knowledge rather than recall facts
Understanding learning, being able to reflect on the learning process and aware of how to learn
Learning with understanding
Why we value understanding seems obvious at first glance and it frequently occurs in the expression of learning goals, but the reasons are, of course, dependent upon what we mean by it. It is quite different from knowing facts, although it requires factual knowledge. Understanding shows in the ability to organise knowledge, to relate it actively to new and to past experience, forming ‘big’ ideas, much in the way that distinguishes ‘experts’ from ‘novices’ (Bransford et al, 1999). Big ideas are ones that can be applied in different contexts; they enable learners to understand a wide range of phenomena by identifying the essential links between different situations without being diverted by superficial features. Merely memorising facts or a fixed set of procedures does not support this ability to apply learning to contexts beyond the ones in which it was learned. Knowledge that is understood is thus useful knowledge, something that pupils appreciate when they constantly ask for their learning to be relevant to their lives.
Learning how to learn
We value the development of students’ awareness and understanding of the process of learning – the development of meta-cognition - because we recognise the need to prepare young people for life and work in the rapidly changing society of today and tomorrow. Pupils need to be able to think about and reflect on their learning as part of the learning process and acquire a set of effective learning practices.

10. About reliability
Tests and examinations are assumed to be reliable, but
Strong research evidence that up to 30% pupils are given incorrect grades or levels
Teachers’ own tests may be even less reliable and result in wrong decisions being made
Published data on the reliability of tests and examinations is limited, but what there is indicates that an unacceptable proportion of students are placed in the wrong levels or grades eg up to 30% according the Wiliam (2001) (Reliability, validity, and all that jazz. Education 3-13, 29 (3) 17-21)
Study of some tests created by teachers suggests that these are of considerably lower reliability, which could have serious consequences for pupils if they are used to decide future course opportunities.

11. Dominant impact of ‘external’ assessment
Reasons:
Often ‘high stakes’ for individual students
‘High stakes’ for teachers when results used for accountability
Serious impact because of:
Influence on teachers’ own summative assessments
Influence on the use of formative assessment
Preference for using tests & examinations

12. The need for change
If current ways of assessing outcomes are not including important learning outcomes, are not as reliable as assumed and have negative impacts, then we must look for alternatives
Most users of assessment want evidence of both academic and non-academic achievements
Because of the strong influence of what is assessed on what is taught, we must find ways of assessing important outcomes such as learning with understanding and understanding learning.
In addition, most users of assessment information want to see evidence of both academic and non-academic achievement. Pupils and parents want all students’ achievements, including what is learned outside the classroom, to be reported and credited. Both employers and higher education admission tutors want to be able to identify students who are independent learners, who show initiative and perseverance and have learned how to learn. Reporting such outcomes when students leave school is not enough; the progress towards them needs to be monitored throughout school. Consequently such outcomes must be included in valid summative assessment.

13. Some advantages of using assessment by teachers
Potential for the full range of goals to be included when teachers collect evidence as part of normal work with students
Can relieve the pressure of terminal tests and examinations
Teachers can use information about students formatively as well as summatively
Can release time and other costs for alternative use
How well the assessment reflects the range of achievements that it is intended should be included will depend not only on what evidence is collected but on how it is collected. Although some outcomes can be assessed by written tests most require evidence of different kinds, some from open-ended tasks and others from evidence gathered over time. When developing tests, concern for reliability of marking tends to restrict the range of tasks. Examples are the science practical examinations taking the form of routine procedures making little cognitive demand on students, but easily scored. This threat to validity can be minimised by using a wider range of evidence including teachers’ judgements across a range of activities.
Use of evidence gathered and judged by teachers can improve the match between the range of intended learning and the information provided in the assessment since:
as part of their regular work, teachers can build up a picture of students’ attainment across the full ranges of activities and goals. This gives a broader and fuller account of achievement than can be obtained through tests, which can only include a restricted range of items.
Freedom from test anxiety means that the assessment is a more valid indication of students’ achievement.
Evidence of learning can be used formatively as well as summarised when required to summative purposes
Time and money can be saved, as we will see later.

14. Key points: from experience in countries in & outside the UK
Where teachers have become dependent upon external tests - at least two years of trial and evaluation needed for the value of new practices to be properly judged
 Top-down approaches are not as effective as ones involving teachers in building up necessary procedures
 Effective moderation and professional development are key factors in establishing confidence
The project gathered information about the ways in which assessment by teachers is featuring in current practice in, and future plans for, assessment systems in the four countries of the UK, including the assessment of vocational skills, and in countries outside the UK. The experience in Queensland, over 32 years, of using assessment by teachers as the basis for the Senior Certificate was described by Graham Maxwell. The Berkeley Evaluation and Assessment Research (BEAR) system was presented by Mark Wilson. These main points from the discussions helped in identifying basic principles that the project considers should guide the development of an assessment system in which teachers’ assessment has a substantial role, listed later.

15. What about the disadvantages?
Teachers’ judgements often perceived as being unreliable
Increase in work load for teachers
Can lead to the same distortion of teaching as testing if used for high stakes accountability
Each of these points will be considered in more detail. Starting with reliability.

16. What can be done - about reliability ?
Evidence of unreliability of teachers’ judgements comes from studies where no guidance or training was given
Research shows that, with appropriate pd and moderation procedures, teachers’ assessment can be highly reliable
The training and moderation required have benefits beyond reliability of results; they enhance the quality of teaching and learning
Access to a bank of well designed tasks
As noted, experience of implementing summative assessment based on teachers’ judgments in other countries shows that it is possible to reach a satisfactory degree of reliability. Moderation and training are the key factors.
The research evidence of low reliability comes from studies of situations where no attempt was made to prepare teachers for taking a major role in summative assessment. However, the research also shows that when criteria are well specified (and understood) the teachers are able to make judgments of acceptable reliability. The moderation procedures that are required for quality assurance can be conducted in a way that provides quality enhancement of teaching and learning. There is no doubt, however, that robust and permanent procedures for quality assurance and quality control of teachers’ judgments are needed.
Further, teachers can be helped by access to a bank of well-designed tasks, with marking criteria, which exemplify activities through which pupils can work towards important goals, such as critical reasoning and the application of knowledge in new situations. Good tasks can exemplify objectives in operation
Can supplement teachers’ judgements for individuals where existing evidence is insufficient
Can complement teachers’ judgements by providing uniform evidence for all pupils
Must be seen as helping teachers to come to their judgements not supplanting them.

17. What can be done – about workload ?
Teachers already spend a large proportion of their time on assessment
In England, this amounts to
nearly 400 hours/year in Y6, of which almost half is on internal or external tests or test preparation; for pupils about 84 hours/year or almost four weeks
in Y7, 8 & 9 the time is about 100 hours per class for subject teachers; for pupils 20 hours per subject per year
Saving half of this would more than compensate for extra time on moderation
Estimating costs of assessment is notoriously difficult; the uncertainly of concepts and the complexity of variables mean that all reports have to be treated with great caution. The evidence available from three reports and some examples from individual schools provides some very tentative estimates of the time and direct costs of summative assessment currently in England.
(Sources: Financial Modelling of the English Exams System , report from PriceWaterhouseCoopers (PWC) for the QCA
The Cost of Assessment. Centre for Science Education, Sheffield Hallam University 2003
Survey of secondary schools by the Secondary Heads Association (SHA))
At KS 1& 2 Teachers’ time spent specifically on testing (both internal and external) varies from zero in Y1 to 20 hours per year in Y6. The actual time spent on national testing is much smaller than the time spent on regular tests given by the teacher. Other evidence would suggest that this is the result of teachers giving students practice tests and of assuming that tests are necessary, in preference to their own judgements. Teachers already spend about hours per year (upto an hour a week on moderation).
At KS 3 national tests and other tests such as end of module tests take about 30 to 35 hours per subject and other assessments about 54 hours (with some further time being taken in report writing and parents’ evenings). Currently little time is spent on moderation in KS3. Fewer tests and more assessment by teachers would increase the need for moderation. However, the time saved in testing more than compensates for the 3% of time necessary to provide half a day every three weeks for moderation activities.

18. What can be done – about the ‘high stakes’ ?
Judging schools based on test results
does not reflect all that a school strives for
results in disproportional attention to ‘borderline’ pupils
encourages teaching to the test
More valid and reliable methods for school evaluation taking context into account should be used
High stakes should be transferred to how well school meets a range of goals
The high stakes for teachers and schools of test results comes from using them to set targets and to judge schools by how many pupils reach certain levels. But this is a flawed approach, since it is not providing the information required for accountability. Further, the ‘naming and shaming’ of schools for not meeting targets has a severely detrimental impact rather than improving standards (Hattersley, Guardian, 21 Feb 2006).
Not only are the results unlikely to reflect the full range of educational outcomes which a school strives for and for which it should be held accountable, but to reach the target, attention is focused on those students who are close to the required level, with less attention to those who are either too far below or are already above the target level. A third point is the focus on the narrow requirements of passing the test or examination.
For a more positive impact, accountability is best based on information about a range of student achievements and learning activities, judged by reference to the context and circumstances of the school and used positively to improve students’ opportunities for learning. The information used in accountability should include information about the curriculum and teaching methods and relevant aspects of students’ backgrounds and of their learning histories. Some good examples exist in various school self-evaluation guidelines.
( For example: How Good is Our School in Scotland, in England the emphasis on schools self-evaluation in ‘A New Relationship with Schools’ (DfES and Ofsted, 2004) and in Wales Guidance on the Inspection of Primary and Nursery Schools (September 2004) and Guidance on the Inspection of Secondary Schools (September 2004) (Estyn).)
[1]

19. Conclusions
We want a system capable of providing reliable information about a wide range of pupil competences
Systems depending primarily on test results do not provide this
No approach to summative assessment is without problems and some negative impacts on pupils and teachers
Research evidence suggests that a system making appropriate use of assessment by teachers has far fewer negative consequences than one based on tests
…contd

20. …continued
Procedures are needed to help teacher understand and use criteria consistently also benefit teaching
Assessment procedures need to be transparent to gain the confidence of users
Summative assessment should be carried out only when needed to report progress, at other time assessment should have a formative function
Procedures should enable evidence used for formative assessment to be reviewed against summative criteria …Contd

21. …continued
To reduce the ‘high stakes’ for schools that lead to distortion of the curriculum and teaching
Systems for school accountability should not depend solely on pupils achievement results
To provide more valid and useful information about national and regional standards
Achievement should be monitored by assessing a sample of pupils using a wide evidence base

22. Implications for action: policy
Recognise the short-coming of current assessment policies in relation to validity, reliability, cost and impact on the curriculum, and teaching
Consider replacing national testing, where it exists, by moderated teachers’ judgments
Divert resources from tests into quality assurance and enhancement
Review the role of teachers’ assessment in examinations for year olds
Promote openness in assessment procedures

23. Implications for action: school management
School assessment policy should require summative assessment only when necessary for reporting progress, not more frequently
Establish, maintain and protect time for quality assurance procedures for internal summative assessment,
Ensure parents understand the formative and summative use of assessment
Resist pressure for test data and encourage positive discourse about assessment

24. Implications for action: teachers
Ensure that assessment is always used to help learning
Only when a summative assessment report is needed, ensure that best evidence is reliably judged against relevant criteria
Involve pupils in self-assessment and help them to understand the assessment criteria
Take part in moderation of judgments and other quality assurance procedures.
Use tests only when most appropriate, not as routine.

25. Implications: inspectors and advisers
Help schools to establish assessment policies that encourage formative use of assessment and moderation procedures for summative assessment
For summative assessment encourage use of a range of pupils’ achievements
Ensure that appropriate professional development in assessment is available
Help schools set targets based on self-evaluation across a range of data not only on levels achieved by pupils

26. Implications: teacher educators
Ensure that courses allow adequate time for
discussion of the different purposes of assessment and the uses made of assessment data
trainees and participants to identify, sample and evaluate different ways of gathering evidence of pupils’ performance
giving experience of generating assessment criteria linked to specific learning goals
considering evidence of bias and other sources of error in assessment and how they can be minimised

27. Some references
See the ARG website for information and reports from the ASF project:
Recent relevant ARG publications
The Role of Teachers in the Assessment of Learning (2006) available on the ARG website and from the CPA Office, Institute of Education, London WC1H 0AL
Gardner, J (Ed) Assessment and Learning (2006) London: Sage

28. Formative and summative assessment
working together
Lesson goals
Learning
activities
Criteria for
reporting levels a b c d
etc
Moderation