1. Race to the Top Assessment Program Consultation Carol Campbell Atlanta, Georgia November 17, 2009

2. 2 Assessment Design Principles Effective Assessment Systems Designed around a clear vision of purpose and learning goals Align with and advance common standards, curriculum expectations, learning objectives and instructional strategies Encompass and combine assessment of, for and as learning Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning Embody high expectations to engage, motivate, support and stretch all students to progress and achieve Build and develop professional capacity Assess a range of content, knowledge, skills and performance in authentic, applied and appropriate ways Are part of a clear and flexible learning trajectory across subjects and over education/career pathways Support data- informed decision-making for local and system improvement Place quality of assessment over quantity

3. 3 1. Clear Vision of Purpose and Learning Goals - Specifying 21 st century skills, college and career- readiness  Example: High school examinations in England combine: Subject/curricular knowledge with generic skills Functional skills – “essential for life, learning and work” – practical skills in English, mathematics and information and communication technology” (ICT). Example – Math level 1 - “A member of a reprographics team is able to look at statistics and graphs detailing paper usage and identify trends that might suggest that equipment requires servicing” Personal learning and thinking skills (PLTS) – “These are skills that will equip people for successful employment and lifelong learning…PLTS require learners to be:  Independent enquirers  Creative thinkers  Reflective learners  Team workers  Self-managers  Effective participants”

4. 4 Assessing 21 st Century Skills – England, Ages 14 - 19 Assessment criteria for Functional Skills – “Assessment must, according the skill areas:  Provide realistic contexts, scenarios and problems  Specify tasks that are relevant to the context  Require application of knowledge, skills and understanding for a purpose  Require problem solving  Assess process skills and the outcome of their application in different contexts Generally assessments externally defined (exam boards) and internally assessed (teachers), including controlled assessments in classroom and projects. Combinations of open and fixed response items varies by subject area and level. Performance assessments increasingly used in range of qualifications – coursework, projects, extended writing Opportunities for accredited work experience in some qualifications, e.g. Diploma composite qualification includes 10 days work experience

5. 5 Ontario, Canada - Achievement Chart – English, Grades 9-12 – Thinking Skills Thinking – The use of critical and creative thinking skills and/or processes Categories50-59% (L1)60-69% (L 2)70-79% (L 3)80-100% (L4) The student: Use of planning skills (e.g., generating ideas, gathering information, focusing research, organizing information) uses planning skills with limited effectiveness … some effectiveness … considerable effectiveness … high degrees of effectiveness Use of processing skills (e.g. drawing inferences, interpreting, analysing, synthesizing, evaluating) uses processing skills with limited effectiveness … some effectiveness … considerable effectiveness … high degrees of effectiveness Use of critical/creative thinking processes (e.g. oral discourse, research, critical analysis, critical literacy, metacognition, creative process) uses critical/creative thinking processes with limited effectiveness … some effectiveness … considerable effectiveness … high degrees of effectiveness

6. 6 2. Align with and advance common standards, curriculum expectations, learning objectives and instructional strategies  Example: Ontario, Canada  Content Standards Curriculum for Grades 1-12 for all subjects and courses  Developed through extensive research, benchmarking, and consultation with all stakeholders  All teachers are required to teach the curriculum expectations  Performance Standards (criterion-referenced) Criteria are described for student achievement of the curriculum expectations for Grades 1-12  Four levels of achievement  Four categories of knowledge and skills: Knowledge and Understanding; Thinking; Communication; Application  Consistency: Exemplars of student work for all curriculum Standard provincial report cards - aligned with content and performance standards

7. 7 3. Encompass and combine Assessment of, for and as learning Assessment… Of learning - summative assessments to confirm student knowledge, skills, understanding of curriculum, proficiency For learning – formative assessments using information throughout the learning process to investigate student learning, inform instruction and provide feedback to support student progress As learning – assessment as a process of meta- cognition involving student self-assessment and understanding of own learning processes and knowledge

8. 8 Summative and Formative Assessments: Example – The National Assessment System, Scotland, UK FORMATIVE SUMMATIVE INTERNAL EXTERNAL Formative assessment Personal learning planning Involving learners, and parents And other adults in the learning process Local authority collection and analysis of information to inform provision and improvement Inspection feedback and subject/quality/improving reports Follow-through inspection activities Teachers’ judgements and reports, With local moderation and National Assessments as part of understanding And sharing standards Scottish Survey of Achievement P3, P5, P7, S2 National Qualifications (SQA) International studies HMIE inspections and reports on Authorities and schools

9. 9 4. Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning Multi-faceted assessments used internationally include:  Assessments of knowledge (recall & analysis) and assessments of performance (demonstration of ability to apply knowledge in practice)  Multiple-choice, constructed-response, extended tasks and projects  On-demand and curriculum-embedded elements  Externally-developed (by teachers and developers) and classroom-developed/managed  Scoring made consistent through training, moderation, calibration, and auditing

10. 10 Examples of Assessment Types: Ontario and England Types of TaskExample External standardized test – multiple choice response Approx. 50% of questions in Grade 9 and 10 provincial assessments, Ontario Not used in England External standardized test – open-ended response Approx. 50% of questions in Grade 9 and 10 provincial assessments, Ontario Used for KS 2, parts of GCSE (40-100% of final mark) and A-level in England Performance assessment, e.g. projectControlled assessments (coursework) for GCSE (up to 60% of overall mark) and Extended Project for A-level (equivalent to half an A-level) Interaction and observations between teacher and student, e.g. discussion, conference, classroom and homework tasks and tests Systematic Teacher assessments reported to parents/guardians annually and data collected nationally at key points (England, Ontario) Cumulative assessments, e.g. portfoliosNational Vocational Qualifications and Diplomas (England)

11. 11 Singapore GCE A-Level Examinations 3 hour duration; 2 to 4 papers per H2 subject Open-ended essays, structured questions, case studies, source-based questions Externally set and marked by SEAB/CIE Time-based Written Papers Longer duration of about 6 months Product (e.g. Artwork or design task), Oral Presentation, Independent Study Tasks set by SEAB/CIE, internally marked by teachers, externally moderated by SEAB/CIE) School- based Coursework

12. 12 Systematic Achievement Reports at the End of Key Stages, England, UK  Key Stage 1 (age 6-7): Classroom evidence is combined with centrally-developed open-ended tests and tasks in English and mathematics, taken when children are ready and marked by teachers, with moderation.  Key Stage 2 (age 8-11): Teachers’ summary judgments of student progress from classroom evidence are combined with open-ended tests in English, mathematics and science; externally marked and results collected nationally.  Key Stage 3 (age 14): External tests were recently abolished. Moderated teacher assessments are used to report achievement levels in all subjects, nationally collected.  No mandatory high school assessments – although most students will participate in GCSEs (age 15-16) and progress to other academic (A/AS level), vocational (diploma) and flexible pathways for progression

13. 13 5. Embody high expectations to engage, motivate, support and stretch all students to progress and achieve  Example: England, UK:  Making Good Progress Pilot: Progression Targets to increase the number of pupils who make two levels of progress throughout a key stage Progression Premium to reward schools which help students who entered a key stage behind national expectations and make good progress

14. 14 The Assessment for Learning Strategy, England, UK Our aims are that:  Every child knows how they are doing, and understands what they need to do to improve and how to get there. They get support they need to be motivated, independent learners on a ambitious trajectory of improvement  Every teacher is equipped to make well-founded judgement about pupils’ attainment, understands the concepts and principles of progression, and knows how to use their assessment judgements to forward plan, particularly for pupils who are not fulfilling their potential  Every school has in place structured and systematic assessment systems for making regular, useful, manageable and accurate assessments of pupils, and for tracking their progress  Every parent and carer knows how their child is doing, what they need to improve, and how they can support the child and their teachers.

15. 15 Applying Knowledge: A Literacy Task, Grade 10 Ontario Task: Write a minimum of three paragraphs expressing an opinion on the topic below. Develop your main idea with supporting details (proof, facts, examples, etc.).  Purpose and Audience: an adult who is interested in your opinion  Length: The lined space provided for your written work indicates the approximate length of the writing expected. Topic: Are today’s famous people good role models for young people?

16. 16 6. Assess a range of content, knowledge, skills and performance in authentic, applied and appropriate ways Example: General Certificate of Secondary Education (GCSE) (age 15-16), England  GCSEs are available in over 50 subjects: students choose.  GCSEs normally take 2 years to complete. Many courses offer assessment in each unit rather than only at the end.  GCSEs use constructed response items, and usually include structured classroom-based tasks that are more extended.  Increased emphasis on extended writing and course work.  Controlled assessments are taken under supervised conditions and are either set by the awarding body and marked by teachers or set by teachers and marked by the awarding body. Teachers determine the timing of controlled assessments. Controlled assessments comprise up to 60% of a grade, depending on the subject.  Introduction of assessment of functional skills and personal learning and thinking skills across subjects  Cost of assessing one unit = $10 approx.  Average costs of a GCSE course assessment = $40 approx.

17. 17 ICT Controlled Assessment Task, GCSE England Litchfield Promotions works with over 40 bands and artists to promote their music and put on performances in the UK. The number of bands they have on their books is gradually expanding. Litchfield Promotions needs to be sure that each performance will make enough money to cover all the staffing costs and overheads as well as make a profit. Many people need to be paid: the bands; sound engineers; and, lighting technicians. There is also the cost of hiring the venue. Litchfield Promotions needs to create an ICT solution to ensure that they have all necessary information and that it is kept up to date. Their solution will show income, outgoings and profit. Candidates will need to:  Work with others to plan and carry out research to investigate how similar companies have produced a solution (The company does not necessarily have to work with bands and artists or be a promotions company.)  Clearly record and display your findings  Recommend a solution that will address the requirements of the task  Produce a design brief, incorporating timescales, purpose and target audience

18. 18 ICT Task (cont.) Produce a solution, ensuring that the following are addressed: it can be modified to be used in a variety of situations it has a friendly user interface it is suitable for the target audience it has been fully tested. You will need to:  Incorporate a range of: software features, macros, modeling, and validation checks (used appropriately).  Obtain user feedback  Identify areas that require improvement, recommending improvement, with justification  Present information as an integrated document  Evaluate your own and others’ work.

19. 19 England: AS and A Levels - Changes to Assessments  A broad range of question types to assess a wide range of skills  Questions requiring extended answers give students the opportunity to demonstrate the full breadth and depth of their understanding  Synoptic assessments that will test students’ understanding of the subject as a whole, including ability to: Develop a broader and deeper understanding of the connections between the knowledge and understanding set out in the specifications as a whole Draw together some of the key insights from their studies Make purposeful use of these in undertaking assessment tasks  Introduction of an extended project as a separate qualification for A level students to add to their study program.

20. 20 7. Build and Develop Professional Capacity  Example: Teacher collaboration on assessment design and marking, Alberta, Canada  Test design: Identify student characteristics Assist in exam blueprint document Ensure curricular “fit” of the exam Pilot prototype multiple-choice and written response forms Help develop writing assignments and their scoring Item building: Item development sessions held throughout the province involve teachers in the creation of new multiple-choicce and written responses.

21. 21 Teacher Collaboration – Alberta, Canada Reviewing Tests: Each new examination form is reviewed by a committee that includes classroom teachers. The committee examines both the written response and multiple choice sections to ensure that the examination is fair, and demonstrates fidelity to the curriculum. Confirming Standards: Before written responses are marked a committee of teachers meets with examination branch staff to select student work for use in marker training (example papers, training papers and reliability review papers).

22. 22 Teacher Collaboration – Alberta, Canada The Marking Process: Each January and June approximately 1300 teachers meet to mark the written responses in a process with tight calibration to produce consistency in scoring. Examination Advisory Committee: Once yearly representatives of various stakeholder groups, including the Alberta Teachers’ Association meet with branch staff to review examination results from the previous school year, and to offer suggestions for the improvement of the examination program.

23. 23 Scoring of Assessments - Ontario  Grade 9 Mathematics – raw score points = 46% multiple-choice, 54% open- response  Ontario Secondary School Literacy Test – raw score points = 48% multiple- choice, 52% open-response  Multiple choice items are machine scored  Written open-ended responses systematically scored by qualified scorers (mostly Ontario educators)  Use of generic rubric to describe levels of performance, item-specific rubric for each item, and anchors to illustrate descriptors for the score points to the rubric  Committee of educators define range of performance within each code of scoring rubric – ‘range finding’ development  Every scorer participates in extensive training  System of checks and monitoring applied throughout process  Quality assurance and statistical techniques to validate interrater reliability and scoring validity  Use of equating process to place student scores on common scale over two adjacent years and to determine cut scores

24. 24 Teaching-Learning Critical Pathway. Gathering Evidence Area of Greatest Need Current Practice Rubric, Data Wall, Culminating Task Professional Learning Community Action Moderated Marking Share Findings, Communication Using Assessments to Inform Improvements: Example: Ontario

25. 25 8. Are part of a clear and flexible learning trajectory across subjects and over education/career pathways  Example: Exam boards in England ensure consistency of standards within subjects while allowing flexibility of approach across subjects.  Five exam boards - governed by specifications, criteria, frameworks and oversight of Office of the Qualifications and Examinations (OFQUAL) Regulator  Exam boards:  set out the course syllabus & specifications to meet criteria set by the regulator  oversee writing of exam questions and how exam papers should be marked  check coursework/controlled assessments  provide training and quality control for examiners, including use of ‘standardization meetings’ to compare marking across examiners  set grade boundaries to align marks and standards

26. 26 National Qualifications Framework, England: Learner Pathways

27. 27 9. Support data-informed decision-making for local and system improvement Assessment for Learning: School and Classroom (Assessment Reform Group,1999) Assessment for Learning and Data- Informed Decision Making: Systemic (Campbell and Levin, 2007) Assessment for learning should be part of effective planning of teaching and learning. Assessment of and for learning, plus relevant contextual, student and school data, is part of effective strategic and operational planning for state/province, district, school and classroom actions for improved teaching and learning. Assessment for learning should focus on how students learn. Assessment for learning focuses on how systems, schools and students improve their progress, achievements and outcomes.

28. 28 Assessment for Learning: School and Classroom (Assessment Reform Group,1999) Assessment for Learning and Data- Informed Decision Making: Systemic (Campbell and Levin, 2007) Assessment for learning should be recognised as central to classroom practice. Data-informed decision making, including assessment for learning, is recognised as central to educational practices at all levels of the system – province/state, district, school and classroom. Assessment for learning should be regarded as a key professional skill for teachers. Skill and capacity to use, understand and apply data to inform improved actions and outcomes should be regarded as a key professional skills for all educators, including teachers, principals, district staff and state/provincial officials. Assessment for learning should be sensitive and constructive because any assessment has an emotional impact. Careful consideration of which data to use, by whom and for what purpose is required to ensure the sensitive and constructive use of data, including assessment, to support improvement for all not to rank or judge unfairly.

29. 29 Assessment for Learning: School and Classroom (Assessment Reform Group,1999) Assessment for Learning and Data- Informed Decision Making: Systemic (Campbell and Levin, 2007) Assessment for learning should take account of the importance of learner motivation. The use of data, including assessment for learning, to generate motivation to improve should place an emphasis on developing respectful partnerships to engage educators at all levels of the system to work together. Assessment for learning should promote commitment to learning goals and a shared understanding of the criteria by which they are assessed. Shared goals for improvement, and indicators of success towards these goals, are developed and understood to generate a common commitment to improvement targets and learning outcomes. Learners should receive constructive guidance about how to improve. Building capacity for improvement involves providing feedback, strategies, resources and supports to enhance both professional learning and student learning.

30. 30 Assessment for Learning: School and Classroom (Assessment Reform Group,1999) Assessment for Learning and Data- Informed Decision Making: Systemic (Campbell and Levin, 2007) Assessment for learning develops learners’ capacity for self-assessment so that they can become reflective and self-managing. Data-informed approaches involve combining assessment of and for learning to balance external and internal accountability, while building the professional capacity for educators to implement self-evaluation, improvement planning and monitoring strategies. Assessment for learning should recognise the full range of achievements of all learners. Assessment for learning, and the use of related data, recognises the full range of achievements of all learners and schools, including not only achievement results but also progress over time and equity of outcomes for closing gaps in performance while also raising the bar overall.

31. 31 Students School Boards/Authorities Teachers, Principals & Administrators  Funding  Policy/Programs  Public Reporting Ontario School Information System (OnSIS) Staging Area Board Validation & Sign-off by Director Ministry Validation Encryption & Depersonalization Elementary Secondary Data Warehouse (ESDW) Statistics Canada EQAO Other Sources Other Sources Data Data repository or data warehouse Student Management Systems (SMS) Decision Support System EQAO Stats Can Other Board Administrators Province-Wide Information Data M i SA G i ARE Info Ministry of Education

32. 32 10. Place quality of assessment over quantity Examples:  Australia – National standards (underway) State Syllabi, State and Local Assessments (3 grades plus high school)  Canada – Provincial standards, syllabi & assessments (2 grades plus high school)  England – National standards & curriculum School-based assessments + national tasks (1 grade plus high school) Five secondary examination boards  Singapore – National standards and curriculum National exams + school-based tasks (1 grade plus high school)

33. 33 Common Practices Across Countries  Assessments are part of a tightly integrated system of standards, curriculum, instruction, assessment, and teacher development at the state or national (in small countries) level  Assessments include evidence of actual student performance on challenging tasks that evaluate 21 st century standards of learning  Teachers are integrally involved in the development and scoring of assessments  Assessments are designed to continuously improve teaching and learning  Use of multiple-choice testing limited and combined with other forms of assessment  Emphasis on criterion-referenced and curriculum-embedded assessments  Not same approach for every student, every grade, every year  Increasing use of teacher assessments in national reporting, which external summative assessments in specific years/transition points

34. 34 PrinciplesProposals  Designed around a clear vision of purpose and learning goals  Align with and advance common standards, curriculum expectations, learning objectives and instructional strategies  Start with the end in mind – clarify and confirm vision for US high school graduates as engaged citizens, active contributors within the global economy and future leaders. Requires combination of functional skills, generic and specialist knowledge, performance competencies, and personal learning strategies  Integrate assessment design with common standards for consistency of curriculum expectations and clarity of teaching and learning standards. Involves criterion-referenced and curriculum-embedded assessment items. Attention to consistency within curricular areas. Vertical alignment over grade levels to support progression of learning.

35. 35 PrinciplesProposals  Encompass and combine assessment of, for and as learning  Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning  Develop an assessment system focused on learning where summative assessments are one element of a wider repertoire of assessment information. Increase use and consistency of teachers’ assessments in reporting and monitoring student achievement and progress.  Use range of item-types and assessment approaches, including use of performance assessments, extended writing and project work. Use, but limit, time on standardized multiple-choice items and test taking.

36. 36 PrinciplesProposals  Embody high expectations to engage, motivate, support and stretch all students to progress and achieve  Build and develop professional capacity  Range of assessments, including items and tasks that involve authentic experience, critical thinking and applied performance of students. Attention to student personal learning plans and progression targets. Growth and cohort analysis. Develop capacity for student self- and peer- assessment, including group work.  Educator capacity building is critical and requires time, resources (human, financial, material) and support. Teacher moderation is a powerful strategy to support teaching and learning throughout school year. Assessment items, constructs, interval data and other technical advice are required to be available and accessible to identify linkages between assessments, curriculum expectations, instructional strategies and interventions for specific students.

37. 37 PrinciplesProposals  Assess a range of content, knowledge, skills and performance in authentic, applied and appropriate ways  Are part of a clear and flexible learning trajectory across subjects and over education/career pathways  Implement repertoire of formative and summative assessments with variety of approaches within and across grade levels to demonstrate knowledge, understanding and application.  Support clarity of learning pathways through high school and into college and career readiness, with accreditation options and credit accumulation requirements to indicate when a student is on or off track for success

38. 38 PrinciplesProposals  Support data-informed decision-making for local and system improvement  Place quality of assessment over quantity  Implement data infrastructure (technology and systems) and culture (human capacity and data access) to enable collection, analysis and use of data at all levels of system  Shift emphasis from multiple-choice standardized tests as main approach for every student, every grade, every year. Combine external tests with coursework and other approaches to assess learning over time and in applied contexts. Develop consistency across assessment aligned with common standards, quality assurance mechanisms, and attention to building consistency in educator understanding and application of assessment strategies, including teacher moderation.