Monitoring and Pre-scoring Activities Alternative and Alternate Assessments Virginia Department of Education September 2017

Purpose The purpose of this presentation is to provide guidance to school divisions on monitoring and conducting pre-scoring activities for VAAP collections of evidence and VSEP course work compilations.

Monitoring Prevent the “MAD DASH” before the COE and CWC are due. Periodic and systematic review of evidence-based assessments in the development process with opportunities for feedback and intervention. Why Monitor? What Monitoring is Not? To create better COE and CWC. To increase the alignment between the student’s level of achievement and the evidence presented in the COE or CWC. To catch potential problems early. Checking with the teacher Flipping through an evidence based assessment Review without feedback Prevent the “MAD DASH” before the COE and CWC are due.

Why Collections of evidence and Course Work Compilations are not successful Missing or incomplete SEI tags Ungraded or incorrectly graded evidence Use of unacceptable evidence Missing evidence to support the SOL or ASOL Use of evidence not aligned with the SOL or ASOL Evidence does not support the level of performance for the ASOL Evidence does not support the depth of knowledge for the SOL

When will monitoring occur? Who will receive feedback? Monitoring Decisions Who will monitor? How will you train them? When will monitoring occur? Who will receive feedback? How will feedback be given? What are your intervention options?

Components of Monitoring Trained Reviewers Feedback Loops Intervention Options

Trained Reviewers Trained reviewers may include, Persons with Content Knowledge and Special Education Experience School Test Coordinator Assigned or Hired Staff Building Level Teams Central Office Level Teams

Feedback Loops Teachers preparing collections Principals Building Level Coordinators (VAAP/VSEP) Trainers/Technical Assistance Staff Service Providers

Intervention Options Attendance at local training Access to state and local SOL or assessment resources Peer or small group sessions Individualized consultation and support

A Sample Division Monitoring Plan School-based Review Teams created and trained Teachers turn in evidence-based assessments every nine-weeks All evidence-based assessments reviewed by team according to division pacing chart Team report s shared with principal and central office Central office assigns Content Teams to address concerns with evidence.

A Sample Building Level Monitoring Plan Select and train school based monitoring teams Monitoring teams review COEs and CWCs every nine weeks Monitoring teams report to the school principal and school test coordinator. Feedback is provided to teachers Interventions are scheduled as needed

What Pre-scoring is Not? A detailed review of evidence-based assessments prior to submission for scoring. Why Pre-score? What Pre-scoring is Not? To provide a last check to ensure that all the scoring rules have been followed To ensure that no last- minute detail has been overlooked To provide an opportunity to make changes before the COE/ CWC is submitted for scoring A “mad dash” to create a collection of evidence The time to provide instruction

Pre-scoring Decisions Who will pre-score? How will you train them? When will pre-scoring occur in relation to the submission date? Who will receive feedback? How will feedback be given? What are your intervention options?

Rule #1- Evidence Must Be Student Generated/Evidence Must Show Individual Student Achievement VAAP VSEP Does the evidence show any level of achievement for the ASOL being defended? Are all of the ASOL addressed according to the designated level of performance? Is the evidence student generated? (not copied from the blackboard, textbook, or computer, etc.) If the evidence is from a group project, is the achievement of the particular student clearly identified? If the evidence is a worksheet, are there examples or directions that provide answers?

Rule #2 & #4 - Evidence Must Be Labeled with SEI Tag Does each piece of evidence have a completed SEI tag with: Content area SOL/ASOL Bullet(s) “Demonstrated” or “Inferred” checked (VSEP) Level of Performance (VAAP)

Scoring Rule #3 & #4 - VSEP Rule # 3: Evidence must clearly demonstrate student performance at or above the level required by the SOL being addressed. Rule #4: Evidence must clearly demonstrate the depth of knowledge expected of the student in accordance with the SOL test blueprint. Is the evidence on grade level? Are all components of the SOL addressed (stems and bullets)? Does evidence demonstrate the expected depth of knowledge in accordance with the SOL test blueprint? Is the evidence the best representation of the student’s knowledge and skills of the SOL?

Scoring rules VSEP: Rule #5 Evidence must be graded Vaap: Rule #7 Evidence must clearly address photographs, captions, and grading as appropriate. VSEP: Rule #5 Evidence must be graded Has all evidence submitted been graded with correct and/or incorrect answers clearly identified? If audio, video, or interview evidence is submitted, does this evidence include a statement of accuracy? Has all evidence submitted been graded with correct and/or incorrect answers clearly identified? If audio, video, or interview evidence is submitted, does this evidence include a statement of accuracy? Is a descriptive statement of the activity and student’s level of accuracy during the activity included in the captioned photograph? Do all anecdotal records include a statement of accuracy?

Pre-Scoring Media Considerations Organizational Considerations Do videotapes and audiotapes have transcripts? Are videotapes and audiotapes clearly labeled with SEI tags? SEI tags should be on the case, not on the DVD. Is equipment/software needed to review the media clearly identified on the case? Is the evidence organized according to the scoring worksheet? Are all required state and local forms completed?

Sample Pre-scoring Plan PHASE 2 - Central Office Review of COE and Checklist Submitted from schools Reviewed by central office teams Return to schools for correction of identified errors or omissions PHASE 1 - Building Level Review of COE and Checklist Teacher submits School-based oversight teams review Building Administrator review and sign-off Return to Teacher to address concerns

Tools for Prescoring VSEP Implementation Manual Teacher Checklist for Course Work Compilation ……………55 Administrator Checklist for Course Work Compilation…….56 - 57 Completed CWC Submission Checklist……………..…………..58 VAAP Implementation Manual Teacher Checklist for Collections of Evidence…………………79 - 81 Administrator Checklist for Collections of Evidence……….82 - 83 Completed COE Submission Checklist…………………………..84 - 85

Sample # 1 ASOL HSE-CN1 b: The student will determine central idea of the nonfiction text and select details to support it. This was the only piece of evidence submitted for this ASOL and reporting category. Received a score of 0.

Scenario 1: Student A VAAP collection of evidence is developed and submitted for a grade 8 student. All of the evidence submitted in the student’s collection of evidence reflects grade 7. SEI Tag reflects grade 7 ASOLs A number of worksheets submitted state “Grade 7” What was missed in the monitoring process for this VAAP collection of evidence A Step Further…. Student A’s collection of evidence is scored during the division’s local scoring event. Scores are submitted and validated in PearsonAccess. Student A’s collection of evidence is selected for audit. What was missed during the pre-scoring process for this VAAP collection of evidence? What do you think will happen in the audit?

Scenario #2: Teacher B Teacher B has four VAAP collections of evidence currently in development. Teacher B uses several of the same worksheets for all students. Student responses to all questions are identical on these worksheets. What was missed in the monitoring process for these VAAP collections of evidence? A Step Further…. These collections of evidence are scored during the division’s local scoring event. Scores are submitted and validated in PearsonAccess. Two of these collections of evidence are selected for audit. What was missed during the pre-scoring process for these VAAP collections of evidence? What do you think will happen in the audit?

Scenario # 3 Teacher C Teacher C has one VAAP collection of evidence currently in development. The ASOLs listed on the VAAP Content Area Cover Sheet do not match the ASOLs listed on the SEI tags. There are three pieces of evidence without SEI tags. What was missed in the monitoring process for this VAAP Collection of evidence? A Step Further…. This collection of evidence is scored during the division’s local scoring event. Scores are submitted and validated in PearsonAccess. The collection of evidence is selected for audit. What was missed during the pre scoring process for this VAAP collection of evidence? What do you think will happen in the audit?

Scenario #4 Teacher D Teacher D has one VSEP collection of evidence currently in development for Earth Science. Evidence for one SOL addresses all bullets, but does not address the stem. What was missed in the monitoring process for this VSEP Course Work Compilation? A Step Further…. This Course Work Compilation is submitted for scoring. What do you think will happen in scoring?

Monitoring & Pre-scoring Reduces or Eliminates COEs and CWCs with: Incomplete or mislabeled SEI tags Missing evidence Inaccurate or ungraded evidence Unacceptable evidence (copied from textbooks, homework, hand-over-hand)

Monitoring & Prescoring Eliminates Surprises! Ensures collections required for students educated outside of the division are completed Prevents incomplete collections in the division Prevents collections not driven by IEP, 504 or LEP Student Assessment Participation Plans

Questions? Division of Student Assessment and School Improvement (804) 225-2102 Student_Assessment@doe.virginia.gov