1. Monitoring and Pre-scoring Activities
Alternative and Alternate Assessments
Virginia Department of Education
September 2017

2. Purpose
The purpose of this presentation is to provide guidance to school divisions on monitoring and conducting pre-scoring activities for VAAP collections of evidence and VSEP course work compilations.

3. Monitoring Prevent the “MAD DASH” before the COE and CWC are due.
Periodic and systematic review of evidence-based assessments in the development process with opportunities for feedback and intervention.
Why Monitor?
What Monitoring is Not?
To create better COE and CWC.
To increase the alignment between the student’s level of achievement and the evidence presented in the COE or CWC.
To catch potential problems early.
Checking with the teacher
Flipping through an evidence based assessment
Review without feedback
Prevent the “MAD DASH” before the COE and CWC are due.

4. Why Collections of evidence and Course Work Compilations are not successful
Missing or incomplete SEI tags
Ungraded or incorrectly graded evidence
Use of unacceptable evidence
Missing evidence to support the SOL or ASOL
Use of evidence not aligned with the SOL or ASOL
Evidence does not support the level of performance for the ASOL
Evidence does not support the depth of knowledge for the SOL

5. When will monitoring occur? Who will receive feedback?
Monitoring Decisions
Who will monitor?
How will you train them?
When will monitoring occur?
Who will receive feedback?
How will feedback be given?
What are your intervention options?

6. Components of Monitoring
Trained Reviewers
Feedback Loops
Intervention Options

7. Trained Reviewers
Trained reviewers may include,
Persons with Content Knowledge and Special Education Experience
School Test Coordinator
Assigned or Hired Staff
Building Level Teams
Central Office Level Teams

8. Feedback Loops
Teachers preparing collections
Principals
Building Level Coordinators (VAAP/VSEP)
Trainers/Technical Assistance Staff
Service Providers

9. Intervention Options
Attendance at local training
Access to state and local SOL or assessment resources
Peer or small group sessions
Individualized consultation and support

10. A Sample Division Monitoring Plan
School-based Review Teams created and trained
Teachers turn in evidence-based assessments every nine-weeks
All evidence-based assessments reviewed by team according to division pacing chart
Team report s shared with principal and central office
Central office assigns Content Teams to address concerns with evidence.

11. A Sample Building Level Monitoring Plan
Select and train school based monitoring teams
Monitoring teams review COEs and CWCs every nine weeks
Monitoring teams report to the school principal and school test coordinator.
Feedback is provided to teachers
Interventions are scheduled as needed

12. What Pre-scoring is Not?
A detailed review of evidence-based assessments prior to submission for scoring.
Why Pre-score?
What Pre-scoring is Not?
To provide a last check to ensure that all the scoring rules have been followed
To ensure that no last- minute detail has been overlooked
To provide an opportunity to make changes before the COE/ CWC is submitted for scoring
A “mad dash” to create a collection of evidence
The time to provide instruction

13. Pre-scoring Decisions
Who will pre-score?
How will you train them?
When will pre-scoring occur in relation to the submission date?
Who will receive feedback?
How will feedback be given?
What are your intervention options?

14. Rule #1- Evidence Must Be Student Generated/Evidence Must Show Individual Student Achievement
VAAP
VSEP
Does the evidence show any level of achievement for the ASOL being defended?
Are all of the ASOL addressed according to the designated level of performance?
Is the evidence student generated? (not copied from the blackboard, textbook, or computer, etc.)
If the evidence is from a group project, is the achievement of the particular student clearly identified?
If the evidence is a worksheet, are there examples or directions that provide answers?

15. Rule #2 & #4 - Evidence Must Be Labeled with SEI Tag
Does each piece of evidence have a completed SEI tag with:
Content area
SOL/ASOL
Bullet(s)
“Demonstrated” or “Inferred” checked (VSEP)
Level of Performance (VAAP)

16. Scoring Rule #3 & #4 - VSEP
Rule # 3: Evidence must clearly demonstrate student performance at or above the level required by the SOL being addressed.
Rule #4: Evidence must clearly demonstrate the depth of knowledge expected of the student in accordance with the SOL test blueprint.
Is the evidence on grade level?
Are all components of the SOL addressed (stems and bullets)?
Does evidence demonstrate the expected depth of knowledge in accordance with the SOL test blueprint?
Is the evidence the best representation of the student’s knowledge and skills of the SOL?

17. Scoring rules VSEP: Rule #5 Evidence must be graded
Vaap: Rule #7 Evidence must clearly address photographs, captions, and grading as appropriate.
VSEP: Rule #5 Evidence must be graded
Has all evidence submitted been graded with correct and/or incorrect answers clearly identified?
If audio, video, or interview evidence is submitted, does this evidence include a statement of accuracy?
Has all evidence submitted been graded with correct and/or incorrect answers clearly identified?
If audio, video, or interview evidence is submitted, does this evidence include a statement of accuracy?
Is a descriptive statement of the activity and student’s level of accuracy during the activity included in the captioned photograph?
Do all anecdotal records include a statement of accuracy?

18. Pre-Scoring
Media Considerations
Organizational Considerations
Do videotapes and audiotapes have transcripts?
Are videotapes and audiotapes clearly labeled with SEI tags? SEI tags should be on the case, not on the DVD.
Is equipment/software needed to review the media clearly identified on the case?
Is the evidence organized according to the scoring worksheet?
Are all required state and local forms completed?

19. Sample Pre-scoring Plan
PHASE 2 - Central Office
Review of COE and Checklist
Submitted from schools
Reviewed by central office teams
Return to schools for correction of identified errors or omissions
PHASE 1 - Building Level
Review of COE and Checklist
Teacher submits
School-based oversight teams review
Building Administrator review and sign-off
Return to Teacher to address concerns

20. Tools for Prescoring VSEP Implementation Manual
Teacher Checklist for Course Work Compilation ……………55
Administrator Checklist for Course Work Compilation……
Completed CWC Submission Checklist……………..…………..58
VAAP Implementation Manual
Teacher Checklist for Collections of Evidence…………………
Administrator Checklist for Collections of Evidence………
Completed COE Submission Checklist…………………………

21. Sample # 1
ASOL
HSE-CN1 b: The student will determine central idea of the nonfiction text and select details to support it.
This was the only piece of evidence submitted for this ASOL and reporting category.
Received a score of 0.

22. Scenario 1: Student A
VAAP collection of evidence is developed and submitted for a grade 8 student.
All of the evidence submitted in the student’s collection of evidence reflects grade 7.
SEI Tag reflects grade 7 ASOLs
A number of worksheets submitted state “Grade 7”
What was missed in the monitoring process for this VAAP collection of evidence
A Step Further….
Student A’s collection of evidence is scored during the division’s local scoring event.
Scores are submitted and validated in PearsonAccess.
Student A’s collection of evidence is selected for audit.
What was missed during the pre-scoring process for this VAAP collection of evidence?
What do you think will happen in the audit?

23. Scenario #2: Teacher B
Teacher B has four VAAP collections of evidence currently in development.
Teacher B uses several of the same worksheets for all students.
Student responses to all questions are identical on these worksheets.
What was missed in the monitoring process for these VAAP collections of evidence?
A Step Further….
These collections of evidence are scored during the division’s local scoring event.
Scores are submitted and validated in PearsonAccess.
Two of these collections of evidence are selected for audit.
What was missed during the pre-scoring process for these VAAP collections of evidence?
What do you think will happen in the audit?

24. Scenario # 3 Teacher C
Teacher C has one VAAP collection of evidence currently in development.
The ASOLs listed on the VAAP Content Area Cover Sheet do not match the ASOLs listed on the SEI tags.
There are three pieces of evidence without SEI tags.
What was missed in the monitoring process for this VAAP Collection of evidence?
A Step Further….
This collection of evidence is scored during the division’s local scoring event.
Scores are submitted and validated in PearsonAccess.
The collection of evidence is selected for audit.
What was missed during the pre scoring process for this VAAP collection of evidence?
What do you think will happen in the audit?

25. Scenario #4 Teacher D
Teacher D has one VSEP collection of evidence currently in development for Earth
Science.
Evidence for one SOL addresses all bullets, but does not address the stem.
What was missed in the monitoring process for this VSEP Course Work Compilation?
A Step Further….
This Course Work Compilation is submitted for scoring.
What do you think will happen in scoring?

26. Monitoring & Pre-scoring
Reduces or Eliminates COEs and CWCs with:
Incomplete or mislabeled SEI tags
Missing evidence
Inaccurate or ungraded evidence
Unacceptable evidence (copied from textbooks, homework, hand-over-hand)

27. Monitoring & Prescoring
Eliminates Surprises!
Ensures collections required for students educated outside of the division are completed
Prevents incomplete collections in the division
Prevents collections not driven by IEP, 504 or LEP Student Assessment Participation Plans

28. Questions?
Division of Student Assessment and School Improvement (804)