Developing Monitoring and Pre-Scoring Plans for Alternate/Alternative Assessments Virginia Department of Education Division of Student Assessment and School Improvement August 2010

Purpose of Web Conference To assist school divisions in developing plans to monitor and pre-score Collections of Evidence (COE) for the Virginia Alternate Assessment Program (VAAP) and the Virginia Grade Level Alternative (VGLA) and Course Work Compilations (CWC) for the Virginia Substitute Evaluation Program (VSEP).

Developing Monitoring and Pre-Scoring Plans for Alternate/Alternative Assessments What are monitoring and pre-scoring? Why should divisions develop monitoring and pre-scoring plans? Who should be involved in developing the monitoring and pre-scoring plans? What components should the monitoring plan include? What components should the pre-scoring plan include?

What are monitoring and pre-scoring?

Monitoring A periodic and systematic review of COEs and CWCs while they are in the development process with opportunities for feedback and intervention.

What Monitoring is Not Checking in with the teacher Flipping through COEs or CWCs Reviewing COEs and CWCs without providing feedback

Pre-Scoring A detailed review of COEs and CWCs to address technical errors (i. e., grading, SEI tags) prior to scoring with opportunities for feedback and corrections.

What Pre-Scoring is Not Assigning a rubric score to the evidence Altering student evidence or student responses Judging the instructional quality of the evidence

Why should divisions develop monitoring and pre-scoring plans?

Monitoring Keeps division staff abreast of the status of each COE and/or CWC Ensures that teachers are providing instruction based on the Standards of Learning (SOL) Ensures that instruction is on target with the division’s pacing guide

Monitoring Ensures that technical issues (i.e., grading, SEI tags) are appropriately addressed Provides the opportunity to address issues with ample time for correction and/or intervention

Pre-scoring Provides a final opportunity to correct technical errors and omissions that may result in the loss of credit to students prior scoring Results in the creation of better COEs and CWCs

Who should be involved in developing the monitoring and pre- scoring plans?

Who should be involved in developing the plans? Depends on... Number of COEs and CWCs Available central office resources Available building level resources

Who should be involved in developing the plans? Central Office Staff Division Director of Testing (DDOT) or designee Special Education Director or designee Director of Instruction or designee Title III Coordinator or designee Building Level Staff Principal or designee School Test Coordinator Special Education Chairperson

Questions Questions may be submitted by typing your question into the chat window. Make sure that your questions are sent to “Host and Presenter.”

What components should the monitoring plan include?

Critical Planning Information Develop a Master Participation List for each alternate/alternative assessment program that includes...  Name of student  Grade level of student  School  Content area being assessed by alternate/alternative assessment  Teacher(s) responsible for instruction in the content area assessed

Monitoring Plan Decisions  Who will monitor?  How will monitors be trained?  When will monitoring occur?  Who will receive feedback?  How will feedback be given?  What options are available if corrections are needed?

Remember! Your monitors will provide training and/or technical guidance to teachers submitting COEs and are not allowed to score these COEs.

Who will monitor? Staff with content knowledge and alternate/alternative assessment knowledge  Hired consultants  Special education or testing specialists  Instructional specialists  LEP specialists  School test coordinators  Title I mathematics and reading specialists  Experienced teachers  Building administrators

How will monitors be trained? Provide access to implementation manuals and other resources (i.e. testing memos, VGLA worksheets, SEI tags, Curriculum Frameworks) Conduct local training for monitors to include review and discussion of scoring rules, SOL or ASOL information, issues generated from the previous year’s scoring, etc.

When will monitoring occur? Monthly End of each grading period At the same time as benchmark testing Every other month Other

Who will receive feedback?  Teachers  Principals  Alternate/Alternative Assessment Leaders  Central Office Staff  Trainers/Technical Assistance Providers

How will feedback be given? Post It notes on COEs and CWCs Locally created feedback forms given to teachers, principals, and/or others Meetings with teachers, principals, and/or others about monitoring results Other

What options are available if corrections are needed? Individualized consultation and support Peer or small group training sessions Division-level professional development Access to state and local SOL and/or ASOL assessment resources Other

A Sample Monitoring Plan for 500 COEs School-based Review Teams will be created in each school. DDOT and Special Education Director will train School-based Review Teams using implementation manuals, VDOE power points, division pacing charts, and other resources. Teachers submit COEs or CWCs to the School-based Review Team at the end of each nine-weeks.

A Sample Monitoring Plan School-based Review Teams will report their finding on each COE or CWC using a locally created form. A copy of the form will be distributed to the teachers, principals and central office staff. Central office will dispatch instructional specialist and assessment specialist to schools in need of additional support and training.

A Sample Monitoring Plan 1 Select And Train School-based Review Teams 3 Instructional and assessment specialists dispatched to schools based on 9 Week Reports 2 School-based Review Teams check COEs every 9 weeks according to Pacing Chart and report to principals and central office staff

A Sample School-Based Review Team 9-Week Monitoring Form – Grade 4 Reading Student % of SOL Evidenced % of SOL Evidenced % of SOL Evidenced Less than 50% of SOL Evidenced Ted Smith X Kay Blue X Joe Davis X Ann Jones X

A Sample Teacher Review Form VGLA 9 Week Review Sheet School-Based Teams Teacher: ____________________ Student: ___________________________ Grade Level:______ Content Area:_________________________________________ REVIEW for: ____ 1st Nine Weeks ____ 3rd Nine Weeks ____ 2nd Nine Weeks ____ 4th Nine Weeks Reviewed By:____________________________ Date:________ Collection Status: Address the following questions: (1) Is there evidence for all of the standards for the nine weeks according to the Pacing Chart? (2) Does the work submitted align with the standards satisfactorily? (2) Does the evidence demonstrate student mastery? (3) Has the student work been graded accurately? (4) Other? Recommendation(s): Be as specific as possible Follow-up review needed for implementation of recommendations: ___Yes ____No Follow –up Review Date: ________________

Monitoring Questions Questions may be submitted by typing your question into the chat window. Make sure that your questions are sent to “Host and Presenter.”

Pre-Scoring Decisions  Who will pre-score?  How will pre-scorers be trained?  When will pre-scoring occur in relation to submission date?  Who will receive feedback?  How will feedback be given?  What options are available if corrections are needed?

Persons selected for pre-scoring can not serve as scorers. Pre-scorers should be knowledgeable about all scoring rules and the SOL and/or ASOL content Pre-scorers should be detail oriented. Who will pre-score?

Remember! Your pre-scorers will provide training and/or technical guidance to teachers submitting COEs and are not allowed to score these COEs.

Training should include a list of specific issues to look for in each COE or CWS  grading  correctly completed SEI tags  detailed anecdotal records  captioned photographs How will pre-scorers be trained?

Other helpful questions to ask in Pre-scoring Is the evidence organized according to the scoring worksheet? Are all required state and local forms included and completed?

Select a time to pre-score: that will give ample time for COEs or CWCs to be complete or near completion. that will give opportunity to return COEs or CWCs to teachers for corrections and returned by local due date. When will pre-scoring occur in relation to local due date?

Teacher(s) submitting COE or CWC. Building level principals Others Who will receive feedback?

Post-it Notes on COEs and CWCs Local created forms given to teachers, principals, and/or others Meetings with teachers, principals, and/or others Other How will feedback be given?

Return COEs to teachers for corrections. Assign staff to assist teacher in corrections such as grading evidence, organizing evidence, completing SEI tags. Options may be limited or non-existent due to time constraints Other What options are available if corrections are needed?

Sample Pre-scoring Plan Pre-scoring will be conducted by teachers, building level staff and central office staff in three phases  Teacher Level  Building Level  Central Office Level Scoring rules will be reviewed with all persons involved in pre-scoring. Checklists will be provided.

Sample Pre-Scoring Plan PHASE 1 – Teacher Review  Conducted April  Submitting Teacher reviews COEs or CWCs using Teacher Checklists in Implementation Manuals PHASE 2 – School-based Review  Conducted April 19 – 26  COEs or CWCs using notes and local forms gathered  Building Administrator reviews COEs or CWCs using Administrator Checklist in Implementation Manuals

Sample Pre-Scoring Plan Phase 3 – Central Office pre-scoring Team Review Conducted April 27 – May 7  Central Office team reviews COEs and CWC against scoring rules. Findings noted on Post-It notes placed on evidence.  Central Office teams return COEs or CWCs to schools to correct errors and omissions.  COEs or CWCs are corrected by teachers. Teachers sign affidavits  COEs or CWCs submitted to principals or designee. Principal or designee signs affidavits.  COEs or CWC submitted for scoring.

Monitoring & Pre-Scoring Results Reduces or Eliminates COEs and CWCs with:  Missing Evidence or SEI tags  Unacceptable Evidence (textbooks, homework)  Inaccurate or Ungraded Evidence

Monitoring & Pre-Scoring Results Eliminates Surprises :  COEs or CWCs not done for students educated outside of the division  Incomplete COEs or CWCs in the division  COEs or CWCs not driven by IEP, 504 or LEP Plans

Questions Questions may be submitted by typing your question into the chat window. Make sure that your questions are sent to “Host and Presenter.”

Thank you for participating in this web conference. You will receive an evaluation form from Pearson regarding this presentation. Please complete and submit. We appreciate your feedback.