1. EOY MOSL Assessment Administration and Scoring Process & MOSL Updates
NYC DOE Office of Academic Quality
EOY MOSL Assessment Administration and Scoring Process & MOSL Updates
March, 2014
Welcome. Have participants introduce themselves.
Today we will be discussing a high-level overview of the administration and scoring process for all of the assessments used as part of the Measures of Student Learning component of Advance. This will be an interactive session where you will have opportunities to engage with the resources and learn where the answers to your questions can be found.
Each school has made different MOSL selections and so different parts of this presentation and materials will be most relevant.

2. Agenda
Orientation to the Measures of Student Learning (MOSL) End-of-Year Assessment, Administration, and Scoring Handbook
Review MOSL Selections & EOY Assessments
Updates
K-2 NYC Performance Test ELA Texts
Math Double Testing Waiver for Grade 7 & 8 Students in Accelerated Courses
Calendar Change Requests
Advance Implementation Funding: SAM 60
Teacher Level Selection Tool
Create an EOY Advance Implementation Plan
Questions
Here is a quick preview of what we will cover today.
I have DRAFT copies of the EOY Handbook that are confidential and I will be collecting them at the end.
First we will review all of the EOY Assessment Administration and Scoring processes from start to finish.
This 7-step process looks slightly different for different assessments so we will have time to dive more deeply into the assessments that are most relevant to your school/network.

3. Session Approach and Norms
High Engagement, Low Tech – Please turn cell phones on silent.
Contribute to the Learning – Throughout the afternoon, write any outstanding questions on index cards and place them in the center of your table. Q&A at the end of the day will address these and any other lingering questions.
Maximize Our Time Together – Be Mindful of “Airtime”— Use this day to collaborate, troubleshoot and brainstorm with your colleagues. Save complex, individual questions for other avenues of communication and support.
Before we begin, let’s pause to discuss how we approach this session.
**discuss the norms with participants**

4. Objectives
Participants will gain a high-level understanding of the step-by-step process schools must complete to successfully administer and score EOY assessments centered on the EOY Assessment Administration Checklist 
Participants will create an End-of-Year MOSL Implementation Plan 
Review objective

5. EOY MOSL Assessment Administration Handbook
Where to Begin?
EOY MOSL Assessment Administration Handbook
Online guide, located on the Intranet
One-stop, comprehensive resource that organizes all guidance related to administering and scoring MOSL assessments.
Numerous hyperlinks to all relevant resources.
Organized according to implementation step and assessment:
Plan for EOY administration
Prepare and administer
Score and submit
This EOY MOSL Assessment Administration and Scoring Handbook will be the primary starting point for understanding all the information and processes related to EOY Assessment administration.
Review the bullets in the callout box.
I am sure that as you begin to explore this document you will have many questions. Please note these questions for later. We will walk through this process step-by-step as a group. Take five minutes now to just navigate through the document and explore its structure. Don’t worry about the details as we will have plenty of time to discuss those in this session.
Ask all participants to orient themselves to the EOY handbook. Give participants several minutes to skim this document on their own. Ask them to pay close attention to the Table of Contents and the Executive Summary. Tell them to note any questions they have but to wait until dive deeper to ask those questions.
Take 3-4 minutes to orient yourself to the handbook.
Please note that information connected to specific page numbers in the EOY Handbook and related materials are subject to change pending the final release of the EOY MOSL Assessment Administration Handbook.

6. Review MOSL Selections & EOY MOSL Assessments
Review your school specific MOSL Selections
Review the MOSL Assessments in yellow on the right of your spreadsheet
The yellow highlighted area identifies the specific MOSL assessments your school is required to administer by grade based on your MOSL selections

7. Understanding EOY Administration and Scoring1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials
Keeping your MOSL Selections & EOY Assessments in mind, take 10 minutes to independently review the:
Overview for Principals (P. 2-3)
End-of-Year Assessment Calendar for Advance (P. 4-5)
NYC Performance Assessments- End-of-Year Assessment Topics
Administrator EOY MOSL Assessment Checklist (P. 7-11)
Note any questions that arise.
Which steps might be the most challenging for your school?  2 3 4 5
Let’s start by discussing the 7-step process that schools must take to get this work done. The first section in the EOY Assessment Handbook is the Administrator Checklist. Before we discuss the checklist, take a moment to independently review these steps.
Consider which steps will be the most challenging for your schools and networks?
**They could review this in the electronic copies of the Handbook, but it is recommended that participants have a hard copy of the Administrator’s Checklist. 6 7

8. Navigating the EOY Assessment Administration and Scoring Handbook1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials
Overview of the Handbook’s Structure
EOY Checklist- The entire Handbook is organized into this 7-step process. The Checklist can help you navigate to the information you need, serving as a table of contents for the document. You can click on the orange headers to link to the assessment specific information you need.
Each section of the Handbook serves as a stand-alone resource:
Overview for Principals
Calendar Planning Tool
Four administration and scoring guidance documents:
NYC Performance Assessments
Running Records
FAST
Scantron Performance Series 2  3 4 5
The Overview for Principals contains high-level information about new or updated policies.
The EOY Checklist and supporting guidance in the Handbook contains detailed instructions for Assessment Coordinators.
As indicated, the sections of the handbook can be printed as separate documents.
NOTE: The administration and scoring guidance for NYSED assessments is found in a different resource on the Intranet. This resource covers the administration and scoring guidance for all other MOSL assessments. NYSED Assessments, AP exams, NYC Performance Assessments, and 3rd Party Assessments may all be used as a Measure of Student Learning (MOSL) this year; it will be important to coordinate administration and scoring processes across these assessments. 6 7

9. Step 1: Plan for EOY Administration and Scoring
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials
Review windows for printing, administration, norming and scoring If necessary, consider additional dates and times for norming and scoring EOY assessments.
If relevant, consider content of NYC Performance Assessments and ensure assessment schedule is aligned with course curricula.
Identify and schedule key EOY administration tasks and assign roles and responsibilities, including:
Perform roster verification
Confirm/ update Teacher-Level MOSL Selections in the Advance Web Application (functionality tentatively available March 24th)
Share Teacher Assessment Administration Checklist with teachers
Schedule assessment printing, administration, norming and scoring activities 2 3 4 5
Refer to the checklist and follow along as we discuss each step in detail. We will also pause to discuss new and updated resources that correspond to specific steps in the checklist.
Let’s discuss the first step: Planning for Administration and Scoring.
Review the bullets.
Discuss the significance of roster verification. Point to the additional handout published on Advance. 6 7

10. New Resource Spotlight:
Calendar Planning Tool Pgs 1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials
Designed to support principals and assessment coordinators to create an end-of-year assessment implementation calendar for their school.
Contains action steps and templates that will help identify key tasks and dates.
Includes a calendar template with key milestones highlighted to support school leaders integrate these EOY MOSL tasks into existing timelines and structures. 2 3 4 5
One new resource that helps principals execute this first step of the checklist is the EOY Calendar Planning Tool. Refer to this on page 10. 6
We will look at the Calendar Planning Tool more closely later in the training and allow time for you to create an EOY Advance implementation plan. 7

11. New Resource Spotlight:
Teacher Checklist 1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials
Similar to the EOY MOSL Assessment Checklist (P. 7-11), but shares relevant EOY assessment information tailored to teachers
Contains specific action items aligned to each of the 7 steps.
Teacher Checklist has been released on the Advance intranet page and is included in your folder. 2 3 4 5
The EOY Handbook is meant to be a resource for School Leaders and Network staff. An additional resource designed to share with teachers the information that is most relevant to them has also been created. It’s called the Teacher Checklist. 6 7

12. Step 2: Prepare Assessments
Each assessment requires different preparation steps.
Depending on the assessment, specific tasks may include the following:
Select and download certain online assessment(s).
Print or request assessment materials (FAST)
Receive or select texts
Print ATS score sheets (when necessary)
Ensure teachers have the relevant materials necessary to administer the assessment (assessment tasks, texts, etc.)
Check technical and audio requirements for specific assessments (Scantron Performance Series)
Obtain log-in information (FAST & Scantron Performance Series) 1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials 2 3 4 5
You’ll notice in the Handbook that step 1 is the same for all assessments. However steps 2-7 vary slightly depending on the assessment. The handbook has been designed to walk you through these steps for each assessment individually. For this part of the presentation we will be speaking generally, about all assessments, but keep in mind there are specific considerations that are only relevant to specific assessments. 6 7
Locate and review Step 2 for at least one of your EOY MOSL Assessments and review the steps for “Prepare Assessments”. (Refer to P. 8 in the handbook to identify the page for the assessment you’re looking for.)

13. Step 3: Administer Assessments1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials
Each assessment requires different administration steps.
Depending on the assessment, specific administration tasks may include the following:
Ensure the security of testing materials
Follow administration guidelines in the Handbook and or in the assessment system
Collect assessment materials.
Note: Even if multiple teachers in your school are evaluated on the same assessment, students should only take each assessment once. 2 3 4 5
Answer: Since each assessment has slightly different requirements, these steps are presented together in the handbook and within each step the assessment-specific tasks are further explained. 6 7
Locate and review Step 3 for at least one of your EOY MOSL Assessments and review the steps for “Administer Assessments”. (Refer to P. 9 in the handbook to identify the page for the assessment you’re looking for.)

14. Steps 4 and 5: Prepare for Scoring and Score
Scoring Overview:
Scoring requirements vary greatly from one assessment to another.
Depending on the assessment, specific scoring tasks may include the following:
Norm scoring practices
(NYC Performance Assessments, FAST, Running Records)
Review scoring policies
Finalize scoring schedule
Coordinate “school-based distributed scoring” for all assessments (except FAST and K-2 Math NYC Performance Assessment)
For assessments that do not require distributed scoring, administrators must review and approve assessment results. Approval is indicated on an Assessment Results Approval Form (P. 78) . (FAST, K-2 Math Performance Assessment) 1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials 2 3 4 5 6 7

15. Steps 4 and 5: Prepare for Scoring and Score
Distributed Scoring:
State teacher evaluation law stipulates that teachers with a “vested interest” cannot score the assessments of students they directly teach. Whenever “practicable” schools must establish processes whereby other teachers would score a teacher’s MOSL assessments.
Because this scoring policy is not practicable for FAST or K-2 Math NYC Performance Assessments, teachers using these assessments as their MOSL are permitted to score their own students' assessments.
The NYCDOE has outlined a process for administrator review of these scores (Assessment Results Approval Form P. 78). 1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials 2 3 4 5
NOTE: Running Records and FAST are considered NYC Performance Assessments but for the purpose of this training when we refer to NYC Performance Assessments, we are referring to the written NYC Performance Assessments.
Point out that Scantron Performance Series is NOT MANUALLY SCORED and therefore does not require any sort of distributed scoring. There is some confusion in the field about this. 6 7 15

16. Steps 4 and 5: Prepare for Scoring and Score1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials
Distributed Scoring- Who is Eligible to Score?
It is preferred that assessments be scored by teachers who hold a license in the subject area and/ or are currently teaching the subject area. Principals, APS, Achievement coaches or other network staff may participate in scoring at the discretion of the network and principal.
Teachers may score assessments from grades that they are not currently teaching. For example, a 9th grade ELA teacher may score a 10th grade ELA assessment. 2 3 4 5
Let’s discuss who is actually eligible to score assessments.
*review the bullets. 6 7 16

17. Steps 4 and 5: Prepare for Scoring and Score1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials
Scoring Guidelines:
In accordance with the state regulations, the DOE recommends that schools choose from one of the following scoring models:
Option 1: Teachers in the same grade and subject swap to score end-of-year assessments.
Option 2: Teachers of the same subject in different grade levels swap to score end-of-year assessments.
Option 3: At the principal’s discretion, teachers who are properly normed in an end-of-year assessment may score.
Clusters and networks will establish structures to support schools where the options listed above are not feasible (i.e. schools where there is only one content area teacher for the whole school). 2 3 4 5
Here are a few recommendations to provide schools as they implement school-based distributed scoring. If these options are absolutely infeasible, schools should reach out to their networks for individualized support. 6 7 17

18. Steps 4 and 5: Prepare for Scoring and Score
Group Norming Sessions for NYC Performance Assessments*:
Schools should conduct norming sessions after administering but before scoring the assessment in order to ensure that all teachers have a common understanding of the rubric and increase fairness of the scoring process.
Group norming sessions should follow this process (approx. 2-3 hours):
Step 1: Review the rubric and discuss expected performance at each performance level for each rubric trait.
Step 2: Review annotated sample student work and discuss the evidence for each score.
Step 3: Individually score one EOY assessment, then discuss as a group the evidence for each score.
Step 4: Collectively score several students’ NYC Performance Assessments and come to consensus on a score for each trait.  The scorer ID for these collectively scored papers can be
*Norming scoring practices are also recommended for Running Records and FAST." 1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials 2 3 4 5
Explain what norming is and why it is important: Increases fairness, adds instructional value, focuses on a shared understanding of the rubric, etc.
Note that this is the process for NYC Performance Assessments but that the NYC DOE recommends norming scoring practices for all manually scored assessments (i.e. Running Records and FAST).
Norming for Running Records and FAST will need happen in advance of assessment administration because these assessments are administered and scored at the same time. 6 7

19. Steps 4 and 5: Prepare for Scoring and Score1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials
Review the norming steps found on page 32 for the writing prompt, pg. 46 for running recordsin the
EOY Assessment Handbook.
Why is it so critical that teachers scoring the NYC Performance Assessment are properly normed?
Why is this also considered a best practice for Running Records and FAST? 2 3 ? 4 5 ?
Note that this is the process for NYC Performance Assessments but that the NYC DOE recommends norming scoring practices for all manually scored assessments (i.e. Running Records and FAST in addition to NYC Performance Assessments).
This is so critical for NYC Performance Assessments because they are subject to distributed scoring. Since teachers without a vested interest will be scoring the assessments of other teachers, it is critical that all teachers share a clear understanding of the rubric and a common definition of the various scores. This will increase the perception of fairness. 6 7
Locate and review Steps 4 & 5 for at least one of your EOY MOSL Assessments and review the steps for “Prepare for Scoring & Score”. (Refer to P in the handbook to identify the pages for the assessment you’re looking for.)

20. Archive Assessment Materials
Step 6: Submit Scores 1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials
Submitting Scores- Overview:
Scores for different assessments are submitted through different systems.
Depending on the assessment, specific submission tasks may include the following:
Scan ATS Score Sheets
Submit Scores in respective data systems
AssessmentPro
STARS Classroom
SANDI & FAST Online Portal
Generate reports and fix errors. 2 3 4 5 6 7
Locate and review Step 6 for at least one of your EOY MOSL Assessments and review the steps for “Submit Scores”. (Refer to P. 10 in the handbook to identify the pages for the assessment you’re looking for.)

21. Step 7: Archive Assessment Materials1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials
Sign the MOSL Assessment Material Security Form (P. 72)
Archive Assessment Materials
It is the responsibility of the school to retain and store all Advance assessment-related materials. Assessments, student work, data collection tools, scanning confirmation pages and Assessment Results Approval Forms should be organized by course and section and stored in an accessible, secured location by the principal, assessment coordinator or designee for a minimum of three years. 2 3 4 5 6 7
Locate and review Step 7 for at least one of your EOY MOSL Assessments and review the steps for “Administer Assessments”. (Refer to P. 11 in the handbook to identify the pages for the assessment you’re looking for.)

22. Updates
K-2 NYC Performance Test ELA Texts
Math Double Testing Waiver for Grade 7 & 8 Students in Accelerated Courses
Calendar Change Requests
Advance Implementation Funding: SAM 60
Teacher Level Selection Tool

23. Activity: Creating an EOY MOSL Implementation Plan
Sample MOSL Selections
Review the Calendar Planning Tool and the Advance EOY calendar.
Begin planning for End-of-Year MOSL assessments at your school.
Additional Planning Support: MOSL End-of-Year (EOY) Assessment Analysis and Next Steps Tool  1
Plan for Administration and Scoring
Prepare Assessments
Administer Assessments
Prepare for Scoring
Score
Submit Scores
Archive Assessment Materials
Grade/ Subject
Local Measure
State Measures
6-8 ELA
NYC Performance Assessments –
ELA
State ELA Tests
6-8 Math
State Math Tests (Lowest Third)
State Math Tests
6-8 Social Studies
NYC Performance Assessments – Social Studies (lowest third)
NYC Performance Assessments – Social Studies
6-7 Science
NYC Performance Assessments – Science (lowest third)
NYC Performance Assessments - Science
8 Science
NYC Performance Assessments – 8th Grade Science
State Grade 8 Science Test 2 3 4 5
The Handbook includes the windows for Printing, Scoring, and Submitting. It is necessary to thoughtfully consider how schools would accomplish this work.
If participants are struggling, push them to consider the following questions:
Where, in the given windows, would it make the most sense to schedule administration?
When might you schedule per session scoring time to accommodate heavy PBA burden, etc.
Some of these points are not immediately apparent so make sure that participants are reaching the following points in their table discussions.
Some factors to consider:
You might need extra time for distributed scoring here given the heavy performance assessment burden, however it is important to consider when the content is being taught.
The EOY performance assessments are aligned to scope and sequence for these content areas so it is important to make sure that teachers have covered this material BEFORE they administer the assessments. 6 7

24. Thank you!
Lori Ann O’Connor, Senior MOSL Implementation Specialist Office of Academic Quality, Cluster 6
Office (718)
Cell (646)
This concludes the EOY Assessment Administration and Scoring Turnkey presentation.