1 Regents Scanning Pilot –CFN 201 Amy Reynolds, Assessment Liaison Damarine Reyes, Data/IT Liaison

22 Agenda >Overview of new Regents data requirements & DOE’s plan to meet these requirements >Scanning Process  Key points for each step in the process  Method >Next Steps: Using Regents data to inform instructional decisions  ATS reports overview  Activity >Discussion / Questions >Support

3 Regents Exam Data New York State Department of Education : New Requirement: Beginning in June 2011, NYSED will require that ALL Districts submit item-level data for select Regents exams. This includes the following subjects: English Comprehensive (not included for January pilot) Integrated Algebra Earth Science Living Environment US History & Government Global Studies & Geography Impact: Student responses need to be scanned to efficiently capture the required item-level data without impacting the turnaround time for results required by the schools. Note: Teachers must still score constructed response portions of exams Other Regents exams and RCTs will still use the traditional scoring process

4 DOE Data Capture Solution To leverage existing technology and processes currently in use in schools to capture item-level information & provide real-time results. This includes: Answer Documents: Generated & printed at school-level Scoring: Teachers to score Constructed Response items only Scanning: Using schools’ attendance scanners, locally scan documents on-site Reporting Results: Automated transfer from ATS to STARS Benefits: Increased automation of time-consuming manual processes Less overall time required to complete scoring by teachers No further data entry requirements of aggregate scores Reduction in clerical scoring errors eliminate manual scoring of MC items adding totals from different sections application of conversion tables Districts/schools/teachers will be able to better utilize item-level results to help inform instruction, similar to Grade 3-8 exams

STARSATS Attendance Scanner ATS STARS (and beyond) 1. Schedule students for Regents Exams in STARS 2. Generate & Print answer documents for each scheduled student 3. Answer documents printed for each scheduled student 6. Scan answer documents for upload to ATS 7.ATS receives data, applies scoring key & conversion chart for scale score instantly. 8.ATS passes data nightly to STARS for Report Cards and Transcripts and for reporting to NYSED Exam Day Scoring 4. Student uses pre-slugged answer document 5.Scoring committee marks Open-Ended items and bubbles in score on answer document. Attendance Scanner Re-scanning (if needed) Scanning Process Overview 5

STARS – (Review. Nothing has changed) Schedule students for Regents exam in STARS Who is responsible for STARS scheduling? Program chair of each school usually handles STARS scheduling process STARS scheduling process remains exactly the same as before Major Consideration: Should I schedule one section or multiple sections for each Regents exam? One section: Shorter scheduling process Use “Master Schedule” function Multiple Sections: Allows you to break students up into certain groups, assign each group to a certain room, print attendance rosters by exam room, and print exam invitation reports Use “Master Schedule Builder” function For support with using STARS, contact your network’s Data/IT person 6

ATS Print answer documents from ATS to attendance scanner Who is responsible for ATS printing? Pupil accounting secretary Print answer documents up to a week before Regents week There are three printing options: Regents Scanning Handbook / ATS Screenshots, page 20 1.“Print scheduled students not previously printed”- these students have been scheduled in STARS  Print all answer sheets that have not been printed yet 2.“Print by student ID”  Print an answer sheet for any student one at a time. This option could be for another public school student taking the exam at your school 3.“Print by name”  Print an answer sheet for a non-public school student who is taking the exam at your school For support with using ATS, contact your network’s Data/IT person. 7

Log in to ATS. On the screen that appears below type: REGT 8 ATS Screenshots TAB to RGPR and press RIGHT/ CTRL/RED ENTER

9 ATS Screenshots

Exam Day Use printed answer documents for test taking & scoring Students use pre-slugged answer documents on the day of the exam to answer multiple choice questions Students and Teachers need to use PENCIL for marking responses on answer documents Responses are only to be marked in the new answer document Proctors ensure that student answer documents are signed and bubbled in appropriately. Scoring team bubbles in their scores for the open-ended questions on the teacher portion of the answer document Teacher portion for all January Regents exams is page 2, except for the one-page Integrated Algebra form 10

Attendance scanner Scan answer documents for upload to ATS (p.27 Handbook) Do not scan until you’ve received notice from the DOE Use your school’s attendance scanner Scan the answer documents face up & bar code to the left (bottom first) Scan both pages for 2-page answer documents Batch page 1 documents separately from page 2 documents Scanner will print a report of how many documents were scanned successfully; for those not scanned successfully the document will be reprinted with an error message on it Rescan answer sheets if necessary Audit trail captured of re-scans and changes to the item responses Scanning must be complete by the end of the Regents window 11 For support with using the attendance scanner, contact your network’s Data/IT person or call the DIIT Helpdesk at (718)

ATS ATS processes scanned data and makes it available for use ATS receives data, applies scoring key and conversion chart for scale score instantly If both pages of a 2-page answer document are not scanned, scale score will not be displayed (Incomplete Record) ATS passes data nightly to STARS for Reports Cards and Transcripts, and for reporting to NYSED, etc. 12

Accessing Results ATS Reports Available for Printing Document Status Report Ensure that answer documents have been properly scanned. Item Report Analyze the Regents results for individual students and inform decisions on how to move these particular students forward. Review error report – multiples and omissions Item Distribution Report Analyze the Regents results for the student group as a whole in order to inform instructional decisions for these students as well as future students. 13

Document Status Report Allows you to ensure that answer documents have been properly scanned. 14 Student Name and ID Raw and Scale Score Scan Date and Time # of Answer Sheets Generated # of Answer Sheets Scanned Errors Warnings

Item Report Allows you to analyze the Regents results for individual students and inform decisions on how to move these particular students forward. 15 Student Name Student ID # Raw and Scale score Question number Answer key Student answer choices Open-ended question scores Dash = correct answer

Item Distribution Report Allows you to analyze the Regents results for the student group as a whole in order to inform instructional decisions for these students as well as future students. 16 Question Number For each answer choice, number and percentage of students that selected it Total number of responses for each question

Data Analysis Activity Examine sample data using worksheets… Distribution Report Item Report Reflect on… Uses for the reports Next Steps Share… 17

Using data to inform instructional decisions ITEM REPORT Questions this report could help you to answer Which students scored the highest? Which students scored the lowest? Which students will have to retake this Regents exam? Which questions did individual students have trouble with? Are there any patterns in terms of what skills these questions focus on? Next Steps Plan instructional interventions for students who will need to retake the Regents Support students who passed the Regents with further mastery of the areas that they had trouble with **Keep in mind that you will need to have a copy of the corresponding Regents exam to see the actual questions which correspond to the data. 18

Quality Assurance – Data Validation 5% manual verification Schools need to manually verify the scores of 5% of the answer documents for each exam have been scored properly. Identify two people at school level perform the data verification CFN Assessment Liaison will collect signed document from each pilot school Document will indicate that the school has performed the 5% manual verification

Review Scanning Process & Regents Scanning Implementation Checklist Pre-Exam - Regents Scanning Implementation Checklist STARS – Schedule students for Exams Supplies: Paper, toner Printing answer documents Exam Day - Regents Scanning Implementation Checklist Personnel available to printout extra answer sheets if necessary Students use PENCIL on the answer documents Post-Exam - Regents Scanning Implementation Checklist Complete score / open-ended questions. Wait for official notice from DOE to begin Scanning Accessing reports Quality Assurance – Complete Data Validation Documentation 20

21 Ongoing Support for Assessment Liaisons Technical Support Scheduling, ATS and STARS >Contact your Data/IT Liaison Scanner and printer >Contact the DIIT helpdesk at (718) Overall Process Support Contact your Borough Assessment Implementation Director (AID) Queens – Barbara Marcisak Training Documents Access documents from today’s training in ARIS Connect’s Summative Assessment Implementation Community: