1. Children First Focus on Accountability Accountability Initiative July 19, 2006 CONFIDENTIAL

2. 1 Accountability Initiative –Vision and Overview –Achievement Reporting and Innovation System (ARIS) Accountability Initiative Implementation Timeline CONTENTS

3. 2 THE PROBLEM Number of Schools = 52 Number of Schools = 82 Number of Schools = 113 Number of Schools = 189 Number of Schools = 205

4. 3 THE SOLUTION: A CYCLE OF CONTINUOUS IMPROVEMENT Leadership / clear goals Innovation / evaluation Adjustment SCHOOLS Enablers / Tools Continuous learning and support (e.g. Empowerment Schools Intensive) Periodic assessments ARIS Accountabilities / Measures Progress Reports Quality Reviews

5. 4 INNOVATIONS IN ACCOUNTABILITY New accountability measurementsOld accountability measurements Snapshot of performance Movement only across single proficiency drivers- ignores outcomes on high/ low ends Only grades schools with challenging populations: SINI, SURR Incentives to focus on students least in need Progress: what the school adds All amounts of improvement in student achievement Universal application: All schools are graded: A, B, C, D, F Extra credit for gains by students most in need School improvement plan and target setting Leadership change School restructuring School closure 4 Year Cycle of Consequences

6. ACCOUNTABILITIES / MEASURES: SCATTER CHART

7. 6 ACCOUNTABILITIES / MEASURES: QUALITY REVIEWS Gather data Plan and set goals Align instruction Build and align capacity Monitor and revise Continuous data-driven improvement Understand what each student knows and is able to do monitor, and Measure the students progress over time. Set each students next learning step. Collaborate with student / parents to set goals for improving teaching practice and accelerating student learning Align school instructional activity and resources around focused plans for accelerating individual student learning Align instructional skills and capacity development around established goals for accelerating individual student learning Update documentation / best practices on effectiveness of its improvement plans and flexibility of change processes

8. 7 Identify high leverage points for intervention to improve student performance Align instructional practice and resources, access external resources to support change Assess progress and make mid- course corrections Start by analyzing schools performance data to ID areas that need improvement Create urgency with publicized results Select a point of focus Analyze student- level data, instructional practices, and adult learning needs ENABLERS / TOOLS: CREATE A CYCLE OF CONTINUOUS LEARNING Top-down qualitative analysis from Quality Review Bottom-up quantitative view from Assessments Publish Progress Reports Deeply informed interventions

9. 8 ENABLERS / TOOLS: A SET OF DIAGNOSTIC TOOLS-PERIODIC PROGRESS MEASURES

10. 9 ENABLERS / TOOLS: DIAGNOSTIC ASSESSMENT METHODOLOGIES October 2006Jessica Holloway Edward Smith Raul Fuentes Summarizing Main Idea Fact vs. Opinion Predicting Context Clues Paraphrasing December 2006Jessica Holloway Edward Smith Raul Fuentes Summarizing Main Idea Fact vs. Opinion Predicting Context Clues Paraphrasing

11. 10 ENABLERS / TOOLS: PERIODIC ASSESSMENT – A MENU OF OPTIONS THAT SUIT THE NEEDS OF SCHOOL AND STUDENT POPULATION Periodic assessments DOE and vendor provided assessments Default system based on citywide ELA and Math curricula Math #1 Math #2 Math #7 ELA #1 ELA #5 School customized assessments School created assessments Customized version provided to align with different curriculum, sequence or desired outcome (more predictive) Standards-based alternative assessment system that.. Is developed by school, network or region in collaboration with a partner Ensures equal rigor, measures progress and allows for cross- group comparison At every grade level, a set of diagnostic assessment tools to help inform interventions and instruction to ensure an individualized instructional plan for every student

12. 11 ENABLERS / TOOLS: PERIODIC ASSESSMENT – A MENU OF OPTIONS THAT SUIT THE NEEDS OF SCHOOL AND STUDENT POPULATION DOE and vendor provided assessments Default system based on citywide ELA and Math curricula Math #1 Math #2 Math #7 ELA #1 ELA #5 School customized assessments School created assessments Customized version provided to align with different curriculum, sequence or desired outcome (more predictive) Standards-based alternative assessment system that.. Is developed by school, network or region in collaboration with a partner Ensures equal rigor, measures progress and allows for cross- group comparison ARIS

13. 12 Accountability Initiative –Vision and Overview –Achievement Reporting and Innovation System (ARIS) Accountability Initiative Implementation Timeline CONTENTS

14. 13 OBJECTIVES AND SCOPE OF ARIS Objectives Ensure access to the information and tools necessary to enable Longitudinal and detailed analysis and reporting of achievement and performance data Best-in-class assessment design and implementation processes to extract data from multiple sources, including designing own reports and the capability to slice and dice the data Tools to improve learning of students with a range of identified needs; Sharing best practices and collaborating across schools to encourage cultures of continuous school improvement and professional learning Scope of the ARIS Provide principals, teachers and parents with online information on student achievement, including periodic assessments Support development of longitudinal and detailed analysis Develop an integrated portal, including a dashboard to analyze key environment factors and achievement metrics Enable real-time prediction of school performance against year-end targets Generate standard reports for specific end- users Develop scorecards to enable drill-downs by student, assessment, strand and sub-strand analysis Provide knowledge management tools that capture teaching and assessment content generated at school level with an ability to approve, publish and share with the networks or the broader district. As well as capturing new quality review data

15. 14 WHAT ARIS IS AND WHAT IT IS NOT ARIS is…ARIS is not… A solution that… –Analyzes and reports –Assesses –Innovates (assessments, responsive interventions) –Collaborates and personalizes –Shares knowledge (sharing) and manages content A solution that may not be available off- the-shelf A solution that may require a partnership or consortium of Bidders to offer, not by a single vendor or a service provider A collaborative effort between the Vendor and NYCDOE to innovate, develop and execute A system that will replace all of the current applications now in operation at the DOE (e.g. ATS, HSST). Users will still need to update and print reports very much in the same way they do today but the access to all of the key systems will now all be well integrated through a single interface An Enterprise Data Warehouse (EDW) that will have all the current available data in DOE (assessment, Financial, HR, etc.). ARIS may be the foundation for an EDW in the long run, but initially the data warehouse will only contain student achievement data and the necessary information to analyze by several key criteria (e.g., age, school, grade, background, etc.) An opportunity to replace desktops and/or network infrastructure

16. 15 OVERVIEW OF REQUIRED FUNCTIONALITY Generate reports and analyze achievement/performance data Use best-in class assessments to improve student achievement Innovate the professional learning / share best practices across all schools Analysis + Reporting Canned and customized reports Drill-down, roll-up, filtering Ad-hoc queries User-generated databases Adding new data to central repository Data viewing in 3 rd party tools (e.g., Excel) Advanced analysis (for selected users) Advanced statistics Pattern detection for target setting Knowledge and plan management Library of templates for different document types (e.g., lesson plans, curriculums) Document creation and upload to repository Document tagging Collaborative document creation Publishing approval Document linking to reporting (surveys) Knowledge and plan management (continued) Searching documents in the repositories Document viewing and download Pattern analysis across text documents Collaboration Posting announcements; setting alerts Online discussions; online messaging Custom workspaces, integration w/email Periodic assessments For school- or center-created –Library of templates/idea bank for school-created tests (e.g., rubric-based, standardized multiple choice) –Test creation and upload For 3 rd party-created –Assessment content upload into online testing, reporting and analysis For all –Online testing and scoring –Score data upload into reporting and analysis Personalization Pre-defined home page based on user role Personalization of home page (HP) by user Adding new tools to HP from a tool catalog Auditing (ie.,usage instances, frequency, modification history tracking) 123

17. 16 KEY TECHNICAL AND SYSTEM REQUIREMENTS Implications for ARIS RFP / requirements Context and challenges Data issues –Significant archive size –Uncertain data quality –Critical data element gaps Legacy applications –Significant number, fragmented functionality –Diverse but emerging preferences for platforms –Goal of having a single Enterprise Data Warehouse Infrastructure –Uneven / unclear quality of... Desktop standards / availability Network access –Stringent performance requirements –Funding constraints Data issues Vendors, based on their experience, to work with DOE on developing a processes and tools for… –Data cleaning and migration –Filling critical data element gaps –Improving data entry quality Legacy applications Bidders to.. –Develop new data sources for new accountability functions (e.g., QR) –Offer a scalable and evolvable data model –Comply with strongly preferred, but not mandatory standards Infrastructure –Bidders have the option to host externally, or suggest hardware for DOE-hosted solutions –Optimized for mobile/ existing infrastructure –Funding for infrastructure outside of ARIS contract scope

18. 17 DOE ARIS PREPAREDNESS TASK FORCE FOR KEY TECHNICAL AND SYSTEM REQUIREMENTS Context Objectives Data Integrity: Create solutions and define processes to ensure and maintain confidence in the quality of data and validity of ARIS outputs Infrastructure: Define the infrastructural improvements for maximizing usage and effectiveness of ARIS Data Source Inventory: Identify alternate types of approaches/ applications schools use to host data Communications: Craft messaging tailored to user experience NYCDOE has set up a Task Force to set ARIS for success by preparing internal DOE systems and processes to complement vendor solution The task force will… Identify level of cleanliness of existing data 1 Identify key causes for data inaccuracy or incompleteness Design and pilot solutions to clean existing source systems feeding ARIS 2 Recommend processes for maintaining data quality going forward Concentrate on transactional systems (ATS, HSST, etc.)

19. 18 OVERVIEW OF TRAINING APPROACH 1Includes select admin staff, select teachers 2Includes 25 Central Staff/Accountability Office; 10 power users drawn from schools 3Includes 150 network/region; 4200 principals and other school achievement leaders 4Primarily teachers Central/DAA (50) Network / Regions (280) Principals (1,400) Other school achievement leaders 1 (5,600) Teachers (87,000) Support staff / DIIT (?) Parents (~100s K) Initial training (role-based, project- and site-based) –One-on-one (for power and advanced) –Train-the-trainer (for standard) - school achievement leaders train other teachers Ongoing need-based training –Phone support service –Online simulation exercises Initial training –Train-the-trainer – select volunteers to train others Ongoing need-based training –Phone support service –Online simulation exercises Phase 1 (Sep. 2007)Phase 2 (Sep. 2008) User segments Training approach Role-based view Skill-based view ( _ ) Segment size Power (35 2 ) Advanced (~4,350 3 ) Standard (~90,000 4 ) Standard (~ 100s K) RFP requires the vendors to submit plans for measuring training effectiveness

20. 19 Accountability Initiative –Vision and Overview –Achievement Reporting and Innovation System (ARIS) Accountability Initiative Implementation Timeline CONTENTS

21. 20 ACCOUNTABILITY INITIATIVE IMPLEMENTATION TIMELINE 2006-072007-08 Empowerment Schools Other SchoolsAll Schools Progress Report (Previous Year)YesNoYes Progress Report (Current Year)Yes School TargetsYesNoYes School GradeYes Quality ReviewYes Rewards/ ConsequencesYesNoYes Data Management SystemSpring 2007NoYes Periodic AssessmentsYesNoYes Support for Continuous LearningYesNoYes