1. 1 The Evolution of Accountability Reporting: Lessons Learned Over Nine Years By a State Education Agency Mindy Crain-Dorough, PhD Southeastern Louisiana University mindy.dorough@selu.edu Robyn Madison-Harris, EdD Louisiana Department of Education robyn.madison@la.gov

2. 2 Overview of presentation Background on report development and what Louisiana does History of Accountability Changes Problems/Solutions Recommendations for future directions

3. 3 Purpose of this study To compile a list of lessons learned as a reference for other states To document the history of accountability reporting in Louisiana To explain the complexities of the report development process To provide recommendations for improvements accountability reporting in Louisiana and other states

4. 4 Report development – Not an easy thing to do Present complicated information and data in a format that is aesthetically pleasing and understandable Negotiate many conflicting opinions about content Turn complicated and detailed policy into computer programs that calculate scores Coordinating many large data files Extensive checking of all aspects of calculations and report generation has to be conducted. All of these must be done in a very rigid timeframe.

5. 5 Why do we go to all this trouble? Not just to meet federal and state mandates Parents make decisions about which schools to which they will send their children and even where to live. Parents are also driven to take action at their children’s schools. Teachers, principals, and district staff use the results to make decisions about school improvement efforts and funding priorities

6. 6 Louisiana’s choices Reports produced at the state level for districts and schools School Report Card for Parents in paper format and sent home with every child Electronic PDFs of reports are available on LDE website Assessment and accountability reporting are separate LDE works with a contractor to produce reports

7. 7 Back to the beginning Louisiana had report cards as far back as 1996-1997 as part of a legislative mandate Transformed report cards into accountability reports in 1998-1999

8. 8 School Report Card for Parents

9. 9 Inside

10. 10 Back

11. 11 Changes to reports over the years YearChange IncurredImpact on Reports SY 1998-1999First year of school accountabilityComplete overhaul 2 new reports SY 1999-2000Interim year – no new labels reported Science and Social Studies data reported Addition of new tables and text SY 2000-2001Grades 9-12 entered accountability First year growth was evaluated Complete overhaul SY 2001-2002District Accountability data first reported (1 year lag) New district reports SY 2002-2003Incorporated NCLB mandatesComplete overhaul 1 new report SY 2003-2004Graphic design overhaul of all reportsComplete overhaul SY 2004-2005No major changes SY 2005-2006Adjustments for hurricanes Incorporation of iLEAP Complete overhaul SY 2006-2007Incorporation of graduation rate and index Revised school sanction classifications Addition of new tables and text

12. 12 Along comes NCLB First reports containing NCLB mandates in 2002- 2003 Louisiana had to incorporate NCLB into policy and thus the existing reports A dual system created – SPS Component and Subgroup Component Conducted an evaluation of existing reports for what was and was not working Prioritized contents Put highlights in report cards and details in a new report

13. 13 Incorporation of NCLB 2001-2002 Reports Added in 2002-2003 Intended Audience NCLB Requirements Included School Report Card for Parents ParentsGeneral information on dual system, table with Subgroup AYP information School Report Card for Principals Principals, faculty, and district staff More details than Parent report card on dual system and Subgroup AYP School Accountability Report (“Press Report”) Media, researchers, and the general public Major changes to format Subgroup Component Report Principals, faculty, and district staff All other NCLB requirements

14. 14 School Report Card for Parents

15. 15 Inside

16. 16 Back

17. 17 Methodology Qualitative – authoethnographic, case study, and phenomenological approaches Data Collection Authoethnographic recount by authors Interviews of current and former accountability staff Document analysis Data Analysis – cross-case examination and content analysis strategies

18. 18 Problems/Solutions Accountability system changing every year Be organized Planning meetings with key parties Preparation for planning meetings “Regrouping” at the end of each year

19. 19 Problems/Solutions Errors in the reports Improve the checking process every year using a log of problems found in previous year(s) Include cross-report consistency checking in the process Create thorough processing rules, “data maps,” and record layouts that work together Communication between report developers, data calculation staff, and the contractor on these issues is key Allow districts to review data in aggregated form before inclusion in reports, especially for new indicators Have emergency plans in place should errors occur Use everyone’s time effectively – hire temps if necessary to do more tedious checking

20. 20 Problems/Solutions Too many conflicting opinions about content Use focus groups to obtain evidence of what works best for consumers Develop a protocol for making decisions regarding conflicting viewpoints

21. 21 Problems/Solutions Miscommunication within a state agency Is the agency structure conducive to functional report development? Develop an issues database to track issues and their resolutions as well as a process for maintaining the database Develop a thorough timeline with a process for ensuring everyone is on task and a protocol for revising the timeline should unexpected events arise Find a method of meeting regularly to discuss issues and timelines that is most effective for all involved – we are currently having weekly conference calls with the contractor and impromptu meetings in-between

22. 22 Problems/Solutions Vendor relations Good vendor qualities include flexibility, availability to devote extensive time to the project, advanced technical skills, and good quality assurance practices Balance of working for us and working with us Establish protocol for staff turnovers and management changes Set standards and roles for quality assurance procedures and processes Regular communication and established communication procedures

23. 23 Problems/Solutions New mandates (internal and external to state) Evaluate the current reports for what works and what doesn’t Prioritize contents of the report (current and anticipated) Take advantage of the opportunity to make enhancements to the reports that are undergoing major revision anyway Look at what other states are doing to get ideas

24. 24 Problems/Solutions Disasters occur Meet with those most impacted by the events to obtain all information before making decisions Consider all aspects of the accountability system and brainstorm possible impacts

25. 25 Problems/Solutions Giving one individual too much responsibility Have a team of report developers Rotate lead responsibility for individual reports every two or three years

26. 26 Problems/Solutions Promoting and supporting report release Use Press Conferences, written press releases, and radio announcements Create a “Help Desk” to answer questions about accountability and other agency programs Provide accountability trainings on policy changes, the importance of data quality, and description of report contents to school and district staff

27. 27 Problems/Solutions Making reports useful Brainstorm areas in the agency that can be highlighted in the report cards Provide a message to parents in the report card from the superintendent to inspire parents to become more involved in their child’s education

28. 28 Validity and reliability Validity Confirm valid use of reports through focus groups and training Reliability Thorough and detailed checking process

29. 29 The current “Parent Report Card”

30. 30 Recommendations Transition to interactive accountability reporting and tie to school improvement efforts Combine assessment and accountability data Reduce the number of releases and release the Parent Report Cards earlier Re-evaluate the content and purposes of the current reports Have an “Accountability Czar”