2. A Brief History of the NRS American Institutes for Research February 2005

3. A Brief History of the NRS 2 Wave 1: Development of the NRS Pre-NRS: Emergence of accountability Development process Wave 2: Implementation Refinement of measures and procedures (1999–2001) State implementation(2000–2003) Training and T/A on requirements, data quality, data use Wave 3: Focus on Improvement Overview

4. A Brief History of the NRS 3 General trend toward accountability for education and employment programs (late 1980s—early 1990s) Quality Indicators (1991–92) Required by National Literacy Act Federal (model) indicators State indicators Emergence of Accountability

5. A Brief History of the NRS 4 Assessed local program performance in: program planning recruitment retention student learning professional development curriculum and instruction support services Quality Indicators

6. A Brief History of the NRS 5 Primarily qualitative measures, focused on program process—what a program does Less emphasis on student outcomes Lacked uniform, quantitative measures No standards No rewards/incentives Not fully implemented in states—voluntary Quality Indicator Approach

7. A Brief History of the NRS 6 1993: Government Performance and Results Act (GPRA)—required all Federal agencies to develop indicators of program performance to demonstrate they are meeting their goals. 1995: Adult education program threatened with elimination into workforce development block grant system. Lack of data on program effectiveness cited. 1996: State directors and U.S. Department of Education develop a framework for a system of program accountability. 1997: National stakeholder meeting to validate framework and identify outcome measures. Emergence of NRS

8. A Brief History of the NRS 7 Included all state directors Formal process to identify goals of adult education Identified measures for each goal Endorsed development of formal accountability system, leading to first NRS development project 1997 Development Meeting

9. A Brief History of the NRS 8 Educational improvement and lifelong learning Enhancement of employability Improve civic participation Improve literacy and education within the family Goals and Measures Identified

10. A Brief History of the NRS 9 Collaborative process of DAEL, state directors, other stakeholders Broad group identified basic program goals and measures Advisory board refined measures and procedures Key decisions brought to all state directors before final resolution NRS Development Project: 1997–98

11. A Brief History of the NRS 10 Goals—reflect program purposes Measures—tied to goals Performance Standards—establish level of performance on measures Rewards and/or Sanctions—for exceeding or falling below standards Key Components of Accountability Systems

12. A Brief History of the NRS 11 Board included state directors, literacy volunteer agencies, experts in accountability Concluded there were too many measures, some difficult to collect Applied criteria for inclusion Refinement by Advisory Board (1997–98)

13. A Brief History of the NRS 12 Meet WIA requirements Maintain state and local control, flexibility to meet learner needs Minimize burden Accommodate methodology to produce valid and reliable national data Definitions and procedures compatible with other agencies Criteria for Including Measures

14. A Brief History of the NRS 13 Required four types of measures—from NRS development work Requires measures tied to state performance standards. Rewards for states exceeding standards Allows states sanctions—states must “consider” local performance. WIA Passage

15. A Brief History of the NRS 14 Include other measures that state directors had identified and considered Could not be required due to WIA Also concern on increased burden, difficulty in collecting and defining Optional tables created, with possible formal inclusion if widespread state use Optional Measures and Tables

16. A Brief History of the NRS 15 Transitional year 1999 Guidelines published, national training, state technical assistance Performance standard setting process Refinements to measures and procedures Definitions, procedures, tables refined Data quality checklist Data quality guide and training Wave 2: Implementation

17. A Brief History of the NRS 16 Implement new polices and procedures: WIA reauthorization Continue to refine NRS procedures and measures Stronger focus on state monitoring of local programs Use data for program administration and improvement Wave 3: Improvement