1. Overview and Discussion of NRS Changes

2. Massachusetts Department of Elementary and Secondary Education 2 NRS Changes for FY13 AGENDA  Review changes on follow-up outcomes and new data requirements  Review criteria for cohorts  Review changes to reporting timeline and reporting tables  Review questions and answers posted on the ACLS website  Respond to participant questions

3. Massachusetts Department of Elementary and Secondary Education 3 Time Period Definitions  Program Year (PY)– Twelve month period from July 1 through June 30 defined by the calendar year at start of period. ( federal terminology )  For example, July 1, 2012 through June 30, 2013 is the time period corresponding to PY12.  Fiscal Year (FY)– Twelve month period from July 1 through June 30 defined by the calendar year at end of period. ( state terminology)  For example, July 1, 2012 through June 30, 2013 is the time period corresponding to FY13.

4. Massachusetts Department of Elementary and Secondary Education 4 Changes to Follow-up Outcomes: Automatic Cohort Designation Program Year (FY2012): Participants identified based on self-reported goals (program – level) Program Year (FY2013): Participants automatically identified based on “cohort” which are based on demographic variables in the MIS system (state – level) The definition for “cohort” varies by outcome.

5. Massachusetts Department of Elementary and Secondary Education 5 Implications of “Automatic Cohort Designation”  Students belong to a group to be followed for NRS post-program outcomes based on pre-defined characteristics.  Following students for post program outcomes will no longer be goal dependent.  As has always been the case, we are only concerned with students having at least 12 hours of attendance.

6. Massachusetts Department of Elementary and Secondary Education 6  YES!!  Local staff should continue to set goals with students  Goal setting is an essential component of a good program and good instruction but….  Students goals will no longer be used as a way to identify students for the NRS follow-up measures Do Student Goals Still Matter?

7. Massachusetts Department of Elementary and Secondary Education 7

8. 8 Secondary Credential Cohort Obtain Secondary Credential Cohort  Learners who take all GED tests, or  Are enrolled in adult high school at the high ASE level, or  In the assessment phase of the EDP, and  Who exit during the program year Entry into cohort occurs only for those who take all 5 GED tests by the end of reporting period (December 31).

9. Massachusetts Department of Elementary and Secondary Education 9 Q: Do students have to still be enrolled when they take the tests?  No, students are included in the cohort if they take all five test before the end of the reporting period for the program year in which they exit (December 31) Q: Must a student pass all GED tests to be included as achieving the outcome?  Yes, students must pass all GED tests to be included as achieving the outcome Question: Cohort for Exited Students and GED Tests

10. Massachusetts Department of Elementary and Secondary Education 10  Example 1: A student who enrolls in fall of 2012, exits in March 2013, takes 3 tests then, takes 2 more in September 2013, would be included in the cohort for 2012-13  Example 2: A student who enrolls in October 2012, takes 2 GED tests in May 2013, exits in November 2013 and takes 3 GED tests would be included in the cohort for 2013-14  Example 3: A student who enrolls in December 2012, exits in March 2013, takes 3 tests then, takes 2 more in January 2014, would not be included in any cohort Passing GED Tests: Examples

11. Massachusetts Department of Elementary and Secondary Education 11 Postsecondary Cohort All learners who  have earned a secondary credential while enrolled  hold a secondary credential  or are enrolled in a class specifically designed for transitioning to community college and who exit during the program year

12. Massachusetts Department of Elementary and Secondary Education 12 Questions: Foreign Credential and Transition class Q: Are students with a secondary credential, earned in another country, in this cohort? Are ESOL students to be included?  Yes, these students are included in the cohort Q: How is a transition class defined ?  A class that has a specific purpose to prepare students for entry into postsecondary education, training or an apprenticeship program.  T he rate based class focus would be Bridge to College/Math or Bridge to College/Writing. The non rate based class focus would be Bridge to College/Math, Bridge to College/Writing, or College and Career Readiness.

13. Massachusetts Department of Elementary and Secondary Education 13 Q: Please clarify multi-year reporting of entry into postsecondary education  Students who exit during program year achieve this outcome if they enter postsecondary any time from exit through the end of the following program year (June 30)  Follow students and report for two years, in two rows on Table 5 (unduplicated)  First report (2012-13): report same as of now  Future years: Update prior year—keep cohorts separate  First two year reporting on 2013 report (due December 31, 2014) Reporting of Entry into Postsecondary--Timing

14. Massachusetts Department of Elementary and Secondary Education 14  Entered Employment Cohort  Learners who are not employed at time of entry and in the labor force who exit during the program year Employment Status in SMARTT:  Employed  Unemployed and Looking for Work  Unemployed and Not Looking for Work  Retired or Otherwise Not Looking for Work  Homemaker Programs should exit students who are no longer in attendance by filling out the “Exit Student” screen in SMARTT. Employment Cohorts

15. Massachusetts Department of Elementary and Secondary Education 15  Retained Employment Cohort  Learners who were not employed at time of entry and in the labor force, who are employed by the first quarter after exit quarter  Learners employed at entry who exit during the program year Employment Status in SMARTT:  Employed  Unemployed and Looking for Work  Unemployed and Not Looking for Work  Retired or Otherwise Not Looking for Work  Homemaker Programs should exit students who are no longer in attendance by filling out the “Exit Student” screen in SMARTT. Employment Cohorts

16. Massachusetts Department of Elementary and Secondary Education 16  Students, in entered employment cohort, that are employed the first quarter after exit are also in the retained employment cohort  The timeframe is the same as always–it has not been changed  States must use students’ employment status at entry to determine cohort status for employment outcomes. Reminder on Retained Employment

17. Massachusetts Department of Elementary and Secondary Education 17 Q: How is “unemployed, in the labor force defined?”  NRS definition (Unchanged): Learners who are not employed but are seeking employment, make specific efforts to find a job and are available for work Employment Status in SMARTT:  Unemployed and Looking for Work Question: Definition of Unemployed in Labor Force

18. Massachusetts Department of Elementary and Secondary Education 18  Entered Employment  Employed in the first quarter after exit.  Retained Employment  Employed in the third quarter after exit. Timing of Employment Measure

19. Massachusetts Department of Elementary and Secondary Education 19  Collection: One quarter after exit for entered employment, three quarters after exit for retained employment  Reporting lag (previous year) to accommodate lag in Unemployment Insurance (UI) wage records Review: Employment Data Collection and Reporting

20. Massachusetts Department of Elementary and Secondary Education 20 Exit QuarterMeasure Collect Report July 1–September 30, 2012 (First Quarter) Entered Employment October 1–December 31, 2012 (Second Quarter FY 2013) FY 2013 Retained Employment April 1–June 30, 2013 (Fourth Quarter FY 2013) FY 2013 October 1–December 31, 2012 (Second Quarter) Entered Employment January 1–March 31, 2013 (Third Quarter FY 2013) FY 2013 Retained Employment July 1–September 30, 2013 (First Quarter FY 2014) FY 2014 January 1–March 31, 2013 (Third Quarter) Entered Employment April 1–June 30, 2013 (Fourth Quarter FY 2013) FY 2013 Retained Employment October 1–December 31, 2013 (Second Quarter FY 2014) FY 2013 April 1–June 30, 2013 (Fourth Quarter) Entered Employment July 1–September 30, 2013 (First Quarter FY 2014) FY 2014 Retained Employment January 1–March 31, 2014 (Third Quarter FY 2014) FY 2014

21. Massachusetts Department of Elementary and Secondary Education 21  Enter Employment/Get a Job -- Those with SSN and signed release form. For all others, survey method must be used.  Retain Employment -- Those with SSN and signed release form. For all others, survey method must be used. Survey form may be accessed at: http://www.doe.mass.edu/acls/smartt/nrssurvey _followup.pdf Data Matching Department of Revenue

22. Massachusetts Department of Elementary and Secondary Education 22  Obtain GED -- All students (regardless if SSN provided and release form signed since match will be with the Oklahoma Scoring Service)  Enter Postsecondary Education -- -- All students (regardless if SSN provided and release form signed since match will be with the National Student Clearinghouse) Data Matching Oklahoma Scoring Service, and National Student Clearinghouse

23. Massachusetts Department of Elementary and Secondary Education 23 Follow-up Outcome Timing Secondary Credential  Obtained credential before the end of the reporting period (December 31) Postsecondary Education  Entered any time from exit until the end of the following program year (June 30)  Report for two years, in two rows on Table 5 (unduplicated) Programs should exit students who are no longer in attendance by filling out the screen found by clicking on the “Exit Student” menu option within SMARTT.

24. Massachusetts Department of Elementary and Secondary Education 24 Reporting Changes: Table 5  Table 5 revised to include instructions for automatic cohort designation  New row added for postsecondary transition: Report two years (unduplicated)  Update prior year  First two year reporting on 2013 report (Due December 31, 2014)

25. Massachusetts Department of Elementary and Secondary Education 25 Reporting Postsecondary Example in 2013-14  In 2013-14, 125 students in cohort exit, state matches 100 and finds 50 have entered postsecondary  State runs match on 2012-13 cohort again by June 30, 2014 and finds another 20 students have entered postsecondary in addition to the 40 from previous year

26. Massachusetts Department of Elementary and Secondary Education 26 Example of Table 5: 2013-14 Core Follow-up Outcome Measures Number of Participants in Cohort Number of Participants Included in Survey Sample Number of Participants Responding to Survey or Used for Data Matching Response Rate or Percent Available for Match Number of Participants Achieving Outcome Percent Achieving Outcome (A)(B)(C)(D)(E)(F)(G) Entered Postsecondary Education or Training -- current program year 120 10083% (100/120) 4040% (40/100) Entered Postsecondary Education or Training – prior program year Leave entire row blank

27. Massachusetts Department of Elementary and Secondary Education 27 Example of Table 5: 2014-15 Core Follow-up Outcome Measures Number of Participants in Cohort Number of Participants Included in Survey Sample Number of Participants Responding to Survey or Used for Data Matching Response Rate or Percent Available for Match Number of Participants Achieving Outcome Percent Achieving Outcome (A)(B)(C)(D)(E)(F)(G) Entered Postsecondary Education or Training -- current program year 125 10080%5050% Entered Postsecondary Education or Training – prior program year 120 10083%60 (prior year was 40, now 40+20) 60% (60/100)

28. Massachusetts Department of Elementary and Secondary Education 28 Implications for Implementation  The number of learners to follow will increase.  Obtaining SSN’s and signed release forms will lessen the follow-up burden.  Local staff will need to follow-up by survey method for cases where SSN’s and signed release forms were not obtained for those within any of the employment cohorts.

29. Massachusetts Department of Elementary and Secondary Education 29 Q: How should we determine years of student education? Do we need to check?  Self-reported by student, NRS does not require verification but the state may choose to do so Q: How do we count education in other countries that is different than our system?  Use years of schooling  Use in-country definition for secondary credential Questions: Reporting Student Education

30. Massachusetts Department of Elementary and Secondary Education 30 Q: Can students belong to multiple cohorts?  Yes, for example, an unemployed student in the labor force, with secondary credential who exits, would be in two cohorts Question: Multiple Cohorts

31. Massachusetts Department of Elementary and Secondary Education 31 Q: Do we have to re-identify cohort membership for students continuing the next year?  Yes, for example, employment status or secondary credential status may change  But only collect measure at exit Question: Continuing Students

32. Massachusetts Department of Elementary and Secondary Education 32 ADDITIONAL QUESTIONS??