1. An Administrators Guide to Student Follow-Up and Data Quality Presented by: Dean Smith Lennox McLendon Kathi Polis

2. McLendon and Polis2 Training Objectives To review NRS guidelines for determining what participant populations need to participate in follow-up surveys To outline the seven-step process for conducting NRS-approved participant surveys To examine strategies for training telephone interviewers

3. McLendon and Polis3 Training Objectives To develop a local plan for conducting NRS-compliant follow up To review and provide input into the development of data quality standards for local programs To discuss and clarify MAERS-related data collection and reporting questions and issues

4. McLendon and Polis4 Introduction to Student Follow-Up Requirements established by National Reporting System Defines: Outcomes you must collect Methodologies for collecting them State reporting requirements

5. McLendon and Polis5 Core Measures Educational Gains – completion of EFLs Follow-Up Measures Entered employment Retained/improved employment Earned a secondary school diploma or GED Entered postsecondary education or job training

6. McLendon and Polis6 Two Follow-Up Methodologies Data Match (recommended) Most cost and labor effective method Student Survey Michigan 2005 - 2006 = student survey 2006 – 2007 = exploring data match for employment and GED measures

7. McLendon and Polis7 What Participants to Survey Only participants with follow-up goals Enter, retain, improve employment Enroll in postsecondary Earn a GED/diploma High school diploma completion can be verified through program records. GED can be verified through local GED Examiner (with participants permission granted on Follow-Up Notice)

8. McLendon and Polis8 When to Survey for Program Year 2005 – 2006 Postsecondary, HS Diploma, GED Anytime prior to October 25, 2006 (for 2005- 2006 year) Tips Postsecondary/job training – know the colleges enrollment schedule to determine most appropriate time (e.g., mid-September for fall semester college enrollments) GED – allow as much time as possible for participants to complete test and receive their test scores (e.g., mid- September for previous program year)

9. McLendon and Polis9 When to Survey – The Plot Thickens! Participants with entered employment follow- up goals Timing is the challenge! Must determine TWO outcomes 1. If they got a job by the 1 st quarter after exit quarter AND 2. If they were still employed by the 3 rd quarter after exit

10. McLendon and Polis10 When to Survey for 2005 - 2006 Exit QuarterConduct Entered Employment Follow-Up by end of: Collect Retained/Improved Employment by end of: First Quarter July 1 – Sept. 30, 2005 Second QuarterFourth Quarter Second Quarter Oct.1 – Dec. 31, 2005 Third QuarterFirst Quarter, Next Program Year (July 1 – Sept. 2006) Third Quarter Jan. 1 – March 31, 2006 Fourth QuarterNot reported for 2005 – 2006 Fourth Quarter April 1,2006 – June 30, 2006 First Quarter, Next Program Year (July 1 – Sept. 2006) Not reported for 2005 - 2006

11. McLendon and Polis11 Activity 1 – Match Game Lets see how well you do at matching the right information together. Refer to Activity 1 Handout.

12. McLendon and Polis12 The Survey Process Getting Ready: 1. Select and train telephone interviewers. 2. Determine your telephone survey schedule. Participant Intake: 3. Inform participants of the survey. 4. Ensure accurate contact information. Data Collection: 5. Identify survey respondents. 6. Conduct the telephone survey. 7. Record the results.

13. McLendon and Polis13 Getting Ready: Step 1: Select and train telephone interviewers. Selecting telephone interviewers May be you, your teachers, volunteers, other staff You need folks who: Speak clearly Understand the program and reasons for follow- up Understand the adult learner population Are polite and courteous Are patient yet persistent AND Who WANT to be telephone interviewers!

14. McLendon and Polis14 Getting Ready Training telephone interviewers Training should include How to use the script, including what to say to participants to introduce the survey and get their cooperation Ways to avoid refusals How to ask the survey questions How to record responses How to answer participant questions about the survey.

15. McLendon and Polis15 Getting Ready Training telephone interviewers Go over every question to ensure interviewers understand: the purpose, what is being asked, and what responses are desired. Include mock interviews and other practice. Manual – Appendix C: Training Strategies for Telephone Interviewers

16. McLendon and Polis16 Program Survey Appendices D1 – D6 Standardized script for each follow-up measure/goal Must follow script precisely Three sections 1 – includes general participant information 2 – relates to follow-up goal 3 – requests participant feedback and recommendations

17. McLendon and Polis17 Activity 2: Can We Talk? Divide into pairs. One person will be the student; the other will play the role of interviewer. For your designated follow-up goal, conduct the telephone interview. What was the result? Did you see any problems you might encounter? If so, what were they and how might you handle them?

18. McLendon and Polis18 Step 2: Determine your telephone schedule. Employment goals – quarterly Dont forget Participants with enter employment goals will need to be contacted twice (first for employment, second for job retention) Postsecondary and GED goals Depends on college schedules, GED test dates, etc. – You be the judge. All follow-up must be entered into MAERS no later than October 25!

19. McLendon and Polis19 Step 3: Inform participants of the survey. At intake Follow-Up Notice – Appendix A Explains follow-up process Requests their participation Provides the participants permission to be contacted for follow-up Provides permission to verify GED completion with state or local GED testing examiner Given to all participants with follow-up goals (or potential to convert to follow-up goals)

20. McLendon and Polis20 Step 4: Ensure accurate contact information. Important to collect alternative contact information Addresses and phone numbers of friends, relatives, etc. Use the Follow-Up Contact Information Form – Appendix B Can input into MAERS Update Alternative Contact screen (Appendix E) Encourage participants to keep their contact information up-to-date

21. McLendon and Polis21 Step 5: Identify survey respondents. CRITICAL – Input participant data into MAERS at least quarterly! Enables MAERS to generate a Follow-Up Contact List Lists all participants that need to be contacted and when.

22. McLendon and Polis22 Activity 3: Beginning Your Follow- Up Plan Refer to the Student Follow Up Plan Review the questions for Steps 1 – 5. Complete the Response Plan column for any items that you can answer right now.

23. McLendon and Polis23 Step 6: Conduct the telephone survey. Ask for the individual by first name or nickname. If teacher is not making the call, have follow- up staff state they are calling on behalf of the participants teacher. Must use the Contact Log (Appendix F). Must attempt minimum of four contacts. No answer after four attempts – mark as a non-respondent

24. McLendon and Polis24 Step 7: Record the results. Record follow-up results in MAERs Follow-Up Information screen (at least quarterly). File program survey in participants paper file. Non-respondents – file blank survey Adapt the Participant Follow-Up Checklist (Appendix G) for your use.

25. McLendon and Polis25 Activity 4: Completing Your Follow Up Plan Refer to the Student Follow Up Plan. Complete the Response Plan column for Steps 6 and 7.

26. McLendon and Polis26 Data Quality Standards Improving the accuracy of our data U.S. Department of Education developed data quality standards that states must address annually A guide to the policies, processes, and materials that need to be in place for quality data Data foundation and structure Data collection and verification Data analysis and reporting Staff development Levels of performance range from acceptable to exemplary

27. McLendon and Polis27 Data Quality Standards Michigans Draft Data Quality Standards Adapted the national data quality standards to fit local programs Draft form only – want your input Recommendations will be forwarded to the taskforces for further review and submission to State Office

28. McLendon and Polis28 Data Quality Standards Michigans Draft Data Quality Standards Acceptable level standards will go into effect July 1, 2006. Superior level – July 1, 2007 Exemplary level – July 1, 2008 Training and technical assistance will be made available.

29. McLendon and Polis29 Data Quality Standards Format Three columns 1 st column – states standard or process 2 nd column – provides examples of sample verification 3 rd column – requires yes or no response A few questions require brief narrative response.

30. McLendon and Polis30 Data Quality Standards Process Program director submits completed checklist and signed certification page to State Office by September 30, 2007 (for previous program year). Written improvement plan for unmet standards Describe new policies or procedures you will put in place to meet the standards, Identify barriers to moving to a higher quality level, and Describe the technical assistance needed to implement the plan. DLEG will provide customized technical assistance to help you.

31. McLendon and Polis31 Activity 5: Getting Your Input Group 1 tables will review A and C. Group 2 tables will review B and D. Make a list of any concerns, questions, or recommendations for the standards or verification samples.

32. McLendon and Polis32 Deans Corner Let the games begin!!!!

33. McLendon and Polis33 Always willing to help… Dean Smith SmithD20@michigan.gov Lennox McLendon lmclendon@naepdc.org Kathi Polis polis123@adelphia.net

34. McLendon and Polis34 This project was developed by National Human Resources Development, Inc. (NHRD) and the National Adult Education Professional Development Consortium in cooperation with the Michigan Department of Labor and Economic Growth and funded through a grant under Section 222(a)(2) State Leadership Activities of the Adult Education and Family Literacy Act, Title II of the Workforce Investment Act of 1998, amended. For more information visit: http:www.maepd.org