Table 5 and Follow-up Collecting and Reporting Follow-up Data Or What’s Behind Table 5? American Institutes for Research February 2005

Table 5 and Follow-up 2 Table 5: Outcomes Earned a High School Diploma or GED Entered post-secondary School Entered Employment Retained Employment

Table 5 and Follow-up 3 Table 5: Outcomes States submit six (6) data points for each of the four outcomes: Core Follow- up Outcome Measures # Participants With Main or Secondary Goal # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (A)(B)(C)(D)(E)(F)(G)

Table 5 and Follow-up 4 Table 5: Column (B) All participants who had the outcome as a goal and who exited during the program year. Core Follow- up Outcome Measures # Participants With Main or Secondary Goal # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (A)(B)(C)(D)(E)(F)(G)

Table 5 and Follow-up 5 Table 5: Column (C) The number of participants who were surveyed to assess whether they had achieved the outcome. Note: C should be equal to B unless sampling was used, in which case C would be less than B. For data matching C = [BLANK] Core Follow- up Outcome Measures # Participants With Main or Secondary Goal # Participant s Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (A)(B)(C)(D)(E)(F)(G)

Table 5 and Follow-up 6 Table 5: Column (D) Of those surveyed or data matched, the number who responded or were used for data match. Note: D should be equal to or less than B and C each. Core Follow- up Outcome Measures # Participants With Main or Secondary Goal # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (A)(B)(C)(D)(E)(F)(G)

Table 5 and Follow-up 7 Table 5: Column (E) Percent of respondents or participants successfully matched; column D divided by column B. E = D/B * 100 Should be > 50%, but likely not 100% Core Follow- up Outcome Measures # Participants With Main or Secondary Goal # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (A)(B)(C)(D)(E)(F)(G)

Table 5 and Follow-up 8 Table 5: Column (F) Number of participants achieving outcome. Note: F must be equal to or less than D. Core Follow- up Outcome Measures # Participants With Main or Secondary Goal # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (A)(B)(C)(D)(E)(F)(G)

Table 5 and Follow-up 9 Table 5: Column (G) Percent achieving outcome. G= F/D * 100 Core Follow- up Outcome Measures # Participants With Main or Secondary Goal # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (A)(B)(C)(D)(E)(F)(G)

Table 5 and Follow-up 10 Table 5: Example Core Follow- up Outcome Measures # Participants With Main or Secondary Goal # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (A)(B)(C)(D)(E)(F)(G) Entered Employment 1, %20025%

Table 5 and Follow-up 11 Real Data – PY 2002–03 Examples State # Participants With Main or Secondary Goal # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (A)(B)(C)(D)(E)(F)(G) State 1 1, %35637% What type of follow-up was used?

Table 5 and Follow-up 12 Real Data – PY 2002–03 Examples State # Participants With Main or Secondary Goal # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (A)(B)(C)(D)(E)(F)(G) State 1 1, %35637% Why is this number less than column B?

Table 5 and Follow-up 13 Real Data – PY 2002–03 Examples State # Participants With Main or Secondary Goal # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (A)(B)(C)(D)(E)(F)(G) State 1 1, %35637% What data should be reported here?

Table 5 and Follow-up 14 Real Data – PY 2002–03 Examples State # Participants With Main or Secondary Goal # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (A)(B)(C)(D)(E)(F)(G) State 1 1, %35637% Is this percent computed correctly?

Table 5 and Follow-up 15 Exercise Questions For states 2–6: method of follow-up What method of follow-up did each state use? correctly Which states, if any, filled in the table correctly? Which did not? errors Do you see any errors in the table? confidence Do you have confidence in these data? Why or why not?

Table 5 and Follow-up 16 Exercise Example State # Participants With Main or Secondary Goal # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (A)(B)(C)(D)(E)(F)(G) State 1 1, %35637% State 2 3,325NOTAVAIL.0%1,06232% State 3 8,644000%4,06747% State 4 31, %21,77969% State 5 9,6333,8531,15630%54313% State %9659%

Table 5 and Follow-up 17 Top Errors for Table 5 Missing Data Put “0” for # surveyed in Column (C) Put “0” for # responded or matched in Column (D) Unreasonably high response rates Many 100% Unreasonably high data matching Many 100% (DOL goal for matching = 90%) Unreasonable matching of # who had goal to # who responded Column (D) = Column (B)

Table 5 and Follow-up 18 Top Errors for Table 5 (cont’d.) Incorrectly computed percentages: Response rates calculated as D/C instead of D/B (B and C should be the same unless using random sampling, however) Number achieving outcome computed as F/B instead of F/D

Table 5 and Follow-up –03 Data Collection Methods

Table 5 and Follow-up 20 Data Matching Methodology Produces better quality data: More student coverage Better data validity Better response rate Matches Title I—better for OMB common measures 28 states use it for employment measures—we need more states! What are the reasons for not data matching? How can we promote use of this methodology?

Table 5 and Follow-up 21 What’s Behind the Data? State responsibility to ensure accuracy and quality of follow-up measures Requires monitoring of local program procedures Key steps: Identify students—goal setting and tracking Uniform data collection Training Reporting

Table 5 and Follow-up 22 Percent of Students with Goals Measure # with Goal Total # Students % with Goal Entered Employment —Average ALL states 6,90452,54810%* State ,2841% State 22,624138,1842% State 31,609 70,8932% Entered post-secondary— Average ALL states 2,91952,5486% State ,0620% State ,5741% State 614,523565,3113% High School Completion— Average ALL states 7,20752,54820%** State 77,192114,0086% State 82,76832,4929% State 933,118387,7109% * 38% Unemployed (Table 6) ** 18% Enrolled in ASE

Table 5 and Follow-up 23 Quality Control: Surveys Identify students Set student goals appropriately By exit quarter for employment measures Data Collection Quarterly collection for employment State survey instrument Sufficient resources (staff, time, funds) Sampling method (if applicable)

Table 5 and Follow-up 24 Quality Control: Surveys (cont’d.) Training Trained staff on survey procedures Improving response rate Reporting Quarterly reporting Database to link outcomes to programs and students

Table 5 and Follow-up 25 Quality Control: Data Matching Procedure for collecting and validating Social Security numbers Report all with goal, prior to eliminating invalid or missing numbers (Table 5, Col. D) Data have exit quarter and employment records matched after correct exit quarter Only students with goal matched and/or reported—should not have 100% with outcome

Table 5 and Follow-up 26 Discussion: Issues and Strategies Improving reporting and collecting Resources Training on Monitoring