1. Employment and Training Administration DEPARTMENT OF LABOR ETA Data Validation ARRA Performance Accountability Forum San Francisco, California December 9-10, 2009 Keith Rowe Karen Staha U.S. Department of Labor Employment and Training Administration

2. Employment and Training Administration DEPARTMENT OF LABOR ETA Session Outline 1.Getting to Square One 2.Data Validation Results 3.Source Documentation Requirements 4.Common DV Review Findings 5.Issues/Challenges/Considerations

3. Employment and Training Administration DEPARTMENT OF LABOR ETA Reports and Information Data Collection Data Processing Data validation affects every aspect of performance management. “Performance is your reality. Forget everything else.” - Harold Geneen Performance Management

4. Employment and Training Administration DEPARTMENT OF LABOR ETA The Data Validation Initiative Purpose of USDOL’s Initiative To support improved management of federal programs and respond to data quality issues cited by oversight agencies Data Validation (DV) Directives  Eleven guidance letters/notices to date  Initial Guidance—TEN 14-02 (5/28/03)  Latest Guidance—TEN 08-09 (08/25/09)  All ETA performance-related guidance is at: www.doleta.gov/performance

5. Employment and Training Administration DEPARTMENT OF LABOR ETA What Does Data Validation Mean? In the context of the USDOL DV Initiative for federal employment and training programs:  Report Validation (RV) Pertains to aggregate reports and ensures performance calculations are accurate (e.g., ETA 9091 or WIA Annual Report); and  Data Element Validation (DEV) Pertains to individual data elements within required reports and ensures the data used in the calculations are accurate (e.g., Date of Birth). Both processes are required to ensure performance data are reliable!

6. Employment and Training Administration DEPARTMENT OF LABOR ETA Report Validation (RV) Programs that submit year-end aggregate reports must validate their reports prior to submission.  WIA  Wagner-Peyser/Jobs for Veterans’ State Grants  [Not Trade] Report validation is largely a technical function, performed at state level.

7. Employment and Training Administration DEPARTMENT OF LABOR ETA How RV Works States import their data into specialized software to compare the software’s calculated performance outcomes to the State’s calculated outcomes.  Validation counts are the figures computed from the DRVS software; reported counts are what was reported by the State.  The software computes error rates.  ALL states must upload RV results to ETA’s online management information system (EBSS)  Note: States that use DART software to generate Labor Exchange (LX or Wagner-Peyser/VETS) program reports do not need to perform LX RV.

8. Employment and Training Administration DEPARTMENT OF LABOR ETA Data Element Validation (DEV)  Pertains to all programs, but is minimal for the LX (Wagner-Peyser/VETS) program.  Involves checking data in participant records against allowable source documentation to verify compliance with federal definitions. Elements either “pass” or “fail” validation.  Except in case of LX, DEV is labor- intensive, because it involves state staff conducting reviews of a sample of participant records from across the state.

9. Employment and Training Administration DEPARTMENT OF LABOR ETA How DEV Works The DRVS software generates a sample of participant records to be “inspected” by state staff.  A random sample for Workforce Investment Act Programs (WIA) and Trade Act programs  25 total records for Wagner-Peyser Employment Services (LX or Labor Exchange) “DEV Worksheets” for each record are also generated. The worksheets contain elements selected for validation that apply to the participant.

10. Employment and Training Administration DEPARTMENT OF LABOR ETA

11. Employment and Training Administration DEPARTMENT OF LABOR ETA How DEV Works (cont.) State validators use appropriate guidance to note allowable source documentation & check the accuracy of each data element. Documentation must either MATCH the element or SUPPORT the element. Most source documentation is located at the One-Stop level. Some documentation is stored at state-level only (e.g., wage record information must be checked against the state’s wage record files).

12. Employment and Training Administration DEPARTMENT OF LABOR ETA Think of it this way... Scenario:  A state reports an Adult Entered Employment Rate (EER) of 78% based on a numerator of 975 and a denominator of 1250  The EER calculation is based on the percentage of adults not employed at participation who were employed in 1 st quarter after exit.

13. Employment and Training Administration DEPARTMENT OF LABOR ETA Data Quality from the Perspective of RV For instance:  How do we know the 78% Adult EER is correct?  Does the denominator consist of the “right” exiters (e.g., those not employed at participation)?  Are all Transitioning Service Members (TSMs) included in calculations as required? In other words, are the calculations correct? Did the state follow federal reporting specifications for the numerator and denominator?

14. Employment and Training Administration DEPARTMENT OF LABOR ETA Data Quality from the Perspective of DEV For instance:  How do we know those individuals identified as Transitioning Service Members (TSMs) were within 12 months of separation or 24 months of retirement from the service?  For those “employed in the 1 st quarter after exit,” what if the exit date was actually in a prior quarter? In other words, were the data used for the EER calculation or even program eligibility correct to begin with?

15. Employment and Training Administration DEPARTMENT OF LABOR ETA The Essence of Data Validation It’s all about Data Quality and Performance Management!  Are the data used in the calculations accurate?  Are the calculations reported by the state accurate based on federal reporting specifications?

16. Employment and Training Administration DEPARTMENT OF LABOR ETA Putting DV Into a Program Context (Trade Act) What data? What do they report?  Each quarter, states submit a TAPR (Trade Activity Participant Report) consisting of individual records; USDOL generates aggregate performance results on a quarterly and annual basis.  TAA Programs are responsible for DEV only (no RV).

17. Employment and Training Administration DEPARTMENT OF LABOR ETA Putting DV Into a Program Context (Labor Exchange) What data? What do they report?  Each quarter, states submit the ETA 9002A—E and VETS 200 A—C ; these are aggregate reports of participants, services and outcomes.  WP/VETS Programs are responsible for RV and a minimal DEV (25 records), based on the final fourth quarter report submission.

18. Employment and Training Administration DEPARTMENT OF LABOR ETA Putting DV Into a Program Context (WIA) What data? What do they report?  States validate WIA Annual Reports and use individual WIASRD records to conduct Data Element Validation.  States are not required to validate WIA Quarterly Reports.

19. Employment and Training Administration DEPARTMENT OF LABOR ETA National Data Validation Results PY 2007 DEV Results  WIA Data Element Validation  Top 5 elements with the highest Overall Error Rate  Elements chosen with at least 1000 records sampled  50 states submitted results

20. Employment and Training Administration DEPARTMENT OF LABOR ETA FSElementValidated Cases Errors AdultProgram Exit Date 135101192 DWProgram Exit Date 1959 701 NEGProgram Exit Date 4762 340 OYProgram Exit Date 6835 513 YYProgram Exit Date 10462 863 Program Exit Date

21. Employment and Training Administration DEPARTMENT OF LABOR ETA FSElementValidated CasesErrors OYDateDegree/ Certificate 6835875 YYDateDegree/ Certificate 104621635 Date of Degree or Certificate

22. Employment and Training Administration DEPARTMENT OF LABOR ETA Date Exited Training FSElementValidated Cases Errors AdultDateExitTraining135101213 DWDateExitTraining11959864 NEGDateExitTraining4762190

23. Employment and Training Administration DEPARTMENT OF LABOR ETA Date of Dislocation FSElementValidated Cases Errors DWDislocationDate119591317 NEGDislocationDate4762597

24. Employment and Training Administration DEPARTMENT OF LABOR ETA FSElementValidated Cases Errors NEGNEGProject14762281 NEG Project 1

25. Employment and Training Administration DEPARTMENT OF LABOR ETA National Data Validation Results PY 2008 Report Validation Results  33 states utilized DRVS for reporting WIA RV results.  11 states had significant (>2%) percent differences for at least one report element.  Top 5 report elements had the most instances of differences > 2%.  All but 6 states utilize ETA (DRVS) or ETA- approved software for LX RV results. Only one state showed any significant differences.

26. Employment and Training Administration DEPARTMENT OF LABOR ETA Older Youth 12 Months Earnings Change Numerator and Rate (Table L)  6 out of 11 states had significant differences.  Range of percent difference was between 31% and -77. Older Youth Out of School Earnings Change (Table I)  5 out of 11 states had significant differences.  Range of percent difference was between -2% and -22%. Older Youth Wages at Entry Numerator (Table L)  5 out of 11 states had significant differences.  Range of percent difference was between 38% and -48% Report Validation Results

27. Employment and Training Administration DEPARTMENT OF LABOR ETA Literacy Numeracy Denominator  5 out of 11 states had significant differences.  Range of percent difference was between 6% and -7%. WIA Adult Participants  5 out of 11 states had significant differences.  Range of percent difference was between -2% and -13632%. Report Validation Results (con’t)

28. Employment and Training Administration DEPARTMENT OF LABOR ETA Source Documentation Whether electronic or paper, the purpose of source documentation is to have an auditable trail that documents eligibility, services delivered and outcomes received. It’s about accountability!

29. Employment and Training Administration DEPARTMENT OF LABOR ETA Guidance  Location! Location! Location!  WIA PY08—TEN 8-09 Attachment A  Trade FY09—TEN 8-09 Attachment D  LX The State verifies the accuracy of data elements using its source databases (State MIS, UI/WRIS/FEDES cross-match).

30. Employment and Training Administration DEPARTMENT OF LABOR ETA Validation Rules  Match – the data on the worksheet must be the same as the data on the source document. For example, if the worksheet shows the birth date to be 7/1/1975, the source documentation must show the exact same date.  Support – the data on the worksheet must be similar to the data in the source documentation.  This instruction is used, when the information must be processed or interpreted.  For example, if the worksheet shows a participant to be basic skills deficient, the source documentation must support this interpretation.

31. Employment and Training Administration DEPARTMENT OF LABOR ETA Allowable Source Documentation Four common types of source documentation are:  MIS (or Management Information System),  Cross-Match,  Self-Attestation, and  Case Notes.

32. Employment and Training Administration DEPARTMENT OF LABOR ETA MIS MIS refers to specific, detailed information that is stored in the state’s management information system that supports an element. An indicator, such as a checkmark on a computer screen, is not acceptable source documentation in and of itself.

33. Employment and Training Administration DEPARTMENT OF LABOR ETA Example of MIS - Date of Exit DRVS# 35 What information would you look for in your MIS to pass this data element?  The services provided  Date of last service  There were no other qualifying services provided within 90 days following this service.  The participant was not scheduled for a future service (no gap in service per TEGL 17-05).

34. Employment and Training Administration DEPARTMENT OF LABOR ETA Cross Match Cross match requires accessing a non- WIA MIS to find detailed supporting evidence for the data element. An indicator or presence of a SSN or checkmark in a non-WIA database is not sufficient evidence.

35. Employment and Training Administration DEPARTMENT OF LABOR ETA Example of Cross Match - Youth Placement Information (DRVS #154) State XX uses a cross-match with their Higher Education Administration as the source document for verifying placement into post-secondary education. What information would you need to pass this data element?  Name or some other personal identifier  Information showing enrollment in post-secondary education  Date that indicates the youth was enrolled in the first quarter after the exit quarter

36. Employment and Training Administration DEPARTMENT OF LABOR ETA Self-Attestation Self-attestation is when a participant states his/her status for a particular data element, and then signs and dates a form acknowledging this status. The key elements for self-attestation are:  The participant identifying his/her status for permitted elements, and  Signing and dating a form attesting to this self- identification.

37. Employment and Training Administration DEPARTMENT OF LABOR ETA Example of Self-Attestation – Displaced Homemaker DRVS# 24 What type of information/language must be included on a self-attestation form to be considered an acceptable source document for this data element? A. The individual:  Has been providing unpaid services to family members in the home, and  Has been dependant on the income of another family member, and  Is no longer supported by that income, and  Is unemployed or underemployed, and  Is experiencing difficulty in obtaining or upgrading employment  Applicant signature, date signed

38. Employment and Training Administration DEPARTMENT OF LABOR ETA Case Notes Case notes refer to either paper or electronic statements by the case manager that identify, at a minimum, the following:  a participant’s status for a specific data element,  the date on which the information was obtained, and  the case manager who obtained the information.

39. Employment and Training Administration DEPARTMENT OF LABOR ETA Types of Source Documentation Misc.  Only one of the listed source documents is required to validate a data element - Beware of conflicting information!  State Source Documentation Requirements

40. Employment and Training Administration DEPARTMENT OF LABOR ETA Common DV Findings

41. Employment and Training Administration DEPARTMENT OF LABOR ETA Failure to Follow Federal DV Requirements  Record Retention  Source Documentation  Validation Rules - Match or Support  Failure to Submit RV and DEV Summaries

42. Employment and Training Administration DEPARTMENT OF LABOR ETA Failure to Follow Federal DV Requirements Record Retention  UI/WRIS wages  DEV worksheets  Local case files

43. Employment and Training Administration DEPARTMENT OF LABOR ETA Failure to Follow Federal DV Requirements Source Documentation Issues  Using documents that are not on ETA’s list of acceptable source documents  Youth who needs additional assistance  Date of birth

44. Employment and Training Administration DEPARTMENT OF LABOR ETA Failure to Follow Federal DV Requirements  Validation Rules - Match or Support  Insufficient information  Conflicting information  Failure to Submit RV and DEV Summaries  DRVS used for producing federal reports  States that use DART for LX reporting  Failure to Validate all Required Data Elements

45. Employment and Training Administration DEPARTMENT OF LABOR ETA Other DV Review Findings  Failure to accurately record Participation and Exit Dates  Poor case notes  Separation of duties

46. Employment and Training Administration DEPARTMENT OF LABOR ETA Issues, Challenges, Considerations  Policy changes affecting definitions and calculations  Software changes and releases to keep up with policy changes  Handbooks not updated  Overwhelming demand for technical assistance (TA)  State resource requirements

47. Employment and Training Administration DEPARTMENT OF LABOR ETA Follow Up For questions about data validation, please send to etaperforms@dol.gov.etaperforms@dol.gov Check out www.doleta.gov/performance.www.doleta.gov/performance Keith Rowe at rowe.keith@dol.govrowe.keith@dol.gov Karen Staha at staha.karen@dol.govstaha.karen@dol.gov