1. 1 New Perspectives on the Quality of Administrative Data COPAFS Quarterly Meeting September 21, 2012 Bill Iwig, USDA/NASS

2. 2 Outline FCSM Subcommittee on Statistical Uses of Administrative Data: Where Weve Been Data Quality Working Group: Addressing the Elusive Quality Issue Development of the Data Quality Assessment Tool Status of Pilot Testing Next Steps Discussion

3. 3 FCSM Subcommittee on Statistical Uses of Administrative Data Created in late 2007 to identify and document: Agency experiences and practices Areas of needed research Barriers that preclude or raise the cost of administrative records projects Opportunities to collaborate

4. Subcommittee Products Profiles in Success of Statistical Uses of Administrative Data (Prell, et al.; 2009) Model Agreement to Share Administrative Records (Cornman, et al.; 2010): Under OMB Review Informed Consent: Requirements and Practices for Statistical Uses of Administrative Data (Nickerson, et al.; 2012): Under OMB Review 4

5. Data Quality Working Group: Addressing the Elusive Quality Issue A consistent and major concern of administrative data users. Survey organizations have an understanding about the error properties of their survey data – but not of administrative data. Other non-statistical quality dimensions also included in the Data Quality Frameworks of most major statistical organizations. 5

6. Development of the Data Quality Assessment Tool Data Quality ~ Fitness for Use Intended to facilitate decisions by a potential data user about the suitability of the data for identified purpose. Intended to prompt additional questions. Based on the information gathered, the data user may decide to: Use the data as planned Alter plans Accommodate some quality weakness in the data Not use the data 6

7. Development of the Data Quality Assessment Tool Multi-Dimensional Relevance Accessibility Interpretability Coherence Accuracy Institutional Environment Questions within each dimension are designed to address quality characteristics of micro-level administrative data. Example answers provided based on characteristics of a Supplemental Nutrition Assistance Program (SNAP) data set. 7

8. Development of the Data Quality Assessment Tool Discovery Phase Evaluation period leading to approval for developing a data-sharing MOU. Initial Acquisition Phase Evaluation period from MOU development approval to first- time receipt of data. Repeated Acquisition Phase Continuing periodic receipt of data. 8

9. Putting the Phases and Dimensions Together Discovery Phase: 12 questions covering Relevance, Accessibility, and Interpretability, including a request for a Data Dictionary Initial Acquisition Phase: 29 questions covering Accessibility, Interpretability, Coherence, Accuracy, and Institutional Environment Repeated Acquisition: 11 questions covering Interpretability (updated Data Dictionary), Coherence, Accuracy, and Institutional Environment 9

10. What it is and What it isnt Evaluates quality characteristics of an administrative data file. Doesnt address quality characteristics of linked data. Doesnt provide guidance for making fitness for use decisions. Doesnt provide quality improvement recommendations. Not specifically designed to address quality of commercial data. 10

11. Status of Pilot Testing Obtained feedback from three federal/state reviewers prior to pilot testing. Pilot tested on: Child Care data files from the Office of Child Care Information Systems at DHHS – Initial Acquisition Phase Wisconsin Child Care Payment System files – All Phases Expecting a response for two additional files. 11

12. General Comments Completing the Data Quality Assessment Tool is perceived to be burdensome to the administrative agency. Some agencies may be concerned about data quality implications. Many administrative data files are compiled from various data sources. Availability of a data dictionary is very important. Some questions will be addressed in the Data Sharing MOU. Key issue is to understand the content, structure, strengths, and weaknesses of the data. Based on this information, the user then decides if the data can be used for the intended purpose. Census Bureau plans to add responses to their Meta-Data Discovery Site for access by researchers exploring potential uses of administrative data. 12

13. Next Steps Presented at the FCSM/COPAFS Statistical Policy Seminar in December. Update based on feedback. Make available to the public. Pursue opportunities to promote use. 13

14. Thank You Bill Iwig National Agricultural Statistics Service Bill.Iwig@nass.usda.gov 202-720-3895/3918 14