1. 1 Meeting Data Collection Challenges of the Future James Griffith Ted Socha Thomas Weko Postsecondary Studies Division National Center for Education Statistics

2. 2 Purpose of the Presentation To show how our “technological innovations” have responded to challenges facing sample survey and longitudinal study data collections.

3. 3 Overview Provide overview of postsecondary sample survey and longitudinal studies as prelude to … Major challenge – Encourage participation of institutions and students Technological innovations to meet this challenge Future challenges, concluding remarks

4. 4 Postsecondary Longitudinal and Sample Survey Studies National Postsecondary Student Aid Study (NPSAS) Beginning Postsecondary Students Longitudinal Study (BPS) Baccalaureate and Beyond Longitudinal Study (B&B) National Study of Postsecondary Faculty (NSOPF)

5. 5 Primary Purposes NPSAS NPSAS:08, includes 125,000 undergraduate and 13,000 graduate/first-professional students enrolled in 1,963 institutions Describes how students and their families pay for postsecondary education and the role of federal student aid Describes undergraduate and graduate student populations, including topics such as civic participation, community service, educational aspirations, life goals, disabilities Provides rich database for postsecondary research and policy analysis, data gathered on over 1,100 variables Responds to emerging federal policy interests, such as debt level, use of private loans, training in STEM majors, and state-level financial aid issues

6. 6 Primary Purposes BPS Includes about 18,500 first-time beginning students identified in NPSAS, followed 2 years and 5 years later Provides data on student “flow” in and out of postsecondary education, such as persistence, transfer behavior, attainment Gathers data on initial work experiences of those who obtain certificates or 2-year degrees B&B Includes about 23,500 bachelor degree completers identified in NPSAS, followed 1 year, 5 years, and 10 years later Provides important information on graduate education, work, and career through retrospective information on: -- postsecondary experiences -- paths taken to the degree award, time to degree completion Focuses on “teacher pipeline,” interest in teaching, preparation, and job search

7. 7 Chronology of PLSSS Studies BPS = Beginning Postsecondary Students Longitudinal StudyB&B = Baccalaureate and Beyond Longitudinal Study

8. 8 Major Data Collections NPSAS occurs in staged sampling periods requiring data collection at several levels: Institution data collection Student interview data Other administrative data

9. 9 Data Collections by Study NPSASBPS B&B Institution-Level Data Institution characteristics (IPEDS data) ××× Student records (institutional and state aid) × Transcripts ×× Student Interview Data ××× Other Administrative Data Central Processing System (e.g., FAFSA) ××× NSLDS (Pell and loan files) ××× National Student Clearinghouse (enrollment) ××× ACT / SAT / Praxis ××

10. 10 The Challenge

11. 11 Declining participation of institutions and students Missing data at the case-level and item-level Complicated by … -- Diversity in access and use of technology across institutions (about 6,700) and students (about 19 million) -- Data privacy/security concerns Government = Misuse of study data Institutions = Risks of releasing student information Families = Privacy, identity theft -- Tight schedule for data collection, processing, and delivery – from lists of students gathered in May to data delivery in January -- Students as transient population GOAL – To maximize the usefulness and quality of the data vis-à-vis the challenge and complicating factors …

12. 12 Institutional Data Collection Challenge: Maximize institution participation by acquiring enrollment lists for sampled institutions. Innovations: Contacted early (fall contact for spring collection) Established “Study Coordinator” (IRP as initial commitment) Provided Help Desk (10-12 staff, Mon-Fri work hours) Provided easy access through Website (information, secure login, CADE) Ensured confidentiality / security (FERPA documents, IRBs) Offered multiple options for participation (secure website, encrypted fax, hard copy)

13. 13 Institutional Data Collection Challenge: Minimize institution nonresponse. Innovations: Developed real-time monitoring of characteristics of institutions (IMS) Performed bias analysis for < 85% participation overall and/or within strata, NCES standard Made weight adjustment to reduce bias at the institution/unit level

14. 14 Real-time Monitoring System We can see our response statistics in real-time. We then target “under-responding” institutions for participation in near real-time.

15. 15 Institutional Data Collection Challenge: Maximize student record collection. Innovations: Provided accessible Web-based instrument (CADE) Provided Help Desk (Mon-Fri during normal work hours) Negotiated reimbursement to “incentivize” participation

16. 16 Example – Web CADE Student Record Abstraction

17. 17 CADE abstraction method Institutions providing CADETotal students 1 NumberPercent 2 NumberPercent 2 Total1,300100.0103,620100.0 Abstraction method Web-CADE86065.948,86047.2 Data-CADE28021.133,21032.0 Field-CADE17013.121,55020.8 Student Record Abstraction Method – NPSAS:04 1 The total represents the number of students sampled from institutions that completed computer-assisted data entry (CADE) and may include students who were classified as study nonrespondents. 2 Percentage of total number of eligible institutions/students. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study (NPSAS:04). Web CADE preferred choice

18. 18 Institutional Data Collection Challenge: Maximize student transcript collection. Innovations: Addressed confidentiality/privacy concerns through secure fax, encrypted email, FTP transferAddressed confidentiality/privacy concerns through secure fax, encrypted email, FTP transfer Negotiated reimbursement to “incentivize” participation

19. 19 Example – Web Transcript Collection

20. 20 Student Interview Data Collection Challenge: Maximize student interview collection. Innovations: Redefined completed case Employed multiple tracing vendors, e.g., CPS, NCOA, Telematch, Equifax, Experian,TransUnion Offered response mode options fitting to the population (self-administered via Web, telephone, face-to-face) Provided accessible Web-based instrument “Incentivized” for early response Provided Help Desk (7 days a week)

21. 21 Cases Have Multiple Data Sources … Primary sources Institution records (CADE)95% Student interviews 70% Federal aid applications (CPS)60% Combinations of source All three primary sources40% Two sources50% One source10% Additional sources Federal loans & Pell Grants (NSLDS)50%

22. 22 Allowing for Redefining a Case Student case is a “complete” if valid data exist for: -- Student type -- Birth date or age -- Gender, -- And at least 8 of the 15 variables: Dependent status, marital status, any dependents, income, expected family contribution, degree program, class level, first- time beginner, months enrolled, tuition, received financial aid, received non-federal aid, student budget, race, and parent education.

23. 23 Difficult-to-Locate Students

24. 24 Even so, Some Are Locatable

25. 25 Example –B&B:08/09 Self-Administered Web Interview

26. 26 Student Mode Choice for Interview Completed interviews NumberWeighted percent Total62,220100.0 Self-administered28,71046.7 Early response17,10027.5 With prompting11,61019.2 Interviewer-administered33,51053.3 NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study (NPSAS:04). Almost ½ respond via web. 1/3 receive incentive. Growing trend.

27. 27 NPSAS:04 Experience with Incentives

28. 28 “Incentivized” Respondents $20 vs. $30 Web rises in preferred choice

29. 29 Student Interview Data Collection Challenge: Minimize bias from case-level and item-level missing data. Innovations: Made weight adjustment at case or student-level Imputed item-level missing data by: -- Logical imputation or -- Statistical imputation with nearest similar neighbor donor

30. 30 Who Responds and Who Doesn’t

31. 31 Nonresponse bias was estimated using statistical modeling. Dependent variable = Responded or Not responded. Predictor variables = Variables having values for both respondents and nonrespondents, thought to be predictive of response status, e.g.: -- institution type; region; institution enrollment from IPEDS file (categorical); student type; FTB status; etc. Weight adjustments = Coefficients for predictor variables used for weighting. Nonresponse Procedure

32. 32 Example Weight Adjustments Already seen -- hard to locate and under- respond. So, “double-up” responding case.

33. 33 Imputation Procedure Logical – use other data sources to determine missing variable values for given case (e.g., substitute FAFSA for institution CADE). Statistical –separate cases into dissimilar groups (using a defined set of variables) such that respective group members are alike. Membership in final group determines donor candidates or “nearest neighbor.” For cases having missing variable values, such values are “borrowed” from the “nearest neighbor.”

34. 34 Item-Level Missing Data

35. 35 Future Challenges to Data Collections Increased concerns about security / confidentiality -- both institutions and students Reduced access to students --Increased liability concerns of institutions to release student-level contact information --Greater cell phone usage Raised respondent expectations– offering $ incentives

36. 36 Products Data for analysis Data analysis system, http://nces.ed.gov/dasol Restricted data file, http://nces.ed.gov/pubsearch/licenses.asp Reports On methods … http://nces.ed.gov/pubsearch/pubsinfo.asp?pu bid=2006180 On content … http://nces.ed.gov/pubsearch/pubsinfo.asp?pu bid=2006186

37. 37 How to be Informed of Products http://ies.ed.gov/newsflash

38. 38 Contact Information Tom.Weko@ed.gov 202-502-7643 James.Griffith@ed.gov 202-502-7387 Ted.Socha@ed.gov 202-502-7383