1. Longitudinal Approaches to Research and Evaluation in Adult Education Professor Stephen Reder Portland State University Charles Darwin University October 9, 2014

2. Outline of Session Flexible & negotiable depending on your needs and interests Flexible & negotiable depending on your needs and interests Longitudinal Research Methods Longitudinal Research Methods Applications to Research and Program Evaluation in Adult Education Applications to Research and Program Evaluation in Adult Education Example applications in my own longitudinal study of adult learning Example applications in my own longitudinal study of adult learning

3. Longitudinal Methods Texts Singer, J. D. & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. Oxford & New York: Oxford University Press. Singer, J. D. & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. Oxford & New York: Oxford University Press. Magnusson, D., Bergman, L. R., Rudinger, G., et al. (Eds.) (1994). Problems and methods in longitudinal research: Stability and change. New York: Cambridge University Press. Magnusson, D., Bergman, L. R., Rudinger, G., et al. (Eds.) (1994). Problems and methods in longitudinal research: Stability and change. New York: Cambridge University Press. Saldaña, J. (2003). Longitudinal qualitative research: Analyzing change through time. Lanham, MD: AltaMira Press. Saldaña, J. (2003). Longitudinal qualitative research: Analyzing change through time. Lanham, MD: AltaMira Press.

4. Longitudinal Adult LLN Research Text Reder, S. & Bynner, J. (Eds.) (2009). Tracking adult literacy and numeracy skills: Findings from longitudinal research. New York & London: Routledge. Reder, S. & Bynner, J. (Eds.) (2009). Tracking adult literacy and numeracy skills: Findings from longitudinal research. New York & London: Routledge.

5. Many Fundamental Research Questions in Adult Education Involve Change Over Time How adults learn How adults learn How participation in programs influences adults’ acquisition of knowledge and skills How participation in programs influences adults’ acquisition of knowledge and skills How adults’ educational development interacts with their social and economic performance How adults’ educational development interacts with their social and economic performance How adults’ private and public situation in the family and community may enhance or obstruct their progress in learning How adults’ private and public situation in the family and community may enhance or obstruct their progress in learning

6. Methodological Concepts and Issues in Longitudinal Research Sample attrition and missing data Sample attrition and missing data Longitudinal stability of measures and scales Longitudinal stability of measures and scales Methods for analyzing change in longitudinal data Methods for analyzing change in longitudinal data

7. Sample Attrition and Missing Data Missing item and unit data Missing item and unit data Ignorable and Non-Ignorable Missing Ignorable and Non-Ignorable Missing Missing Completely at Random (MCAR) Missing Completely at Random (MCAR) Missing at Random (MAR) Missing at Random (MAR)

8. Constructing Measures and Scales That Are Longitudinally Stable Essential to know that the same construct is being measured at different time points Essential to know that the same construct is being measured at different time points Scales having good psychometric properties for measurement on a single occasion do not necessarily have good psychometric properties for measuring change across multiple occasions Scales having good psychometric properties for measurement on a single occasion do not necessarily have good psychometric properties for measuring change across multiple occasions Cannot determine longitudinal stability in advance, must use longitudinal data to examine measurement stability Cannot determine longitudinal stability in advance, must use longitudinal data to examine measurement stability

9. Some Methods For Analyzing Change in Longitudinal Data Growth Curve Modeling Growth Curve Modeling Event Occurrence Modeling Event Occurrence Modeling Structural Equation Modeling Structural Equation Modeling Treatment Effects Models Treatment Effects Models Dynamic Panel Models Dynamic Panel Models

10. The Longitudinal Study of Adult Learning (LSAL) funded by U.S. Department of Education funded by U.S. Department of Education National Institute for Literacy Portland State University

11. LSAL Perspective Look at how programs fit into the lifelong and life-wide landscape of adults’ learning, rather than at how adults fit into LLN programs as students Look at how programs fit into the lifelong and life-wide landscape of adults’ learning, rather than at how adults fit into LLN programs as students We’ll see that things look considerably different from this vantage point We’ll see that things look considerably different from this vantage point

12. LSAL Design Decade-long panel study of Portland (Oregon) early school leavers, age 18-44 at the beginning of the study Decade-long panel study of Portland (Oregon) early school leavers, age 18-44 at the beginning of the study Representative sample of ~1,000 drawn from local rather than national population of dropouts Representative sample of ~1,000 drawn from local rather than national population of dropouts Includes both program participants and nonparticipants Includes both program participants and nonparticipants Examines program participation and other learning activities, social and economic changes, and changes in literacy skills, literacy practices & technology use over time Examines program participation and other learning activities, social and economic changes, and changes in literacy skills, literacy practices & technology use over time Periodic in-home interviews and literacy assessments and SSN- linked administrative data (with individuals’ permission) Periodic in-home interviews and literacy assessments and SSN- linked administrative data (with individuals’ permission) Smaller-scale qualitative components Smaller-scale qualitative components

13. LSAL Realized Sample N = 940 N = 940 496 from RDD Frame 496 from RDD Frame 444 from Student Frame 444 from Student Frame High level of diversity in sample High level of diversity in sample 90% sample retention over 8 years 90% sample retention over 8 years 39 additional pilots for instrument development, training & qualitative studies 39 additional pilots for instrument development, training & qualitative studies

14. Some LSAL Demographics Average age is 28 (at Wave 1) Average age is 28 (at Wave 1) 50 % female and male 50 % female and male 35 % minority 35 % minority 9 % foreign-born 9 % foreign-born 34 % live in poverty 34 % live in poverty 29 % report a learning disability 29 % report a learning disability 34 % took special education 34 % took special education Broad range of assessed basic skills Broad range of assessed basic skills

15. LSAL Timeline wave 1 wave 1 1998 – 1999 wave 2 wave 2 1999 – 2000 wave 3 wave 3 2000 – 2001 wave 4 wave 4 2002 – 2003 wave 5 wave 5 2004 – 2005 wave 6 wave 6 2006 – 2007

16. Repeated Measures of Engagement in Literacy Practices

17. Scaling of Literacy Practices Started with 14 parallel questions, such as:Started with 14 parallel questions, such as: How often do you read the news section of the newspaper? How often do you read the news section of the newspaper? never / rarely / less than once a week / once a week never / rarely / less than once a week / once a week a few times a week / every day a few times a week / every day Confirmatory factor analyses of wave 1 data were used to examine internal reliability & factor structures Confirmatory factor analyses of wave 1 data were used to examine internal reliability & factor structures Two factors emerged, one assessing frequency of engagement in literacy practices and one in numeracy practices Two factors emerged, one assessing frequency of engagement in literacy practices and one in numeracy practices Structural equation models indicate these factors have stationary measurement properties over time Structural equation models indicate these factors have stationary measurement properties over time

18. Literacy Growth Curves With multiple time points and repeated measures, we can develop and test models of individual change in literacy With multiple time points and repeated measures, we can develop and test models of individual change in literacy We can examine how individual characteristics (e.g., years of schooling, nativity), key life events (e.g., birth of a child) and time-varying experiences (e.g., program participation) affect the growth process We can examine how individual characteristics (e.g., years of schooling, nativity), key life events (e.g., birth of a child) and time-varying experiences (e.g., program participation) affect the growth process Linear growth curve models closely fit LSAL data on literacy proficiency and literacy practices Linear growth curve models closely fit LSAL data on literacy proficiency and literacy practices Individuals’ literacy proficiency and practices do change substantially after leaving school Individuals’ literacy proficiency and practices do change substantially after leaving school Heterogeneity of individual change is crucial Heterogeneity of individual change is crucial

20. Person-specific intercept (“starting level”) Person-specific slope (“rate of change”)

21. Change in Proficiency (Wave 1-4) by Age at Wave 1

23. NALS Document Literacy by Age (1992 national data corrected for education & disabilities)

24. Similar results in ALLS & PIAAC data from Australia

25. Key Findings Measures of literacy proficiency and literacy practices show systematic change over time across the adult lifespan Growth curve models of these literacy measures show impressive heterogeneity of change: some adults show literacy growth, others little change, and others literacy loss over time Age, birthplace, parental education, intergenerational reading practices, K-12 schooling experiences, and health systematically influence adult literacy development Age, birthplace, parental education, intergenerational reading practices, K-12 schooling experiences, and health systematically influence adult literacy development

26. Key Findings (con’t) Key life history events – such as changes in family composition and employment changes -- influence adult literacy development Key life history events – such as changes in family composition and employment changes -- influence adult literacy development The dynamics of change are quite different for literacy proficiency and literacy practices: e.g., program participation directly affects literacy practices measures but not literacy proficiency

27. Outcomes & Program Impact Literacy proficiency growth over relatively short periods of time is not affected by program participation Literacy proficiency growth over relatively short periods of time is not affected by program participation Pre-post test accountability data, that apparently show systematic gains in participants’ proficiency, do not contrast participants’ gains with those of comparable non-participants; LSAL indicates those gains are equivalent Pre-post test accountability data, that apparently show systematic gains in participants’ proficiency, do not contrast participants’ gains with those of comparable non-participants; LSAL indicates those gains are equivalent Literacy practices growth over relatively short periods of time is, on the other hand, directly affected by program participation Literacy practices growth over relatively short periods of time is, on the other hand, directly affected by program participation These findings are reinforced by cross-sectional research (e.g., Smith & Sheehan-Holt) and by classroom studies (e.g., Purcell- Gates, Jacobson & Degener) These findings are reinforced by cross-sectional research (e.g., Smith & Sheehan-Holt) and by classroom studies (e.g., Purcell- Gates, Jacobson & Degener) Proficiency measures thus do not reflect the impact that programs have or support evidence-based program and policy improvement processes Proficiency measures thus do not reflect the impact that programs have or support evidence-based program and policy improvement processes

29. Identifying Causal Relationships Temporal ordering Temporal ordering Including all relevant variables Including all relevant variables Analytical methods for experimental designs Analytical methods for experimental designs Analytical methods for non-experimental designs Analytical methods for non-experimental designs Dealing with selection bias Dealing with selection bias Standard regression methods Standard regression methods Propensity score matching methods Propensity score matching methods Instrumental variables methods Instrumental variables methods Dealing with omitted variables bias Dealing with omitted variables bias Fixed effects panel regression Fixed effects panel regression

30. Estimating Participation Impact Adults decide whether to participate in basic skills programs, so participants and nonparticipants are not usually comparable (selection bias) Adults decide whether to participate in basic skills programs, so participants and nonparticipants are not usually comparable (selection bias) Several analytical methods can be used to address selection bias in comparing program participants & nonparticipants: Several analytical methods can be used to address selection bias in comparing program participants & nonparticipants: Treatment effects (propensity score matching) Treatment effects (propensity score matching) Difference-in-differences (propensity score matching) Difference-in-differences (propensity score matching) Fixed effects panel regressions Fixed effects panel regressions

31. Propensity Score Matching Compares participants and nonparticipants matched on their likelihood of participating based on observed pre-participation characteristics: Compares participants and nonparticipants matched on their likelihood of participating based on observed pre-participation characteristics: AgeGender Race/Ethnicity Education AgeGender Race/Ethnicity Education Immigration status Income Immigration status Income Learning disabilities Parents’ education Learning disabilities Parents’ education

32. Income Growth in Matched Participants (100+ hours) & Nonparticipants

33. Treatment Effects Models Estimates average treatment effect on treated by comparing 2007 incomes of matched participants and nonparticipants Estimates average treatment effect on treated by comparing 2007 incomes of matched participants and nonparticipants With participation defined as any amount of attendance, there is no significant difference between groups With participation defined as any amount of attendance, there is no significant difference between groups With participation defined as 25 or more hours of attendance, there is no significant difference between the groups’ 2007 incomes With participation defined as 25 or more hours of attendance, there is no significant difference between the groups’ 2007 incomes With participation defined as 75 or more hours of attendance, there is a nearly significant (p=0.053) difference between the groups’ 2007 incomes With participation defined as 75 or more hours of attendance, there is a nearly significant (p=0.053) difference between the groups’ 2007 incomes With participation defined as 100 or more hours of attendance, there a statistically significant difference: participants average $9,621 more in annual income over what they would have received if they had not participated (in 2013 USD) With participation defined as 100 or more hours of attendance, there a statistically significant difference: participants average $9,621 more in annual income over what they would have received if they had not participated (in 2013 USD)

34. Treatment Effects of Participation on 2007 Earnings Model Minimum Hours of Attendance to Be Considered “Treated” Number of Propensity -Score- Matched “Treated” Number of Propensity -Score- Matched “Controls” Average Treatment Effect on the “Treated” (ATET) tp Effect Size A13961051.1111.4110.159n.s. B25216820.7890.9530.342n.s. C75216802.0141.9420.053n.s. D100197682.1422.4690.0140.45 E150154622.2722.0850.0390.48

35. Difference-in-Differences (DID) Model Compares income changes over a decade (1997-2007) between matched participants and nonparticipants Compares income changes over a decade (1997-2007) between matched participants and nonparticipants There is no statistically significant DID between groups if participation is defined as any amount of attendance There is no statistically significant DID between groups if participation is defined as any amount of attendance If participation is defined as 100 or more hours of attendance, there is a statistically significant DID If participation is defined as 100 or more hours of attendance, there is a statistically significant DID Despite different statistical assumptions, estimates 2007 incomes to average $10,179 more because of participation, comparable to the treatment effects estimate of $9,621 (in 2013 USD) Despite different statistical assumptions, estimates 2007 incomes to average $10,179 more because of participation, comparable to the treatment effects estimate of $9,621 (in 2013 USD)

36. Difference-in-Differences

37. DIFFERENCE-IN-DIFFERENCES ESTIMATION 100+ hours --------------------- ------------ BASELINE --------- ----------- FOLLOW-UP ---------- --------- Outcome Variable | Control | Treated | Diff(BL) | Control | Treated | Diff(FU) | DID ---------------------+---------+-----------+----------+---------+-----------+----------+--------- Log Income | 7.651 | 6.126 | -1.525 | 5.386 | 6.935 | 1.549 | 3.074 Std. Error | 0.513 | 0.468 | 0.695 | 0.617 | 0.513 | 0.802 | 1.061 t | 14.91 | 4.39 | -2.19 | 3.98 | 9.85 | 2.31 | 2.90 P>|t| | 0.000 | 0.000 | 0.029** | 0.000 | 0.000 | 0.054* | 0.004*** ------------------------------------------------------------------------------------------------- * Means and Standard Errors are estimated by linear regression. **Inference: *** p<0.01; ** p<0.05; * p<0.1

38. Fixed Effects Panel Regression Model Within-subject models of year-to-year variations in income in relation to year-to-year program participation and other life events Within-subject models of year-to-year variations in income in relation to year-to-year program participation and other life events Eliminates selection bias due to observed and unobserved time-invariant individual characteristics Eliminates selection bias due to observed and unobserved time-invariant individual characteristics Reveals how temporal details of participation -- intensity, duration and elapsed time – are reflected in observed changes in economic status Reveals how temporal details of participation -- intensity, duration and elapsed time – are reflected in observed changes in economic status

39. Fixed Effects Panel Regression (con’t) Results consistent with other models Results consistent with other models Only when participation involves about 100 or more hours of attendance does it have a significant & substantial impact on future earnings Only when participation involves about 100 or more hours of attendance does it have a significant & substantial impact on future earnings Concentrated hours have a larger impact on earnings than hours distributed over years Concentrated hours have a larger impact on earnings than hours distributed over years The impact of participation on earnings takes several years to develop after program exit The impact of participation on earnings takes several years to develop after program exit

40. Fixed Effects Panel Regressions of Log Annual Earnings Model Participation Variables Coeff. in Panel Regr. Robust Std. Err. t ACUMHOURS.0032.0014 2.39* BYEARS.1221.1756 0.70 CYRSCUM100.4005.1510 2.65** D YRSCUM100 CUMHOURS.3106.0020.1411.0013 2.20* 1.55

41. Pulse, Step, Growth: The Shape of Program Impact

42. <= PULSE High burst, short-lived impact High burst, short-lived impact Example in LSAL data: effects of receiving GED credential shows a short-lived “brushing up” of proficiency Example in LSAL data: effects of receiving GED credential shows a short-lived “brushing up” of proficiency

43. <= STEP Abrupt, qualitative & lasting impact Abrupt, qualitative & lasting impact Some changes in literacy practices seem to have this temporal shape Some changes in literacy practices seem to have this temporal shape

44. <= GROWTH Slow, steady & progressively accumulating impact Slow, steady & progressively accumulating impact This is the shape of program impact on proficiency This is the shape of program impact on proficiency This is the shape of program impact on earnings This is the shape of program impact on earnings Life history events -- such as the birth of children, taking on or losing a partner, or a significant change in employment -- have similarly shaped impacts on the course of literacy development Life history events -- such as the birth of children, taking on or losing a partner, or a significant change in employment -- have similarly shaped impacts on the course of literacy development

45. Summary: Impact of Participation on Earnings Multiple methods of controlling for selection bias all indicate that participation in LLN programs has a significant positive impact on adults’ future earnings Multiple methods of controlling for selection bias all indicate that participation in LLN programs has a significant positive impact on adults’ future earnings The significance of the impact requires a minimum amount of program attendance, about 100 hours in the LSAL data The significance of the impact requires a minimum amount of program attendance, about 100 hours in the LSAL data The earnings premium grows over time and becomes substantial 5-6 years after program exit: the annual premium was nearly half (0.45) a standard deviation of 2007 incomes The earnings premium grows over time and becomes substantial 5-6 years after program exit: the annual premium was nearly half (0.45) a standard deviation of 2007 incomes The impact of participation is not at all evident in short-term follow- ups to program participation The impact of participation is not at all evident in short-term follow- ups to program participation Post-program learning, proficiency growth, and postsecondary education and training may all play a role mediating the continuing impact of participation on labor market outcomes Post-program learning, proficiency growth, and postsecondary education and training may all play a role mediating the continuing impact of participation on labor market outcomes

46. Fixed Effects Panel Regressions of Log Annual Income Model Participation Variables Coeff. in Panel Regr. Robust Std. Err. t E CUMHOURS HAVEGED.0030.4793.0013.4648 2.25** 1.03 F CUMHOURS YEARSGED.0028.4160.0013.1394 2.17* 2.98** G YRSCUM100 HAVEGED.3848.5360.1478.4542 2.60** 1.18 H YRSCUM100 YEARSGED.3025.3821.1437.1364` 2.10* 2.80**