Douglas N. Harris Associate Professor of Economics University Endowed Chair in Public Education Tulane University Presentation at Hualien Dec.17, 2012 Using Randomized Trials and Behavioral Economics to Address Problems in U.S. College Market

Two Recent Advances in Economics Behavioral economics Randomized trials The two advances are connected: Most behavioral economics theories are rooted in psychology research and randomized trials are more common there 20 years ago, this field was called “experimental economics” reflecting the methodological focus

Experiments vs. Quasi-Experiments Quasi-experiments have high external validity We tend to prefer more complex techniques and randomized trials involve less statistical complexity The real advantage is that estimates may generalize to other “real- world” settings However, quasi-experiments can be biased (low internal validity) and often have low statistical power Randomized trials have low bias (high internal validity) However, the situations are often unrealistic and therefore have low external validity Some would say that internal validity is more important because we cannot generalize something to other situations if it is not accurate about the particular setting where it occurred

Financial Incentives for College a Case in Point U.S. spends $155 billion annually on financial aid programs for low- and middle-income people Until about five years ago, we only had quasi- experiments of aid These suggest aid effects of about 3-4 percentage point on entry per $1,000 in aid There is debate about whether these are small or large, however, note that if we compare financial aid to other programs, this does not appear cost-effective

Stylized Facts of U.S. College Market

Two Initial Stylized Facts The return to human capital investment (education) is much higher than the return on other forms of capital investment (i.e., people seem to under-invest in education) Underinvestment is especially common among students from low-income families They invest less even when they have similar academic ability as those from higher income families (and there are no credit constraints)

College attendance among year old males (Carneiro & Heckman, 2002)

B.A. degree completion by age 24 (conditional on entry) Source: Postsecondary Education Opportunity

College attendance by income and test scores (Carneiro & Heckman, 2002)

Theories that might explain these patterns

Old Neoclassical Theories Risky returns: people are risk-averse and, obviously, not everyone obtains the average return (Heckman, 2006) However, people would have to be extremely risk-averse for this to be the only explanation (Palacios-Huerta, 2006) Option value When investments are irreversible and timing is flexible, risk-neutral agents may delay if they can obtain more information in the future (Jacobs, 2007) Psychic costs Education is a consumption good and some do not like it Misinformation: under-estimate return or over-state costs Unlikely; people over-state both the costs and the benefits of higher education by similar amounts (Kane & Avery)

Apparently New Explanations from within Neoclassical Model Variation on risky returns: Uncertainty leads students to invest less in learning during primary and secondary school (perhaps because of misinformation or risk) Uncertainty for all but especially low-income families Initial under-investment worsens the consumption disutility; i.e., people who do poorly in school are more likely to dislike college because it is too hard (they may not anticipate this) Related to Heckman’s idea that investments have to be made early for cognitive reasons

Behavioral Economic Theories Loss aversion General meaning: The marginal disutility of a loss is larger (in absolute value) than the utility of an equal-sized gain In human capital: People may fear taking the chance that they will lose what they have (or lose what they think they can obtain with no education) Endogenous preferences (peer effects) General meaning: People’s preferences are influenced by their friends and family In human capital: Students’ preferences about education are influenced by classmates and parents (hard to separate parent influence from availability)

More Behavioral Economic Theories Availability General meaning: People over-state risks of negative events for which they have vivid examples in mind Human capital investment: Low-income students are more likely to have parents who failed in college; they have a negative example available Ambiguity aversion General meaning: People prefer lotteries where the probabilities are easy to assign (holding constant actual risk) Human capital investment: Probabilities are hard to assign because there are many forms of uncertainty (cost, psychic cost, returns)

Theory Summary Above discussion suggests many possible explanations for underinvestment How can we test them?

Empirical Evidence from Two Randomized Trials

College Student Experiment: Wisconsin Scholars Longitudinal Study Low-income four-year college students randomized to receive $17,500 for college if they stay in school full-time Students located in many public colleges throughout the state of Wisconsin We have data from the first three years

Impact on Aid Package: Year 1 ( ) CONTROLTREATMENT IMPACT ($3,500) Total Aid$11,426 $1,665*** Pell (%) State Grant (%) *** SEOG (%) *** ACG (%) (3.0) Institutional Aid (%) Work Study (%) * Sub. Staff (%) *** Unsub Staff (%) Total loans ($)

Average Impacts of the Grant on Enrollment: Control MeanTreatment Effect Total # terms enrolled (f/s, %) (.09) Ever transferred (%) (2.7) Ever attended 2-year college (%) (2.2) Completed associates degree (%) (1.1)

Average Impacts of the Grant on Credits: Control MeanTreatment Effect Credit Accumulation Average credits completed (1.0) Earned 1-29 credits (%) (2.4) Earned credits (%) (2.4) Earned credits (%) (3.1) *** Earned 60+ credits (%) (2.8) ** Progress toward 4-year Degree 60+ credits, GPA (0.9) 60+ credits, GPA (1.6) 60+ credits, GPA (2.0) ** 60+ credits, 3.5+ GPA (1.4) *

Results from College Student Experiment No effect on college persistence Small positive effects on GPA and number of credits Pattern of effects is not predicted by the neoclassical model Results look slightly better in more recent cohorts Perhaps most interesting is evidence of loss aversion Some students lost the grant (arguably exogenously) and these students had worse outcomes than the control group

High School Student Experiment: Degree Project 2,500 low-income high school students in Milwaukee, Wisconsin were randomly assigned to receive $12,000 for college Main requirements: 2.5 GPA and 90% attendance rate Program has only been operating for one year and students are now in 10 th grade As an aside, the methodology is also interesting: a paired cluster RCT

Distinctive Experiment Cluster RCT Paired randomization Baseline Equivalence Standard error calculations Statistical power associated with correct std errors Generalizability

Paired Randomization When we get the standard errors right, we often have insufficient power in a cluster randomized trial Paired randomization can be a good solution, especially in this case when pairing on a lagged value of the dependent variable But this affects everything about how the analysis is carried out (more to come)

Statistical Power The main concern with cluster RCTs is that, with correct standard errors, power is insufficient to identify plausible effects Pairing combined with paired randomization largely solves this problem Power in this design is the same as an individual- level RCT with the same N, but no covariates Only slightly worse than individual-level RCT with the same covariate R 2

Generalizability with Power This design has the best of both worlds: strong statistical power with improved generalizability Nothing is really generalizable but cluster RCTs are almost certainly an improvement We really can have our cake and eat it too Key is to have lagged dependent variable and use these for both the pairing and for covariate adjustment Significant implications for the design of program evaluations

Analysis Issues Treatment stays with student even if switch schools Cluster RCT means analysis“at level of cluster” (j) Ambiguous terminology; can still use student-level data (i) to (slightly) increase power (Bloom) Pairing means cannot simply use diff-in-means tests For statistical power, we wouldn’t want to anyway y ij =α+βT j +κ p +γ 1 P j +γ 1 X ij +ε ij Outcome Treatment Pairing var. Stud/sch char. Pair effects

Standard Errors Ignoring clustering in cluster RCT yields std errors 1/5 to 1/2 actual size The “unit of analysis is the unit of randomization” is usually interpreted to mean that analysis is at the cluster level of aggregation and that this yields correct standard errors In reality, estimating at cluster level yields std errors that are too small, though still need to account for clustering

Correct Standard Errors Stata “cluster” command helps, but still yields std errors that are too small Bootstrapping is generally a good solution (Cameron, Gelbach, & Miller, 2008) Re-sample with replacement to obtain thousands of “new” samples and re-estimate But there are no prior examples of bootstrapping with paired cluster RCT In short, bootstrap the pairs, not the clusters

Results of High School Student Experiment So far, no effect on GPA or attendance But large positive effects on college expectations Summary: effects on perceptions but not behavior We are just now formulating ways of testing the behavioral economic assumptions

Back to the Theory

Tests of Behavioral Economics Theories Using Behavioral Responses Loss aversion We already have some direct evidence of this as noted above Peer effects We have exogenous variation in the number of classmates students have who received the scholarship If there are peer effects, we should see that students with recipient classmates are affected more

More Tests with Surveys Survey questions below asked in college experiment and we are asking similar ones in high school exper. Consumption value/psychic cost “How much fun is college?” “How interesting are your classes?” “How difficult is the material in your classes?” “How much do you like the freedom of being in college?” “How much do you enjoy the people you go to college with?”

More Tests (cont.) Combined risk-aversion during high school and psychic costs Ask consumption value questions similar to above Also, compare the effects of the two experiments; if there is very little difference in effects, then this theory is rejected

Tests of Behavioral Economic Theories Using Surveys Availability We ask whether students’ parents had a “bad experience” in college We also ask the level of education, so we can distinguish other ways in which parents education might matter Ambiguity aversion More difficult to test No question planned at this time

Conclusion To be announced However, these experiments provide a unique opportunity to test multiple theories The experiments will also have broader implications for financial aid and incentive programs and for our general understanding of human behavior