Inference by College Admission Departments: Bayesian or Cursed? Michael Conlin Michigan State University Stacy Dickert-Conlin Michigan State University Cornell University– October 2009

Optional SAT I Policy Colleges with optional SAT I policy allow applicants to submit their SAT I score but do not require them to submit their score. (As of Spring 2009, more than 800 colleges have SAT- or ACT- optional policies.)

Research Question What does the admission department at the college infer when an applicant chooses not to submit her SAT I score? This inference influences whether or not the school accepts the applicant.

Bayesian vrs. Cursed Equilibrium Most game theoretic papers assume the inference is based on Bayes Rule. Eyster & Rabin (Econometrica 2005) document extensive experimental evidence and develop a simple model of how players underestimate the relationship between other players’ actions and their private information. They call this psychologically motivated equilibrium concept a cursed equilibrium.

Eyster and Rabin (page 1633) “ The primary motivation for defining cursed equilibrium is not based on learning or any other foundational justification, but rather on it pragmatic advantages as a powerful empirical tool to parsimoniously explain data in a variety of context.”

Why is this important? Think of any private information game – including those resulting in adverse selection, signaling and screening. Most of these games apply Bayesian Nash Equilibria to predict outcome. The Bayesian Nash Equilibrium Concept assumes that players apply Bayes Rule at all information sets that are reached with positive probability. Expected outcomes could change significantly if this were not the case.

Example using Spence Education Model Nature Firm e=0 High Productivity Worker Suppose a separating equilibrium exists where the low productivity type gets 0 years of education and the high productivity type gets 16 years of education. Low Productivity Worker e=12 e=16 e=0 e=12 e=16

Voluntary Disclosure/ Verifiable Cheap Talk Example Student i has the following probability distribution in term of SAT I scores. When disclosure is costless, Bayesian Nash Equilibrium results in every type except the worst disclosing and the worst being indifferent between disclosing and not disclosing. SAT I ScoreProbability

Voluntary Disclosure / Verifiable Cheap Talk Models Comments:  Distribution depends on student characteristics that are observable to the school such as high school GPA.  With positive disclosure costs, the “unraveling” is not complete and only the types with the lower SAT I scores do not disclose. Assumptions:  Common Knowledge.  Colleges use Bayesian Updating to Infer SAT I Score of those who do not Submit/Disclose  Colleges’ incentives to admit an applicant is only a function of his/her actual SAT I score (not whether the applicant submits the score)

Eyster and Rabin applied to Voluntary Disclosure / Verifiable Cheap Talk College correctly predicts the distribution of the applicant’s actions but underestimates the degree these actions are correlated with the applicant’s SATI score (i.e., private information). SAT I ScoreProbability “Fully” Cursed Equilibrium (χ=1)– College infers if applicant doesn’t disclose that his/her expected SAT I score is 1300(.2)+1200(.4)+1100(.3)+1000(.1)=1170 “Partially” Cursed Equilibrium (χ=.4 for example)– College infers if applicant doesn’t disclose that his/her expected SAT I score is (.4) (1-.4) [(1100(.3)+1000(.1))/.4] = 1113

Why is this important in the context of SATI optional policy? If, (1) a college underestimates the relationship between an applicant’s decision to not submit and their SATI score (i.e., private information) and (2) certain types of applicants are less likely to submit, then the SATI optional policy is likely to affect the demographic characteristics of the student body.

Overview of Environment SAT EXAMS: College Board administers SAT I & SAT II: Subject exams 7 times/year between October and June. SAT I was a 2-part verbal and math exam. There are 20 SAT II exams. EARLY DECISION: Submit application by mid-November & sign a written agreement (with guidance counselor) that states that upon admittance the applicant will attend if she can “afford it” (agreement not legally binding). Notified in December of acceptance. The deadline for regular admission applications is January 1 st and these applicants are notified of acceptance or denial between March and April. An accepted applicant that chooses to enroll fills out enrollment forms and makes a deposit by May 1st.

Applicant’s Early Decision and SAT Submission Decisions Applicant i’s expected utility conditional on attending the college is μ(Z i, ε ap, ε en )= μ(Z i ) + ε ap + ε en where, μ(Z i ) captures individual–specific preferences that depends on observable variables (Z i ), ε ap represents unobservable individual–specific preferences known to the applicant at the time she submits the application, and ε en represents unobservable individual–specific preferences not known to the applicant until she makes the enrollment decision. Applicant i’s expected utility if doesn’t attend U R if apply regular admission U R -C if apply early decision

Applicant’s Early Decision and SAT Submission Decisions Applicant i’s expected utility from not applying early decision to the college and submitting SATI scores is: where, is the applicant’s expectation of the probability she will attend, ε s represents the “cost” of submitting which is known to the applicant at the time she submits application, and K represents fixed cost of applying.

Applicant’s Early Decision and SAT Submission Decisions Assuming U R =0 (a normalization) and E(ε en )=0, an applicant applies early decision and does not submit if :

CONDITION 1 CONDITION 2 CONDITION 3 is not binding

College’s Objective Function To account for the college’s concern for the quality of its current and future students and the understanding that future student quality depends on the college’s ranking, we allow the college’s objective function to depend on the perceived ability of the incoming students, the “reported” ability of these students, the demographic characteristics of the student body and yield rate.

College’s Objective Function where, Π P (Z P ) captures how the college’s payoff is affected by the perceived student body’s ability level, Π R (Z R ) captures how the college’s payoff is affected by the reported student body’s ability level, Π D (Z D ) captures how the college’s payoff is affected by the demographic characteristics of the student body, f(YR) captures how the yield rate affects the college’s payoff

College’s Acceptance Decision College accepts applicant i if: where, denotes the expected probability an accepted applicant i enrolls where k equals ed (ned) if applicant i applies early decision (regular decision) and l equals s (ns) if the applicant i submits (does not submit) her SATI score, Z +i, Z -i, Z ri are expected characteristics of student body if applicant i is accepted and enrolls, is accepted but does not enroll, and is rejected.

College’s Acceptance Decision The probability applicant i is accepted if she applies early and does not submit is: with the expected student body characteristics being the same if applicant i is accepted but chooses not to enroll (Z -i ) or if applicant i is rejected (Z ri ).

Applicant’s Enrollment Decision An accepted applicant i who does not apply early decision will enroll if: μ(Z i ) + ε ap + ε en > U R An accepted applicant i who does apply early decision will enroll if: μ(Z i ) + ε ap + ε en > U R – C

Applicant’s Enrollment Decision The probability an early decision applicant i, who does not submit, enrolls conditional on being accepted is: P e (Z i,e,ns)= Prob[ε ap +ε en >-μ(Z i )-C, Conditions 1,2&3 hold] / Prob[Conditions 1,2&3 hold].

Generating a Likelihood Function 1. Parameterize the cursed equilibrium concept proposed by Eyster and Rabin; 2. Make assumptions about the applicants’ expectations of attending and the college’s expectations of enrollment; 3. Impose functional form restrictions on the college’s and applicants’ objective functions; 4. Make distributional assumptions for the random components.

Parameterizing cursed equilibrium We assume college’s belief of applicant i’s SAT score if she does not submit is SAT i,un + (1- )SAT i,cond where, SAT i,un is the belief of applicant i’s score if the college does not condition on her choosing not to submit and SAT i,cond is the belief if the college does condition on her choosing not to submit.

Parameterizing Eyster and Rabin’s cursed equilibrium concept We construct SAT i,un regress SATI scores for applicants who submit on their observable characteristics. use the resulting coefficient estimates to construct fitted values for all applicants, i, who do not submit. assume the fitted value for applicant i is what would be inferred if the college believes that applicant i’s decision to not submit did not depend on her actual SATI score. We assume SAT i,cond is applicant i’s actual SATI score.

Eyster and Rabin applied to Voluntary Disclosure / Verifiable Cheap Talk College correctly predicts the distribution of the other players’ actions but underestimates the degree these actions are correlated with the other players’ private information. SAT I ScoreProbability “Fully” Cursed Equilibrium (χ=1)– College infers if applicant doesn’t disclose that his/her expected SAT I score is 1300(.2)+1200(.4)+1100(.3)+1000(.1)=1170 “Partially” Cursed Equilibrium (χ=.4 for example)– College infers if applicant doesn’t disclose that his/her expected SAT I score is (χ)SAT un + (1-χ)SAT cond = (.4) (1-.4) [(1100(.3)+1000(.1))/.4] = 1113

Applicants’ expectations of attending

College’s expectations of enrollment Assume college uses the information they have on each applicant to accurately predict their enrollment probabilities. For those who submit, For applicants who chose not to submit their SATI scores, the college’s expected probability of enrollment is similar to the predicted probability except the college infers an SATI score of SAT i,un + (1- )SAT i,cond while the predicted probability, P e (Z,k,l), is based on their actual score.

Functional Form Restrictions μ(Z i ) = β μ Z i f(YR ai ) = β YR YR ai, f(YR ri ) = β YR YR ri Π P (Z +i P )= β P Z +i P, Π P (Z -i P )= β P Z -i P Π R (Z +i R )= β R Z +i R, Π R (Z -i R )= β R Z -i R and Π D (Z +i D )= β D Z +i D, Π D (Z -iD )= β D Z -i D

Functional Form Restrictions Assuming the college’s expectations of Z -i j, YR ri, and N sb do not vary across applicants, these functional form restrictions result in: where β P’ = β P /(N sb +1), β R’ = β R /(N sb +1), β D’ =β D /(N sb +1), β YR’ = β YR YR ri /(N+1) and β int is a function of Z -i j (for all j), YR ri, N sb and N.

Functional Form Restrictions Note that not all the parameters are identified. For example, suppose high school GPA affects both perceived quality and ranking, then can only identify β P GPA +β R GPA. We can identify β P and β R associated with the SATI scores because some applicants submit while others do not.

Distributional Assumptions ε ap, ε en, ε s and ε qi are standard normal and independent.

Likelihood Function Goal is to estimate a vector of structural parameters θ = {, β μ, β e, β k s, β k sat, β int, β P’, β R’, β D’, β YR’ and C}. Derive parametric expressions, as a function of observed data and structural parameters, for  the probabilities an applicant applies early decision and/or submits her SATI score  the probability the college accepts an applicant  the probability an accepted applicant enrolls. Expressed with standard normal and standard multivariate normal distributions.

Data Application data for 2 liberal arts schools in the north east Each with approximately 1800 students enrolled. Both report a typical SAT I score in the upper 1200s/1600 College X: 2 years ≈ 5 years after the optional policy was instituted. College Y the year after the optional policy was instituted.

College Board Data SAT scores for those who elected not to submit them to the college. Student Descriptive Questionnaire (SDQ) SAT II Scores Self Reported income High school GPA High school activities

Descriptive Statistics for College X Number of Applicants Early Decision Applicants more likely accepted Early Decision Applicants more likely to enroll Applicants that apply early decision and don’t submit have lower SATI scores

Descriptive Statistics for College X Applicants who apply early are weaker students Female applicants from private high school less likely to submit

Descriptive Statistics for College X Legacies and high income applicants are more likely to apply early decision

Descriptive Statistics for College Y Similar to College X except: 1. Early Decision applicants are less likely to be accepted (~.65 compared to ~.85 for College X) and enroll conditional on acceptance (~.92 compared to ~.98 for College X) 2. Measure of academic performance are slightly worse for applicants who do not submit their SAT I scores.

Results from reduced form papers (Dickert-Conlin & Chapman; Conlin,Dickert-Conlin & Chapman) 1. Applicants are behaving strategically by choosing not to reveal their SAT I scores if they are below a value one might predict based on their other observable characteristics.

Results from reduced form papers SATI un

Results from reduced form papers (Dickert-Conlin & Chapman; Conlin,Dickert-Conlin & Chapman) 1. Applicants are behaving strategically by choosing not to reveal their SAT I scores if they are below a value one might predict based on their other observable characteristics. 2. College admission departments are behaving strategically by rewarding applicants who do submit their SAT I scores when their scores will raise the college’s average SAT I score reported to U.S. News and World Report and rewarding applicants who do not submit when their SAT I scores will lower the college’s reported score.

Results from reduced form papers Set of regressors include SATII score, ACT score, private high school, female, HS GPA, HS class rank, family income bracket, legacy, early decision indicator, race, geographic location, HS extracurricular activities. Applicants who submit their SAT I score are less likely to be accepted by College X if their SAT I score is below 1388 and are more likely to be accepted if their score is above Average SAT I score at College X is Subsequent measures of college performance are not correlated with scores and submission decisions in a similar manner.

Table 2: Structural Parameter Estimates College X has more years of experience with optional policy but admission director is relatively new. Expect probability of acceptance to increase with expectation of enrollment but difference in coefficient is worrisome. Estimates large as expected based on enrollment probabilities of early decision and early decision enrollment probability higher for College X.

Table 2 Structural Parameter Estimates Parameters of College’s Payoff Function Utility Parameters

Table 2 Structural Parameter Estimates

Table 3: Model Fit College X

Table 3: Model Fit College Y Model fits the other observables well in terms of whose accepted and those who enrolled.

Table 6: Counterfactuals : Student Body Composition for Applicants who don’t Submit

Table 7: Counterfactuals : Student Body Composition Not much difference because of inability of model to fit acceptance and enrollment probabilities.

Table 8: Counterfactuals College X Student Body Performance Measures for Applicants who do not Submit

Better students transfer to Cornell

Table 8: Counterfactuals College Y Student Body Performance Measures for Applicants who do not Submit

Conclusions 1. The empirical results suggest that the colleges underestimates the relationship between an applicant’s decision to submit her SATI score and her actual SATI score. 2. For those applicants who choose not to submit, this affects their acceptance decision and also the demographic characteristics of the expected student body.