1. Examining the Enrollment and Persistence of Students with Discrepant High School Grades and Standardized Test Scores Anne Edmunds, Ed.D. Higher Education Administrator Edgar Sanchez, Ph.D. Research Associate ACT, Inc.

2. 2

3. 3 High GPA + high test scores = more likely to enroll/persist 3 Low GPA + low test scores = less likely to enroll/persist High GPA + low test scores = ? Low GPA + high test scores = ? THE PROBLEM

4. 4 Session Overview Single Institution Case Study National Replication Discussion and Questions 4

5. Single Institution Case Study Research Questions Literature Review Methodology Data Analysis Findings and Implications

6. 6 Research Question 2 Do significant differences exist in the rate of four-year graduation for students with discrepant high school grades and standardized test scores compared to students with non-discrepant high school grades and standardized test scores? Research Question 1 Do significant differences exist in the rate of persistence to the sophomore year for students with discrepant high school grades and standardized test scores compared to students with non-discrepant high school grades and standardized test scores?

7. 7 Research Question 3 What influence do high school grades and standardized test scores have on the likelihood of persistence to the sophomore year for students with discrepant high school grades and standardized test scores as compared to students with non-discrepant high school grades and standardized test scores? Research Question 4 What influence do high school grades and standardized test scores have on the likelihood of four-year graduation for students with discrepant high school grades and standardized test scores as compared to students with non-discrepant high school grades and standardized test scores?

8. 8 Precollege characteristics used to make admissions decisions Can’t predict what one student will do, but rather looking for patterns with policy implications to make decisions on individual applicants LITERATURE REVIEW High school grades and test scores most common criteria College success often defined as FY GPA, Cum GPA, persistence, graduation

9. 9 Conceptual Framework Astin’s I-E-O Model: Inputs – Environment – Outcomes o Relationship A: Input – Environment o Relationship B: Environment – Outcome o Relationship C: Input – Outcome

10. 10 Conceptual Framework 40+ years of CIRP data o More than half of variance in retention rates and more than two- thirds of variance in degree completion rates attributed to inputs rather than environment o Differences in retention and graduation rates more likely due to student characteristics than institutional effects o Selective institutions control inputs and thus have more control on quality of outcomes Purpose of this study was to examine relationship between two inputs (high school grades and test scores) and two outcomes (persistence and four-year graduation)

11. 11 METHODOLOGY Data Set: Belmont University freshman cohorts 2003-08 Input Variables Considered High School GPA: 4.0 scale, cumulative, unweighted and recalculated by admissions counselors Test Scores: converted to Best ACT o If only ACT scores, highest composite score = Best ACT o If only SAT scores, highest composite SAT (critical reading and math) converted to ACT = Best ACT o If ACT and SAT scores, highest composite SAT converted to ACT, best of original ACT and converted ACT = Best ACT o Other variables considered but not discussed here Outcome Variables Considered Persistence to sophomore year: Persist and Not Persist Four year graduation: Graduate and Not Graduate

12. 12 HS GPATest Score Percentiles CohortnM25th75thM25th75th Fall 2003 5863.443.173.8025.082328 Fall 2004 7083.513.243.8325.322328 Fall 2005 7703.503.243.8325.592328 Fall 2006 7503.553.313.8525.792328 Fall 2007 7883.503.213.8025.672328 Fall 2008 9123.523.253.8325.862428 Total4,5143.513.243.8325.582328 Descriptive Statistics of HS GPA and Test Score by Cohort

13. 13 CohortnPersistence RateGraduation Rate Fall 2003 58678.5%51.5% Fall 2004 70880.1%51.3% Fall 2005 77078.4%53.2% Fall 2006 75079.5%na Fall 2007 78880.3%na Fall 2008 91282.1%na Total 4,51480.0%52.1% Persistence and Four-Year Graduation Rates by Cohort

14. 14 Data Coding Two different methods used to determine discrepant students o Standard Deviation Groups: Based upon relationship between GPA and Test Score o Bottom 25 th Percentile Groups : Based upon relationship between input variables and the respective data distribution for cohort (not discussed here)

15. 15 Standard Deviation Groups: Based upon relationship between GPA and Test Score Consistent Group – High School GPA and Test Score within one standard deviation of each other High School Grades Higher – High School GPA was one standard deviation or more than Test Score Test Score Higher – Test Score was one standard deviation or more than High School GPA

16. 16 CohortnConsistentHigh School Grades HigherTest Score Higher Fall 2003 586 391 97 98 Fall 2004 708 455121132 Fall 2005 770 486141143 Fall 2006 750 490132128 Fall 2007 788 497154137 Fall 2008 912 596161155 Total4,5142,915806793 Categorization of Cohorts by Standard Deviation Group

17. 17 Persistence and Four-Year Graduation Rates by Discrepant Group Discrepant GroupnPersistence Rate %nGraduation Rate % Consistent2,91580.51,33253.1 High School Grades Higher 80680.8 35956.0 Test Score Higher 79377.2 37344.8

18. 18 DATA ANALYSIS Research questions 1 and 2 Binomial tests of proportion used to test for significant differences in rates of persistence Research questions 3 and 4 Logistic regression used to consider influence of inputs on likelihood of persistence

19. 19 Research Question 1 Do significant differences exist in persistence rates for discrepant students compared to non-discrepant students? Test Score Higher persisted at rate lower than, and significantly different from Consistent Difference in persistence rate of High School Grades Higher and Consistent not statistically significant

20. 20 Research Question 2 Do significant differences exist in 4 year graduation rates for discrepant students compared to non-discrepant students? Test Score Higher graduated at rate lower than, and significantly different from Consistent Difference in graduation rate of High School Grades Higher and Consistent not statistically significant

21. 21 Research Question 3 What influence do High School GPA and Test Scores have on likelihood of persistence for discrepant students compared to non-discrepant students? High school grades were found to significantly predict persistence for both discrepant and non-discrepant students Test scores found to predict persistence for neither the discrepant nor non-discrepant students

22. 22 Research Question 4 What influence do High School GPA and Test Scores have on likelihood of four-year graduation for discrepant students compared to non-discrepant students? High school grades predicted graduation for non-discrepant and discrepant students, with a very significant influence on the likelihood of graduation for Test Score Higher students Test scores found to be significant for non-discrepant students, but not as strong of a predictor of graduation as high school grades

23. 23 Key Findings Discrepant students are less likely to persist than non- discrepant students Consistent with the research, high school grades are a stronger predictor of persistence than test scores

24. 24 Implications High school grades stronger predictor of persistence and graduation than test scores, suggesting giving more weight to high school grades in the admission process Difference in persistence and graduation rates of discrepant students suggests consideration in the admission process Lower persistence and graduation rates should be taken in consideration with other factors o Dilemma of admitting high test scores to help rankings initially at the risk of hurting rankings the following year Beyond admission: focus retention efforts on discrepant students for possible intervention

25. 25 Discussion Environment not considered in this study Consistent did not mean both high – could both be low Opportunity to repeat study at other institutions Look at range of sub-scores and whether they are discrepant from each other Need further predictive validity research, specifically institutional validity studies, as recommended by the NACAC Testing Commission

26. National Discrepant Achievement Study Study Goals What do we want to learn? Characteristics of students with discrepant achievement Enrollment and Persistence Hierarchical Multinomial Analysis Findings Discussion

27. 27 Examine the differences between students who have discrepant achievement and those who don’t. Explore the potential impact of not taking these differences into account. Goals

28. 28 2012 ACT-tested sample: –1,085,771 students –9,776 high schools –2,579 postsecondary institutions Enrollment and persistence data were provided by the National Student Clearinghouse. Student data set

29. 29 ACT-tested graduating class of 2012 sample: Discrepant HSGPA & ACTC Scores Consistent HSGPA & ACTC Scores

30. 30 Used Standardized ACTC and HSGPA to create three distinct groups: –ACTC discrepant group: ACTC higher than their HSGPA –HSGPA discrepant group: HSGPA higher than their ACTC –Consistent group: Similar ACTC and HSGPA What does it mean to have discrepant HSGPA and ACT Composite scores?

31. 31 ACT Composite by high school GPA discrepancy groups 0.00.20.40.60.81.01.21.41.61.82.02.22.42.62.83.03.23.43.63.84.0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 Consistent Group HSGPA Discrepant Group ACT Discrepant Group ACT Discrepant Group

32. 32 What does this look like for the 2012 ACT- tested sample? – ACTC discrepant group: 9% (~82K) –HSGPA discrepant group: 5% (~51K) –Consistent group: 85% (~821K)

33. 33 1.Do student characteristics and enrollment and persistence rates differ for students with and without discrepant HSGPA or ACTC scores? 2.How does the likelihood of enrollment or persistence vary depending upon student subgroup membership (i.e. gender, race/ethnicity, income, & type of discrepancy)? What do we want to learn?

34. 34 ACT Discrepant HSGPA Discrepant Consistent Race / Ethnicity African American82513 White633161 Hispanic113213 What are the characteristics of students with discrepant scores? Ethnicities that were comparable across discrepancy groups are not displayed

35. 35 ACT Discrepant HSGPA Discrepant Consistent Gender Female406457 Male603643 Family Income < $36K225128 $36K-$100K474149 > $100K31823 What are the characteristics of students with discrepant scores?

36. 36 ACT Discrepant HSGPA Discrepant Consistent Expected Educational Attainment Bachelor’s Degree434948 Beyond Bachelor’s Degree443442 What are the characteristics of students with discrepant scores? Education levels that were comparable across discrepancy groups are not displayed

37. 37 What are the characteristics of students with discrepant scores? ACT Discrepant HSGPA Discrepant Consistent Math Course Taking Pattern Less than Alg I, Geom, & Alg II11 7 Alg I, Geom, & Alg II313932 Beyond Alg II564558 Over 96% of each discrepancy group had taken English 9, 10, & 11. Mathematics coursework patterns that were comparable across discrepancy groups are not displayed

38. 38 What are the characteristics of students with discrepant scores? ACT Discrepant HSGPA Discrepant Consistent Natural Science Course Taking Pattern BIO121612 BIO, CHEM424349 BIO, CHEM, PHYS393334 Science coursework patterns that were comparable across discrepancy groups are not displayed

39. 39 What are the mean achievement levels of students with discrepant scores? ACTC Discrepant HSGPA Discrepant Consistent HSGPA2.73.63.3 ACT Composite251622

40. 40 What are the enrollment and persistence rates of students with discrepant scores? ACT Discrepant HSGPA Discrepant Consistent Enrollment Rate Total696275 Enrolled Institution Type Two-Year233723 Four-Year776377 Enrolled Institution Affiliation Private251221 Public758879 Persistence Rate Total736573

41. 41 Variables included in the Enrollment Models: –Gender –Minority membership –Income –Expected educational attainment –Math & Science course taking pattern –HSGPA –ACT Composite (ACTC) –ACTC school mean –HSGPA school mean Hierarchical multinomial model of enrollment

42. 42 Significant predictors of enrollment ACTCdHSGPAdConsistent All Combined Gender Minority Status IncomeNot Sig Degree Expectation Math Coursework Science Coursework ACT Composite HSGPANot Sig ACTC * HSGPA Interaction Minority * HSGPA Interaction Income * HSGPA InteractionNot Sig Gender * ACTC Interaction Income * ACTC InteractionNot Sig School HSGPA Mean School ACTC Mean All Significant All Significant

43. 43 Probability of enrollment for students with a 2.9 HSGPA Two-Year Four-Year Not Enrolled

44. 44 Probability of enrollment for students with a 3.3 HSGPA Two-Year Four-Year Not Enrolled

45. 45 Probability of enrollment for students with a 3.7 HSGPA Two-Year Four-Year Not Enrolled

46. 46 Probability of enrollment across student subgroups ACTCdHSGPAdConsistent Gender Male0.770.570.78 Female0.800.620.81 Race/Ethnicity White0.780.570.80 Minority0.800.630.81 Family Income Low-Income0.740.580.76 Mid-Income0.780.630.81 High-Income0.820.670.83 Degree Aspiration Associate’s Degree/Voc-Tech0.540.380.57 Bachelor’s Degree0.800.620.80 Beyond Bachelor’s Degree0.800.650.82 3 3 5 5 3 3 2 2 6 6 1 1 4 4 5 5 5 5 26 24 23 8 8 9 9 7 7 26 27 25

47. 47 Probability of enrollment across student subgroups ACTCdHSGPAdConsistent Math Coursework Taken Less than Algebra I, Geometry, & Algebra II 0.710.500.72 Algebra I, Geometry, & Algebra II 0.760.600.78 Beyond Algebra II0.810.640.82 Science Coursework Taken Biology 0.730.550.75 Biology & Chemistry 0.780.620.80 Biology, Chemistry, & Physics 0.800.620.81 5 5 10 6 6 5 5 7 7 5 5 14 10 7 7 6 6 7 7

48. 48 Variables included in the models: –Gender –Minority membership –Income –Expected Educational attainment –Math & Science Coursework –HSGPA –ACT Composite (ACTC) –Enrolled Institution Selectivity –ACTC School Mean –HSGPA School Mean Hierarchical multinomial model of persistence

49. 49 Significant predictors of persistence to the sophomore year ACTCdHSGPAdConsistentTotal Gender Minority StatusNot Sig SelectivityNot Sig IncomeNot Sig Degree AspirationNot Sig Science CourseworkNot Sig ACT CompositeNot Sig HSGPANot Sig Selectivity * Income InteractionNot Sig Selectivity * Science Coursework InteractionNot Sig Selectivity * ACTC InteractionNot Sig Selectivity * HSGPA InteractionNot Sig* Minority Status * HSGPA InteractionNot Sig Gender * ACTC InteractionNot Sig School HSGPA Mean School ACTC Mean Not Sig All Significant

50. 50 Probability of persistence for students with a 3.4 HSGPA Persist Transfer Drop-out

51. 51 Probability of persistence by student characteristic ACTCdHSPGAdConsistent Gender Male 0.740.680.75 Female 0.770.730.77 Race/Ethnicity Minority 0.750.690.75 White 0.760.730.77 Family income Less than $30,000 0.730.710.74 $30,000 to $100,000 0.750.730.76 More than $100,000 0.770.710.77 2 2 5 5 3 3 2 2 4 4 1 1 2 2 2 2 2 2 4 4 0 0 2 2

52. 52 Probability of persistence by student characteristic ACTCdHSPGAdConsistent Degree Expectation Associates Degree/Voc-Tech 0.750.700.74 Bachelor’s Degree 0.760.720.76 Beyond Bachelor’s Degree 0.750.720.76 Science Coursework Biology 0.740.690.74 Biology, Chemistry 0.760.720.76 Biology, Chemistry, Physics 0.760.720.77 2 2 2 2 1 1 0 0 2 2 2 2 2 2 3 3 2 2 2 2 3 3 3 3

53. 53 Differences exist between students who have a higher ACTC score than HSGPA, higher HSGPA than ACTC score, and Consistent achievement. –Possible explanations include course taking pattern. Recall that Students in the ACT discrepant group are more likely than students in the HSGPA discrepant group to take more advanced Math and Science courses. –Another possible explanation, not explored here, are non- cognitive factors such as student motivation and self-efficacy. The current study further demonstrates that predictors of enrollment and persistence differ between groups and for models that don’t differentiate discrepant achievement. Remember, these are average effects across the country. Key Points

54. 54 Thank You! Anne Edmunds, Ed.D. Higher Education Administrator Anne.Edmunds@gmail.com 615-429-6933 Edgar I Sanchez, Ph.D. Research Scientist Edgar.Sanchez@act.org 319-341-2385