1. Eastern Michigan University
Student Success Between Honors and Non-Honors Students: A Comparative Analysis Using Matching Case Method
Bin Ning & Meng Chen
Eastern Michigan University
MIAIR, November 3, 2016

2. Outline
Context
Analysis and Results
Model Evaluation
Conclusion

3. Higher Education Context
Student Success, frequently measured by retention and graduation rates for quantitative research purposes, is undoubtedly one of the most critical subjects in higher education
Many institutions put great effort into improving student retention and graduation rates through a variety of programs
Establishing or expanding honors program has become an important tool to improve overall student success
Does honors program actually improve student success? The question remains unanswered

4. Institutional Context
EMU: ~18,000 UG and 4,000 GR
Sizes of entering FTIAC cohort fluctuated around 2,700
Carnegie: before 2016—master’s large; starting 2016—Doctoral R3
Relatively good 1st-year retention rate 72-77%
Low 6-year graduation rate: 36-40%
Fast expansion of honors programs from 875 (AY2012) to 1,686 (AY2016)
As a primary result from the expansion, EMU expects the improvement of overall student success

5. EMU’s Investment in Honors College
Staffing
Acquisition of properties
Major marketing campaign
Financial Aid

6. About the Study In this study, we want to test the hypothesis:
Whether students in honors program presents higher first-to-second year retention and six-year degree completion rates or not than non-honors students, after controlling other input factors such as high school GPA, ACT score, and other demographic variables

7. Research Questions
Are higher 1st-year retention and 6-year graduation rates for honors students due to participation in honors college or students with better starting qualifications?
Is there a significant difference on aforementioned rates between honors and non-honors students?

8. Dataset
The study included approximately 13,000 records from six FTIAC cohorts between 2005 and 2010
For the purpose of conducting matching case analysis, records with missing values were removed
A total of 9,436 valid individual records were used
Missing value has to be manually removed. The model does not recognize the missing values.

9. Research Design
Experimental Group: 951 FTIAC honors students were enrolled and stayed in honors college for at least 3 semesters
Control Group: 8,485 non-honors FITAIC students

10. Direct Comparison Results
Cohort N
Retained after Year 1
Retention Rate
Honors
951
930
97.8%
Non-Honors
8,485
7,085
83.5%
Cohort N
Graduated in Six Years
Graduation Rate
Honors
951
794
71.8%
Non-Honors
8,485
3,213
37.9%

11. Matching Case Comparison
Propensity score matching
Four matching variables
High School GPA
ACT Composite Score
Ethnicity
First-generation
McNemar’s test
McNemar: paired non-parametric test, similar to chi-square and fisher’s exact test (for small sample size).
Strength: the concept is similar to paired t test.

12. Propensity Score Matching
Propensity score matching is a statistical matching technique that one treatment case is matched with one or more control cases
The matching can strengthen causal-effect arguments in quasi- experimental and observational data by reducing selection bias
In this study, the propensity score matching was performed in R by using package: MatchIt
Both“ exact” and “nearest” match methods have been selected and results have been compared

13. Matching Methods
Exact Matching: matches each treated unit with a control unit that has exactly same value on each covariate
Sub-classification Matching: distributions of the covariates are similar in each subclass
Nearest Neighbor Matching: control units are closest in terms of distance measure
Optimal Matching: minimizing the average absolute distance across all match pairs
Genetic Matching: using computationally intensive genetic search algorithm to match
Coarsened Exact Matching: matches on a covariate while maintaining the balances of other covariates

14. Exact Match
HONORS~HIGHSCHOOL_GPA+ACT+ETHNICITY+FIRST_GENERATION (method="exact")
Sample sizes:
Non-Honors
Honors
All
8,485
951
Matched
362
329
Discarded
8,123
622

15. Nearest Match
HONORS~HIGHSCHOOL_GPA+ACT+ETHNICITY+FIRST_GENERATION (method="nearest")
Honors: non-honors=1:3
Sample sizes:
Non-Honors
Honors
All
8,485
951
Matched
2,853
Unmatched
5,632 __

16. Retention Rates Graduation Rates
Results
Retention Rates Graduation Rates
Honors= 96.96%
Non-Honors= 81.49%
Method= “exact”
Honors= 97.79%
Non-Honors= 78.27%
Method= “nearest”
Honors= 82.07%
Non-Honors= 63.54%
Method= “exact”
Honors= 83.49%
Non-Honors= 54.15%
Method= “nearest”

17. McNemar’s Test McNemar’s test
Statistical test used to determine consistency in responses across two paired variables
Basically a paired version of Chi-square test
McNemar’s test usually considered paired binary response data displayed in a 2X2 contingency table
It can be extended to a 3X3 or higher square tables by expanding the test statistic to include the sum of values obtained from all possible pairs of 2X2 tables

18. McNemar’s Test (Retention $ “Exact” Method)
McNemar's chi-squared = , df = 1, p-value < 2.2e-16
College
Retained
Non-Retained
Honors
319 10
Non-Honors
295 67
McNemar’s test is frequently used in life sciences

19. McNemar’s Test (Graduation $ “Exact” Method)
McNemar's chi-squared = 100, df = 1, p-value < 2.2e-16
College
Graduated
Non-Graduated
Honors
270 59
Non-Honors
230
132

20. McNemar’s Test (Retention $ “Nearest” Method)
McNemar's chi-squared = , df = 1, p-value < 2.2e-16
College
Retained
Non-Retained
Honors
930 21
Non-Honors
2,233
620

21. McNemar’s Test (Graduation $ “Nearest” Method)
McNemar's chi-squared = , df = 1, p-value < 2.2e-16
College
Non-Honors
Honors
794
157
1,545
1,308

22. Model Evaluation Mean Diff. 16.1 52.8 45.0 47.5 Data dependent -90.7
Distance
16.1
HIGHSCHOOL_GPA
52.8
ACT
45.0
American Indian/Alaskan Native
47.5
Asian
-90.7
Black/African American
75.6
Hispanic/Latino
100.0
Native Hawaiian/Other Pacific Islander
Race/Ethnicity Unknown
-7.1
White
79.0
First-generation
44.0
Data dependent
Lowest mean differences between groups
Perfect balance improvement
Percent improvement of mean difference between control and experiment group before and after matching
Data for non-honors after match. Data dependent means match process depends on data distribution, both groups must have overlap in each variable; lowest mean difference should be realized after match; after match, the balance of distribution of two groups are similar

23. Model Evalution Data dependent Lowest mean differences between groups
Propensity score = probability

24. Distribution of High School GPA
Before match
“Nearest” match
“ Exact” match
Non-Honors
Honors
Non-Honors
Honors
Non-Honors
Honors

25. Distribution of ACT Composite Score
Before match
“Nearest” match
“ Exact” match
The difference even after match is due to sample size
Non-Honors
Honors
Non-Honors
Honors
Non-Honors
Honors

26. Conclusion
There is a significant difference on both retention rates and graduation rates between honors and non-honors students
Honors program can improve student retention and graduations rate
Differences are significant and are independent from students’ starting qualifications

27. Future Considerations
Other factors to be considered
Gender
Living arrangements
Pell
Scaled GPA (high school vigor)

28. Thoughts on Matching Case Method
Pros
Quasi-experimental design
Strengthen causal-effect arguments
Allow the control of input variables
Open source software available
Cons
Increase of number of matching variables will drastically decrease the sample size