1. Analysis and Communication of US News Rankings using Monte Carlo Simulations: A Comparison to Regression Modeling Presented by Chris Maxwell Purdue University AIR 2010

2. Introduction What changes in submitted data most influence our US News rankings? Identify key data elements Provide realistic expectations of future rank This presentation will focus on the US News graduate program in education rankings Results will also be presented for graduate business and national universities rankings

3. Import into Excel and use ordinary linear regression (OLS) to model the US News score: Initial Analysis Started with US News data from website:

4. OLS Problems Variable rejections Multicollinearity Model variability - which model is “right” ? Counterintuitive results

5. OLS Problems (continued) Models can be extremely accurate, but communication of results becomes very problematic Is there another way to model the score using the same data?

6. US News Methodology US news scores are z-score based: (observation - mean)/standard deviation In general, each institution’s z-scores are: multiplied by the US News weight totaled the highest total is scaled to 100 Not all calculation details are known and some data are missing

7. Monte Carlo Simulation Can a US News-type equation be simulated that calculates the US News scores? 18 unknowns, but 50 observations… The equation framework is input into an iterative Excel VBA program Reasonable ranges are defined for the 18 unknown standard deviations and “means”

8. Monte Carlo Simulation (continued) For each iteration (~40,000) in a run: Randomly chose all unknowns Compute score for each institution Rescale so top score is100 Compute sum of squared errors The best-fit equation is saved, algebraically rearranged, and compared to regression Refine the model and repeat the process

9. Model Comparisons

10. Model Comparisons (continued)

14. Summary Conclusions Cautions Questions / Discussion