1. Reputation, Rankings, and Ratings
Kyle Sweitzer
Data Analyst
Office of Planning & Budgets
Presentation for the University Communicators Network
May 14, 2009

2. Numerous Rankings & Ratings
RANKINGS OF ENTIRE INSTITUTION (Ungergraduate)
US News & World Report “America’s Best Colleges”
The Center for Measuring University Performance
Kiplinger’s Best Values in Public Colleges
Forbes “America’s Best Colleges”
World Rankings:
The Times Higher Education Supplement World University Rankings
Shanghai Jiao Tong Academic Ranking of World Universities

3. Numerous Rankings & Ratings
RANKINGS OF SPECIFIC COLLEGES, SCHOOLS, AND PROGRAMS (Graduate)
US News & World Report “America’s Best Graduate Schools”
National Research Council (1982, 1995, 2009)

4. Numerous Rankings & Ratings
AAU Data Exchange shows which rankings are important among AAU institutions US News and the soon-to-come NRC ratings are clearly tops on the list

5. (Ranking of institution as a whole)
Let’s start with the US News Undergraduate Rankings
(Ranking of institution as a whole)

6. Criteria Used in US News Ranking
Peer Assessment (Reputation) 25% Financial Resources 10% Graduation & Retention Rates 20% Exp’s per student 6-year grad rate (80%) Research Exps Frosh retention rate (20%) Public Service Exps Faculty Resources 20% Instructional Exps Avg faculty salary (35%) Acad Support Exps % classes < 20 (30%) Stud Services Exps % fac highest deg (15%) Instit Support Exps % classes ≥ 50 (10%) Op / Maintenance Ex % faculty full-time (5%) Student-faculty ratio (5%) Alumni Giving Rate 5% Admissions Selectivity 15% % alumni who donate SAT/ACT score (50%) Frosh Top 10% (40%) Grad Rate Performance 5% Acceptance rate (10%) Actual vs predicted rate

7. How MSU does in each category
See handout:
“US News Rankings Criteria”
Shows MSU’s performance on each of the rankings criteria over the past five years.
Also shows averages for Big 10 schools

8. MSU’s ranking & reputation over time
Overall Ranking T2 T2 T2 T2 T Peer Reputation Rating

9. Research on reputation
What variables relate to reputation rating? Volkwein, J.F., & Sweitzer, K.V. (2006). Institutional prestige and reputation among research universities and liberal arts colleges. Research in Higher Education, Vol. 47, No. 2, pp

10. US News peer assessment rating
The measure of reputation employed in the study (DV) is the US News peer assessment rating, which is a reputation score that US News reports from the surveys they administer each spring.
US News surveys the president, provost, and admissions director of each four-year institution in the country, asking them to rate the academic reputation of their peer institutions on a scale from 1 to 5
1 = Marginal
2 = Adequate
3 = Good
4 = Strong
5 = Distinguished

11. Reputation Rating for Research Universities
Variables that relate to US News Peer Assessment Rating
Age of Institution
Govern / Control
Total Enrollment .268*** Standardized Beta Coefficients
Expend / Student from Blocked Set-Wise Regression
Student-Fac Ratio -.156***
Avg Prof Salary .268***
Pct Fac Full-time .052* * level of significance
Median SAT * ** level of significance
Faculty Productivity .103*** *** .001 level of significance
Graduation Rate .267***
Alumni Giving Rate .101**
ADJ R-SQUARE .905

12. Comparing US News to Guidebooks
Barron’s Profiles of American Colleges
College Admissions Selector (Noncompetitive to Most Competitive)
Peterson’s Four-Year Colleges
Entrance Difficulty Index (Noncompetitive to Most Difficult)
Princeton Review’s Complete Book of Colleges
Selectivity Rating (60 to 99)
The Fiske Guide to Colleges
Academic Rating (1 to 5)
Correlations with US NEWS Peer Assessment Rating
Barron’s Admissions Selector .76
Peterson’s Entrance Difficulty Index .74
Princeton Review Selectivity Rating .81
Fiske Academic Rating .85

13. Can We Improve Our Reputation
Can We Improve Our Reputation? The Variables that Relate to Change Over Time in US News Peer Ratings
Kyle Sweitzer
Data Analyst
Michigan State University
Fred Volkwein
Professor Emeritus of Education
Penn State University

14. Background
Most studies of reputation have examined which variables relate to prestige for a given year.
Few studies have explored change over time, and those that do look at changes in the overall rank, as opposed to specifically examining change in reputation.
One such study found that the same 47 schools were ranked in the US News Top 50 every year from 1999 to 2006 in the National University category (research universities)

15. Research Question
What variables, if any, relate to changes in US News’ peer assessment ratings for those institutions which have experienced significant changes in the ratings over the nine-year period from ?

16. Population
There are four broad categories in the US News hierarchy, roughly based on Carnegie classifications.
Institutions that remained in the same US News category between 1999 & 2007 were the starting point for inclusion in the study.
Almost 1100 institutions (1095) remained in the same US News category over those nine years.
This ensures that these schools had the same group of peer institutions to rate them (if not the same person, at least the same position at those schools).

17. Schools that were analyzed
Across the 1095 schools remaining in the same US News category all nine years, the mean difference between an institution’s high and low peer reputation rating over the nine years was 0.24
Only those schools that had an above-average difference between their high and low score were included in the analysis (difference of 0.3 or more between their high and low peer rating).
418 schools had a difference of at least 0.3 (412 analyzed)

18. Stability in reputation, 1999-2007
Difference between Number of
low and high score institutions

19. Independent Variables
Total of 22 predictor variables collected, in 1 of 5 categories: SIZE VARIABLES Student Pop, Faculty Pop, Combined Size Variable FINANCE VARIABLES Total Exps, Total Revs, Exp/Stud, Rev/Stud, Tuition SELECTIVITY VARIABLES SAT, Top 25% HS, Accept Rate, Combined Selectivity Var FACULTY VARIABLES Pubs, Pubs/Fac, Salary, % Fac FT, S-F ratio, % classes <20 STUDENT OUTCOMES VARIABLES Frosh Retn Rt, Grad Rt, Avg Fr & Grad Rt, Alum Giving Rt

20. Change in Reputation One immediate finding just from data collection:
The incredibly disproportionate number of schools that changed in reputation based on which US News (Carnegie) category a school is in.
Of the 412 institutions with above-average change in their peer assessment rating over the nine years:
14 “National Universities” (Research Universities)
25 “Liberal Arts Colleges” (Baccalaureate Colleges—Arts & Sciences)
180 “Universities-Masters” (Master’s Colleges & Universities)
193 “Comprehensive Colleges—Bachelors” (Baccalaureate Colleges – Diverse Fields)

21. Results of the Study
The only variable that remained significant in explaining changes over time in reputation ratings is the admissions selectivity variable (Pct of freshmen in Top 25% of HS class), with coefficients of 0.2 to 0.3
Thus, a 10 percentage-point increase in the percent of freshmen that were in the Top 25% of their HS class relates to an increase in the US News peer assessment rating by 0.02 to (10 percentage point decrease means a 0.02 to 0.03 decrease)
Thus, if a school improves from 60% to 70% of incoming students who were in the Top 25% of their HS class, the school would improve, for example, from a 2.4 to a 2.6 in the US News peer assessment rating

22. Overview of Results
Across ALL schools that were rated by US News between 1999 and 2007, the average peer assessment rating has ranged from a low of 2.85 to a high of 2.90 (thus, no change rounded to one decimal)!! This is despite the fact that over 400 schools have seen above-average change (at least 0.3) during the period. (Note: 62% of schools show relative stability over time in reputation) Examining all institutions over nine years, it’s clear that upward movers have balanced downward movers, resulting in a nullification effect in reputation change. Apparently, the data suggests that academic reputation in US News is a zero-sum game. Raters are (either intentionally or unintentionally) only rating a certain number or percentage of schools at a given rating, and if they rate one school higher than the year before, they rate another lower than the previous year.

23. What does it all mean?
Academic reputation changes very little, if at all, especially for research universities and liberal arts colleges. Reputations change slowly, and where reputations do change, admissions selectivity seems to be the single most-important influence. The pool of talented students is limited, and practically every institution is competing for them!!

24. What does it all mean?
If changes in academic reputation boil down simply to changes in the ability of the students coming in the door, how well does the US News peer assessment rating measure quality in higher education? Perhaps the title of the annual US News magazine, rather than “America’s Best Colleges”, would more accurately be called “America’s Most Selective Colleges.”

25. Reputation Among Graduate Programs: Comparing Correlates of U. S
Reputation Among Graduate Programs: Comparing Correlates of U.S. News Graduate Reputation Ratings Between Five Academic Disciplines

26. Why study graduate program reputation ratings?
Prospective graduate students use graduate program ratings to inform their application and admissions decisions.
Administrators use graduate program ratings to inform resource allocation decisions.
(Ehrenberg and Hurst, 1996)

27. Existing studies on graduate rankings/ratings
Most of the studies have examined institutions’ graduate programs as a whole, via aggregating individual program ratings (Volkwein, 1986; Grunig, 1997).
Few studies have examined graduate program ratings at the department or school level.
Even fewer have looked at the U.S. News graduate school ratings (most have examined the NRC ratings).

28. Research Questions
What variables relate to the U.S. News peer assessment ratings of graduate programs in the professional school disciplines of business, education, engineering, law, and medicine?
Are there variables relating to prestige that are common across all of the disciplines in the study, and are there variables that are specific to certain disciplines?
How does the concept of prestige compare across professional school disciplines?

29. Population
Schools/Colleges appearing in the lists of “The Top Schools” in business, education, engineering, law, and medicine in the 2008 edition of U.S. News’ America’s Best Graduate Schools.
50 Schools of Business
52 Schools of Education
51 Schools of Engineering
104 Schools of Law
51 Schools of Medicine

30. Variables / Data Sources
DEPENDENT VARIABLE – Peer assessment survey of deans, faculty, program directors
INDEPENDENT VARIABLES – Data from U.S. News
--standardized admissions tests
--program acceptance rates
--full-time graduate enrollment in the school
--non-resident tuition
--student/faculty ratio
--undergraduate GPA
--variables specific to a discipline

31. Variables / Data Sources
Research activity was measured in terms of faculty publications.
Institute for Scientific Information Web of Science
Science Citation Index and Social Science Citation Index
Search on “Subject Category” for journals specific to a discipline.

32. Regression Models
A separate regression was estimated for each of the disciplines.
Many studies have suggested that two factors play the most significant role in explaining the variance in reputational ratings – size and admissions selectivity.
These two factors were entered into each regression model, along with other variables.

33. Regression Results, Schools of Business
Standardized Betas of Significant Coefficients
Variables Model 1 Model 2 Model 3
Full-time enrollment .624*** .407*** .267*
Non-resident tuition .330*** .228*
Student-faculty ratio ns
Avg GMAT score *** .253**
Pubs per faculty ns
Starting salary of grads ***
Adjusted R-Square
*Significant at .05 level; **Significant at .01 level; ***Significant at .001 level.
ns = non-significant when entered into model

34. Regression Results, Schools of Education
Standardized Betas of Significant Coefficients
Variables Model 1 Model 2 Model 3
Full-time enrollment .366** .354* .535**
Non-resident tuition .514***
Student-faculty ratio ns
Avg GRE score ns
Pubs per faculty *
Adjusted R-Square
*Significant at .05 level; **Significant at .01 level; ***Significant at .001 level.
ns = non-significant when entered into model

35. Regression Results, Schools of Engineering
Standardized Betas of Significant Coefficients
Variables Model 1 Model 2 Model 3
Full-time enrollment .664*** .576*** .792***
Non-resident tuition .327**
Student-faculty ratio ns
Avg quantitative GRE score *** .226*
Pubs per faculty ***
Adjusted R-Square
*Significant at .05 level; ***Significant at .001 level.
ns = non-significant when entered into model

36. Regression Results, Schools of Law
Standardized Betas of Significant Coefficients
Variables Model 1 Model 2 Model 3
Full-time enrollment .213* .159* .163**
Non-resident tuition .508***
Student-faculty ratio – .207*** – .174***
Median LSAT score *** .530***
Pubs per faculty ***
Employment rate at graduation ns
Adjusted R-Square
*Significant at .05 level; **Significant at .01 level; ***Significant at .001 level.
ns = non-significant when entered into model

37. Regression Results, Schools of Medicine
Standardized Betas of Significant Coefficients
Variables Model 1 Model 2 Model 3
Full-time enrollment ns .224* .342***
Non-resident tuition ns
Faculty-student ratio ns .313**
Avg MCAT score *** .637***
Pubs per faculty ***
Adjusted R-Square
*Significant at .05 level; **Significant at .01 level; ***Significant at .001 level.
ns = non-significant when entered into model

38. Summary of Results
The SIZE variable (full-time enrollment) is the only variable that remained significant in the final model for all 5 disciplines.
However, size has the greatest beta coefficient in only 2 of the 5 disciplines (education and engineering).
So for schools of education and engineering, the size of the school matters more to reputation than anything else!

39. Summary of Results
ADMISSIONS SELECTIVITY remains significant in the final model for 4 of the 5 disciplines, and has the greatest beta coefficient for 2 of those 4 – Law schools and Med schools.
So for Law schools and Med schools, the “quality” of the students matters more to the reputation of the school than anything else!
Education is the one discipline for which selectivity does not affect reputation.

40. Summary of Results
FACULTY PRODUCTIVITY (pubs per faculty) also remained significant in 4 of the 5 disciplines, and had the 2nd greatest beta coefficient in all 4.
The 4 disciplines were: engineering, education, law, and medicine.
Not surprising that faculty productivity is significant in explaining graduate reputation.

41. Summary of Results
The business schools may be the most surprising all around --- not only is it the one discipline in which faculty productivity does not influence reputation, but the factor with the greatest influence on reputation is the starting salary of the graduates …..a factor determined by external (market) forces!!

42. Summary of Results
TUITION did not remain significant in the final model for any of the 5 disciplines.
Student-faculty ratio only remained significant in 2 of the 5 disciplines (Law and Med), and was one of the weaker predictors even for them.

43. Summary of Results
So in 4 of the 5 disciplines, either SIZE or ADMISSIONS SELECTIVITY is the biggest determinant of a school’s reputation.
These results confirm prior studies on graduate program reputation that analyzed the 1995 NRC ratings.

44. Summary of Results Variable with the largest beta coefficient:
Business Starting salary of graduates
Education Enrollment size
Engineering Enrollment size
Law Admissions selectivity (LSAT)
Medicine Admissions selectivity (MCAT)

45. Implications
Like with the undergraduate ratings, the question remains as to how well the
U.S. News graduate ratings measure the concept of quality in education.
Is the magazine really determining
“America’s Best Graduate Schools?”

46. US News Graduate Rankings
See two handouts:
Highlights of MSU’s rankings for 2010 edition (rankings that are in the actual magazine – additional rankings are in the online edition)
Summary of Big 10 schools – number of ranked programs over time

47. Other rankings beyond US News
WORLD RANKINGS (entire university ranked)
The Times Higher Education Supplement World University Rankings
(MSU ranked # 203 for 2008 ranking)
Peer review of faculty worldwide (40%)
Employer review (10%)
Citations per faculty (20%)
Faculty / student ratio (20%)
International faculty (5%)
International students (5%)

48. Other rankings beyond US News
WORLD RANKINGS (entire university ranked)
Shanghai Jiao Tong Academic Ranking of World Universities
(MSU ranked # 83 for 2008 ranking)
Faculty winning Nobel & Fields medals (20%)
Highly cited researchers (20%)
Articles published in Nature and Science (20%)
Articles in ISI Science and Soc Sci indices (20%)
Alumni winning Nobel & Fields medals (10%)
Above 5 weighted per FTE academic staff (10%)