Reputation, Rankings, and Ratings Kyle Sweitzer Data Analyst Office of Planning & Budgets kvs@msu.edu Presentation for the University Communicators Network May 14, 2009

Numerous Rankings & Ratings RANKINGS OF ENTIRE INSTITUTION (Ungergraduate) US News & World Report “America’s Best Colleges” The Center for Measuring University Performance Kiplinger’s Best Values in Public Colleges Forbes “America’s Best Colleges” World Rankings: The Times Higher Education Supplement World University Rankings Shanghai Jiao Tong Academic Ranking of World Universities

Numerous Rankings & Ratings RANKINGS OF SPECIFIC COLLEGES, SCHOOLS, AND PROGRAMS (Graduate) US News & World Report “America’s Best Graduate Schools” National Research Council (1982, 1995, 2009)

Numerous Rankings & Ratings AAU Data Exchange shows which rankings are important among AAU institutions US News and the soon-to-come NRC ratings are clearly tops on the list

(Ranking of institution as a whole) Let’s start with the US News Undergraduate Rankings (Ranking of institution as a whole)

Criteria Used in US News Ranking Peer Assessment (Reputation) 25% Financial Resources 10% Graduation & Retention Rates 20% Exp’s per student 6-year grad rate (80%) Research Exps Frosh retention rate (20%) Public Service Exps Faculty Resources 20% Instructional Exps Avg faculty salary (35%) Acad Support Exps % classes < 20 (30%) Stud Services Exps % fac highest deg (15%) Instit Support Exps % classes ≥ 50 (10%) Op / Maintenance Ex % faculty full-time (5%) Student-faculty ratio (5%) Alumni Giving Rate 5% Admissions Selectivity 15% % alumni who donate SAT/ACT score (50%) Frosh Top 10% (40%) Grad Rate Performance 5% Acceptance rate (10%) Actual vs predicted rate

How MSU does in each category See handout: “US News Rankings Criteria” Shows MSU’s performance on each of the rankings criteria over the past five years. Also shows averages for Big 10 schools

MSU’s ranking & reputation over time Overall Ranking 99 00 01 02 03 04 05 06 07 08 09 T2 T2 T2 T2 T2 73 71 74 70 71 71 Peer Reputation Rating 3.5 3.5 3.6 3.6 3.6 3.5 3.5 3.5 3.5 3.5 3.4

Research on reputation What variables relate to reputation rating? Volkwein, J.F., & Sweitzer, K.V. (2006). Institutional prestige and reputation among research universities and liberal arts colleges. Research in Higher Education, Vol. 47, No. 2, pp. 129-148.

US News peer assessment rating The measure of reputation employed in the study (DV) is the US News peer assessment rating, which is a reputation score that US News reports from the surveys they administer each spring. US News surveys the president, provost, and admissions director of each four-year institution in the country, asking them to rate the academic reputation of their peer institutions on a scale from 1 to 5 1 = Marginal 2 = Adequate 3 = Good 4 = Strong 5 = Distinguished

Reputation Rating for Research Universities Variables that relate to US News Peer Assessment Rating Age of Institution ------ Govern / Control ------ Total Enrollment .268*** Standardized Beta Coefficients Expend / Student ------ from Blocked Set-Wise Regression Student-Fac Ratio -.156*** Avg Prof Salary .268*** Pct Fac Full-time .052* * .05 level of significance Median SAT .142* ** .01 level of significance Faculty Productivity .103*** *** .001 level of significance Graduation Rate .267*** Alumni Giving Rate .101** ADJ R-SQUARE .905

Comparing US News to Guidebooks Barron’s Profiles of American Colleges College Admissions Selector (Noncompetitive to Most Competitive) Peterson’s Four-Year Colleges Entrance Difficulty Index (Noncompetitive to Most Difficult) Princeton Review’s Complete Book of Colleges Selectivity Rating (60 to 99) The Fiske Guide to Colleges Academic Rating (1 to 5) Correlations with US NEWS Peer Assessment Rating Barron’s Admissions Selector .76 Peterson’s Entrance Difficulty Index .74 Princeton Review Selectivity Rating .81 Fiske Academic Rating .85

Can We Improve Our Reputation Can We Improve Our Reputation? The Variables that Relate to Change Over Time in US News Peer Ratings Kyle Sweitzer Data Analyst Michigan State University Fred Volkwein Professor Emeritus of Education Penn State University

Background Most studies of reputation have examined which variables relate to prestige for a given year. Few studies have explored change over time, and those that do look at changes in the overall rank, as opposed to specifically examining change in reputation. One such study found that the same 47 schools were ranked in the US News Top 50 every year from 1999 to 2006 in the National University category (research universities)

Research Question What variables, if any, relate to changes in US News’ peer assessment ratings for those institutions which have experienced significant changes in the ratings over the nine-year period from 1999-2007?

Population There are four broad categories in the US News hierarchy, roughly based on Carnegie classifications. Institutions that remained in the same US News category between 1999 & 2007 were the starting point for inclusion in the study. Almost 1100 institutions (1095) remained in the same US News category over those nine years. This ensures that these schools had the same group of peer institutions to rate them (if not the same person, at least the same position at those schools).

Schools that were analyzed Across the 1095 schools remaining in the same US News category all nine years, the mean difference between an institution’s high and low peer reputation rating over the nine years was 0.24 Only those schools that had an above-average difference between their high and low score were included in the analysis (difference of 0.3 or more between their high and low peer rating). 418 schools had a difference of at least 0.3 (412 analyzed)

Stability in reputation, 1999-2007 Difference between Number of low and high score institutions 1.0 1 0.9 2 0.8 1 0.7 10 0.6 6 0.5 46 0.4 107 0.3 245 0.2 403 0.1 260 0.0 14

Independent Variables Total of 22 predictor variables collected, in 1 of 5 categories: SIZE VARIABLES Student Pop, Faculty Pop, Combined Size Variable FINANCE VARIABLES Total Exps, Total Revs, Exp/Stud, Rev/Stud, Tuition SELECTIVITY VARIABLES SAT, Top 25% HS, Accept Rate, Combined Selectivity Var FACULTY VARIABLES Pubs, Pubs/Fac, Salary, % Fac FT, S-F ratio, % classes <20 STUDENT OUTCOMES VARIABLES Frosh Retn Rt, Grad Rt, Avg Fr & Grad Rt, Alum Giving Rt

Change in Reputation One immediate finding just from data collection: The incredibly disproportionate number of schools that changed in reputation based on which US News (Carnegie) category a school is in. Of the 412 institutions with above-average change in their peer assessment rating over the nine years: 14 “National Universities” (Research Universities) 25 “Liberal Arts Colleges” (Baccalaureate Colleges—Arts & Sciences) 180 “Universities-Masters” (Master’s Colleges & Universities) 193 “Comprehensive Colleges—Bachelors” (Baccalaureate Colleges – Diverse Fields)

Results of the Study The only variable that remained significant in explaining changes over time in reputation ratings is the admissions selectivity variable (Pct of freshmen in Top 25% of HS class), with coefficients of 0.2 to 0.3 Thus, a 10 percentage-point increase in the percent of freshmen that were in the Top 25% of their HS class relates to an increase in the US News peer assessment rating by 0.02 to 0.03 (10 percentage point decrease means a 0.02 to 0.03 decrease) Thus, if a school improves from 60% to 70% of incoming students who were in the Top 25% of their HS class, the school would improve, for example, from a 2.4 to a 2.6 in the US News peer assessment rating

Overview of Results Across ALL 1300+ schools that were rated by US News between 1999 and 2007, the average peer assessment rating has ranged from a low of 2.85 to a high of 2.90 (thus, no change rounded to one decimal)!! This is despite the fact that over 400 schools have seen above-average change (at least 0.3) during the period. (Note: 62% of schools show relative stability over time in reputation) Examining all institutions over nine years, it’s clear that upward movers have balanced downward movers, resulting in a nullification effect in reputation change. Apparently, the data suggests that academic reputation in US News is a zero-sum game. Raters are (either intentionally or unintentionally) only rating a certain number or percentage of schools at a given rating, and if they rate one school higher than the year before, they rate another lower than the previous year.

What does it all mean? Academic reputation changes very little, if at all, especially for research universities and liberal arts colleges. Reputations change slowly, and where reputations do change, admissions selectivity seems to be the single most-important influence. The pool of talented students is limited, and practically every institution is competing for them!!

What does it all mean? If changes in academic reputation boil down simply to changes in the ability of the students coming in the door, how well does the US News peer assessment rating measure quality in higher education? Perhaps the title of the annual US News magazine, rather than “America’s Best Colleges”, would more accurately be called “America’s Most Selective Colleges.”

Reputation Among Graduate Programs: Comparing Correlates of U. S Reputation Among Graduate Programs: Comparing Correlates of U.S. News Graduate Reputation Ratings Between Five Academic Disciplines

Why study graduate program reputation ratings? Prospective graduate students use graduate program ratings to inform their application and admissions decisions. Administrators use graduate program ratings to inform resource allocation decisions. (Ehrenberg and Hurst, 1996)

Existing studies on graduate rankings/ratings Most of the studies have examined institutions’ graduate programs as a whole, via aggregating individual program ratings (Volkwein, 1986; Grunig, 1997). Few studies have examined graduate program ratings at the department or school level. Even fewer have looked at the U.S. News graduate school ratings (most have examined the NRC ratings).

Research Questions What variables relate to the U.S. News peer assessment ratings of graduate programs in the professional school disciplines of business, education, engineering, law, and medicine? Are there variables relating to prestige that are common across all of the disciplines in the study, and are there variables that are specific to certain disciplines? How does the concept of prestige compare across professional school disciplines?

Population Schools/Colleges appearing in the lists of “The Top Schools” in business, education, engineering, law, and medicine in the 2008 edition of U.S. News’ America’s Best Graduate Schools. 50 Schools of Business 52 Schools of Education 51 Schools of Engineering 104 Schools of Law 51 Schools of Medicine

Variables / Data Sources DEPENDENT VARIABLE – Peer assessment survey of deans, faculty, program directors INDEPENDENT VARIABLES – Data from U.S. News --standardized admissions tests --program acceptance rates --full-time graduate enrollment in the school --non-resident tuition --student/faculty ratio --undergraduate GPA --variables specific to a discipline

Variables / Data Sources Research activity was measured in terms of faculty publications. Institute for Scientific Information Web of Science Science Citation Index and Social Science Citation Index Search on “Subject Category” for journals specific to a discipline.

Regression Models A separate regression was estimated for each of the disciplines. Many studies have suggested that two factors play the most significant role in explaining the variance in reputational ratings – size and admissions selectivity. These two factors were entered into each regression model, along with other variables.

Regression Results, Schools of Business Standardized Betas of Significant Coefficients Variables Model 1 Model 2 Model 3 Full-time enrollment .624*** .407*** .267* Non-resident tuition .330*** .228* Student-faculty ratio ns Avg GMAT score .388*** .253** Pubs per faculty 2001-2005 ns Starting salary of grads .596***   Adjusted R-Square .736 .807 .878 ---------------------------------------------------------------------------------------------------------------- *Significant at .05 level; **Significant at .01 level; ***Significant at .001 level. ns = non-significant when entered into model

Regression Results, Schools of Education Standardized Betas of Significant Coefficients Variables Model 1 Model 2 Model 3 Full-time enrollment .366** .354* .535** Non-resident tuition .514*** Student-faculty ratio ns Avg GRE score ns Pubs per faculty 2001-2005 .421*   Adjusted R-Square .368 .377 .474 ---------------------------------------------------------------------------------------------------------------- *Significant at .05 level; **Significant at .01 level; ***Significant at .001 level. ns = non-significant when entered into model

Regression Results, Schools of Engineering Standardized Betas of Significant Coefficients Variables Model 1 Model 2 Model 3 Full-time enrollment .664*** .576*** .792*** Non-resident tuition .327** Student-faculty ratio ns Avg quantitative GRE score .443*** .226* Pubs per faculty 2001-2005 .468***   Adjusted R-Square .447 .619 .721 ----------------------------------------------------------------------------------------------------------------- *Significant at .05 level; ***Significant at .001 level. ns = non-significant when entered into model

Regression Results, Schools of Law Standardized Betas of Significant Coefficients Variables Model 1 Model 2 Model 3 Full-time enrollment .213* .159* .163** Non-resident tuition .508*** Student-faculty ratio – .207*** – .174*** Median LSAT score .712*** .530*** Pubs per faculty 2001-2005 .264*** Employment rate at graduation ns   Adjusted R-Square .397 .795 .849 ----------------------------------------------------------------------------------------------------------------- *Significant at .05 level; **Significant at .01 level; ***Significant at .001 level. ns = non-significant when entered into model

Regression Results, Schools of Medicine Standardized Betas of Significant Coefficients Variables Model 1 Model 2 Model 3 Full-time enrollment ns .224* .342*** Non-resident tuition ns Faculty-student ratio ns .313** Avg MCAT score .701*** .637*** Pubs per faculty 2001-2005 .374***   Adjusted R-Square .016 .540 .653 ---------------------------------------------------------------------------------------------------------------- *Significant at .05 level; **Significant at .01 level; ***Significant at .001 level. ns = non-significant when entered into model

Summary of Results The SIZE variable (full-time enrollment) is the only variable that remained significant in the final model for all 5 disciplines. However, size has the greatest beta coefficient in only 2 of the 5 disciplines (education and engineering). So for schools of education and engineering, the size of the school matters more to reputation than anything else!

Summary of Results ADMISSIONS SELECTIVITY remains significant in the final model for 4 of the 5 disciplines, and has the greatest beta coefficient for 2 of those 4 – Law schools and Med schools. So for Law schools and Med schools, the “quality” of the students matters more to the reputation of the school than anything else! Education is the one discipline for which selectivity does not affect reputation.

Summary of Results FACULTY PRODUCTIVITY (pubs per faculty) also remained significant in 4 of the 5 disciplines, and had the 2nd greatest beta coefficient in all 4. The 4 disciplines were: engineering, education, law, and medicine. Not surprising that faculty productivity is significant in explaining graduate reputation.

Summary of Results The business schools may be the most surprising all around --- not only is it the one discipline in which faculty productivity does not influence reputation, but the factor with the greatest influence on reputation is the starting salary of the graduates …..a factor determined by external (market) forces!!

Summary of Results TUITION did not remain significant in the final model for any of the 5 disciplines. Student-faculty ratio only remained significant in 2 of the 5 disciplines (Law and Med), and was one of the weaker predictors even for them.

Summary of Results So in 4 of the 5 disciplines, either SIZE or ADMISSIONS SELECTIVITY is the biggest determinant of a school’s reputation. These results confirm prior studies on graduate program reputation that analyzed the 1995 NRC ratings.

Summary of Results Variable with the largest beta coefficient: Business Starting salary of graduates Education Enrollment size Engineering Enrollment size Law Admissions selectivity (LSAT) Medicine Admissions selectivity (MCAT)

Implications Like with the undergraduate ratings, the question remains as to how well the U.S. News graduate ratings measure the concept of quality in education. Is the magazine really determining “America’s Best Graduate Schools?”

US News Graduate Rankings See two handouts: Highlights of MSU’s rankings for 2010 edition (rankings that are in the actual magazine – additional rankings are in the online edition) Summary of Big 10 schools – number of ranked programs over time

Other rankings beyond US News WORLD RANKINGS (entire university ranked) The Times Higher Education Supplement World University Rankings (MSU ranked # 203 for 2008 ranking) Peer review of faculty worldwide (40%) Employer review (10%) Citations per faculty (20%) Faculty / student ratio (20%) International faculty (5%) International students (5%)

Other rankings beyond US News WORLD RANKINGS (entire university ranked) Shanghai Jiao Tong Academic Ranking of World Universities (MSU ranked # 83 for 2008 ranking) Faculty winning Nobel & Fields medals (20%) Highly cited researchers (20%) Articles published in Nature and Science (20%) Articles in ISI Science and Soc Sci indices (20%) Alumni winning Nobel & Fields medals (10%) Above 5 weighted per FTE academic staff (10%)