Presentation on theme: "Reputation, Rankings, and Ratings"— Presentation transcript:
1 Reputation, Rankings, and Ratings Kyle SweitzerData AnalystOffice of Planning & BudgetsPresentation for the University Communicators NetworkMay 14, 2009
2 Numerous Rankings & Ratings RANKINGS OF ENTIRE INSTITUTION (Ungergraduate)US News & World Report “America’s Best Colleges”The Center for Measuring University PerformanceKiplinger’s Best Values in Public CollegesForbes “America’s Best Colleges”World Rankings:The Times Higher Education Supplement World University RankingsShanghai Jiao Tong Academic Ranking of World Universities
3 Numerous Rankings & Ratings RANKINGS OF SPECIFIC COLLEGES, SCHOOLS, AND PROGRAMS (Graduate)US News & World Report “America’s Best Graduate Schools”National Research Council (1982, 1995, 2009)
4 Numerous Rankings & Ratings AAU Data Exchange shows which rankings are important among AAU institutions US News and the soon-to-come NRC ratings are clearly tops on the list
5 (Ranking of institution as a whole) Let’s start with the US News Undergraduate Rankings(Ranking of institution as a whole)
6 Criteria Used in US News Ranking Peer Assessment (Reputation) 25% Financial Resources 10% Graduation & Retention Rates 20% Exp’s per student 6-year grad rate (80%) Research Exps Frosh retention rate (20%) Public Service Exps Faculty Resources 20% Instructional Exps Avg faculty salary (35%) Acad Support Exps % classes < 20 (30%) Stud Services Exps % fac highest deg (15%) Instit Support Exps % classes ≥ 50 (10%) Op / Maintenance Ex % faculty full-time (5%) Student-faculty ratio (5%) Alumni Giving Rate 5% Admissions Selectivity 15% % alumni who donate SAT/ACT score (50%) Frosh Top 10% (40%) Grad Rate Performance 5% Acceptance rate (10%) Actual vs predicted rate
7 How MSU does in each category See handout:“US News Rankings Criteria”Shows MSU’s performance on each of the rankings criteria over the past five years.Also shows averages for Big 10 schools
8 MSU’s ranking & reputation over time Overall Ranking T2 T2 T2 T2 T Peer Reputation Rating
9 Research on reputation What variables relate to reputation rating? Volkwein, J.F., & Sweitzer, K.V. (2006). Institutional prestige and reputation among research universities and liberal arts colleges. Research in Higher Education, Vol. 47, No. 2, pp
10 US News peer assessment rating The measure of reputation employed in the study (DV) is the US News peer assessment rating, which is a reputation score that US News reports from the surveys they administer each spring.US News surveys the president, provost, and admissions director of each four-year institution in the country, asking them to rate the academic reputation of their peer institutions on a scale from 1 to 51 = Marginal2 = Adequate3 = Good4 = Strong5 = Distinguished
11 Reputation Rating for Research Universities Variables that relate to US News Peer Assessment RatingAge of InstitutionGovern / ControlTotal Enrollment .268*** Standardized Beta CoefficientsExpend / Student from Blocked Set-Wise RegressionStudent-Fac Ratio -.156***Avg Prof Salary .268***Pct Fac Full-time .052* * level of significanceMedian SAT * ** level of significanceFaculty Productivity .103*** *** .001 level of significanceGraduation Rate .267***Alumni Giving Rate .101**ADJ R-SQUARE .905
12 Comparing US News to Guidebooks Barron’s Profiles of American CollegesCollege Admissions Selector (Noncompetitive to Most Competitive)Peterson’s Four-Year CollegesEntrance Difficulty Index (Noncompetitive to Most Difficult)Princeton Review’s Complete Book of CollegesSelectivity Rating (60 to 99)The Fiske Guide to CollegesAcademic Rating (1 to 5)Correlations with US NEWS Peer Assessment RatingBarron’s Admissions Selector .76Peterson’s Entrance Difficulty Index .74Princeton Review Selectivity Rating .81Fiske Academic Rating .85
13 Can We Improve Our Reputation Can We Improve Our Reputation? The Variables that Relate to Change Over Time in US News Peer RatingsKyle SweitzerData AnalystMichigan State UniversityFred VolkweinProfessor Emeritus of EducationPenn State University
14 BackgroundMost studies of reputation have examined which variables relate to prestige for a given year.Few studies have explored change over time, and those that do look at changes in the overall rank, as opposed to specifically examining change in reputation.One such study found that the same 47 schools were ranked in the US News Top 50 every year from 1999 to 2006 in the National University category (research universities)
15 Research QuestionWhat variables, if any, relate to changes in US News’ peer assessment ratings for those institutions which have experienced significant changes in the ratings over the nine-year period from ?
16 PopulationThere are four broad categories in the US News hierarchy, roughly based on Carnegie classifications.Institutions that remained in the same US News category between 1999 & 2007 were the starting point for inclusion in the study.Almost 1100 institutions (1095) remained in the same US News category over those nine years.This ensures that these schools had the same group of peer institutions to rate them (if not the same person, at least the same position at those schools).
17 Schools that were analyzed Across the 1095 schools remaining in the same US News category all nine years, the mean difference between an institution’s high and low peer reputation rating over the nine years was 0.24Only those schools that had an above-average difference between their high and low score were included in the analysis (difference of 0.3 or more between their high and low peer rating).418 schools had a difference of at least 0.3 (412 analyzed)
18 Stability in reputation, 1999-2007 Difference between Number oflow and high score institutions
19 Independent Variables Total of 22 predictor variables collected, in 1 of 5 categories: SIZE VARIABLES Student Pop, Faculty Pop, Combined Size Variable FINANCE VARIABLES Total Exps, Total Revs, Exp/Stud, Rev/Stud, Tuition SELECTIVITY VARIABLES SAT, Top 25% HS, Accept Rate, Combined Selectivity Var FACULTY VARIABLES Pubs, Pubs/Fac, Salary, % Fac FT, S-F ratio, % classes <20 STUDENT OUTCOMES VARIABLES Frosh Retn Rt, Grad Rt, Avg Fr & Grad Rt, Alum Giving Rt
20 Change in Reputation One immediate finding just from data collection: The incredibly disproportionate number of schools that changed in reputation based on which US News (Carnegie) category a school is in.Of the 412 institutions with above-average change in their peer assessment rating over the nine years:14 “National Universities” (Research Universities)25 “Liberal Arts Colleges” (Baccalaureate Colleges—Arts & Sciences)180 “Universities-Masters” (Master’s Colleges & Universities)193 “Comprehensive Colleges—Bachelors” (Baccalaureate Colleges – Diverse Fields)
21 Results of the StudyThe only variable that remained significant in explaining changes over time in reputation ratings is the admissions selectivity variable (Pct of freshmen in Top 25% of HS class), with coefficients of 0.2 to 0.3Thus, a 10 percentage-point increase in the percent of freshmen that were in the Top 25% of their HS class relates to an increase in the US News peer assessment rating by 0.02 to (10 percentage point decrease means a 0.02 to 0.03 decrease)Thus, if a school improves from 60% to 70% of incoming students who were in the Top 25% of their HS class, the school would improve, for example, from a 2.4 to a 2.6 in the US News peer assessment rating
22 Overview of ResultsAcross ALL schools that were rated by US News between 1999 and 2007, the average peer assessment rating has ranged from a low of 2.85 to a high of 2.90 (thus, no change rounded to one decimal)!! This is despite the fact that over 400 schools have seen above-average change (at least 0.3) during the period. (Note: 62% of schools show relative stability over time in reputation) Examining all institutions over nine years, it’s clear that upward movers have balanced downward movers, resulting in a nullification effect in reputation change. Apparently, the data suggests that academic reputation in US News is a zero-sum game. Raters are (either intentionally or unintentionally) only rating a certain number or percentage of schools at a given rating, and if they rate one school higher than the year before, they rate another lower than the previous year.
23 What does it all mean?Academic reputation changes very little, if at all, especially for research universities and liberal arts colleges. Reputations change slowly, and where reputations do change, admissions selectivity seems to be the single most-important influence. The pool of talented students is limited, and practically every institution is competing for them!!
24 What does it all mean?If changes in academic reputation boil down simply to changes in the ability of the students coming in the door, how well does the US News peer assessment rating measure quality in higher education? Perhaps the title of the annual US News magazine, rather than “America’s Best Colleges”, would more accurately be called “America’s Most Selective Colleges.”
25 Reputation Among Graduate Programs: Comparing Correlates of U. S Reputation Among Graduate Programs: Comparing Correlates of U.S. News Graduate Reputation Ratings Between Five Academic Disciplines
26 Why study graduate program reputation ratings? Prospective graduate students use graduate program ratings to inform their application and admissions decisions.Administrators use graduate program ratings to inform resource allocation decisions.(Ehrenberg and Hurst, 1996)
27 Existing studies on graduate rankings/ratings Most of the studies have examined institutions’ graduate programs as a whole, via aggregating individual program ratings (Volkwein, 1986; Grunig, 1997).Few studies have examined graduate program ratings at the department or school level.Even fewer have looked at the U.S. News graduate school ratings (most have examined the NRC ratings).
28 Research QuestionsWhat variables relate to the U.S. News peer assessment ratings of graduate programs in the professional school disciplines of business, education, engineering, law, and medicine?Are there variables relating to prestige that are common across all of the disciplines in the study, and are there variables that are specific to certain disciplines?How does the concept of prestige compare across professional school disciplines?
29 PopulationSchools/Colleges appearing in the lists of “The Top Schools” in business, education, engineering, law, and medicine in the 2008 edition of U.S. News’ America’s Best Graduate Schools.50 Schools of Business52 Schools of Education51 Schools of Engineering104 Schools of Law51 Schools of Medicine
30 Variables / Data Sources DEPENDENT VARIABLE – Peer assessment survey of deans, faculty, program directorsINDEPENDENT VARIABLES – Data from U.S. News--standardized admissions tests--program acceptance rates--full-time graduate enrollment in the school--non-resident tuition--student/faculty ratio--undergraduate GPA--variables specific to a discipline
31 Variables / Data Sources Research activity was measured in terms of faculty publications.Institute for Scientific Information Web of ScienceScience Citation Index and Social Science Citation IndexSearch on “Subject Category” for journals specific to a discipline.
32 Regression ModelsA separate regression was estimated for each of the disciplines.Many studies have suggested that two factors play the most significant role in explaining the variance in reputational ratings – size and admissions selectivity.These two factors were entered into each regression model, along with other variables.
33 Regression Results, Schools of Business Standardized Betas of Significant CoefficientsVariables Model 1 Model 2 Model 3Full-time enrollment .624*** .407*** .267*Non-resident tuition .330*** .228*Student-faculty ratio nsAvg GMAT score *** .253**Pubs per faculty nsStarting salary of grads ***Adjusted R-Square*Significant at .05 level; **Significant at .01 level; ***Significant at .001 level.ns = non-significant when entered into model
34 Regression Results, Schools of Education Standardized Betas of Significant CoefficientsVariables Model 1 Model 2 Model 3Full-time enrollment .366** .354* .535**Non-resident tuition .514***Student-faculty ratio nsAvg GRE score nsPubs per faculty *Adjusted R-Square*Significant at .05 level; **Significant at .01 level; ***Significant at .001 level.ns = non-significant when entered into model
35 Regression Results, Schools of Engineering Standardized Betas of Significant CoefficientsVariables Model 1 Model 2 Model 3Full-time enrollment .664*** .576*** .792***Non-resident tuition .327**Student-faculty ratio nsAvg quantitative GRE score *** .226*Pubs per faculty ***Adjusted R-Square*Significant at .05 level; ***Significant at .001 level.ns = non-significant when entered into model
36 Regression Results, Schools of Law Standardized Betas of Significant CoefficientsVariables Model 1 Model 2 Model 3Full-time enrollment .213* .159* .163**Non-resident tuition .508***Student-faculty ratio – .207*** – .174***Median LSAT score *** .530***Pubs per faculty ***Employment rate at graduation nsAdjusted R-Square*Significant at .05 level; **Significant at .01 level; ***Significant at .001 level.ns = non-significant when entered into model
37 Regression Results, Schools of Medicine Standardized Betas of Significant CoefficientsVariables Model 1 Model 2 Model 3Full-time enrollment ns .224* .342***Non-resident tuition nsFaculty-student ratio ns .313**Avg MCAT score *** .637***Pubs per faculty ***Adjusted R-Square*Significant at .05 level; **Significant at .01 level; ***Significant at .001 level.ns = non-significant when entered into model
38 Summary of ResultsThe SIZE variable (full-time enrollment) is the only variable that remained significant in the final model for all 5 disciplines.However, size has the greatest beta coefficient in only 2 of the 5 disciplines (education and engineering).So for schools of education and engineering, the size of the school matters more to reputation than anything else!
39 Summary of ResultsADMISSIONS SELECTIVITY remains significant in the final model for 4 of the 5 disciplines, and has the greatest beta coefficient for 2 of those 4 – Law schools and Med schools.So for Law schools and Med schools, the “quality” of the students matters more to the reputation of the school than anything else!Education is the one discipline for which selectivity does not affect reputation.
40 Summary of ResultsFACULTY PRODUCTIVITY (pubs per faculty) also remained significant in 4 of the 5 disciplines, and had the 2nd greatest beta coefficient in all 4.The 4 disciplines were: engineering, education, law, and medicine.Not surprising that faculty productivity is significant in explaining graduate reputation.
41 Summary of ResultsThe business schools may be the most surprising all around --- not only is it the one discipline in which faculty productivity does not influence reputation, but the factor with the greatest influence on reputation is the starting salary of the graduates …..a factor determined by external (market) forces!!
42 Summary of ResultsTUITION did not remain significant in the final model for any of the 5 disciplines.Student-faculty ratio only remained significant in 2 of the 5 disciplines (Law and Med), and was one of the weaker predictors even for them.
43 Summary of ResultsSo in 4 of the 5 disciplines, either SIZE or ADMISSIONS SELECTIVITY is the biggest determinant of a school’s reputation.These results confirm prior studies on graduate program reputation that analyzed the 1995 NRC ratings.
44 Summary of Results Variable with the largest beta coefficient: Business Starting salary of graduatesEducation Enrollment sizeEngineering Enrollment sizeLaw Admissions selectivity (LSAT)Medicine Admissions selectivity (MCAT)
45 ImplicationsLike with the undergraduate ratings, the question remains as to how well theU.S. News graduate ratings measure the concept of quality in education.Is the magazine really determining“America’s Best Graduate Schools?”
46 US News Graduate Rankings See two handouts:Highlights of MSU’s rankings for 2010 edition (rankings that are in the actual magazine – additional rankings are in the online edition)Summary of Big 10 schools – number of ranked programs over time
47 Other rankings beyond US News WORLD RANKINGS (entire university ranked)The Times Higher Education Supplement World University Rankings(MSU ranked # 203 for 2008 ranking)Peer review of faculty worldwide (40%)Employer review (10%)Citations per faculty (20%)Faculty / student ratio (20%)International faculty (5%)International students (5%)
48 Other rankings beyond US News WORLD RANKINGS (entire university ranked)Shanghai Jiao Tong Academic Ranking of World Universities(MSU ranked # 83 for 2008 ranking)Faculty winning Nobel & Fields medals (20%)Highly cited researchers (20%)Articles published in Nature and Science (20%)Articles in ISI Science and Soc Sci indices (20%)Alumni winning Nobel & Fields medals (10%)Above 5 weighted per FTE academic staff (10%)