1. The Convergence of University Rankings and System Benchmarking An Apparent Paradox of “Rankology”

2. Questions Two approaches: University Rankings System Benchmarking Are they: Complementary? Competing? Consistent? IREG - Warsaw, 16-17 May 20132

3. Outline (1) Background: from ranking to benchmarking (2) Method of investigation (3)Results (4) Interpretation and conclusion IREG - Warsaw, 16-17 May 20133

4. (1) University Rankings IREG - Warsaw, 16-17 May 20134

5. U Rankings: a Polarizing Exercise U Rankings: hated/loved, criticized/commended, threatening/stimulating but proliferating (“here to stay”)  Ph. Albatch’s advice [“Don’t take too much notice of rankings” ( UWN, March 23, 2013 )]: unlikely to be widely followed More pitfalls discovered, uncovered, elucidated  more attempts to improve methods IREG - Warsaw, 16-17 May 20135

6. U Rankings: the Disease Methodological caveats Biases: Research, English, STEM Composite indicators: Weighting=> Elitism Subjective (reputation) /non transparent Dangerous use (“misuses”, “abuses”) Universities:(1) Focus on competition with others instead of own improvement / Affect strategic planning (2) Focus on biased criteria (research) Policy makers: Focus on a few WCUs instead of whole system Students:Impact on university selection Overall: Impact on financing Commercialization(crowded) market IREG - Warsaw, 16-17 May 20136

7. From Ranking to Benchmarking “If Ranking is the Disease, Is Benchmarking the Cure?” (Jamil Salmi, Sunita Kosaraju. Evaluation in Higher Education, Vol. 5 no.1, June 2011) “Rankings: Neither a Disease nor a Cure” (Ph. Albatch, UWN, 2013) IREG - Warsaw, 16-17 May 20137

8. (2) System Benchmarking IREG - Warsaw, 16-17 May 2013 Resources Access Equity TE SYSTEM Governance Quality control Private Providers 8 Economic, Social & Technological Environment

9. Benchmarking: Objective & Criteria Objective: assess strength, health and performance of countries' tertiary education systems Criteria: resources, inputs, governance, outputs and outcomes of the system (access, equity, quality, relevance) IREG - Warsaw, 16-17 May 20139

10. Benchmarking: Main Initiatives SABER: System Approach for Better Education Results (World Bank)  Still under construction U21(Universitas 21/ University of Melbourne) Most recent, comprehensive available case  See below Benchmarking University Governance (World Bank – MENA):  Hybrid AHELO: Assessment of Higher Education Learning Outcomes (OECD)  Still under experimentation IREG - Warsaw, 16-17 May 201310

11. Hypothesis Benchmarking developed in reaction to Rankings Objectives, level of observation and criteria of Benchmarking and Ranking are quite different ==  Shouldn’t they yield different results? IREG - Warsaw, 16-17 May 201311

12. Method (1) 1/ Select 4 of the more popular university rankings: ARWU, THE, QS, WEBOmetrics 2/ Pick the most recent system benchmarking: U21 3/ Compare their results IREG - Warsaw, 16-17 May 201312

13. Method (2) Issue: How to compare U and Systems? Solution:Translate U rankings into Country Rankings Method: From: number of top universities to: number of tertiary aged youths in one country potentially served by top universities in that country (e.g. supply of top universities) NB: no correlation between the 2 measures IREG - Warsaw, 16-17 May 201313

14. NB: Number of Top 400 U and Supply of Top 400 U (THE) : Rank) IREG - Warsaw, 16-17 May 201314

15. Method (3) Quick look at the 4 leagues selected The “sample”: Top 400 universities IREG - Warsaw, 16-17 May 2013 THEARWUQSWEBO Nbr of countries with at least one top 400 university in each league 41384541 Nbr of countries with at least one top 400 university found in all 4 leagues (Overlap) 34 Nbr of top 400 universities in the countries with at least one top 400 university found in all 4 leagues (Overlap) 389394378387 15

16. The 34 AustraliaJapan AustriaMexico BelgiumNetherlands BrazilNew Zealand CanadaNorway ChinaPoland Czech RepublicPortugal DenmarkRussian Federation FinlandSingapore FranceSouth Africa GermanySouth Korea GreeceSpain Hong KongSweden IndiaSwitzerland IrelandTaiwan IsraelUnited Kingdom ItalyUnited States

17. Comparing the results of the 4 Rankings (1) Correlation between results of the 4 leagues: (Number of top universities in each country) IREG - Warsaw, 16-17 May 201317

18. Comparing the results of the 4 Rankings (2) Correlation between results of the 4 leagues: (1) number of top universities in each country IREG - Warsaw, 16-17 May 2013 Nbr of Top 400 Universities: R 2 THEQSARWUWEBO THE 0.98 0.96 QS 0.950.93 ARWU 0.98 WEBO 18

19. Comparing the results of the 4 Rankings (3) Correlation between results of the 4 leagues: (2) Supply of top universities IREG - Warsaw, 16-17 May 2013 Density: R 2 THEQSARWUWEBO THE 0.960.870.78 QS 0.830.72 ARWU 0.86 WEBO 19

20. Supply: Nbr of top U/ TE aged population The first five countries QSARWUTHEWEBO 1Finland16.16.911.59.2 2New Zealand14.54.814.52.4 3Switzerland13.411.813.411.8 4Ireland13.38.013.35.3 5Denmark11.59.211.59.2 The last five countries QSARWUTHEWEBO 30Poland0.30.50.50.8 31Mexico0.10.10.10.1 32Brazil0.10.20.10.4 33China0.10.10.10.2 34India0.040.010.020.01

21. Benchmarking: “U 21”Method (1) 1/ A priori selection of 48 countries ( +2) 2/ Assessment of countries’ performance based on one overall indicator and 4 “measures”: (1) Resources (2) Environment (3)Connectivity (4)Output IREG - Warsaw, 16-17 May 201321

22. Benchmarking: Method (2) (1)Resources (25%) : 5 indicators on expenditures (2) Environment (25%) : 2 indicators on gender balance, 1 indicator on data quality, 3 indicators on policy and regulatory environment, 1 homegrown index on internal governance IREG - Warsaw, 16-17 May 201322

23. Benchmarking: Method (3) (3) Connectivity (10%) : 2 indicators on degree of internationalization (students & research) (4) Output (40%): 5 indicators on research, 1 indicator on Probability of a person to attend a top 500 university (*) based on ARWU… 1 indicator on enrollment 1 indicator on tertiary educated population 1 indicator on unemployment among tertiary educated population IREG - Warsaw, 16-17 May 201323

24. Benchmarking: Links between the 5 measures IREG - Warsaw, 16-17 May 2013 Overall Resources (25%) Outputs (40%) Environment ( 25%) Connectivity (25%) Overall 0.880.93 0.630.59 Resources 0.75 0.390.43 Outputs 0.500.38 Environment 0.40 Connectivity 24

25. Comparing Results of Rankings and Benchmarking (1a) Countries Overlap between UR and SB: U21 & THE:37 common countries U21 & QS:40 common countries U21 & ARWU:37 common countries U21 & WEBO:41 common countries  Essentially same pool of countries IREG - Warsaw, 16-17 May 201325

26. Comparing Results of Rankings and Benchmarking (1b) IREG - Warsaw, 16-17 May 201326 Not in U21 Not in one (or more) Ranking ColombiaArgentina EstoniaBulgaria IcelandChile LebanonCroatia OmanHungary PhilippinesIndonesia Saudi ArabiaIran UAEMalaysia Romania Slovakia Slovenia Thailand Turkey Ukraine

27. Comparing Results of Rankings and Benchmarking (2) IREG - Warsaw, 16-17 May 2013 U21THEQSARWUWEBO Overall0.740.730.76 Ressources0.69 0.780.77 Outputs0.550.570.630.62 Environmment0.500.490.450.38 Connectivity0.730.650.630.58 Correlation between U21 Indicators and Rankings (Supply): R 2 27

28. Comparing Results of Rankings and Benchmarking (3) U21 (Overall) and THE Rankings (R 2 = 0.74) IREG - Warsaw, 16-17 May 201328

29. Comparing Results of Rankings and Benchmarking (4) U21 (Resources) & ARWU (Supply): R 2 = 0.78 IREG - Warsaw, 16-17 May 2013 U21 (Resources)ARWU (Supply) U21 (Resources)ARWU (Supply) Canada10060 South Korea6016 Denmark9792 New Zealand5948 Sweden94136 Portugal5813 USA9246 Spain5823 Norway9274 Iran571 Finland8969 UK5660 Switzerland87118 Japan5319 Singapore8248 Poland495 Netherland8090 Italy4731 Austria7546 Czech Rep4712 Ireland7280 Russia431 Belgium6972 Brazil422 France6732 Mexico401 Hong Kong6486 Hungary4012 Israel6480 Argentina392 Germany6446 South Africa353 Taiwan6316 China331 Australia6382 India230.1 Greece6324 29

30. Conclusions /Interpretation 1/Hypothesis not confirmed: a/same set of countries b/similar results 2/Two types of explanations: a/methodological b/structural IREG - Warsaw, 16-17 May 201330

31. Epilogue System Benchmarking ends up ranking countries Boundaries between UR and SB are blurred SB suffers common symptoms with UR Convergence of the two streams of “Rankology” not surprising Benchmarking needs to expand its pool of countries to become more relevant IREG - Warsaw, 16-17 May 201331

32. Take Away IREG - Warsaw, 16-17 May 201332

33. Thank You IREG - Warsaw, 16-17 May 201333 SB UR