Multi-dimensional, field-based rankings Gero Federkeil, CHE, Germany Special Workshop: Introduction to Academic Rankings for the Rectors of Universities of the Republlic of Kazakhstan Astana, 2009/06/13

Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 2 Presentation 1.CHE – Centre for Higher Education Development 2.Rankings and information about higher education 3.The classical ranking-model 4.The CHE ranking approach

3 DEAN Annual Conference / Barcelona Nov I. CHE – Center of Higher Education Development private, not-profit organisation founded in 1994 by Bertelsmann Foundation and German Rectors Conference purpose: promotion of reforms in German higher education Ranking of German universities among founding tasks of CHE; first ranking in 1998 activities: HE policy issues consulting ranking, since 1998 staff: ~ 30 people more information:

II. Users of rankings Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 4 (prospective) students: information about universities and programmes in the field the want to study academics/researchers: comparison with colleagues in their field rectors/university leaders : information about the position of their institution policy makers: information about their national universities (international position, efficiency)  Diverse expectations / needs for information  Rankings have to find a balance between those needs incl. Giving information for users with different knowledge about higher education

5 DEAN Annual Conference / Barcelona Nov III. The „classical“ model: ranking orthodoxy There is a “classical” league table approach of rankings used by most rankings: 1.ranking of whole institutions 2.aggregation indicators into a single composite overall indicator by using fixed weights 3.league table with individual numerical positions (like soccer table)

6 Exampe: THES World Rankings III. „Example: QS World Rankings composite overall score weights of indicators ? But: is Johns Hopkins exactly 92,9 % as good as Harvard? league table with clear rank positions ranking of whole universities

III.Critical remarks: ranking whole institutions Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 7 Example 1: Universities with identical score at a given indicator: University AUniversity B Psychology37,830,0 Sociology15,527,0 Economics23,029,0 Literature17,625,0 Mechanical Engineering26,031,0 Physics25,528,4 Chemistry33,028,9 Biology37,033,0 Medicine45,3 Average29,0

Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 8 University AField average Psychology37,832,0 Sociology15,516,0 Economics23,028,5 Literature17,615,0 Mechanical Engineering26,028,8 Physics25,532,1 Chemistry33,0 Biology37,041,0 Medicine45,350,5 Average29,0 III. Critical remarks: ranking whole institutions Example 2: results in the context of the respective fields:

Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 9 III. Critical remarks: composite indicators U.S. News & World Report Ranking: Weights of indicators: IndicatorWeight Reputation25% Retention Rate20% Faculty resources20% Student selectivity15% Financial resources10% Graduation rate5% Alumni giving5% Total100 % 20% 30% 15% 5% 10% 15% 5% 100 % But why not:

No individual ranks in league tables No overall score from weighted indicators No ranking of whole universities Multidimensional ranking Ranking of single fields / programmes Rank groups top intermediate bottom IV. THE CHE approach – an alternative

labour market, employability city, university students study outcome teachingressources research overall assessment (students, professors) internatio- nalisation IV. The CHE-Ranking: Indicators 20 – 25 indicators...

12 ACA Policy Seminar, 4 April 2008 IV. The CHE-Ranking: Indicators... from different data sources… research  publications /citations (bibliometric analysis)  research grants (faculties/departments)  research reputation (professors survey)

13 ACA Policy Seminar, 4 April 2008 IV. The CHE-Ranking:Indicators... facts as well as judgements teaching  student-staff-ratio (fact)  student assessment of contact between students and professors  student assessment of course organisation

IV. CHE ranking: presentation of results Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 14  selecting a field.. .... or a university Looking at the results is possible either by...

Step 1: Selecting a field Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/ fields, covering 80 % of German students 33 fields, covering 80 % of German students

First overview: Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/ selected indicators Alphabetic list of universities

Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 17 First overview: Sort by indicator Within groups: alphabetical order - no league table! Within groups: alphabetical order - no league table!

Details for a single university : Humboldt Berlin Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 18 Facts as well as

Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 19 subjective views by students and professors (about reputation) and professors (about reputation) Details for a single university : Humboldt Berlin

Online: Interactive, personalised ranking Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 20 STEP 1: Selection of (up to ) 5 indicators according to personal preferences STEP 1: Selection of (up to ) 5 indicators according to personal preferences

Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 21 STEP 2: Decision about personal relevance of indicators STEP 2: Decision about personal relevance of indicators Interactive, personalised ranking

Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/ and the result: A personalised ranking... and the result: A personalised ranking Interactive, personalised ranking

Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/ which looks quite different if we select different indicators

Conclusions Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 24 Rankings should define their aims and target groups... but they have different users anyway (students, researchers etc.) Rankings should adress the specific need for information of different users...which in most cases is about fields/subjects... which differ with regard to the relevance of dimensions/ indicators (teaching, reasearch etc.)

Conclusions Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 25 So rankings should be field-based in the first place multi-dimensional, showing the profile of institutions and leaving the decision about the importance/ weight of indicators to users And, last but not least, they should avoid giving false impressions of exactness of league tables

Berlin Principles Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/ Provide consumers with a clear understanding of all of the factors used to develop a ranking, and offer them a choice in how rankings are displayed. Rankings should:

Thank you very much! More information: or Or www. Special Workshop: Introduction to Academic Rankings for the Rectors of Universities of the Republlic of Kazakhstan Astana, 2009/06/13