U-MULTIRANK Approach to m ultidimensional evaluation of HEI performance Getalo Elena lead expert, Development Programs Office, Tomsk Polytechnic University 28 March, 2014

U-Multirank lead partners : CHE - Centre for Higher EducationCHE - Centre for Higher Education (Germany) CHEPS - Center for Higher Education Policy Studies CWTS - Centre for Science and Technology Studies Elsevier Bertelsmann Stiftung folge3 Consortium for Higher Education and Research Performance Assessment The Project is financed by the European Commission (2009) Partners:

U-Multirank New approach to universities ranking (features)  multi-dimensional (orientation not only on research activity (aims, missions, diversity of universities), transparency  user-driven (possibility for users to develop personalised rankings by selecting indicators in terms of their own preferences (practical orientation)  multi-level (combination institutional ranking (of whole institutions) with field-based rankings that focus on particular academic disciplines or groups of programs.  no calculation of composite overall indicators and league tables  comparison institutions with similar institutional activity profiles

U-Multirank NEW Approach: the comparability of institutions is important; institutions and programs should only be compared when their purposes and activity profiles are sufficiently similar. Two-step-process: 1. identification of the institutions with similar profiles by “mapping” their activities; 2. ranking of “like with like” can only be applied afterwards > Is the information enough obvious to be properly comparable? The information is comparable. =

U-Multirank 1. Dimensions (5 categories of university performance evaluation): -teaching and learning - research - knowledge transfer - international orientation - regional engagement 2. Fields of study: 2011: Mechanical /Electrical engineering, Business 2013: Business, Electrical Engineering, Mechanical Engineering, Physics 2015: Computer Science, Psychology, Sociology/Social Sciences and Music 3. Data collection: - Questionnaire Institutional Data - Questionnaire Field – based Data - Questionnaire Students Survey

DimensionIndicatorDefinition Teaching and Learning Expenditure on teaching The percentage of total institutional expenditure dedicated to teaching activities Degree level focus The number of higher degrees (master and Ph.D.) awarded as a percentage of the total number of degrees awarded ScopeThe number of broad educational subject fields in which a significant number of degrees are awarded Highest degree awarded The highest degree level the institution has awarded degrees at Research Expenditure on research The amount of money spent on research activities in the reference year as a percentage of total expenditure Academic research publications A count of peer reviewed publications of the institution per FTE academic staff Professional publications A count of professional publications of the institution per FTE academic staff U-Multirank Mapping Indicators

DimensionIndicatorDefinition Knowledge Transfer Income from private sources The total amount of external research income and income from knowledge transfer from private sources as a percentage of the total income Patent applications Filed The number of patent applications for which the university acts as an applicant related to number of academic staff International Orientation Income from international sources Income from international sources, including subsidies and contracts for teaching and research, from both public and private international sources as a percentage of total income Foreign degree seeking students The number of degree seeking students with a foreign diploma on entrance as % of total enrolment in degree programs. U-Multirank Mapping Indicators

DimensionIndicatorDefinition Regional Orientation Income from regional sources Institutional income from local/regional authorities, local/regional charities and local/regional contracts as a percentage of total institutional income New entrants from the region The percentage of new entrants to bachelor programs coming from the region General CountryThe country where the legal seat of the institution is located Size of the institution The size of the institution in terms of number of students enrolled Online programs The number of programs offered fully online as a percentage of the total number of programs Legal statusThe public/private character of the institution Age of institution The year the institution or its oldest part was founded U-Multirank Mapping Indicators

DimensionInstitutional ranking Field-based rankings TEACHING & LEARNING Student-staff-ratio X Graduation rate (BA and - separately - MA) XX Percentage of academic staff with PhD X Percentage of students graduating within normative period (BA and –separately - MA) XX Rate of graduate employment XX Inclusion of work experience in degree program X Indicators from the student survey X Overall learning experience X Quality of courses & teaching X Organisation of program X Contact to teachers X Social climate X Facilities (libraries, laboratories, rooms, IT) X Research orientation of teaching program X Inclusion of work experience /practical elements X U-Multirank Indicators for evaluation

DimensionInstitutional ranking Field-based rankings RESEARCH External research income (per fte academic staff)XX Doctorate productivity X Total research publication output (per fte academic staff)* XX Art related outputX Field-normalised citation rate*XX Highly cited research publications *XX Interdisciplinary research publications*XX Research orientation of teaching (student survey) X Number of post-doc positionsX External research income (per fte academic staff)XX * Bibliometric and patent indicators will be analysed based on existing data bases U-Multirank Indicators for evaluation

DimensionInstitutional ranking Field-based rankings KNOWLEDGE TRANSFER Income from private sources (service contracts, consultancies, licenses, royalties, clinical trials, etc.) XX Joint research publications with industry*XX Patents (per fte academic staff)*XX Co-patents with industry (per fte academic staff)*XX Number of spin-offs (average over three year period)X Patent citations to research publications*XX Revenues from Continuous Professional Development X U-Multirank Indicators for evaluation

DimensionInstitutional ranking Field- based rankings INTERNATIONAL ORIENTATION Educational programmes (BA, resp. MA) in foreign language X International orientation of degree programmes X Opportunities to study abroad (student survey) X Student mobility (composite of incoming, outgoing, joint degree students) XX Percentage of international academic staffXX Percentage of PhDs awarded to foreign studentsXX International joint research publications*XX International research grantsXX REGIONAL ENGAGEMENT Percentage of graduates working in the region X X Student internships in regional enterprises X X Degree theses in cooperation with regional industry X Regional joint research publications* XX Income from regional sources XX U-Multirank Indicators for evaluation

U-Multirank Sunburst representation of an institutional performance profile

U-Multirank Results First results : IREG-7 Conference: Employability and Academic Rankings – Reflections and Impacts (14-16 May 2014, London, UK ) Gero Federkeil, CHE - Center for Higher Education; Manager-in-charge of Rankings, Member of U-Multirank Consortium, Vice-President of IREG Observatory, Germany

Practical information: Registration for participation in the 2014 edition of U-Multirank is closed. Pre-Registration for U-Multirank 2015, contact form ( 2015/) 2015/ phone:+ 49 (0) fax:+ 49 (0) web: address: CHE Centre for Higher Education Post Box 105, Gütersloh, Germany Twitter: U-Multirank