Presentation is loading. Please wait.

Presentation is loading. Please wait.

EVOLVING RESEARCH EVALUATION RAE2008 OUTCOMES AND REF PROSPECTS JONATHAN ADAMS 10 NOVEMBER 2009.

Similar presentations


Presentation on theme: "EVOLVING RESEARCH EVALUATION RAE2008 OUTCOMES AND REF PROSPECTS JONATHAN ADAMS 10 NOVEMBER 2009."— Presentation transcript:

1 EVOLVING RESEARCH EVALUATION RAE2008 OUTCOMES AND REF PROSPECTS JONATHAN ADAMS 10 NOVEMBER 2009

2 Evaluation Impact –Indicators vs profiles –Decision support, not substitution What variables are relevant in... –Academic –Economic –Policy –Social... Impact Objective impact evaluation is constrained by a lack of reference data outside ‘academic’

3 Thomson Reuters and Education Web of Science –250 + journal categories –3 relate to ‘education’ Education & educational research – 210 journals Education, disciplines – 37 journals Education, special – 36 journals –Coverage is back-filled –Regional spread is diversifying

4 Sample coverage Thomson Reuters coverage has broadened –Alberta journal of educational research –American journal of education –Asia Pacific journal of education –Australian journal of early childhood –British educational research journal –Chinese education and society –Didactica Slovenica - pedagoska obzorja –Durham and Newcastle research review –Egitim ve bilim - education and science –European journal of Dental education All the way through to –Revista Brasileira de Ensino de Fisica –Zeitschrift fur Soziologie der Erziehung und Sozialisation

5 UK coverage National Citation Report Evidence tracks all author addresses and reconciles to ‘real’ organisations Profiles for each UK university

6 Steady but volatile improvement

7 What is used to represent excellent research outcomes? RAE1996 ScienceEngineeringSocial sciencesHumanities and arts Outputs% % % % Books and chapters5, , , , Conference proceedings2, , , , Journal articles77, , , , Other1, , , , RAE2001 Books and chapters1, , , , Conference proceedings , , Journal articles76, , , , Other , , RAE2008 Books and chapters1, , , Conference proceedings2, Journal articles80, , , , Other2, , ,

8 RAE2008 grade spread Staff FTE4* %3* %2* %1* %U HEIs >= 20% 4* [13] HEIs < 20% 4* [68] Education UoA OVERALL Golden Triangle Russell Group excl Group University Alliance Million

9 Previously among studies by Evidence HEFCs/UUK – Maintaining Research Excellence –Recognition of a peak supported by a research platform –Evolution of 3 to 4 to 5 grade units HEFCE – Role of QR funding –Strategic significance of the block grant Universities UK – Excellence and Diversity –Regional network of research competence –Importance of an evidence base for research policy OST/DIUS/BIS –Diversity as a critical element in research policy

10 RAE2008 – a novel outcome with dispersed rewards Gareth Roberts’ proposals on research profiling –Problem of the ‘cliff edge’ –Recognising dynamic excellence in the research base Unexpected differences between panels –The problem of interpreting methodology and standards –The absence of a commonly understood standard? –The dispersed map of reward-able research Decisions about funding allocations –2* = 1 (x 3 to) 3* = 3 (x 2.33 to) 4* = 7 –Greater relative rewards for modest gains than for very expensive world-class excellence (cf 2001)

11 How can we explore the spread of excellence? We have –Panel outcomes from RAE2008 – but opinions differ, so... –Bibliometric data on a consistent basis from 1981 We split the HE sector into crude groups –All HEIs –Russell Group –1994 Group –Universities with a shorter history of research investment And we created an elite ‘golden triangle’ band –Oxford (C12 th ), Cambridge (1209), UCL (1836), Imperial (C19 th ), LSE (1895)

12 Excellence is linked to selectivity

13 And excellence is concentrated UK output Total articles and reviews, Papers above world average impact Papers with impact exceeding four times world average Papers with at least one co-author from the HE research base 306, ,358 37% of HE total 16, % of HE total Papers with at least one co-author from the Russell Group 204,307 66% of UK HEIs’ total 79,505 70% of HE output 39% of Russell total 11,734 73% of HE output 5.7% of Russell total Papers with at least one co-author from a ’golden triangle’ HEI 87,157 28% of HEIs’ total 37,370 33% of HE output 43% of GT total 6,308 39% of HE output 7.2% of GT total

14 The problem with simplistic indicators They don’t really express the complexity of research performance Average impact (e.g. ‘crown indicator’) can be very misleading –Research Council studies reveal error of interpretation –Skewed data, median much smaller than average –Lots of papers are not cited –The interesting bit is about how much is really, really cited lots So we we prefer Impact Profiles ®

15 UK background and ‘golden triangle’ This is the small but critical excess of really highly cited research output

16 Even smaller differences separate the UK and USA research profiles

17 Excellence extends to other institutions

18 But the differences become very fine

19 And if we distil further, other complexity is revealed

20 Older institutions have no monopoly on quality, but the peak is distinctive For the 1994 Group, there are relatively fewer low-cited papers and relatively more high-cited papers than for many in the Russell Group

21 What does this imply? There is a very concentrated peak of exceptional excellence UK international competitiveness is associated with selectivity that increased concentration There is a regional network of exceptional quality, much in dynamic institutions created in the 1960s The balance that supports diversity & dynamism and the concentrated support of international excellence is very fine Model first, meddle second

22 EVOLVING RESEARCH EVALUATION RAE2008 OUTCOMES AND REF PROSPECTS JONATHAN ADAMS 10 NOVEMBER 2009


Download ppt "EVOLVING RESEARCH EVALUATION RAE2008 OUTCOMES AND REF PROSPECTS JONATHAN ADAMS 10 NOVEMBER 2009."

Similar presentations


Ads by Google