Research Assessment and UK publication patterns Jonathan Adams

June 2005 Evidence Ltd – what does it do? Research performance analysis and interpretation –Founded 2000 –Grew out of Higher Education Policy Unit at University of Leeds UK studies for HEFCE, OST, Gov’t dep’ts, RCs, HEFCW, Royal Society, RDAs International analyses for EC, Austria, New Zealand, Sweden, Switzerland Office of Science & Technology annual PSA target indicators for the UK research base UK Higher Education Research Yearbook Institutional research performance reports

June 2005 The UK’s Research Assessment Exercise – what is it? A research evaluation cycle –1986, 1989, 1992, 1996, 2001, 2008 Considers outputs, training, funding and strategy across higher education institutions Assessment carried out by peer review panels –69 subject-based Units of Assessment (UoAs) Outcome is a quality-related grade –Performance indicator and weighting factor for core research funding (QR) Chancellor has announced shift to metrics post 2008

June 2005 The RAE has led to an increase in UK share of world citations

June 2005 Improvement has happened at all grade levels Data are for core science UoAs, grade at RAE96

June 2005 The RAE is a driver of research activity in universities Changes announced by Chancellor in Budget documents will not affect this Publications are a key measure of performance –RA2 allows 4 items per researcher –These can be books, articles, proceedings or other works –Citation data for serials makes them ideal for evaluating outcomes as well as output This makes publishing patterns a central focus for management and researchers

June 2005 Evidence analysed publications submitted to the RAEs RAE2001 RA2 validation, built on RAE1996 journal analyses and international benchmarking –195,000 items submitted across the 68 UoAs –90% of books, journal articles and conference proceedings were reconciled to real publications –Progressively identify all journal titles/ISSNs and get frequency data by University and UoA Focus on journal articles, initially those on Thomson Scientific (ISI) databases –Track citation performance

June 2005 Research outputs submitted to RAE show variation in output type across subjects Science & EngineeringHumanities & Arts

June 2005 RAE submissions by type and area show shifts between 1996 and 2001 There was a volume decrease in some areas, because of more selective submissions, and there were shifts in output balance

June 2005 Article concentration by journal was similar in 1996 and 2001 There are many journals with fewer than 100 articles across all UoAs

June 2005 Most journals are submitted to only one or a few UoAs but some spread across many subjects

June 2005 Articles were more common in ISI coverage varied.

June 2005 Overall: RAE2001 outputs for science and engineering were usually articles, often in ISI journals, but this is less true in other subject areas

June 2005 There are also HEI factors. Those that submitted more articles tended to have more publications in ISI journals The correlation is highly significant statistically

June 2005 Thomson data cover a high proportion of journals that science researchers believe represent their best work, but this is not always true in the social sciences Note the number of journals. A few are high frequency. Many are singletons and may be marginal to the core subject.

June 2005 Frequently submitted journals are usually ISI journals in science areas. Grade effects are not significant in Physics – is this due to cultural commonality? This example uses data for UoA19. Titles in red are not on Thomson ISI databases

June 2005 Frequently submitted journals are often, but not always, ISI journals in the social sciences. There are grade effects. This example uses data for UoA43. Titles in red are not on Thomson ISI databases

June 2005 Changing cultures? Economic & Social Research Council –Evidence study shows researchers use bibliometric data to evaluate research quality but do so as experts –There is a shift to a trans-Atlantic research culture in which journals will be increasingly significant European Science Foundation/Humanities & Arts –ESF-facilitated panels have been reviewing journal lists Music – no general acceptance of validity History – 1500 national and international serials in 3 bands –Further evidence of expanding role of serials as a standard output mode alongside monographs

June 2005 Publication data throw light on other aspects of the research process Relationship between UoA subject areas is shown by similarity in journal usage Citation impact is correlated with RAE grade when normalised against field When impact is normalised against journal there is less grade variation Publication data can be linked to other factors such as staff numbers and research income

June Clinical Lab Sci... Accountancy Hosp. based... Com. based... Other stud. Pharmacy Biochemistry Biol. sciences Pre-clin. stud. Physiology Pharmacology Anatomy Veterinary sci. Clin. Dentistry Food sci... Agriculture Earth sci. Environ. sci. Geography Archeology Mineral/mining... Chemistry Metallurgy... Physics Chem. eng. Computer sci. Gen. Eng. Mechanical eng... Electrical eng... Civil eng. Pure maths. Applied maths. Statistical res... Nursing Sports related... Psychology Education Politics... Social policy... Sociology Social work Communication... Built environ. Town/country... Economics... Business... stud. Law Library and info... Anthropology Asian stud. Middle east... Theology... American stud. Iberian... European stud. French German, Dutch... English History Italian Russian... Linguistics Classics... Philosophy History of Art... Art and Design Drama, Dance... Music Celtic stud. Journal frequency analyses show the intellectual relationship between UoAs Engineering Medical Physical Maths Bio-Med Environment Social Arts & hum’s This tree diagram illustrates similarity in the frequency with which journals were submitted to RAE1996

June 2005 There is journal differentiation within communities – these data are for UoA13 Psychology Citation impact varies with grade normalised by category But not when normalised by journal impact factor Grade at RAE2001

Research Assessment and UK publication patterns Jonathan Adams