Snowball Metrics Slides from : Anna Clements, University of St Andrews Lisa Colledge, Elsevier Stephen Conway, University of Oxford Keith Jeffery, euroCRIS 10/17/2015 Snowball Metrics Project Partners University of Oxford, University College London, University of Cambridge, Imperial Colledge London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier 1

Snowball Metrics are… Tried and tested methodologies that are available free-of-charge to the higher education sector Defined and agreed by Higher Education Institutions to support their strategic decision making Absolutely clear definitions enable apples-to-apples comparisons 10/17/20152 Snowball Metrics Project Partners University of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

The origins of Snowball Metrics Highlighted recommendations from the study Institutions and funders should work more collaboratively, and develop stronger relationships with suppliers An agreed national framework for data and metric standards is needed Suppliers should participate in the development of data and metric standards Sentiments among English research institutions were the prelude to this partnership Growing recognition of the value of data/metrics to inform and monitor institutions’ research strategies Dissatisfaction with available tools: multiple custom implementations, incompatibility of data systems Frustration over the lack of standard metrics available to give sensible, relevant measurements Frequent data requests from external bodies put pressure on institutions to collect and return data on aspects of performance that are not necessarily best suited to the institutions themselves Imperial College London and Elsevier conducted a study of research information management in England funded by the Joint Information Systems Committee (JISC)study of research information management... which led to Snowball Metrics Snowball Metrics Project Partners University of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

Defining Snowball Metrics A subset of participants initiated a second phase of work that would address and build on these recommendations Vision: Institutions agree on metrics and methodologies as standards to support strategic decision making by enabling benchmarking across institutions globally Approach: Eight UK higher educations institutions working with a supplier, Elsevier, partnered to reach consensus on a set of metrics. Snowball Metrics reflect universities’ view of the world, not Elsevier’s. Elsevier played a supporting role to help higher education institutions articulate their needs, and to test the feasibility of addressing them Outcome: Agreed and tested Snowball Metrics methodologies have been shared freely in the Recipe Book, so that any organization can adopt the framework Next steps: All project partners are committed to facilitating a global definition of these metrics as standards by incorporating other existing standards into the “recipes”, and working with all suppliers to implement these metrics Project Partners Snowball Metrics Project Partners University of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

Snowball Metrics Landscape Snowball Metrics Project Partners University of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier Metrics are sliced at the institutional and discipline / funder type levels Metrics need a combination of institutional, proprietary, and public data

Snowball Metrics shared in Recipe Book 10/17/20156 Snowball Metrics Project Partners University of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier Input Metrics Scholarly Output Citation Count h-index Field-Weighted Citation Impact Publications in Top Percentiles Collaboration Process Metrics Output Metrics Applications Volume Awards Volume Income Volume Market Share

The value of agreeing definitions Certainty that we are comparing like with like Identify common framework for comparisons – Institutional organizational structures are different and not suitable to provide context – Agreed a common framework in the UK to benchmark within: HESA Cost Centres – We needed to assign our data to the HESA Cost Centre framework 10/17/20157 HESA Cost Centre: a grouping of researchers by field that allows meaningful comparisons between different types of data within a discipline HESA: Higher Education Statistics Agency Snowball Metrics Project Partners University of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

Specifying Awards Volume 10/17/20158 Removing ambiguity (extracts from the complete “Recipe”) Snowball Metrics Project Partners University of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier *Ordered according to productivity 2011 (data source: Scopus) Use aggregated values of awards over award lifetime, not the value (to be) spent in any financial year Date used is the date that the award is entered in the institutional grants system Include subsequent financial amendments – supplements and reductions Do not include non-financial amendments such as no-cost extensions

Metrics are tested for feasibility 10/17/2015 Snowball Metrics Project Partners University of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier 9

Metrics are tested for feasibility 10/17/2015 Snowball Metrics Project Partners University of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier 10

Metrics are tested for feasibility 10/17/ Snowball Metrics Project Partners University of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier *Ordered according to productivity 2011 (data source: Scopus)

Our task – to CERIFy the metrics 10/17/2015 Snowball Metrics Project Partners University of Oxford, University College London, University of Cambridge, Imperial Colledge London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier 12

MICE Project: CERIF Architecture

CERIF Indicators Segment

Physical data Structure Indicators

10/17/2015 Snowball Metrics Project Partners University of Oxford, University College London, University of Cambridge, Imperial Colledge London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier *Ordered according to productivity 2011 (data source: Scopus) 16