1. What Does Responsible Metrics Mean?
Bibliometrics in Universities, 1st December 2015
Katie Evans,
Research Analytics Librarian, University of Bath

2. Responsible metrics
“The research community should develop a more sophisticated and nuanced approach to the contribution and limitations of quantitative indicators”
“The problem is that evaluation is now led by the data rather than by judgement”
Hicks, D., Wouters, P., de Rijcke, S., & Rafols, I., The Leiden Manifesto for research metrics. Nature. 520 (7548) doi: /520429a
Wilsdon, J., et al. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. DOI: /RG

3. What are the questions? What are the criteria?
Leiden Manifesto Principles:
Quantitative evaluation should support qualitative, expert assessment
Measure performance against the research mission of the institution, group or researcher

4. What are the questions? What are the criteria?
CC-BY Hakan Dahlstrom
Demonstrate
Discuss
Analyse
Iterate

5. Coverage & transparency
Leiden Manifesto Principles
Protect excellence in locally relevant research
Keep data collection and analytical processes open, transparent and simple
Allow those evaluated to verify data and analysis

6. Coverage Subject Area Coverage Overall 78%
Main Panel A – Medical & Life Sciences
97%-99%
Main Panel B – Science & Engineering
91% -100%
Main Panel C – Social Sciences
41% - 91%
Main Panel D - Humanities
17% - 60%
University of Bath Submissions
48% - 99%
This data comes from:
HEFCE (2015). The Metric Tide: Correlation analysis of REF2014 scores and metrics (Supplementary Report II to the Independent Review of the Role of Metrics in Research Assessment and Management). HEFCE. DOI: /RG
and my own calculations for Bath.
Very low coverage UoAs are Panel C - Law (41%) and Panel D – Classics (17%).

7. Coverage

8. Transparency
CC-BY Sharon Mollerus
Individual or small groups – provide underlying publications list for checking
Larger groups – offer to share SciVal groups etc.

9. Interpretation Leiden Manifesto Principles
Account for variation by field in publication and citation practices
Avoid misplaced correctness and false precision
Quantitative evaluation should support qualitative, expert assessment

10. Source: SCImago Visualize
Genetics
Medicine
Computer Science
UK publications in 30 subject categories, average citations, total citations and number of documents (bubble size)
Shows variation in citation rates
Source: SCImago Visualize

11. ‘Can the REF run on metrics’ Times Higher Education 18th June 2015
Treating different disciplines fairly is one of the main challenges.
But even if you have 100% coverage, disciplinary differences still matter.
This graph is analysis done by Elsevier and published by the THE on the correlation between citation indicators and REF output results. It ranges from very strong (Biology, Chemistry) to not strong, even weak negative for construction engineering and anthropology.
Graph from:
Can the research excellence framework run on metrics? An Elsevier analysis explores the viability of a ‘smarter and cheaper’ model. Times Higher Education 18th June
The Metrics Tide Supplementary Report II has more sophisticated (and independent analysis) of correlation between REF output grades and various indicators, see:
‘Can the REF run on metrics’ Times Higher Education 18th June 2015

12. Supporting interpretation
Get researchers using bibliometric tools
Discussion
Footnotes!

13. Absolute Subject normalised

14. Challenges of subject normalising

15. Assessing individuals
Leiden Manifesto Principle
Base assessment of individual researchers on a qualitative judgement of their portfolio

16. Policy Leiden Manifesto Principles
Recognize the systematic effects of assessment and indicators
Scrutinize indicators regularly and update them

17. “… develop a clear statement of principles on their approach to research management and assessment, including the role of quantitative indicators”
Working on policy in response to Metric Tide recommendation

18. Not just assessment – explore!
CC-BY Thomas Williams