1. The Role of Libraries in the Evaluation of Research
Katie Evans, Research Analytics Librarian
Kate Robinson, University Librarian University of Bath, UK

2. Overview Introducing the University of Bath Roles for the Library
Challenges & Opportunities

3. University of Bath is a mid-sized UK University, located in the UNESCO World Heritage city of Bath.
(Photo is of Bath Abbey not the University of Bath!)

4. Introducing the University of Bath
“Our mission is to deliver world-class research and teaching, …”
£100 million research grants
etc.
15,000
students
We aim for excellence in both teaching and research.
Approx. 15,000 students
We have a strong track record for student satisfaction and are doing well in national leagues tables.
We’re ranked 1st in the Times Higher Education Student Experience Survey 2015 (of UK universities).
Research grants and contracts portfolio is worth around £100 million.
The University of Bath will be 50 years old this year and is currently the top ranked European University in the QS ‘Top 50 under 50’ World University Ranking.

5. University of Bath Library Research Services
Collections
& Archives
Open
Access
Data
Management
Analytics
The Research Services Team in the Library provide services for the University on:
Research collections & archives
Open access – About 80% of recent journal articles by University of Bath authors are open access, available (or will be available after embargo) from our institutional repository
Research data management – We have a growing research data archive, with about 90 datasets available through the archive in its first year
Research analytics service – providing analysis, training and support on the use of publications and citations data across the University. This has enabled the Library to play a growing role in research evaluation.

6. Research Evaluation: Roles for the Library
Enhancing performance in external evaluations
Informing strategic decision makers
Developing publishing strategy
Supporting partnerships
We will discuss 4 ways that the Library contributes to research evaluation at the University of Bath.

7. External evaluations The University’s research is evaluated by:
World university rankings and league tables
National research assessment exercises and government schemes
Research funders
These are all important to the University for reputation, funding, attracting high quality researchers & students etc.
Some of these evaluations make direct use of publications and citations data e.g. World University Rankings.
In some we might choose to present bibliometric analysis as evidence for the University’s research strengths e.g. in grant applications.
Others are qualitative assessments (e.g. the Research Excellence Framework), but we may still find it helpful to use bibliometric analysis to try to understand our performance in these evaluations.
We use bibliometrics to profile and benchmark the University’s publishing – this helps us to understand our performance in external assessments and look for ways to improve.
It’s particularly important to be able to benchmark e.g. against comparator universities or a subject area as a whole. Publications and citations are growing all the time, so we need to benchmark to understand our position in this changing environment.
We have subscribed to SciVal – a research analytics tool from Elsevier – that makes this sort of profiling and benchmarking much easier. We also use other tools and data sources (both subscription e.g. Scopus, Web of Science, Journal Citation Reports and freely available e.g. Google Scholar, Altmetric.com, SCImago, CWTS journal indicators, VoSViewer).
We also provide bespoke analysis to support research funding bids and provide training for academic staff on how they can use bibliometrics to present the strengths of their publishing track record e.g. in grant applications.

8. Example: International collaboation
Uni A
Uni B
Bath
For example, this graph (from SciVal, Aug 2016) shows the percentage of papers that are internationally co-authored for University of Bath and two comparator universities. Over the past 20 years, international collaboration rates have increased dramatically. In recent years, University of Bath hasn’t increased the proportion of our publications including an international co-authorship as quickly as these two comparators.

9. Informing strategic decision makers
What was
the effect of this initiative on publishing?
What are the University’s research strengths?
As well as external evaluations imposed on the University from outside, we also need to understand and evaluate research for the University’s own purposes.
These are examples of questions we’ve been asked by the University’s senior management team.
Providing data on publishing, citations and co-authorship doesn’t completely answer these questions – but it does contribute valuable information.
How do we compare to top universities for internationalisation?

10. Developing publishing strategy
Encourage & inform development of publishing strategy
Department & Group Profiles
Comparator Analysis
Academic Training
In tandem with evaluating research, the University wants to enhance its research publishing. This means doing great research AND publishing it to good effect.
The Library encourages Departments, Research Groups and individual academic staff to reflect on and develop their publishing strategy.
Will their publications reach the right audience? Will they be easily found? Will they be recognised as high quality, relevant research and cited by other researchers?
There are big differences in publishing patterns between subject areas, so this work is best done at the Department/Research Group level rather than looking at the whole University at once.
Profiling a Department’s research and looking at aspirational comparators’ publishing are good starting points for starting a discussion and informing development of publishing strategy.
We run hands-on workshops for research students and academic staff and are regularly invited to give presentations at Department / Research Group team meetings. In these workshops/presentations we cover topics such as:
How do choose where to publish
Good practices for attracting citations (including writing for search engine optimisation and for human readers)
Ways to disseminate and promote your research (e.g. social media, writing for the general public, presenting at conferences, press releases)
Exploring a field – what are the most prominent journals? Who are the most prominent authors and institutions?
Finding potential collaborators
Tools for understanding and managing your research profile online e.g. Google Scholar, Scopus, Web of Science, ResearchGate, ORCID;
Tracking attention to your work – both citations and alternative metrics
Presenting your research strengths

11. Example: Autism & Emotion
For example:
University of Bath has just opened a Centre for Applied Autism Research:
Researchers interested in the topic autism & emotion could inform their publishing strategy by finding out e.g.:
Which journals papers in the area are published in most frequently
Which social psychology papers in this area are highly cited – look at how the titles & abstracts are written
Where (in Poland) is research in this area being carried out, and who’s doing it.
(Screenshots from SciVal, Aug 2016)

12. Supporting partnerships
The University of Bath values international research partnership and is working to grow and develop these relationships.
The Library works closely with the University’s International Relations Office in this area.
Research analytics data tells us about publications co-authored between University of Bath and other institutions around the world – How many? What subjects? Who was involved? What has the academic impact of these publications been (i.e. citations)?
This slide shows a screenshot from SciVal with the number of institutions around the world the University of Bath has co-published with since 2011.
We can use this information to understand and monitor existing partnerships.
We can also see ‘potential collaborators’ i.e. academics working in a particular topic of interest. And get an overview of the subject mix and research strengths of different institutions.
We can use this to strengthen existing relationships and new develop new partnerships. E.g. trying to identify areas of research synergy between University of Bath and partners.
Similarly, partnership with industry is also very important. We can use data on co-publishing with industry to monitor and benchmark these relationships and to evidence the strength of our track record of industry collaboration when bidding for funding and developing new relationships.

13. Challenges & Opportunities
Responsible metrics
Staff knowledge & skills
New working relationships
The roles for the Library in research evaluation we have discussed are relatively new – University of Bath Library established its Research Analytics Service in 2014.
We will now consider some of the challenges and opportunities these new roles present for libraries.

14. Responsible metrics
“The research community should develop a more sophisticated and nuanced approach to the contribution and limitations of quantitative indicators”
“The problem is that evaluation is now led by the data rather than by judgement”
Quotes from:
Hicks, D., Wouters, P., de Rijcke, S., & Rafols, I., The Leiden Manifesto for research metrics. Nature. 520 (7548) doi: /520429a
Wilsdon, J., et al. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. DOI: /RG
We have talked about using publication, citation, co-authorship and alternative metrics in the evaluation of research. However, none of these metrics are a direct measure of research quality.
If the research agenda were driven by bibliometrics alone then there could be undesirable side effects – e.g. discouraging innovative/risky research is new/small/local areas; gender bias; stress, anxiety & low morale caused by blunt/heavy-handed performance management.
Hence the growing awareness and promotion of the idea of ‘responsible metrics’. In particular, in the UK, it has been recommended that universities should develop statement of principles on the approach to research assessment and management, including the role of quantitative indicators (i.e. metrics).
Two key challenges are:
(1) How to account for differences between subject areas?
(2) Is it fair to assess an individual’s research publishing record using bibliometrics?
There are no easy answers to these questions!
This is both a challenge for the Library – to understand the issues and develop expertise in using metrics responsibly and advising others on this
And an opportunity – to be recognised for our expertise in this area and take the lead on working out what responsible metrics use looks like in practice; ensuring that this becomes embedded in University processes and supporting colleagues across the University to use bibliometrics effectively and responsibly.

15. Library staff knowledge & skills
University Strategy
Research Landscape
Statistical Analysis
Metrics & Altmetrics
Scholarly Publishing
Training
Researcher Viewpoint
Tools & Resources
Library staff working in this area need a range of knowledge and skills.
Some of these are areas where the Library has substantial experience e.g. scholarly publishing, providing training
But others may be new to a lot of librarians e.g. understanding academic reward systems and how these affect researchers’ publishing behaviour; dealing with statistics.
And of course this area is developing all the time, so there’s always more to learn.
Because this is a relatively new area, it can be hard to access relevant training.
The University of Bath is actively involved in a couple of initiatives helping Library staff to develop knowledge & skills needed:
1. the ‘LIS-Bibliometrics’ community – a practitioner community centred around the LIS-Bibliometrics JISCmail list, see our blog: The Bibliomagician
Although UK based, we have list members from all around the world and would welcome more. We run one-day conferences for practitioners about once a year. These are opportunities to engage with issues and challenges, share good practice, network and learn from each other.
2. The UK SciVal User-Group – again providing valuable opportunities to share good practice and network.

16. New working relationships
Library
Academics
Research Office
Planning Office
Research Marketing
International Office
The Library has always worked closely with students, academics and other professional services around the University. But our growing role in research evaluation extends these relationships in new ways. e.g.
Working with the research office on funding bids and training/support for researchers on presenting their research track record
Planning office – providing bibliometric data for Key Performance Indicators and contributing to analysis and discussion of our performance in international league tables
Research marketing – working together on supporting academics using social media and other online tools (e.g. writing for ‘The Conversation’) to promote their research; using altmetrics
International office – supporting work on partnerships discussed earlier
Academics – more involvement with senior academic staff with responsibility for publishing strategy in their Department as well as continuing to support individual academics
This can be challenging, because we have to understand the different viewpoints and priorities of all these stakeholders so that we can work with them effectively. But it’s also a great opportunity for the Library to a key player in the University’s research evaluation activities.

17. The Library adds … Word cloud created with https://Tagul.com
This is why I think the Library is well placed to contribute to research evaluation and the other activities we’ve discussed.