1. Tim Wales, Associate Director (E-Strategy)
Royal Holloway, University of London
Bibliometrics and the Ref: What do researchers want or need for the future?
13th November 2009

2. Pro A&I
BRIEF BIO

3. University College London
Source:

4. King’s College London (former Pharmacy Library)
Source:

5. City University
Source:

6. London Business School
Source:

7. Open University

8. Trinity College, Bridgeford University
Royal Holloway, University of London
Source:

9. REF (BIBLIOMETRIC) Update

10. Source: http://www.flickr.com/photos/treacle/66010614/

11. Red card to bibliometrics?
HEFCE Pilot exercise concluded:
“...citation information is not sufficiently robust to be used formulaically or as a primary indicator of quality, but there is consider scope for it to inform and enhance the process of peer review.”
HEFCE Pilot exercise concluded: “...citation information is not sufficiently robust to be used formulaically or as a primary indicator of quality, but there is consider scope for it to inform and enhance the process of peer review.”
Source: HEFCE (2009b), p.3

12. More of a yellow card... Details of citation data use
“Robust” citation data UoAs
Sub-panels decide in advance
# of citations for each output provided
With “appropriate benchmarks”
To “inform and supplement” review
Medicine
Science
Engineering
Arts
Humanities
“others”
Source: HEFCE (2009b), p.3

13. ...with final warning(s)
Clear guidelines on using the data robustly to take account of the known limitations and to avoid bias
Panels will not make judgements about quality of outputs solely on the basis of citation information; expert information must be applied
All submitted outputs will be treated equally, whether or not citation information available
Clear guidelines on using the data robustly to take account of the known limitations and to avoid bias
Panels will not make judgements about quality of outputs solely on the basis of citation information; expert information must be applied
All submitted outputs will be treated equally, whether or not citation information available
Source: HEFCE (2009b), p.3

14. REF BIBLIOMETRICS PILOT

15. Methodology
Thomson Reuters Web of Science and Elsevier Scopus citation data used
22 HEIs took part
35 UoAs from 2008 RAE used
(Coverage greater than 40% in citation databases)
Publications between 2001 and 2006 inclusive + citation data to 2007
3 data gathering models tested
Journal articles + review papers only

16. Bibliometric models Publication data by institutional address
Publication data by staff, all outputs
Publication data by staff, selected outputs

17.  Not individual/UoA specific
Model summary
Institutional address
Papers assigned to HEIs on basis of author address in WoS/Scopus
Also assigned to subject categories based on destination journal
Staff, all outputs
All papers by staff submitted to RAE2008 for the pilot UoAs
Staff, selected outputs
6 most highly cited outputs of staff in 2.
 Not individual/UoA specific
 Too ‘burdensome’ 4 

18. Data gathering issues
There were difficulties for some in drawing together data from disparate sources, some of which were not held electronically.
Many institutions also found it difficult to comprehensively identify all published research produced by their staff (especially former staff).
...further issues arose due to the need to disambiguate papers written by different people with the same name (especially for larger institutions)

19. UoA publishing variations
Tend to publish in conference proceedings?
Source: HEFCE (2009a), p.154

20. Coverage variations (Education 85% not indexed)
Source: HEFCE (2009a), p.158

21. WHAT ABOUT RESEARCHERS?

23. Journals dominate
Source: Communicating knowledge, p.17

24. Monographs less so
Source: Communicating knowledge, p.18

25. RAE distorts behaviour
Source: Communicating knowledge, p.34

26. UK researchers need TLC
Many researchers are confused by the mixed messages they are receiving as to how best to communicate their findings....
Funders, learned societies and publishers may also wish to consider whether they might take more of a lead in helping to devise guidelines on good practice [in attribtion & listing of authors]
Research timescales need to be carefully considered in any arrangements for the assessment of performance.
Source: Communicating knowledge, pp.6-7

27. RESEARCHERS NEED E-A&I
ISB research
RESEARCHERS NEED E-A&I

28. Printed TOCs, indices etc
Power browsers
c.75%
Printed TOCs, indices etc
Source: Tennopir & King, 2008

29. Tennopir & King ISB summary
(3 decade analysis)
US scientists increased number of readings (from libraries) via searching & citations
Broader range of readings thanks to Library subscriptions to online collections
80% of articles >10 years found via online searching or citation (links)
US scientists read many articles for every one that they cite
Choosing the best article to cite may be subject to peer pressure in the form of choosing more often to cite those that are cited by others.
Following citation links in e-journal articles may have proportionately more influence on citation behavior than reading behavior

31. Digital ISBs Horizontal information seeking Viewers Power browsers
bouncing
viewing
volatile
diverse
horizontal
checking
promiscuous
Horizontal information seeking Viewers Power browsers
“60% of e-journal users view no more than 3 pages” “Average time on e-book or e-journal sites: 4-8 mins” “Power browse horizontally through titles, contents pages and abstracts going for quick wins”
Source: CIBER, 2008

32. Nicholas et al. study
Source: Nicholas et al., 2007

33. OK, WHAT’s MY POINT?

34. Given that...
The HEFCE Bibliometric pilot found that HEIs had major difficulties in sourcing and providing accurate publication data efficiently
There is still a limited need for citation data for REF
RAE submission trends and external ISB research indicate an increase in journal article publishing and use across most disciplines
Today’s and tomorrow’s researchers value A&I data from libraries to aid power browsing and identifying articles to cite

35. It follows that...
HEIs and researchers need a REF compliant system that:
contains all of an HEI’s publication data for current and former staff
Provides citation and benchmarking data for those UoAs that will be using it in the REF
Harnesses researchers use of Library A&I databases to good effect
Interfaces with existing bibliographic and web systems, e.g. OA repository to help promote individual outputs and raise profiles

36. RESEARCH INFORMATION SYSTEMS (RIS)
The answer?
RESEARCH INFORMATION SYSTEMS (RIS)

37. Source: http://www.flickr.com/photos/roadsidepictures/244926428/

38. Research Management System
Research Managers
Institutional Repository
Importing of bibliographic data
Researchers
Validation of bibliographic data
Submission of research artefacts
Depositing research artefacts
Research Management System
Library staff
Querying of research output data
HEFCE HR
ID Mgt
CMS
Querying
RHUL Systems
Submission of bibliographic data
WWW
Updating researchers’ websites
Grants
REF
Web of Science
PubMed
etc

39. Assisted HEFCE REF Pilot
RIS products...
CONVERIS
PURE
(Develop in-house solutions such as at University College Dublin)
Assisted HEFCE REF Pilot

43. Standard RIS A&I data ingest
PURE
Symplectic
ISI Web of Science *
Scopus *
PubMed
ArXiv.org
DBLP
Google Books
Google Scholar
British Library
OCLC WorldCat *
ZETOC *
ISI Web of Science *
Scopus *
PubMed
ArXiv.org
DBLP
Google Books
Google Scholar
British Library
OCLC WorldCat *
ZETOC *
* = Institutional subscription required

44. (Harold Macmillan )
EVENTS DEAR BOY, EVENTS

45. St George’s Medical School
2009 Merger
Merge..err?
Royal Holloway
St George’s Medical School

46. RIS example
PURE v3 screenshots

47. Researcher’s view

48. Import A&I data

49. Author matching

50. Full-text/OA archiving

51. Bibliometric reporting

52. Citation analysis

53. Conclusion – Long live A&I
A&I services have a new and vital role to play in HEI REF/general research administration thanks to the new market for RIS
Reduce manual ingest of bibliographic data
Sometimes are the only means of identifying past HEI research
Facilitate output benchmarking thanks to controlled vocabularies and added-value metadata

54. Final thought
“...one of the publishers joked that given the evidence showing the key role of abstracts in today’s crowded information environment, maybe they should reverse their business model and give full-text away and charge for abstracts. They were only half joking.”
A&I
research
Source: Nicholas et al. (2007)

55. References & Thanks
CIBER (2008) Information behaviour of the researcher of the future. Available from: HEFCE (2009a) Report on the pilot exercise to develop bibliometric indicators for the Research Excellence Framework. Available from: HEFCE (2009b) The Research Excellence Framework: A brief guide to the proposals. Available from: Nicholas, D., Huntingdon, P. & Jamali, H.R. (2007) The Use, Users, and Role of Abstracts in the Digital Scholarly Environment. Journal of Academic Librarianship Vol.33(4),pp RIN (2009) Communicating knowledge: how and why researchers publish their findings. Available from: Tennopir, C. & King, D. (2008) Electronic Journals and Changes in Scholarly Article Seeking and Reading Patterns. D-Lib Magazine, Vol.14(11/12). Available from: Thanks to Adrian Joyce (Business Analyst, RHUL) for RIS material PURE screenshots (c) copyright Atira A/S 2009
@timwales