1. The Multi-Dimensional Research Assessment Matrix
Henk F. Moed
Elsevier, Amsterdam, the Netherlands
Seminar Research Evaluation in Practice, National Geographic Society, Washington DC, 17 October 2012

2. Short CV Henk F. Moed Years Position 1981-2009
Staff member at Centre for Science and Technology Studies (CWTS), Leiden Univ.
2009
Professor of Research Assessment Methodologies at Leiden University
2010 – Sept 2012
Elsevier, SciVal Dept. Senior Scientific Advisor
As from Sept 2012
Elsevier, AGRM Dept. Head of Informetric Research Group

3. 1 2 3 Contents Boundaries of the playing field; rules of the game
The multi-dimensional research assessment matrix 3
Towards more sophisticated indicators: combining big data sets

4. (o) Bibliometrics makes sense

5. Which country has these main collaborators?
USA
Main collaborators UK
China
Germany
Canada

6. Which country has these main collaborators?
Brazil
Main collaborators
Argentina
USA
Portugal
France
Chile

7. Which country has these main collaborators?
Malaysia
Main collaborators
Thailand
India
Singapore
Iran UK

8. Which country has these main collaborators?
Romania
Main collaborators
France
Hungary
Germany
Italy
Bulgaria

9. Which country has these main collaborators?
South Africa
Main collaborators UK
China
USA
Nigeria
Australia
Netherlands

10. Boundaries of the playing field; rules of the game 2
Contents 1
Boundaries of the playing field;
rules of the game 2
The multi-dimensional research assessment matrix 3
Towards more sophisticated indicators: combining big data sets

11. (i) Bibliometric research assessment is potentially a proper tool to consolidate academic freedom

12. (ii) Bibliometric tools help establishing a longer term perspective in research funding

13. (iii) One must be cautious using “societal benefit” as an assessment criterion of basic research: it can not be measured in a politically neutral way

14. (iv) Citations measure scientific-scholarly impact rather than quality or validity

15. (v) Citation counts in social sciences and humanities may be influenced by political ideologies

16. Citation impact and ideology
Fall of the Berlin wall in Nov. 1989

17. (vi) The future of research assessment lies in the intelligent combination of metrics and peer review

18. (vii) Case study on funding policies of a National Research Council reveals biases in peer review

19. Affinity Applicants – Evaluation Committee
0 Applicants are/were not member of any Committee
Co-applicant is/was member of a Committee, but not of the one evaluating
First applicant is/was member of a Committee, but not of the one evaluating
Co-applicant is member of the Committee(s) evaluating the proposal
First applicant is member of the Committee(s) evaluating the proposal

20. For 15 % of SUBMITTED applications an applicant is a member of the evaluating Committee (Affinity=3, 4)
% SUBMITTED APPLICATIONS
AFFINITY APPLICANT - COMMITTEE

21. Probability to be granted increases with increasing affinity applicants-Committee
% GRANTED APPLICATIONS
AFFINITY APPLICANT - COMMITTEE

22. Logistic regression analysis: Affinity Applicant-Committee has a significant effect upon the probability to be granted
MAXIMUM-LIKELIHOOD ANALYSIS-OF-VARIANCE TABLE (N=2,499)
Source DF Chi-Square Prob
INTERCEPT
CITATION IMPACT APPLICANT **
Rel transdisc impact applicant
AFFINITY APPLICANT-COMMITTEE **
Sum requested **
Institution applicant **
LIKELIHOOD RATIO

23. (viii) Data must be accurate and verifiable

24. (ix) Assessed researchers must have the opportunity to check data and comment on outcomes

25. (x) A framework is needed to characterize and position bibliometric indicators and products

26. The multi-dimensional research assessment matrix
Contents 1
Boundaries of the playing field and
rules of the game 2
The multi-dimensional research assessment matrix 3
Towards more sophisticated indicators: combining big data sets

27. The Multi-Dimensional Research Assessment Matrix
Expert Group on the Assessment of University-Based Research (AUBR, 2010)

28. Multi‐dimensional Research Assessment Matrix (Part)
Unit of assessment
Purpose
Output dimensions
Bibliometric indicators
Other indicators
Individual
Allocate resources
Research productivity
Publications
Peer review
Research group
Improve performance
Quality, scholarly impact
Journal citation impact
Patents, licences, spin offs
Department
Monitor research programs
Innovation and social benefit
Actual citation impact
Invitations for conferences
Institution
Increase regional engagement
Sustainabi-lity & Scale
Internat. co-authorship
External research income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com-pletion rates

29. Multi‐dimensional Research Assessment Matrix (Part)
Unit of assessment
Purpose
Output dimensions
Bibliometric indicators
Other indicators
Individual
Allocate resources
Research productivity
Publications
Peer review
Research group
Improve performance
Quality, scholarly impact
Journal citation impact
Patents, licences, spin offs
Department
Monitor research programs
Innovation and social benefit
Actual citation impact
Invitations for conferences
Institution
Increase regional engagement
Sustainabi-lity & Scale
Internat. co-authorship
External research income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com-pletion rates
Read
column-
wise

30. Multi‐dimensional Research Assessment Matrix (Part)
Unit of assessment
Purpose
Output dimensions
Bibliometric indicators
Other indicators
Individual
Allocate resources
Research productivity
Publications
Peer review
Research group
Improve performance
Quality, scholarly impact
Journal citation impact
Patents, licences, spin offs
Department
Monitor research programs
Innovation and social benefit
Actual citation impact
Invitations for conferences
Institution
Increase regional engagement
Sustainabi-lity & Scale
Internat. co-authorship
External research income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com-pletion rates

31. Multi‐dimensional Research Assessment Matrix (Part)
Unit of assessment
Purpose
Output dimensions
Bibliometric indicators
Other indicators
Individual
Allocate resources
Research productivity
Publications
Peer review
Research group
Improve performance
Quality, scholarly impact
Journal citation impact
Patents, licences, spin offs
Department
Monitor research programs
Innovation and social benefit
Actual citation impact
Invitations for conferences
Institution
Increase regional engagement
Sustainabi-lity & Scale
Internat. co-authorship
External research income
Research field
Promotion& hiring
Research infrastruct.
citation ‘prestige’
PhD com-pletion rates

32. MD-RAM: Example 1 Publications in international jrnls;
Individual
Hiring/promotion
Productivity & impact
Publications in international jrnls;
Actual citation impact
PhD date, place, supervisor;
Invitations for conferences

33. Multi‐dimensional Research Assessment Matrix (Part)
Unit of assessment
Purpose
Output dimensions
Bibliometric indicators
Other indicators
Individual
Allocate resources
Research productivity
Publications
Peer review
Research group
Improve performance
Quality, scholarly impact
Journal citation impact
Patents, licences, spin offs
Department
Monitor research programs
Innovation and social benefit
Actual citation impact
Invitations for conferences
Institution
Increase regional engagement
Sustainabi-lity & Scale
Internat. co-authorship
External research income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com-pletion rates

34. MD-RAM: Example 2 Publications in international jrnls;
Research group
Monitoring research program
Scientific impact
Publications in international jrnls;
(Trend in) actual citation impact
Collaborations
Topicality

35. Research assessment methodologies: Important considerations
Methodology must be fit-for-purpose
What is the primary “problem” to be solved?
Be aware of unintended effects
Change a methodology every 5-10 years
What is an acceptable “error rate”?
Wrong in individual cases  benificiary for the system as a whole

36. Towards more sophisticated indicators: combining big data sets
Contents 1
Boundaries of the playing field and
rules of the game 2
The multi-dimensional research assessment matrix 3
Towards more sophisticated indicators: combining big data sets

37. Journal articles + citations
Journal full text data
Journal usage data
Unit of assess- ment
Books
Patents
Conference Procs
Trade jrnls
Newspapers
Social media

38. (a) Downloads vs. Citations What do full article downloads measure?

39. Authors vs. readers
Readers
Authors ?

40. Hypothesis on degree of correlation between downloads and citations
Authors
Authors
Readers
Readers
Strong
Weak

41. Usage vs. citations per main field
Scientific ?
PSYCHOL
Societal

42. (b) Patent citations to journal articles: The technological impact of research

43. PATENTS (TotalPatent) 42 LIBRARY SCIENCE JOURNALS (Scopus)
The Technological Impact of Library Science Research: A Patent Analysis [Halevi et al, 2012]
PATENTS (TotalPatent)
Citations by patent examiners and inventors
42 LIBRARY SCIENCE JOURNALS (Scopus)

44. Cited articles: keywords in titles
The articles feature information retrieval and indexing, information and documents management systems which pertain to electronic and digital libraries development
Citing patents: keywords in titles
The patents focus on electronic information administration, navigation, and products and services management in commercial systems.

45. (c) International scientific migration

46. International migration vs. co-authorship
Relationship
Definition
Comment
International
co-authorship
Authors from institutions located in different countries jointly publish a paper
Country relates to where authors work, NOT to their nationality
migration
A scientific author moves from one country to another

47. Map of countries with Ratio migration/collaboration > 1.2

48. TO COUNTRY
FROM Country
No. Co-authored papers
No. Migrating authors
Ratio % Migration / % Co-authorship
PAK
IND
276
118
3.6
PRT
BRA
1,971
423
3.4 96
3.3
NLD
IRN
492 80
3.1
USA
12,013
3,307
2.8
145 21
CHN
TWN
3,979
1,048
2.6
3,039
780
352
2.4
MYS
NGA
122 31
2.1
Language similarity drives migration stronger than it drives co-authorship
Political tensions affect migration less than they affect co-authorship

49. (d) Citation context analysis Combining citation data from Scopus with full text article data from ScienceDirect

50. The use of contextual citations analysis to disclose the thematic and conceptual flow of cross- disciplinary research: the case of the Journal of Informetics (Gali Halevi et al., to be published, 2012)

51. Emerging sectional themes (OUTSIDE DISCIPLINE)
FINDINGS & DISCUSSION
INTRODUCTION
The themes sequence in the word clouds below might suggest that the individual output evaluation done by structured peer review leads to an acknowledgment of the importance and evolution of networks rather than individuals
CONCLUSIONS

52. (e) Book Citation Index Citation flows between books and journals

53. A Scholarly Book Citation Index: Approaches
Add selected book series
Add books from selected publishers
Add selected individual book titles

54. Thank you for your attention!

55. Elsevier Bibliometric Research Program: www.ebrp.elsevier.com