The Multi-Dimensional Research Assessment Matrix Henk F. Moed Elsevier, Amsterdam, the Netherlands Seminar Research Evaluation in Practice, National Geographic Society, Washington DC, 17 October 2012

Short CV Henk F. Moed Years Position 1981-2009 Staff member at Centre for Science and Technology Studies (CWTS), Leiden Univ. 2009 Professor of Research Assessment Methodologies at Leiden University 2010 – Sept 2012 Elsevier, SciVal Dept. Senior Scientific Advisor As from Sept 2012 Elsevier, AGRM Dept. Head of Informetric Research Group

1 2 3 Contents Boundaries of the playing field; rules of the game The multi-dimensional research assessment matrix 3 Towards more sophisticated indicators: combining big data sets

(o) Bibliometrics makes sense

Which country has these main collaborators? USA Main collaborators UK China Germany Canada

Which country has these main collaborators? Brazil Main collaborators Argentina USA Portugal France Chile

Which country has these main collaborators? Malaysia Main collaborators Thailand India Singapore Iran UK

Which country has these main collaborators? Romania Main collaborators France Hungary Germany Italy Bulgaria

Which country has these main collaborators? South Africa Main collaborators UK China USA Nigeria Australia Netherlands

Boundaries of the playing field; rules of the game 2 Contents 1 Boundaries of the playing field; rules of the game 2 The multi-dimensional research assessment matrix 3 Towards more sophisticated indicators: combining big data sets

(i) Bibliometric research assessment is potentially a proper tool to consolidate academic freedom

(ii) Bibliometric tools help establishing a longer term perspective in research funding

(iii) One must be cautious using “societal benefit” as an assessment criterion of basic research: it can not be measured in a politically neutral way

(iv) Citations measure scientific-scholarly impact rather than quality or validity

(v) Citation counts in social sciences and humanities may be influenced by political ideologies

Citation impact and ideology Fall of the Berlin wall in Nov. 1989

(vi) The future of research assessment lies in the intelligent combination of metrics and peer review

(vii) Case study on funding policies of a National Research Council reveals biases in peer review

Affinity Applicants – Evaluation Committee 0 Applicants are/were not member of any Committee Co-applicant is/was member of a Committee, but not of the one evaluating First applicant is/was member of a Committee, but not of the one evaluating Co-applicant is member of the Committee(s) evaluating the proposal First applicant is member of the Committee(s) evaluating the proposal

For 15 % of SUBMITTED applications an applicant is a member of the evaluating Committee (Affinity=3, 4) % SUBMITTED APPLICATIONS AFFINITY APPLICANT - COMMITTEE

Probability to be granted increases with increasing affinity applicants-Committee % GRANTED APPLICATIONS AFFINITY APPLICANT - COMMITTEE

Logistic regression analysis: Affinity Applicant-Committee has a significant effect upon the probability to be granted MAXIMUM-LIKELIHOOD ANALYSIS-OF-VARIANCE TABLE (N=2,499) Source DF Chi-Square Prob ------------------------------------------------------------- INTERCEPT 1 18.47 0.0000 CITATION IMPACT APPLICANT 3 26.97 0.0000 ** Rel transdisc impact applicant 1 0.29 0.5926 AFFINITY APPLICANT-COMMITTEE 2 112.50 0.0000 ** Sum requested 1 45.47 0.0000 ** Institution applicant 4 25.94 0.0000 ** LIKELIHOOD RATIO 199 230.23 0.0638

(viii) Data must be accurate and verifiable

(ix) Assessed researchers must have the opportunity to check data and comment on outcomes

(x) A framework is needed to characterize and position bibliometric indicators and products

The multi-dimensional research assessment matrix Contents 1 Boundaries of the playing field and rules of the game 2 The multi-dimensional research assessment matrix 3 Towards more sophisticated indicators: combining big data sets

The Multi-Dimensional Research Assessment Matrix Expert Group on the Assessment of University-Based Research (AUBR, 2010)

Multi‐dimensional Research Assessment Matrix (Part) Unit of assessment Purpose Output dimensions Bibliometric indicators Other indicators Individual Allocate resources Research productivity Publications Peer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs Department Monitor research programs Innovation and social benefit Actual citation impact Invitations for conferences Institution Increase regional engagement Sustainabi-lity & Scale Internat. co-authorship External research income Research field Promotion, hiring Research infrastruct. citation ‘prestige’ PhD com-pletion rates

Multi‐dimensional Research Assessment Matrix (Part) Unit of assessment Purpose Output dimensions Bibliometric indicators Other indicators Individual Allocate resources Research productivity Publications Peer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs Department Monitor research programs Innovation and social benefit Actual citation impact Invitations for conferences Institution Increase regional engagement Sustainabi-lity & Scale Internat. co-authorship External research income Research field Promotion, hiring Research infrastruct. citation ‘prestige’ PhD com-pletion rates Read column- wise

Multi‐dimensional Research Assessment Matrix (Part) Unit of assessment Purpose Output dimensions Bibliometric indicators Other indicators Individual Allocate resources Research productivity Publications Peer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs Department Monitor research programs Innovation and social benefit Actual citation impact Invitations for conferences Institution Increase regional engagement Sustainabi-lity & Scale Internat. co-authorship External research income Research field Promotion, hiring Research infrastruct. citation ‘prestige’ PhD com-pletion rates

Multi‐dimensional Research Assessment Matrix (Part) Unit of assessment Purpose Output dimensions Bibliometric indicators Other indicators Individual Allocate resources Research productivity Publications Peer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs Department Monitor research programs Innovation and social benefit Actual citation impact Invitations for conferences Institution Increase regional engagement Sustainabi-lity & Scale Internat. co-authorship External research income Research field Promotion& hiring Research infrastruct. citation ‘prestige’ PhD com-pletion rates

MD-RAM: Example 1 Publications in international jrnls; Individual Hiring/promotion Productivity & impact Publications in international jrnls; Actual citation impact PhD date, place, supervisor; Invitations for conferences

Multi‐dimensional Research Assessment Matrix (Part) Unit of assessment Purpose Output dimensions Bibliometric indicators Other indicators Individual Allocate resources Research productivity Publications Peer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs Department Monitor research programs Innovation and social benefit Actual citation impact Invitations for conferences Institution Increase regional engagement Sustainabi-lity & Scale Internat. co-authorship External research income Research field Promotion, hiring Research infrastruct. citation ‘prestige’ PhD com-pletion rates

MD-RAM: Example 2 Publications in international jrnls; Research group Monitoring research program Scientific impact Publications in international jrnls; (Trend in) actual citation impact Collaborations Topicality

Research assessment methodologies: Important considerations Methodology must be fit-for-purpose What is the primary “problem” to be solved? Be aware of unintended effects Change a methodology every 5-10 years What is an acceptable “error rate”? Wrong in individual cases  benificiary for the system as a whole

Towards more sophisticated indicators: combining big data sets Contents 1 Boundaries of the playing field and rules of the game 2 The multi-dimensional research assessment matrix 3 Towards more sophisticated indicators: combining big data sets

Journal articles + citations Journal full text data Journal usage data Unit of assess- ment Books Patents Conference Procs Trade jrnls Newspapers Social media

(a) Downloads vs. Citations What do full article downloads measure?

Authors vs. readers Readers Authors ?

Hypothesis on degree of correlation between downloads and citations Authors Authors Readers Readers Strong Weak

Usage vs. citations per main field Scientific ? PSYCHOL Societal

(b) Patent citations to journal articles: The technological impact of research

PATENTS (TotalPatent) 42 LIBRARY SCIENCE JOURNALS (Scopus) The Technological Impact of Library Science Research: A Patent Analysis [Halevi et al, 2012] PATENTS (TotalPatent) Citations by patent examiners and inventors 42 LIBRARY SCIENCE JOURNALS (Scopus)

Cited articles: keywords in titles The articles feature information retrieval and indexing, information and documents management systems which pertain to electronic and digital libraries development Citing patents: keywords in titles The patents focus on electronic information administration, navigation, and products and services management in commercial systems.

(c) International scientific migration

International migration vs. co-authorship Relationship Definition Comment International co-authorship Authors from institutions located in different countries jointly publish a paper Country relates to where authors work, NOT to their nationality migration A scientific author moves from one country to another

Map of countries with Ratio migration/collaboration > 1.2

TO COUNTRY FROM Country No. Co-authored papers No. Migrating authors Ratio % Migration / % Co-authorship PAK IND 276 118 3.6 PRT BRA 1,971 423 3.4 96 3.3 NLD IRN 492 80 3.1 USA 12,013 3,307 2.8 145 21 CHN TWN 3,979 1,048 2.6 3,039 780 352 2.4 MYS NGA 122 31 2.1 Language similarity drives migration stronger than it drives co-authorship Political tensions affect migration less than they affect co-authorship

(d) Citation context analysis Combining citation data from Scopus with full text article data from ScienceDirect

The use of contextual citations analysis to disclose the thematic and conceptual flow of cross- disciplinary research: the case of the Journal of Informetics 2007 (Gali Halevi et al., to be published, 2012)

Emerging sectional themes (OUTSIDE DISCIPLINE) FINDINGS & DISCUSSION INTRODUCTION The themes sequence in the word clouds below might suggest that the individual output evaluation done by structured peer review leads to an acknowledgment of the importance and evolution of networks rather than individuals CONCLUSIONS

(e) Book Citation Index Citation flows between books and journals

A Scholarly Book Citation Index: Approaches Add selected book series Add books from selected publishers Add selected individual book titles

Thank you for your attention!

Elsevier Bibliometric Research Program: www.ebrp.elsevier.com