Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Multi-Dimensional Research Assessment Matrix Henk F. Moed Elsevier, Amsterdam, the Netherlands Seminar Research Evaluation in Practice, National Geographic.

Similar presentations


Presentation on theme: "The Multi-Dimensional Research Assessment Matrix Henk F. Moed Elsevier, Amsterdam, the Netherlands Seminar Research Evaluation in Practice, National Geographic."— Presentation transcript:

1 The Multi-Dimensional Research Assessment Matrix Henk F. Moed Elsevier, Amsterdam, the Netherlands Seminar Research Evaluation in Practice, National Geographic Society, Washington DC, 17 October 2012

2 Short CV Henk F. Moed YearsPosition 1981- 2009 Staff member at Centre for Science and Technology Studies (CWTS), Leiden Univ. 2009Professor of Research Assessment Methodologies at Leiden University 2010 – Sept 2012 Elsevier, SciVal Dept. Senior Scientific Advisor As from Sept 2012 Elsevier, AGRM Dept. Head of Informetric Research Group

3 Contents 1 Boundaries of the playing field; rules of the game 2 The multi-dimensional research assessment matrix 3 Towards more sophisticated indicators: combining big data sets

4 (o) Bibliometrics makes sense

5 Which country has these main collaborators? Main collaborators UK China Germany Canada USA

6 Which country has these main collaborators? Main collaborators Argentina USA Portugal France Chile Brazil

7 Which country has these main collaborators? Main collaborators Thailand India Singapore Iran UK Malaysia

8 Which country has these main collaborators? Main collaborators France Hungary Germany Italy Bulgaria Romania

9 Which country has these main collaborators? Main collaborators UK China USA Nigeria Australia Netherlands South Africa

10 Contents 1Boundaries of the playing field; rules of the game 2 The multi-dimensional research assessment matrix 3 Towards more sophisticated indicators: combining big data sets

11 (i) Bibliometric research assessment is potentially a proper tool to consolidate academic freedom

12 (ii) Bibliometric tools help establishing a longer term perspective in research funding

13 (iii) One must be cautious usingsocietal benefit as an assessment criterion of basic research: it can not be measured in a politically neutral way

14 (iv) Citations measure scientific- scholarly impact rather than quality or validity

15 (v) Citation counts in social sciences and humanities may be influenced by political ideologies

16 Citation impact and ideology Fall of the Berlin wall in Nov. 1989

17 (vi) The future of research assessment lies in the intelligent combination of metrics and peer review

18 (vii) Case study on funding policies of a National Research Council reveals biases in peer review

19 Affinity Applicants – Evaluation Committee 0Applicants are/were not member of any Committee 1Co-applicant is/was member of a Committee, but not of the one evaluating 2First applicant is/was member of a Committee, but not of the one evaluating 3Co-applicant is member of the Committee(s) evaluating the proposal 4First applicant is member of the Committee(s) evaluating the proposal

20 For 15 % of SUBMITTED applications an applicant is a member of the evaluating Committee (Affinity=3, 4) % SUBMITTED APPLICATIONS AFFINITY APPLICANT - COMMITTEE

21 Probability to be granted increases with increasing affinity applicants-Committee AFFINITY APPLICANT - COMMITTEE % GRANTED APPLICATIONS

22 Logistic regression analysis: Affinity Applicant-Committee has a significant effect upon the probability to be granted MAXIMUM-LIKELIHOOD ANALYSIS-OF-VARIANCE TABLE (N=2,499) Source DF Chi-Square Prob ------------------------------------------------------------- INTERCEPT 1 18.47 0.0000 CITATION IMPACT APPLICANT 3 26.97 0.0000 ** Rel transdisc impact applicant 1 0.29 0.5926 AFFINITY APPLICANT-COMMITTEE 2 112.50 0.0000 ** Sum requested 1 45.47 0.0000 ** Institution applicant 4 25.94 0.0000 ** LIKELIHOOD RATIO 199 230.23 0.0638

23 (viii) Data must be accurate and verifiable

24 (ix) Assessed researchers must have the opportunity to check data and comment on outcomes

25 (x) A framework is needed to characterize and position bibliometric indicators and products

26 Contents 1 Boundaries of the playing field and rules of the game 2The multi-dimensional research assessment matrix 3 Towards more sophisticated indicators: combining big data sets

27 The Multi-Dimensional Research Assessment Matrix Expert Group on the Assessment of University-Based Research (AUBR, 2010)

28 Multi dimensional Research Assessment Matrix (Part) Unit of assessment PurposeOutput dimensions Bibliometric indicators Other indicators IndividualAllocate resources Research productivity PublicationsPeer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs DepartmentMonitor research programs Innovation and social benefit Actual citation impact Invitations for conferences InstitutionIncrease regional engagement Sustainabi- lity & Scale Internat. co- authorship External research income Research field Promotion, hiring Research infrastruct. citation prestige PhD com- pletion rates

29 Multi dimensional Research Assessment Matrix (Part) Unit of assessment PurposeOutput dimensions Bibliometric indicators Other indicators IndividualAllocate resources Research productivity PublicationsPeer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs DepartmentMonitor research programs Innovation and social benefit Actual citation impact Invitations for conferences InstitutionIncrease regional engagement Sustainabi- lity & Scale Internat. co- authorship External research income Research field Promotion, hiring Research infrastruct. citation prestige PhD com- pletion rates Read column- wise

30 Multi dimensional Research Assessment Matrix (Part) Unit of assessment PurposeOutput dimensions Bibliometric indicators Other indicators IndividualAllocate resources Research productivity PublicationsPeer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs DepartmentMonitor research programs Innovation and social benefit Actual citation impact Invitations for conferences InstitutionIncrease regional engagement Sustainabi- lity & Scale Internat. co- authorship External research income Research field Promotion, hiring Research infrastruct. citation prestige PhD com- pletion rates

31 Multi dimensional Research Assessment Matrix (Part) Unit of assessment PurposeOutput dimensions Bibliometric indicators Other indicators Individual Allocate resources Research productivity PublicationsPeer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs DepartmentMonitor research programs Innovation and social benefit Actual citation impact Invitations for conferences InstitutionIncrease regional engagement Sustainabi- lity & Scale Internat. co- authorship External research income Research field Promotion & hiring Research infrastruct. citation prestige PhD com- pletion rates

32 MD-RAM: Example 1 Individual Hiring/promotion Productivity & impact Individual Hiring/promotion Productivity & impact PhD date, place, supervisor; Invitations for conferences PhD date, place, supervisor; Invitations for conferences Publications in international jrnls; Actual citation impact Publications in international jrnls; Actual citation impact

33 Multi dimensional Research Assessment Matrix (Part) Unit of assessment PurposeOutput dimensions Bibliometric indicators Other indicators IndividualAllocate resources Research productivity PublicationsPeer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs Department Monitor research programs Innovation and social benefit Actual citation impact Invitations for conferences InstitutionIncrease regional engagement Sustainabi- lity & Scale Internat. co- authorship External research income Research field Promotion, hiring Research infrastruct. citation prestige PhD com- pletion rates

34 MD-RAM: Example 2 Research group Monitoring research program Scientific impact Research group Monitoring research program Scientific impact Collaborations Topicality Collaborations Topicality Publications in international jrnls; (Trend in) actual citation impact Publications in international jrnls; (Trend in) actual citation impact

35 Research assessment methodologies: Important considerations Methodology must be fit-for-purpose What is the primary problem to be solved? Be aware of unintended effects Change a methodology every 5-10 years What is an acceptable error rate? Wrong in individual cases benificiary for the system as a whole

36 Contents 1 Boundaries of the playing field and rules of the game 2 The multi-dimensional research assessment matrix 3Towards more sophisticated indicators: combining big data sets

37 Journal full text data Journal usage data Patents Trade jrnls Social media Journal articles + citations Books Conference Procs Newspapers Unit of assess- ment

38 (a) Downloads vs. Citations What do full article downloads measure?

39 Authors vs. readers Authors Readers ?

40 Hypothesis on degree of correlation between downloads and citations Readers Authors Readers Authors Strong Weak

41 Usage vs. citations per main field Societal Scientific ? PSYCHOL

42 (b) Patent citations to journal articles: The technological impact of research

43 The Technological Impact of Library Science Research: A Patent Analysis [Halevi et al, 2012] PATENTS (TotalPatent) 42 LIBRARY SCIENCE JOURNALS (Scopus) Citations by patent examiners and inventors

44 The patents focus on electronic information administration, navigation, and products and services management in commercial systems. The articles feature information retrieval and indexing, information and documents management systems which pertain to electronic and digital libraries development Cited articles: keywords in titles Citing patents: keywords in titles

45 (c) International scientific migration

46 International migration vs. co-authorship RelationshipDefinitionComment International co-authorship Authors from institutions located in different countries jointly publish a paper Country relates to where authors work, NOT to their nationality International migration A scientific author moves from one country to another

47 Map of countries with Ratio migration/collaboration > 1.2

48 TO COUNTRY FROM Country No. Co- authored papers No. Migrating authors Ratio % Migration / % Co- authorship PAKIND276118 3.6 PRTBRA1,971423 3.4 INDPAK27696 3.3 NLDIRN49280 3.1 USAIND12,0133,307 2.8 NLDPAK14521 2.8 CHNTWN3,9791,048 2.6 USAIRN3,039780 2.6 BRAPRT1,971352 2.4 MYSNGA12231 2.1 Language similarity drives migration stronger than it drives co-authorship Political tensions affect migration less than they affect co- authorship

49 (d) Citation context analysis Combining citation data from Scopus with full text article data from ScienceDirect

50 The use of contextual citations analysis to disclose the thematic and conceptual flow of cross- disciplinary research: the case of the Journal of Informetics 2007 (Gali Halevi et al., to be published, 2012)

51 Emerging sectional themes (OUTSIDE DISCIPLINE) The themes sequence in the word clouds below might suggest that the individual output evaluation done by structured peer review leads to an acknowledgment of the importance and evolution of networks rather than individuals INTRODUCTION FINDINGS & DISCUSSION CONCLUSIONS

52 (e) Book Citation Index Citation flows between books and journals

53 A Scholarly Book Citation Index: Approaches a)Add selected book series b)Add books from selected publishers c)Add selected individual book titles

54 Thank you for your attention!

55 Elsevier Bibliometric Research Program: www.ebrp.elsevier.com


Download ppt "The Multi-Dimensional Research Assessment Matrix Henk F. Moed Elsevier, Amsterdam, the Netherlands Seminar Research Evaluation in Practice, National Geographic."

Similar presentations


Ads by Google