MEASURING RESEARCHERS: FROM RESEARCH PUBLICATIONS AND COMMUNICATION TO RESEARCH EVALUATION Lucie Vavříková 1

Research Types basic research applied research innovation / industrial research Why do we do research? interest and curiosity societal demand profit 2 How much funding is spent per minute for R&D? $

Sharing research: scientific communication Why? To not reinvent the wheel How? Research outputs (results): Publication Application Other Channel? Journals, conferences, books / international databases of scientific literature Patent offices Other sources 3

Research outputs Dependent on scientific discipline: 4 Science & Medicine Research articles Review articles Letters & notes Technology & Computer science Research articles Conference- papers Patents Social sciences, arts, humanities Books Research articles

Research outputs – details per disciplne 5

What are the roles of scientific communication? What it can do for us? Sharing research Mirroring what’s happening in research…why you want to know what’s happening in research? Personal goals – advancing own research and career Societal – find answers to problems (cancer) Policy / management goals – understand how funding is spent, what it brings, under what conditions Research goals – understand how research is happening, what’s it’s organization, dynamics and characteristics 6

7

Scientific communication key agents: publications and citations Publications: articles in scientific journals Peer–reviewed content Question of the quality of the journal Quality indicators: reputation, Impact Factor, SJR, SNIP Science, Nature, Cell Citations: sign of scientific communication Citation is an indicator of use of knowledge/information Quantity of publications and citations  assumptions on quality of research  research evaluation 8

Research evaluation / measurement In science, quantity and quality are correlated. Dennis, 1954 Two traditions peer review (a process to understand and advance science) responsibility principle (tool for research management) 9

Peer review Traditional way of “doing research” (discourse) Peers: colleagues/experts in the field Challenges: Objectivity Keeping existing structures Use Selecting papers to journals, conferences Selecting projects to be funded Assessing the results of research projects and in general research work Evaluation of institutions, grant schemes etc.

Principle of responsibility (research management tool) Based on the need of policy makers and management Do we do the right thing (relevance)? Do we do it right (implementation)? Are there any results of those activities (impact)? Can we do it better, what should be done next time (efficiency, feedback)? Closing the gap of intervention logic Learning tool Justification of use of public money Challenges Multiple methods assessing impacts still cannot properly address: end user identification, multi-directional impacts (not easy to be isolated), delay in time, accidentality 11

Traditional method of research evaluation: bibliometrics Bibliometrics Use of mathematic and statistical methods to assess scientific communication Network analysis What is evaluated: a set of publications representing: journals, researchers – micro level Institutions, funding programmes – meso level country, region – macro level

Basic bibliometric methods Publication analysis How much is published by a unit in a time, field, region Citation analysis Citedness of publications  citation impact Co-citations Content analysis Co-word analysis

Challenge in assessing the quality of research across disciplines Every discipline differs in output production and its citedness Citation and publication behaviour patterns = benchmark 14 2 citations in medicine 2 citations in mathematics

Where do we take data from? 15

Where do we take data from? Citation indices Web of Science (Thomson Reuters) Scopus (Elsevier) Google Scholar Advanced research management tools Incites (Thomson Reuters) SciVal (Elsevier) SciMagoJR etc. 16

Practical use cases – journal metrics Assign “quality mark” to the journal Helps researchers to navigate in the ocean of journals Serve as a proxy for evaluating quality of output A researcher is that good in how good journal is achieved for the publication Metrics: Journal Impact Factor, SJR, SNIP Misuse and challenges: Expected impact considered instead of real impact Disregarding discipline citation patterns Citation ethics 17

Practical use cases - researcher metrics Help understand the quality of output researcher produce For promotion For academic degrees For building teams For selecting a PhD supervisor For proving sufficient seniority in grant and project applications For salary As a unit and/or proxy for team/institution evaluation Metrics: H-Index Misuse and challenges: Comparing incomparable Not taking into account other roles of researcher (teaching) Disregarding discipline citation patterns 18

Practical use case – national evaluation Helps decision making on funding allocation via non/direct mechanism Takes stock of research in the country to provide input for decision making (intervention logic) Czech Republic – Information System for R&D&I Yearly collects outputs defined in methodology With the help of journal metrics (sic!) assigns points to individual outputs Using peer review assesses the best outputs as nominated by institutions 19

Practical use case – institutional evaluation How would you assess an institution? What outputs would you take into account? Why? Who?..? 20

Live demo ScimagoJR Scopus SciVal 21

Thank you for your attention. Any questions? 22