About use and misuse of impact factor and other journal metrics Dr Berenika M. Webster Strategic Business Manager 23 January 2009, Sydney

2 Outline INTRODUCTION IMPACT FACTOR – WHAT IT CAN CANNOT TELL YOU ABOUT A JOURNAL WHAT OTHER TOOLS AND INDICATORS ARE THERE? CONCLUSIONS

3 How to measure impact/influence of a publication? Measures of volume –Counts of … Measures of impact –Citations Measures of influence –Combine volume and impact Weighted, relative and normalised

4 Journal citation information has been used successfully by… Publishers Librarians Authors Research policy and funding agencies Information scientists Industry

5 Sources of bibliometric indicators of impact/influence of a publication Journal Citation Reports –Impact Factor (2 and 5-year) –Measures of obsolescence (cited half-life) and currency (citing half-life and immediacy index) –Eigenfactor Web of Science indexing and visualisation Citation reports (standard and custom) Indicators – journal, university, national

6 Journal inclusion: factors considered Publishing standards Editorial content International diversity Citation data analysis Self citation rates

7 Impact Factor – some headlines Worshiping false idols: The impact factor dilemma The power of the unrelenting impact factor - Is it a force for good or harm? Impact factor, impact, and smoke and mirrors Impact factor: Is it dragging science off course? The dreaded impact factor is back to haunt us! The impact factor or the fairy tale of impartiality The magic of the impact factor: unmasking of a phenomenon Is the "impact factor" killing the German language? The tyranny of the impact factor Nightmare impact factor

8 Impact Factor – what it does not measure The Journal Impact Factor was not designed for or intended to be used as a measure or proxy for the performance of individual papers or researchers. (Seglen, 1997) The Impact Factor does not measure the impact or influence of a journal, but of an average item published in that journal. (Leydesdorff, 2008) Specific IF requirements (say, n papers in journals with IF > 1) ignore field differences Summing up or combining IFs only make matters worse. As computer scientists say: GIGO

9 Impact Factor – where it can be helpful Global view of internationally influential journals, a vetted corpus Simple-to-understand calculation Does not reward age or size/frequency of output Gives insight to recent performance in field Citation counts matched to journal titles to collect as many citations as possible Long history of journal impact factors allows for time-series studies Generally reasonable results (journals recognised as influential occupy top ranks) Widely available and used over 30+ years (market success implies some utility for users)

10 Scenario 1: Journal editor asks… I want to know how my journal “stacks up” against competition. I want to know who are the most influential authors in my journal’s field in Australia. I want to know which institutions in Australia publish in the field covered by my journal. And which institution’s research has the biggest impact.

11 Scenario 2: Librarian asks… To rationalise my journal collection, I need to know in which journals staff at my university publish and which journals cite their publications. I want to identify the most influential journals in the field of Nanoscience and Nanotechnology in the last years. And important emerging journals.

12 Scenario 3: A researcher asks … I am looking for a post-doc position in the leading institution in the field of Nanotechnology. Who publishes in my field and whose publications have the greatest impact? I am looking for a research group conducting research in Nanotechnology for possible collaboration on my next project.

13 How does my journal stack up against the competition? Basic statistics from JCR (field delineation, influence, currency and obsolescence) Basic statistics in WoS (including h-index) Journal Performance Indicators data (benchmarking against field and world baselines) Essential Science Indicators (10-year perspective)

14 Nature Materials in JCR

15 Impact Factor (IF) Five Year Impact Factor (VIF) Numerator: citations to all items Denominator: citatable items only (arts and revs)

16 Journal self-citations –displays the contribution of journal self-citations to Impact Factor calculations –helps to identify journals with a narrow focus Journal Citation Reports 2007

17 Nature Materials over years Journal Citation Reports 2007

18 Category ranking and quartile distribution

19 Cited / citing journals Nature Materials Journal Citation Reports 2007

20 Journal metrics: WoS analytics

21 Journal Performance Indicators – 5-year view Citations per paper for subject category and Nature Neuroscience

22 Journal Performance Indicators – 5-year view Percentage of cited papers for subject category and Nature Neuroscience

23 Essential Science Indicators – 10 year view

24 Which regions conduct nanotechnology research? National Science Indicators, 2007

25 Which countries in AP publish nanotechnology papers? National Science Indicators, 2007

26 Which institutions in Australia conduct research in my area? Nanotechnology papers and citation in Australian institutions, University Science Indicators, 2007

27 and whose publications have the biggest impact… University Science Indicators, 2007

28 Who are the most influential authors in my field in Australia? National Citation Reports, 2007

29 in more detail… National Citation Reports, 2007

30 Which journals do my university researchers cite and which journals cite them? Citation Reports, custom data

31 Is my library subscribing to the right journals? Journal Performance Indicators, 2007

32 Other bibliometric journal performance measures –IF variants –h-index and its variants –Based on other data sources –Related to national performance Ranking lists –Organisation-based –National –Regional Publisher data –Subscriptions –Downloads

33 Conclusions “My measure is better than your measure” But different measures attempt to answer different questions, emphasize different aspects and nuances of a phenomenon The fallacy of the demand for single-number metrics that are all-revealing A lack of comprehensive, validated standards by which to compare different metrics or to show which combination is superior to others