1. FROM BIOMEDICAL RESEARCH TO HEALTH IMPROVEMENT Grant Lewison grantlewison@aol.co.uk and grant.lewison@kcl.ac.uk evaluametrics Evaluation of research through publication metrics

2. PUTTING BIOMEDICAL RESEARCH INTO PRACTICE evaluametrics Evaluation of research through publication metrics

3. OTHER CITATIONS: CLINICAL GUIDELINES - 2 Example: cancer clinical guidelines (BJC, 98, 1944) Sample of 43 guidelines (up to 2006) Comparison with world cancer research, 1999-2001 because peak PY of cited refs is then: evaluametrics Evaluation of research through publication metrics

4. OTHER CITATIONS: CLINICAL GUIDELINES - 3 UK papers are over-cited by x3, but some other countries’ papers are also over-cited. DE and JP much under-cited: evaluametrics Evaluation of research through publication metrics

5. OTHER CITATIONS: CLINICAL GUIDELINES - 4 Comparisons can also be made of impact of papers from other countries, e.g., six Swedish medical schools: evaluametrics Evaluation of research through publication metrics Göteborg (GOT) papers are much better cited than ones from Karolinska (KAR), probably because the latter are more basic and the former more clinical

6. OTHER CITATIONS: CLINICAL GUIDELINES - 5 What cancer sites are represented in papers cited by UK guidelines, and do they correlate with disease burden? Data on burden for 2004 from WHO. Some cancers (e.g., lung - LUN, prostate - PRO, pancreatic - PAN) appear less among guideline refs than would be expected, and others (breast - MAM, melanoma - MEL) appear more. evaluametrics Evaluation of research through publication metrics

7. OTHER CITATIONS: POLICY DOCUMENTS - 2 Example of erpho report: January 2006 Issue 16 ISSN 1477-7304 HIV and sexually transmitted infections in the East of England Report has 33 references of which 10 journal papers Example of report from No 10: Strategy Unit Alcohol Harm Reduction project. Interim Analytical Report Nov 2003 This report had over 200 references of which ~70 journal papers: mainly social sciences (NOT biomedical), and heavily from UK

8. evaluametrics Evaluation of research through publication metrics OTHER CITATIONS: POLICY DOCUMENTS - 3 UK Department of Health has many reports, e.g.: Sport and Exercise Medicine A Fresh Approach published May 2012, includes 39 refs to journal papers some only have a few refs, e.g.: Improving Outcomes: A Strategy for Cancer January 2011 has 36 refs of which 6 are journal papers, but this is a policy paper signed by Secretary of State so has more weight

9. evaluametrics Evaluation of research through publication metrics OTHER CITATIONS: TEXTBOOKS - 2 Three individual chapters analysed, with FR, IT and UK authors FR: The effect of cell-matrix interactions and aging on the malignant process (2007; basic; RL = 3.4) IT: Lung cancer: does age affect treatment strategy? (2009; clinical; RL = 1.0) UK: Supportive, Palliative and End-of-Life Care for Older Breast Cancer Patients (2010; clinical; RL = 1.1) Examination of their references: age, national distribution, citations

10. evaluametrics Evaluation of research through publication metrics OTHER CITATIONS: TEXTBOOKS - 3 Cumulative %s of papers older than given delay Some references cited by French author are very old (20% are 16+ years). Ones cited by UK author are the youngest – but result may not be typical

11. evaluametrics Evaluation of research through publication metrics OTHER CITATIONS: TEXTBOOKS - 4 Percent of cited references from different countries Each citing author gives preference to own country’s papers, but this is most extreme for France, which over- cites by x 8, more than UK (x 4) and Italy (x3), relative to presence in world cancer research

12. evaluametrics Evaluation of research through publication metrics So the cited references seem to be carefully selected as high- impact papers OTHER CITATIONS: TEXTBOOKS - 5 The references appear to be highly cited papers For the clinical papers cited by IT author, mean ACI in 5 years = 83 cites (cf. world mean in cancer of 22)

13. BBC WEBSITE COVERAGE OF CANCER RESEARCH evaluametrics Evaluation of research through publication metrics Story gives the date, the name of the journalist, the institution carrying out the work and the name of the journal – often also one of the leading researchers – so that the cited paper can be identified easily

14. evaluametrics Evaluation of research through publication metrics COMPARISON OF BBC AND DISEASE BURDEN Note log-log scales Many cancers are under-reported but this may reflect the paucity of research papers. Note relative popularity of breast cancer stories and those on skin and cervical cancer research