1. NICTA Copyright 2013From imagination to impact Identifying Publication Types Using Machine Learning BioASQ Challenge Workshop A. Jimeno Yepes, J.G. Mork, A. R. Aronson Identifying Publication Types Using Machine Learning

2. NICTA Copyright 2013From imagination to impact Publication Types Define the genre of the article, e.g. Review Special type of MeSH Heading that are used to indicate what an article is rather than what it is about Citations can be indexed with more than one PT There are 61 PTs identified in the four MeSH Publication Characteristics (V) Tree top-level sub-trees that the indexers typically use 2

3. NICTA Copyright 2013From imagination to impact Publication Type: Review Example 3 This review attempts to highlight... PMID: 24024204 (September 12, 2013)

4. NICTA Copyright 2013From imagination to impact Publication Types PubMed allows for queries including publication type fields, e.g. Review[pt] PTs are available in the MEDLINE citation XML and ASCII formats 4 Clinical Trial, Phase II Journal Article Randomized Controlled Trial Research Support, N.I.H., Extramural Research Support, Non-U.S. Gov't

5. NICTA Copyright 2013From imagination to impact Motivation Indexing of citations with Publication Type (PT) as part of the Indexing Initiative at the US NLM Recommend PTs as part of the MTI (Medical Text Indexer) support tool MTI performed poorly on PTs in previous attempts and stopped suggesting PTs altogether on November 10, 2004 5

6. NICTA Copyright 2013From imagination to impact MTI in a nutshell 6

7. NICTA Copyright 2013From imagination to impact Machine learning motivation MTI showed poor results in Publication Type (PT) indexing in previous work Indexing of PTs can be seen as a text categorization task We have considered as a binary case. For a given PT the citations indexed with it are considered as positives and the rest as negative 7

8. NICTA Copyright 2013From imagination to impact Data set development Over time the indexing policy changes, consider the most recent indexing Selected citations Date Completed (date indexing was applied to the citation) ranging from January 1, 2009 to December 31, 2011 8

9. NICTA Copyright 2013From imagination to impact Data set development Data set obtained from the 2012 MEDLINE Baseline Repository (MBR) Query Tool http://mbr.nlm.nih.gov MBR allows us to randomly divide the list of PMIDs into Training (2/3) and Testing (1/3) sets 1,784,061 randomly selected PMIDs for Training and 878,718 for Testing 9

10. NICTA Copyright 2013From imagination to impact Data set development Filter out articles requiring special handling OLDMEDLINE, PubMed-not-MEDLINE, articles with no indexing, CommentOn, RetractionOf, PartialRetractionOf, UpdateIn, RepublishedIn, ErratumFor, and ReprintOf. Final data set: 1,321,512 articles for Training and 651,617 articles for Testing 10

11. NICTA Copyright 2013From imagination to impact Test set statistics Citations in test set: 651,617 Imbalance between positives and negatives 11 Publication TypeOccursAbbrevBaseline F 1 Case Reports51,037CR- Clinical Trial6,165CT- Congresses1,954CO 0.3397 Controlled Clinical Trial1,727CC- Editorial11,519ED- English Abstract46,471EA 0.0010 In Vitro4,284IV 0.1679 Meta-Analysis3,467MA 0.2674 Randomized Controlled Trial17,356RC- Review75,298RV-

12. NICTA Copyright 2013From imagination to impact Machine learning algorithms MTI ML: Support Vector Machine Stochastic Gradient Descent based on Hinge Loss (Sgd) Modified Huber Loss (Yeganova et al, 2011) (Mhl) AdaBoostM1 (C4.5 as based method) (Ada) Mallet: Naïve Bayes (NB) and Logistic Regression (LR) 12

13. NICTA Copyright 2013From imagination to impact Features Title and abstract text (Base) Base + Journal Unique Identifier, Author affiliations, Author Names, and Grant Agencies (additional features) (F) Base + bigrams (B) Base + additional features + bigrams (BF) AdaBoostM1 was not trained with bigrams due to time constraints 13

14. NICTA Copyright 2013From imagination to impact Results (F 1 measure) 14 CRCTCOCCEDEAIVMARCRV Mhl0.79480.12040.69970.05780.14520.57700.15490.70930.74640.7324 Mhl-F0.81310.11530.69990.06240.54260.81980.16100.72310.75440.7512 Mhl-B0.82910.09930.71130.01920.22900.63860.11460.76870.78400.7485 Mhl-BF0.83770.09090.70240.01920.55840.83180.11000.77330.79110.7660 Sgd0.80750.00580.69180.01030.08440.58980.07340.74100.77320.7579 Sgd-F0.82580.09430.70040.03800.34610.82550.15050.73100.76830.7685 Sgd-B0.82520.08700.71090.01820.11830.64250.10490.77420.78990.7582 Sgd-BF0.83920.08360.70890.01810.49390.83430.10050.77270.79100.7699 NB0.69850.02810.45080.00090.09100.42150.10560.31250.49360.6355 NB-F0.74610.00320.06520.00000.08890.51800.00120.00050.25440.5452 NB-B0.70070.00000.08820.00000.01480.08570.0000 0.09990.4330 NB-BF0.67470.00900.06520.00000.04430.21630.0000 0.05330.3039 LR0.80140.13190.69540.07540.17270.59180.15580.71000.74440.7466 LR-F0.81550.12470.69890.06330.54690.81980.15860.72690.75810.7473 LR-B0.83540.11160.70570.02800.21930.63570.13030.76550.78680.7592 LR-BF0.84110.10750.70140.02690.54420.83590.12280.77020.79210.7736 Ada0.80420.05750.65640.01020.23830.41800.07290.75180.77090.7088 Ada-F0.80800.05340.67740.01910.42740.78520.06530.75070.77380.7164

15. NICTA Copyright 2013From imagination to impact Methods/features comparison No clear winning method that works best for all of the Publication Types, echoing the findings for MeSH indexing Logistic Regression provides the highest F 1 measures for six of the ten PTs in our study Bigrams and additional features tend to perform better than using just title and abstract tokens 15

16. NICTA Copyright 2013From imagination to impact Naïve Bayes performance Naïve Bayes is far behind all of the other methods This effect already known (Rennie et al. 2003) is more dramatic when there is an imbalance between the classes This effect is more dramatic with a larger set of dependent features 16

17. NICTA Copyright 2013From imagination to impact ML performance indexing PTs Case Reports, Congresses, English Abstract, Meta-Analysis, Randomized Controlled Trial, and Review all have F 1 measures above 0.7 making them promising candidates for future integration into the indexing process The remaining PTs Clinical Trial, Controlled Clinical Trial, Editorial, and In Vitro all have F 1 measures too low for consideration at this time but provide the kernel for further research into improving their performance 17

18. NICTA Copyright 2013From imagination to impact English Abstract PT ML already high performance (F1: 0.8359) Indexing rule already in place: if an article has a title in brackets (meaning it was translated into English) and contains an abstract, it should receive the English Abstract Publication Type This PT is already automatically assigned using this rule and ML algorithms need to add more features explicitly 18

19. NICTA Copyright 2013From imagination to impact In Vitro PT In Vitro is one of the low performing terms In our error analysis, we find that in almost all of the false negatives that we manually reviewed, the information for designating the article as In Vitro was located in the Methods section of the full text of the article 19

20. NICTA Copyright 2013From imagination to impact Conclusions Evaluated the automatic assignment of PTs to MEDLINE articles based on machine learning For the majority (6 of 10) of PTs the performance is quite good with F 1 measures above 0.7 20

21. NICTA Copyright 2013From imagination to impact Conclusions In addition to the title and abstract text, further information provided from fields in the MEDLINE article result in improved performance Extend current work to include most of the remaining frequently used PTs and exploring the use of openly available full text from PubMed Central to see the impact in terms like In Vitro 21

22. NICTA Copyright 2013From imagination to impact Questions? MTI ML package http://ii.nlm.nih.gov/MTI_ML/index.shtml Publication Types data set http://ii.nlm.nih.gov/DataSets/index.shtml#2013_BioASQ 22