1. 1 Open Source Text Mining Text Mining 2003 @ SDM03 Cathedral Hill Hotel, San Francisco Hinrich Schütze, Enkata May 3, 2003

2. 2 Motivation Open source used to be a crackpot idea. Bill Gates on linux (1999.03.24): “I really don't think in the commercial market, we'll see it in any significant way.” MS 10-Q quarterly filing (2003.01.31): “The popularization of the open source movement continues to pose a significant challenge to the company's business model.” Open source is an enabler for radical new things Google Ultra-cheap web servers Free news Free email Free … Class projects Walmart pc for $200

3. 3 GNU-Linux

4. 4 Web Servers: Open Source Dominates Source: Netcraft

5. 5 Motivation (cont.) Text mining has not had much impact. Many small companies & small projects No large-scale adoption Exception: text-mining-enhanced search Text mining could transform the world. Unstructured → structured Information explosion Amount of information has exploded Amount of accessible information has not Can open source text mining make this happen?

6. 6 Unstructured vs Structured Data Prabhakar Raghavan, Verity

7. 7 Business Motivation High cost of deploying text mining solutions How can we lower this cost? 100% proprietary solutions Require re-invention of core infrastructure Leave fewer resources for high-value applications built on top of core infrastructure

8. 8 Definitions Open source Public domain, bsd, gpl (gnu public license) Text mining Like data mining but for text NLP (Natural Language Processing) subdiscipline Has interesting applications now More than just information retrieval / keyword search Usually: some statistical, probabilistic or frequentistic component

9. 9 Text Mining vs. NLP (Natural Language Processing) What is not text mining: speech, language models, parsing, machine translation Typical text mining: clustering, information extraction, question answering Statistical and high volume

10. 10 Text Mining: History 80s: Electronic text gives birth to Statistical Natural Language Processing (StatNLP). 90s: DARPA sponsors Message Understanding Conferences (MUC) and Information Extraction (IE) community. Mid-90s: Data Mining becomes a discipline and usurps much of IE and StatNLP as “text mining”.

11. 11 Text Mining: Hearst’s Definition Finding nuggets Information extraction Question answering Finding patterns Clustering Knowledge discovery Text visualization

12. 12 foodscience.com-Job2 JobTitle: Ice Cream Guru Employer: foodscience.com JobCategory: Travel/Hospitality JobFunction: Food Services JobLocation: Upper Midwest Contact Phone: 800-488-2611 DateExtracted: January 8, 2001 Source: www.foodscience.com/jobs_midwest.html OtherCompanyJobs: foodscience.com-Job1 Information Extraction

13. 13 Knowledge Discovery: Arrowsmith Goal: Connect two disconnected subfields of medicine. Technique Start with 1st subfield Identify key concepts Search for 2nd subfield with same concepts Implemented in Arrowsmith system Discovery: magnesium is potential treatment for migraine

14. 14 Knowledge Discovery: Arrowsmith

15. 15 When is Open Source Successful? “Important” problem Many users (operating system) Fun to work on (games) Public funding available (OpenBSD, security) Open source author gains fame/satisfaction/immortality/community Adaptation A little adaptation is easy Most users do not need any adaptation (out of the box use) Incremental releases are useful Cost sharing without administrative/legal overhead Dozens of companies with significant interest in linux (ibm …) Many of these companies contribute to open source This is in effect an informal consortium A formal effort probably would have killed linux. Same applies to text mining? Also: bugs, security, high-availability, ideal for consulting & hardware companies like IBM

16. 16 When is Open Source Not Successful? Boring & rare problem Print driver for 10 year old printer Complex integrated solutions QuarkXPress ERP systems Good UI experience for non-geeks Apple Microsoft Windows (at least for now)

17. 17 Text Mining and Open Source Pro Important problem: fame, satisfaction, immortality, community can be gained Pooling of resources / critical mass Con Non-incremental? Most text mining requires significant adaptation. Most text mining requires data resources as well as source code. The need for data resources does not fit well into the open source paradigm.

18. 18 Text Mining Open Source Today Lucene Excellent for information retrieval, but not much text mining. Rain/bow, Weka, GTP, TDMAPI Text mining algorithms / infrastructure, no data resources NLTK NLP toolkit, some data resources WordNet, DMOZ Excellent data resources, but not enough breadth/depth.

19. 19 Open Source with Open Data Spell checkers (e.g., emacs) Antispam software (e.g., spamassassin) Named entity recognition (Gate/Annie) Free version less powerful than in-house

20. 20 SpamAssassin: Code + Data

21. 21 Open Data Resources: Examples SpamAssassin Classification model for spam Named entity recognition Word lists, dictionaries Information extraction Domain model, taxonomies, regular expressions Shallow parsing Grammars

22. 22 Code Data ? ProprietaryOpen Source No Resources Needed Significant Resources Needed Code vs Data Text Classification N. Entity Recognition Information Extraction Complex&Integrated SW Good UI Design Linux Web Servers Spam Filtering Spell Checkers

23. 23 Open Source with Data: Key Issues Can data resources be recycled? Problems have to be similar. More difficult than one would expect: my first attempt failed (medline/reuters). Next: case study Assume there is a large library of data resources available. How do we identify the data resources that can be recycled? How do we adapt them? How do we get from here to there? Need incremental approach that is sustained by successes along the way.

24. 24 Text Mining without Data Resources Premise: “Knowledge-poor” text mining taps small part of potential of text mining. Knowledge-poor text mining examples Clustering Phrase extraction First story detection Many success stories

25. 25 Case Study: ODP -> Reuters Case Study: Train on ODP Apply to Reuters

26. 26 Case Study: Text Classification Key Issues for text classification Show that text classifiers can be recycled How can we select reusable classifiers for a particular task? How do we adapt them? Case Study Train classifiers on open directory (ODP) 165,000 docs (nodes), crawled in 2000, 505 classes Apply classifiers to Reuters RCV1 780,000 docs, >1000 classes Hypothesis: A library of classifiers based on ODP can be recycled for RCV1.

27. 27 Experimental Setup Train 505 classifiers on ODP Apply them to Reuters Compute chi 2 for all ODP x Reuters pairs Evaluate n pairs with the best chi 2 Evaluation Measures Area under ROC curve Plot false positive rate vs true positive rate Compute area under the curve Average precision Rank documents, compute precision for each rank Average for all positive documents Estimated based on 25% sample

28. 28 Japan: ODP -> Reuters

29. 29 Some Results

30. 30 BusIndTraMar0 / I76300: Ports

31. 31 Discussion Promising results These are results without any adaptation. Performance expected to be much better after adaptation.

32. 32 Discussion (cont) Class relationships are m:n, not 1:1 Reuters: GSPO SpoBasCol0 SpoBasMinLea0 SpoBasReg0 SpoHocIceLeaNatPla0 SpoHocIceLeaPro0 ODP: RegEurUniBusInd0 (UK industries) I13000 (petroleum & natural gas) I17000 (water supply) I32000 (mechanical engineering) I66100 (restaurants, cafes, fast food) I79020 (telecommunications) I9741105 (radio broadcasting)

33. 33 Why Recycling Classifiers is Difficult Autonomous vs relative decisions ODP Japan classifier w/o modifications has high precision, but only 1% recall on RCV1! Most classifiers are tuned for optimal performance in embedded system. Tuning decreases robustness in recycling. Tokenization, document length, numbers Numbers throw off medline vs. non-medline categorizer (financial classified as medical) Length-sensitive multinomial Naïve Bayes: nonsensical results

34. 34 Specifics What would an open source text classification package look like? Code Text mining algorithms Customization component To adapt recycled data resources Creation component To create new data resources Data Recycled data resources Newly created data resources Pick a good area Bioinformatics: genes / proteins Product catalogs

35. 35 Other Text Mining Areas Named entity recognition Information extraction Shallow parsing

36. 36 Data vs Code What about just sharing training sets? Often proprietary What about just sharing models? Small preprocessing changes can throw you off completely Share (simple?) classifier cum preprocessor and models Still proprietary issues

37. 37 Open Source & Data Sanitized& Enhanced Code+Data Enhanced Code+Data adapt PublicProprietar y Code+Data V1.0 Code+Data V1.1 publish sanitize new release

38. 38 Free Riders? Open source is successful because it makes free riding hard. Viral nature of GPL. Harder to achieve for some data resources Download models Apply to your data Retrain You own 100% of the result Less of a problem for dictionaries and grammars

39. 39 Data Licenses Open Directory License http://rdf.dmoz.org/license.html Bsd flavor Wordnet http://www.cogsci.princeton.edu/~wn/license.shtml Copyright No license to sell derivative works? Some criteria for derivative works Substantially similar (seinfeld trivia) Potential damage to future marketing of derivative works

40. 40 Code vs Data Licenses Some similarity If I open-source my code, then I will benefit from bug fixes & enhancements written by others. If I open-source my data resource, then my classification model may become more robust due to improvements made by others. Some dissimilarity Code is very abstract: few issues with proprietary information creeping in. Text mining resources are not very abstract: there is a potential of sensitive information leaking out.

41. 41 Areas in Need of Research How to identify reusable text mining components ODP/Reuters case study does not address this. Need (small) labeled sample to be able to do this? How to adapt reusable text mining components Active learning Interactive parameter tweaking? Combination of recycled classifier and new training information Estimate performance Most estimation techniques require large labeled samples. The point is to avoid construction of a large labeled sample. Create viral license for data resources.

42. 42 Summary Many interesting research issues Need institution/individual to take the lead Need motivated network of contributors data resource contributors source code contributors Start with small & simple project that proves idea If it works … text mining could become an enabler on a par with linux.

43. 43 More Slides

44. 44 RegAsiJap0JAP0.860.62 RegAsiPhi0PHLNS0.910.56 RegAsiIndSta0INDIA0.850.53 SpoSocPla0CCAT0.600.53 RegEurRus0CCAT0.580.51 RegEurRus0RUSS0.850.51 SpoSocPla0GSPO0.780.42 SpoBasReg0GSPO0.750.33 RegAsiIndSta0MCAT0.560.32 SpoBasPla1GSPO0.800.31 SpoBasCol0GSPO0.780.31 SpoBasCol1GSPO0.740.26 RegEurSlo0SLVAK0.860.25 SpoBasPla0GSPO0.770.24 RegEurRus0MCAT0.490.23 BusIndTraMar0I763000.810.23 SpoHocIceLeaPro0GSPO0.710.20 SpoBasMinLea0GSPO0.710.20 RegMidLeb0LEBAN0.830.19 RecAvi0I364000.740.18 RegSou0BRAZ0.840.18 RegAsiHonBus0HKONG0.660.18 SpoMotAut0GSPO0.670.18 SpoHocIceLeaNatPla0GSPO0.720.17 SocPol0EEC0.850.17 RegAsiIndSta0M140.590.17 RegAsiChiPro0CHINA0.670.17 RecAvi0I36400100.770.17 SpoFooAmeColNca1GSPO0.720.17 SocPol0G150.860.16 RegEurBul0BUL0.720.15 RegAsiIndPro0INDON0.720.13 SpoSocPla0UK0.490.12 RegEurUkr0UKRN0.730.11 RegEurRus0GPOL0.480.11 RegEurPolVoi0POL0.670.11 RegAsiIndSta0M1410.610.10 SpoFooAmeNflPla0GSPO0.650.09 RegEurGerSta0GFR0.560.09 RegEurFra0FRA0.540.09 RegCar0CUBA0.760.09 RegEurUniBusInd0C180.590.08 RegEurUniEngEss0I662000.720.08 RegSou0PERU0.880.08 ComHar0C220.610.08 RegMidTur0TURK0.690.08 RegAsiIndSta0M130.560.08 RegEurUniBusInd0C1810.590.07 RegNorUniCalLocPxx0LATV0.640.07 RegEurRus0GVIO0.520.07 SpoSocPla0ITALY0.580.07 RegEurUniSco0GSPO0.540.07 RegEurNet0NETH0.650.07 RegEurRus0GDIP0.460.07 ArtMusStyCouBan0GENT0.520.07 RegEurRus0BYELRS0.920.06 BusIndTraMar0C240.540.06 BusIndTraMar0I740000.720.06 RegNorMexSta0I763000.580.06 SpoHocIceLeaNatPla0CANA0.540.06 RegSou0MRCSL1.000.06 SocRelBud0GREL0.570.05 RegEurBel0FRA0.490.05 SpoSocPla0FRA0.500.05 RegEurUniBusInd0I65400050.690.05 RegNorCanQueLoc0FRA0.460.05 RegEurGerSta0GSPO0.450.05 RegAsiIndSta0M1310.610.05 RegAsiPak0SHAJH0.760.05 SpoSocPla0GFR0.480.05 RegSou0PARA0.900.04 RegEurUniBusInd0I97411090.590.04 RegSou0BOL0.900.04 RegEurRus0UKRN0.830.04 SpoSocPla0SPAIN0.610.04 NewOnlCnn0BAH0.560.04 ArtAniVoi0I971000.700.03 RegEurRus0NATO0.750.03 RegEurRus0GDEF0.550.03 SpoSocPla0MONAC0.870.03 SciEarPal0GSCI0.420.03 RegEurRom0ROM0.570.03 RegAsiPhi0I850000.660.03 SpoBasReg0SPAIN0.590.03 BusIndTraMar0USSR0.470.03 SpoSocPla0NETH0.540.03 SpoFooAmeNflPla0CANA0.480.03 RegEurRus0AZERB0.940.03 SciBioTaxTaxPlaMagMag0ECU0.540.03 RegNorUniCalLocPxx0I415000.650.02 RegEurRus0TADZK0.950.02 RegEurUniBusInd0I81502060.710.02 RegEurUniBusInd0I815020.580.02 RegSou0URU0.880.02 RegEurUniBusInd0I503000.740.02 RegEurUniBusInd0I371000.790.02 RefFlaReg0GUREP0.690.02 SciBioTaxTaxPlaMagMag0I01001440.580.02 NewOnlCnn0GWEA0.660.02 RegEurUniBusInd0I850000.570.02 ArtCelMxx0I971000.660.02 SpoMotAut0SMARNO0.880.02 RegEurUniBusInd0I50200220.790.02 NewOnlCnn0DOMR0.550.02 ArtMusStyCouBan0GPRO0.450.02 RegEurUniEngEss0I839540.660.02 SpoBasReg0GREECE0.510.02 RegEurRus0GRGIA0.840.02 RegEurRus0KAZK0.820.02 RegEurNet0M1420.450.02 RegEurUniBusInd0I832000.670.01 NewOnlCnn0BELZ0.500.01 RegEurUniBusInd0C340.490.01 RegEurUniEngEss0I820020.560.01 SpoBasReg0ISRAEL0.380.01 RegEurUniBusInd0I834000.730.01 RegEurUniBusInd0I839540.670.01 RegEurPolVoi0FIN0.580.01 RegEurRus0USSR0.820.01 RegEurUniBusInd0I97411050.580.01 RegEurUniBusInd0I328520.800.01 RegEurUniBusInd0I839400.630.01 BusIndTraMar0BUL0.370.01 RegEurUniBusInd0I610000.680.01 BusIndTraMar0ESTNIA0.600.01 NewOnlCnn0GABON0.460.01 NewOnlCnn0CVI0.700.01 SciBioTaxTaxAniChoAve0GENV0.450.01 SpoMotAut0MONAC0.710.01 ArtCelBxx0I971000.640.01 SpoBasReg0TURK0.460.01 BusIndTraMar0PORL0.570.01 SpoBasReg0CRTIA0.480.01 RegEurUniBusInd0I951000.650.01 BusIndTraMar0CRTIA0.410.01 BusIndTraMar0UKRN0.430.01 ArtCelLxx0I971000.600.01 RegEurRus0MOLDV0.780.01 RegSou0SURM0.800.01 BusIndTraMar0LATV0.600.01 BusIndTraMar0ALB0.240.01 BusIndTraMar0LITH0.580.01 ArtCelSxx0I971000.630.01 RegEurUniBusInd0I160000.590.01 SpoBasCol0E710.420.01 SciBioTaxTaxPlaMagMag0BELZ0.530.01 ArtMusStyCouBan0GOBIT0.530.01 BusFinBanBanReg0C1730.680.01 RegEurRus0ARMEN0.850.01 RegEurRus0I224710.660.01 RegEurRus0TURKM0.860.01 BusIndTraMar0ROM0.400.01 BusIndTraMar0TUNIS0.670.00 RegAsiChiPro0I50200060.760.00 ArtTelNet0I97411050.670.00 BusIndTraMar0YEMAR0.490.00 BusIndTraMar0CYPR0.400.00 RefFlaReg0SLVNIA0.570.00 RegEurUniEngEss0I97411050.570.00 RegEurRus0KIRGH0.830.00 RegCar0GTOUR0.550.00 BusIndTraMar0UAE0.480.00 NewOnlCnn0BERM0.520.00 BusIndTraMar0NAMIB0.480.00 BusIndTraMar0JORDAN0.360.00 RecAvi0C3130.420.00 BusIndTraMar0MOZAM0.510.00 RegEurUniBusInd0I662000.660.00 BusIndTraMar0SILEN0.340.00 RegMidLeb0I97411050.540.00 RegAsiHonBus0I814000.610.00 RefFlaReg0WORLD0.430.00 RegNorUniCalLocVxx0C3130.390.00 RegAsiHonBus0I647000.720.00 RefFlaReg0UPVOLA0.580.00 SciBioTaxTaxPlaMagMag0I01002160.660.00 RegAsiHonBus0I36400480.700.00 SciBioTaxTaxAniChoAve0AARCT0.530.00 RegSou0I50200510.840.00 NewOnlCnn0TCAI0.00

45. 45 Resources http://www-csli.stanford.edu/~schuetze (this talk, some additional material) http://www-csli.stanford.edu/~schuetze Source of Gates quote: http://www.techweb.com/wire/story/TWB19990324S0014 http://www.techweb.com/wire/story/TWB19990324S0014 Kurt D. Bollacker and Joydeep Ghosh. A scalable method for classifier knowledge reuse. In Proceedings of the 1997 International Conference on Neural Networks, pages 1474-79, June 1997. (proposes measure for selecting classifiers for reuse) W.Cohen, D.Kudenko: Transferring and Retraining Learned Information Filters, Proceedings of the Fourteenth National Conference on Artificial Intelligence, AAAI 97. (transfer within the same dataset)Transferring and Retraining Learned Information Filters Kurt D. Bollacker and Joydeep Ghosh. A supra-classifier architecture for scalable knowledge reuse. In The 1998 International Conference on Machine Learning, pp. 64-72, July 1998. (transfer within the same dataset) Motivation of open source contributors: http://newsforge.com/newsforge/03/04/19/2128256.shtml?tid =11, http://cybernaut.com/modules.php?op=modload&name=News&f ile=article&sid=8&mode=thread&order=0&thold=0 http://newsforge.com/newsforge/03/04/19/2128256.shtml?tid =11