Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 SIMS 290-2: Applied Natural Language Processing Marti Hearst October 30, 2006 Some slides by Preslav Nakov and Eibe Frank.

Similar presentations

Presentation on theme: "1 SIMS 290-2: Applied Natural Language Processing Marti Hearst October 30, 2006 Some slides by Preslav Nakov and Eibe Frank."— Presentation transcript:

1 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst October 30, 2006 Some slides by Preslav Nakov and Eibe Frank

2 2 Today The Newsgroups Text Collection WEKA: Exporer WEKA: Experimenter Python Interface to WEKA

3 3 Source: originally collected by Ken Lang Content and structure: approximately 20,000 newsgroup documents –19,997 originally –18,828 without duplicates partitioned evenly across 20 different newsgroups we are only using a subset (6 newsgroups) Some categories are strongly related (and thus hard to discriminate): 20 Newsgroups Data Set comp.sys.mac.hardware sci.crypt sci.electronics misc.forsaletalk.politics.misc talk.politics.guns talk.politics.mideast talk.religion.misc alt.atheism soc.religion.christian computers

4 4 Sample Posting: “talk.politics.guns” From: (C. D. Tavares) Subject: Re: Congress to review ATF's status In article, (Larry Cipriani) writes: > WASHINGTON (UPI) -- As part of its investigation of the deadly > confrontation with a Texas cult, Congress will consider whether the > Bureau of Alcohol, Tobacco and Firearms should be moved from the > Treasury Department to the Justice Department, senators said Wednesday. > The idea will be considered because of the violent and fatal events > at the beginning and end of the agency's confrontation with the Branch > Davidian cult. Of course. When the catbox begines to smell, simply transfer its contents into the potted plant in the foyer. "Why Hillary! Your government smells so... FRESH!" -- --If you believe that I speak for my company, OR write today for my special Investors' Packet... reply from subject signature Need special handling during feature extraction… … writes:

5 5 The 20 Newsgroups Text Collection WEKA: Exporer WEKA: Experimenter Python Interface to WEKA WEKA: Real-time Demo

6 6 Slide adapted from Eibe Frank's WEKA: The Bird Copyright: Martin Kramer (, University of Waikato, New

7 7 Slide by Eibe Frank WEKA: the software Waikato Environment for Knowledge Analysis Collection of state-of-the-art machine learning algorithms and data processing tools implemented in Java Released under the GPL Support for the whole process of experimental data mining Preparation of input data Statistical evaluation of learning schemes Visualization of input data and the result of learning Used for education, research and applications Complements “Data Mining” by Witten & Frank

8 8 Slide by Eibe Frank Main Features 49 data preprocessing tools 76 classification/regression algorithms 8 clustering algorithms 15 attribute/subset evaluators + 10 search algorithms for feature selection 3 algorithms for finding association rules 3 graphical user interfaces “The Explorer” (exploratory data analysis) “The Experimenter” (experimental environment) “The KnowledgeFlow” (new process model inspired interface)

9 9 Slide by Eibe Frank Projects based on WEKA Incorporate/wrap WEKA GRB Tool Shed - a tool to aid gamma ray burst research YALE - facility for large scale ML experiments GATE - NLP workbench with a WEKA interface Judge - document clustering and classification Extend/modify WEKA BioWeka - extension library for knowledge discovery in biology WekaMetal - meta learning extension to WEKA Weka-Parallel - parallel processing for WEKA Grid Weka - grid computing using WEKA Weka-CG - computational genetics tool library

10 10 Slide by Eibe Frank The WEKA Project Today (2006) Funding for the next two years Goal of the project remains the same People 6 staff 2 postdocs 3 PhD students 3 MSc students 2 research programmers

11 11 Slide adapted from Eibe Frank's WEKA: The Software Toolkit Machine learning/data mining software in Java GNU License Used for research, education and applications Complements “Data Mining” by Witten & Frank Main features: data pre-processing tools learning algorithms evaluation methods graphical interface (incl. data visualization) environment for comparing learning algorithms

12 12 WEKA: Terminology Some synonyms/explanations for the terms used by WEKA, which may differ from what we use: Attribute: feature Relation: collection of examples Instance: collection in use Class: category

13 13 Slide adapted from Eibe Frank's WEKA GUI Chooser java -Xmx1000M -jar weka.jar

14 14 Slide adapted from Eibe Frank's Our Toy Example We demonstrate WEKA on a simple example: 3 categories from “Newsgroups”: –, –, – 20 documents per category features: – words converted to lowercase – frequency 2 or more required – stopwords removed

15 15 Slide adapted from Eibe Frank's Explorer: Pre-Processing The Data WEKA can import data from: files: ARFF, CSV, C4.5, binary URL SQL database (using JDBC) Pre-processing tools (filters) are used for: Discretization, normalization, resampling, attribute selection, transforming and combining attributes, etc.

16 16 List of attributes (last: class variable) Frequency and categories for the selected attribute Statistics about the values of the selected attribute Classification Filter selection Manual attribute selection Statistical attribute selection Preprocessing The Preprocessing Tab

17 17 Slide adapted from Eibe Frank's Explorer: Building “Classifiers” Classifiers in WEKA are models for: classification (predict a nominal class) regression (predict a numerical quantity) Learning algorithms: Naïve Bayes, decision trees, kNN, support vector machines, multi-layer perceptron, logistic regression, etc. Meta-classifiers: cannot be used alone always combined with a learning algorithm examples: boosting, bagging etc.

18 18 Choice of classifier The attribute whose value is to be predicted from the values of the remaining ones. Default is the last attribute. Here (in our toy example) it is named “class”. Cross-validation: split the data into e.g. 10 folds and 10 times train on 9 folds and test on the remaining one The Classification Tab

19 19 Choosing a classifier

20 20

21 21 False: Gaussian True: kernels (better) displays synopsis and options numerical to nominal conversion by discretization outputs additional information

22 22

23 23

24 24 all other numbers can be obtained from it different/easy class accuracy

25 25 Contains information about the actual and the predicted classification All measures can be derived from it: accuracy: (a+d)/(a+b+c+d) recall: d/(c+d) => R precision: d/(b+d) => P F-measure: 2PR/(P+R) false positive (FP) rate: b/(a+b) true negative (TN) rate: a/(a+b) false negative (FN) rate: c/(c+d) These extend for more than 2 classes: see previous lecture slides for details Confusion matrix predicted –+ true –ab +cd

26 26 Outputs the probability distribution for each example Predictions Output

27 27 Probability distribution for a wrong example: predicted 1 instead of 3 Naïve Bayes makes incorrect conditional independence assumptions and typically is over-confident in its prediction regardless of whether it is correct or not. Predictions Output

28 28 Error Visualization

29 29 Error Visualization Little squares designate errors Axes show example number

30 30 Running on Test Set

31 31 Slide adapted from Eibe Frank's Find which attributes are the most predictive ones Two parts: search method: –best-first, forward selection, random, exhaustive, genetic algorithm, ranking evaluation method: –information gain, chi-squared, etc. Very flexible: WEKA allows (almost) arbitrary combinations of these two Explorer: Attribute Selection

32 32 Individual Features Ranking

33 33 Individual Features Ranking

34 34 ??? random number seed Individual Features Ranking

35 35 Slide adapted from Jakulin, Bratko, Smrke, Demšar and Zupan's feature correlation 2-Way Interactions Feature Interactions (Advanced) C BA category feature importance of feature B importance of feature A

36 36 Slide adapted from Jakulin, Bratko, Smrke, Demšar and Zupan's 3-Way Interaction: What is common to A, B and C together; and cannot be inferred from pairs of features. Feature Interactions (Advanced) C BA category feature importance of feature B importance of feature A

37 37 Slide adapted from Guozhu Dong's Feature Subsets Selection Problem illustration Full set Empty set Enumeration Search Exhaustive/Complete (enumeration/branch&bounding) Heuristic (sequential forward/backward) Stochastic (generate/evaluate) Individual features or subsets generation/evaluation

38 38 Features Subsets Selection

39 39 17,309 subsets considered 21 attributes selected Features Subsets Selection

40 40 Saving the Selected Features All we can do from this tab is to save the buffer in a text file. Not very useful... But we can also perform feature selection during the pre-processing step... (the following slides)

41 41 Features Selection on Preprocessing

42 42 Features Selection on Preprocessing

43 43 Features Selection on Preprocessing 679 attributes: 678 + 1 (for the class)

44 44 Features Selection on Preprocessing Just 22 attributes remain: 21 + 1 (for the class)

45 45 Run Naïve Bayes With the 21 Features higher accuracy 21 Attributes

46 46 different/easy class accuracy (AGAIN) Naïve Bayes With All Features ALL 679 Attributes (repeated slide)

47 47 WEKA has weird naming for some algorithms Here are some translations: Naïve Bayes: weka.classifiers.bayes.NaiveBayes Perceptron: weka.classifiers.functions.VotedPerceptron Decision tree: weka.classifiers.trees.J48 Support vector machines: weka.classifiers.functions.SMO k nearest neighbor: weka.classifiers.lazy.IBk Some of these are more sophisticated versions of the classic algorithms e.g. the classic Naïve Bayes seems to be missing A good alternative is the Multinomial Naïve Bayes model Some Important Algorithms

48 48 The 20 Newsgroups Text Collection WEKA: Explorer WEKA: Experimenter Python Interface to WEKA WEKA: Real-time Demo

49 49 Slide adapted from Eibe Frank's Experimenter makes it easy to compare the performance of different learning schemes Problems: classification regression Results: written into file or database Evaluation options: cross-validation learning curve hold-out Can also iterate over different parameter settings Significance-testing built in! Performing Experiments

50 50 Experiments Setup

51 51 Experiments Setup

52 52 Experiments Setup CSV file: can be open in Excel datasets algorithms

53 53 Experiments Setup

54 54 Experiments Setup

55 55 Experiments Setup

56 56 Experiments Setup

57 57 Experiments Setup accuracy SVM is the best Decision tree is the worst SVM is statistically better than Naïve Bayes Decision tree is statistically worse than Naïve Bayes

58 58 Experiments: Excel Results are output into an CSV file, which can be read in Excel!

59 59 The Newsgroups Text Collection WEKA: Explorer WEKA: Experimenter Python Interface to WEKA

60 60 Slide adapted from Eibe Frank's @relation heart-disease-simplified @attribute age numeric @attribute sex { female, male} @attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina} @attribute cholesterol numeric @attribute exercise_induced_angina { no, yes} @attribute class { present, not_present} @data 63,male,typ_angina,233,no,not_present 67,male,asympt,286,yes,present 67,male,asympt,229,yes,present 38,female,non_anginal,?,no,not_present... WEKA File Format: ARFF Other attribute types: String Date Numerical attribute Nominal attribute Missing value

61 61 Value 0 is not represented explicitly Same header (i.e @relation and @attribute tags) the @data section is different Instead of @data 0, X, 0, Y, "class A" 0, 0, W, 0, "class B" We have @data {1 X, 3 Y, 4 "class A"} {2 W, 4 "class B"} This is especially useful for textual data (why?) WEKA File Format: Sparse ARFF

62 62 Python Interface to WEKA This is just to get you started Assumes the newsgroups collection Extracts simple features currently just single word features –Uses a simple tokenizer which removes punctuation uses a stoplist lowercases the words Includes filtering code currently eliminates numbers Features are weighted by frequency within document Produces a sparse ARFF file to be used by WEKA

63 63 Python Interface to WEKA Allows you to specify: Which directory to read files from which newsgroups to use the number of documents for training each newsgroup the number of features to retain

64 64 Python Interface to WEKA Things to (optionally) add or change: an option to not use stopwords an option to retain capitalization regular expression pattern a feature should match other non-word-based features morphological normalization a minimum threshold for the number of time a term occurs before it can be counted as a feature tf.idf weighting on terms your idea goes here

65 65 Python Interface to WEKA TF.IDF: t ij  log(N/n i ) TF –t ij : frequency of term i in document j –this is how features are currently weighted IDF: log(N/n i ) –n i : number of documents containing term i –N: total number of documents

66 66 Python Weka Code

67 67 Python Weka Code

68 68 Python Weka Code

69 69 Python Weka Code

70 70 Python Weka Code

71 71 Python Weka Code

72 72 Python Weka Code

73 73 Python Weka Code

74 74 ARFF file …

75 75 ARFF file …

76 76 Assignment Due November 13. Work individually on this one Objective is to use the training set to get the best features and learning model you can. FREEZE this. Then run one time only on the test set. This is a realistic way to see how well your algorithm does on unseen data.

77 77 Next Time Machine learning algorithms

Download ppt "1 SIMS 290-2: Applied Natural Language Processing Marti Hearst October 30, 2006 Some slides by Preslav Nakov and Eibe Frank."

Similar presentations

Ads by Google