1 How to use Weka How to use Weka

2 WEKA: the software Waikato Environment for Knowledge Analysis Collection of state-of-the-art machine learning algorithms and data processing tools implemented in Java Released under the GPL Support for the whole process of experimental data mining Preparation of input data Statistical evaluation of learning schemes Visualization of input data and the result of learning Used for education, research and applications Complements “Data Mining” by Witten & Frank

3 Main Features 49 data preprocessing tools 76 classification/regression algorithms 8 clustering algorithms 15 attribute/subset evaluators + 10 search algorithms for feature selection 3 algorithms for finding association rules 3 graphical user interfaces “The Explorer” (exploratory data analysis) “The Experimenter” (experimental environment) “The KnowledgeFlow” (new process model inspired interface)

4 WEKA: The Software Toolkit Machine learning/data mining software in Java GNU License Used for research, education and applications Complements “Data Mining” by Witten & Frank Main features: data pre-processing tools learning algorithms evaluation methods graphical interface (incl. data visualization) environment for comparing learning algorithms

5 WEKA: Terminology Some synonyms/explanations for the terms used by WEKA, which may differ from what we use: Attribute: feature Relation: collection of examples Instance: collection in use Class: category

6 WEKA GUI Chooser java -Xmx1000M -jar weka.jar

7 Our Toy Example We demonstrate WEKA on a simple example: 3 categories from “Newsgroups”: misc.forsale, rec.sport.hockey, comp.graphics 20 documents per category features: words converted to lowercase frequency 2 or more required stopwords removed

8 Explorer: Pre-Processing The Data WEKA can import data from: files: ARFF, CSV, C4.5, binary URL SQL database (using JDBC) Pre-processing tools (filters) are used for: Discretization, normalization, resampling, attribute selection, transforming and combining attributes, etc.

9 List of attributes (last: class variable) Frequency and categories for the selected attribute Statistics about the values of the selected attribute Classification Filter selection Manual attribute selection Statistical attribute selection Preprocessing The Preprocessing Tab

10 Explorer: Building “Classifiers” Classifiers in WEKA are models for: classification (predict a nominal class) regression (predict a numerical quantity) Learning algorithms: Naïve Bayes, decision trees, kNN, support vector machines, multi-layer perceptron, logistic regression, etc. Meta-classifiers: cannot be used alone always combined with a learning algorithm examples: boosting, bagging etc.

11 Choice of classifier The attribute whose value is to be predicted from the values of the remaining ones. Default is the last attribute. Here (in our toy example) it is named “class”. Cross-validation: split the data into e.g. 10 folds and 10 times train on 9 folds and test on the remaining one The Classification Tab

12 Choosing a classifier

13

14 False: Gaussian True: kernels (better) displays synopsis and options numerical to nominal conversion by discretization outputs additional information

15

16

17 all other numbers can be obtained from it different/easy class accuracy

18 Contains information about the actual and the predicted classification All measures can be derived from it: accuracy: (a+d)/(a+b+c+d) recall: d/(c+d) => R precision: d/(b+d) => P F-measure: 2PR/(P+R) false positive (FP) rate: b/(a+b) true negative (TN) rate: a/(a+b) false negative (FN) rate: c/(c+d) These extend for more than 2 classes: see previous lecture slides for details Confusion matrix predicted –+ true –ab +cd

19 Outputs the probability distribution for each example Predictions Output

20 Probability distribution for a wrong example: predicted 1 instead of 3 Naïve Bayes makes incorrect conditional independence assumptions and typically is over-confident in its prediction regardless of whether it is correct or not. Predictions Output

21 Error Visualization

22 Error Visualization Little squares designate errors Axes show example number

23 Running on Test Set

24 Find which attributes are the most predictive ones Two parts: search method: best-first, forward selection, random, exhaustive, genetic algorithm, ranking evaluation method: information gain, chi-squared, etc. Very flexible: WEKA allows (almost) arbitrary combinations of these two Explorer: Attribute Selection

25 Individual Features Ranking

26 misc.forsale comp.graphics rec.sport.hockey Individual Features Ranking

27 misc.forsale comp.graphics rec.sport.hockey ??? random number seed Individual Features Ranking

28 feature correlation 2-Way Interactions Feature Interactions (Advanced) C BA category feature importance of feature B importance of feature A

29 3-Way Interaction: What is common to A, B and C together; and cannot be inferred from pairs of features. Feature Interactions (Advanced) C BA category feature importance of feature B importance of feature A

30 Feature Subsets Selection Problem illustration Full set Empty set Enumeration Search Exhaustive/Complete (enumeration/branch&bounding) Heuristic (sequential forward/backward) Stochastic (generate/evaluate) Individual features or subsets generation/evaluation

31 Features Subsets Selection

32 misc.forsale comp.graphics rec.sport.hockey 17,309 subsets considered 21 attributes selected Features Subsets Selection

33 Saving the Selected Features All we can do from this tab is to save the buffer in a text file. Not very useful... But we can also perform feature selection during the pre-processing step... (the following slides)

34 Features Selection on Preprocessing

35 Features Selection on Preprocessing

36 Features Selection on Preprocessing 679 attributes: (for the class)

37 Features Selection on Preprocessing Just 22 attributes remain: (for the class)

38 Run Naïve Bayes With the 21 Features higher accuracy 21 Attributes

39 different/easy class accuracy (AGAIN) Naïve Bayes With All Features ALL 679 Attributes (repeated slide)

40 WEKA has weird naming for some algorithms Here are some translations: Naïve Bayes: weka.classifiers.bayes.NaiveBayes Perceptron: weka.classifiers.functions.VotedPerceptron Decision tree: weka.classifiers.trees.J48 Support vector machines: weka.classifiers.functions.SMO k nearest neighbor: weka.classifiers.lazy.IBk Some of these are more sophisticated versions of the classic algorithms e.g. the classic Naïve Bayes seems to be missing A good alternative is the Multinomial Naïve Bayes model Some Important Algorithms

41 The Newsgroups Text Collection WEKA: Explorer WEKA: Experimenter Python Interface to WEKA

age sex { female, chest_pain_type { typ_angina, asympt, non_anginal, cholesterol exercise_induced_angina { no, class { present, 63,male,typ_angina,233,no,not_present 67,male,asympt,286,yes,present 67,male,asympt,229,yes,present 38,female,non_anginal,?,no,not_present... WEKA File Format: ARFF Other attribute types: String Date Numerical attribute Nominal attribute Missing value

43 Python Interface to WEKA This is just to get you started Assumes the newsgroups collection Extracts simple features currently just single word features Uses a simple tokenizer which removes punctuation uses a stoplist lowercases the words Includes filtering code currently eliminates numbers Features are weighted by frequency within document Produces a sparse ARFF file to be used by WEKA