1. Comparing the Parallel Automatic Composition of Inductive Applications with Stacking Methods Hidenao Abe & Takahira Yamaguchi Shizuoka University, JAPAN {hidenao,yamaguti}@ks.cs.inf.shizuoka.ac.jp

2. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 20032 Contents Introduction Constructive meta-learning based on method repositories Case study with common data sets Conclusion

3. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 20033 Overview of meta-learning scheme 学習 Learning Algorithms Meta-Learning Executor データセット Data Set Meta Knowledge A Better result to the given data set than its done by each single learning algorithm Meta-learning algorithm

4. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 20034 Selective meta-learning scheme Integrating base-level classifiers, which are learned with different training data sets generating by –“Bootstrap Sampling” (bagging) –weighting ill-classified instances (boosting) Integrating base-level classifiers, which are learned with different algorithms, with –simple voting (voting) –constructing meta-classifier with a meta data set and a meta-level learning algorithm (stacking, cascading) They don’t work well, when no base-level learning algorithm works well to the given data set !! -> Because they don’t de-compose base-level learning algorithms.

5. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 20035 Contents Introduction Constructive meta-learning based on method repositories –Basic idea of our constructive meta-learning –An implementation of the constructive meta-learning called CAMLET –Parallel execution and search for CAMLET Case Study with common data sets Conclusion

6. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 20036 De-composition& Organization Search & Composition + C4.5 VS CS AQ15 Analysis of Representative Inductive Learning Algorithms Basic Idea of our Constructive Meta-Learning

7. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 20037 CS AQ15 Analysis of Representative Inductive Learning Algorithms Basic Idea of Constructive Meta-Learning De-composition& Organization Search & Composition +

8. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 20038 Analysis of Representative Inductive Learning Algorithms Automatic Composition of Inductive Applications De-composition& Organization Search & Composition + Basic Idea of Constructive Meta-Learning Organizing inductive learning methods, control structures and treated objects.

9. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 20039 Analysis of Representative Inductive Learning Algorithms Version Space AQ15 ID3 C4.5 Boosted C4.5 Bagged C4.5 Neural Network with back propagation Classifier Systems We have analyzed the following 8 learning algorithms: We have identified 22 specific inductive learning methods.

10. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200310 Inductive Learning Method Repository

11. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200311 Data Type Hierarchy Organization of input/output/reference data types for inductive learning methods for inductive learning methods classifier-set data-set If-Then rules tree (Decision Tree) network (Neural Net) training data set test data set Objects validation data set

12. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200312 Identifying Inductive Learning Methods and Control Structures Generating Training & Validation Data Sets Generating a classifier set Evaluating set Modifying training and validation data sets Modifying a classifier set set Evaluating classifier sets classifier sets to test data set Modifying a classifier set set

13. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200313 CAMLET CAMLET: a Computer Aided Machine Learning Engineering Tool Method Repository Data Type Hierarchy Control Structures Compile Go & Test Construction Instantiation Go to the goal accuracy? No Yes Data Sets, Goal Accuracy Inductive application, etc. Refinement User

14. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200314 CAMLET CAMLET with a parallel environment Method Repository Data Type Hierarchy Control Structures Compile Go & Test Execution Level Construction Instantiation Go to the goal accuracy? No Yes Data Sets, Goal Accuracy Inductive application, etc. Composition Level Refinement Specifications Data Sets Results User

15. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200315 Parallel Executions of Inductive Applications Constructing specs Sending specs Initializing R: receiving,Ex: executing, S: sending Composition Level is necessary one process Execution Level is necessary one or more process elements RRR R Waiting Executing Ex Receiving a result Receiving Ex S Refining the spec Sending refined one Refining & Sending Ex R

16. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200316 Refinement of Inductive Applications with GA t Generation Selection Parents Crossover and Mutation Executed Specs & Their Results X Generation execute & add 1.Transform executed specifications to chromosomes 2.Select parents with “Tournament Method” 3.Crossover the parents and Mutate one of their children 4.Execute children’s specification, transforming chromosomes to specs * If CAMLET can execute more than two I.A at the same time, some slow I.A will be added to t+2 or later Generation. t+1 Generation Children

17. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200317 CAMLET CAMLET on a parallel GA refinement Method Repository Data Type Hierarchy Control Structures Compile Go & Test Execution Level Construction Instantiation Go to the goal accuracy? No Yes Data Sets, Goal Accuracy Inductive application, etc. Composition Level Refinement Specifications Data Sets Results User

18. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200318 Contents Introduction Constructive meta-learning based on method repositories Case Study with common data sets –Accuracy comparison with common data sets –Evaluation of parallel efficiencies in CAMLET Conclusion

19. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200319 Set up of accuracy comparison with common data sets We have used two kinds of common data sets –10 Statlog data sets –32 UCI data sets (distributed with WEKA) We have input goal accuracies to each data set as the criterion. CAMLET has output just one specification of the best inductive application to each data set and its performance. –CAMLET has searched about 6,000 inductive applications for the best one, executing up to one hundred inductive applications. 10 “Execution Level” PEs have been used to each data set. Stacking methods implemented in WEKA data mining suites

20. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200320 Statlog common data sets To generate 10-fold cross validation data sets’ pairs, We have used SplitDatasetFilter implemented in WEKA.

21. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200321 Setting up of Stacking methods to Statlog common data sets J4.8 (with pruning, C=0.25) IBk (k=5) Bagged J4.8 (Sampling rate=55%, 5 Iterations) Bagged J4.8 (Sampling rate=55%, 10 Iterations) Boosted J4.8 (5 Iterations)Boosted J4.8 (10 Iterations) Part (with pruning, C=0.25) Naïve Bayes J4.8 (with pruning, C=0.25) Naïve Bayes Classification with Linear Regression (with all of features) Meta-level Learning Algorithms Base-level Learning Algorithms

22. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200322 Evaluation on accuracies to Statlog common data sets Inductive applications composed by CAMLET have shown us as good performance as that of stacking with Classification via Linear Regression.

23. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200323 Setting up of Stacking methods to UCI common data sets J4.8 (with pruning, C=0.25) IBk (k=5) Bagged J4.8 (Sampling rate=55%, 10 Iterations) Boosted J4.8 (10 Iterations) Part (with pruning, C=0.25) Naïve Bayes J4.8 (with pruning, C=0.25) Naïve Bayes Linear Regression (with all of features) Meta-level Learning Algorithms Base-level Learning Algorithms

24. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200324 Evaluation on accuracies to UCI common data sets AlgorithmJ4.8NBLRMax. of 3CAMLET Average81.9281.9781.0783.7983.23

25. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200325 Analysis of parallel efficiency of CAMLET We have analyzed parallel efficiencies of executions of the case study with “Statlog common data sets”. –We have used 10 processing elements to execute each inductive applications. Parallel Efficiency is calculated as the following formula: 1<=x<100, n:#fold, e:Execution Time of Each PE

26. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200326 Parallel Efficiency of the case study with StatLog data sets Parallel Efficiencies of 10-fold CV data sets Parallel Efficiencies of training-test data sets

27. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200327 Why the parallel efficiency of “satimage” data set was so low? Because: - CAMLET could not find satisfactory inductive application to this data set within 100 executions of inductive applications. - In the end of search, inductive applications with large computational cost badly affected to parallel efficiency.

28. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200328 Contents Introduction Constructive meta-learning based on method repositories Case Study with common data sets Conclusion

29. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200329 Conclusion CAMLET has been implemented as a tool for “Constructive Meta-Learning” scheme based on method repositories. CAMLET shows us a significant performance as a meta-learning scheme. –We are extending the method repository to construct data mining application of whole data mining process. The parallel efficiency of CAMLET has been achieved greater than 5.93 times. –It should be improved with methods to equalize each execution time such as load balancing or/and three-layered architecture. with more efficient search method to find satisfactory inductive application faster.

30. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200330 Thank you!

31. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200331 Stacking: training and prediction Training phasePrediction phase Training Set (Int.) Test Set (Int.) Training SetTest Set Meta-level Training Set Meta-level classifier Meta-level learning ベースレベル 分類器 ベースレベル 分類器 Base-level classifiers Learning of Base-level classifiers Transformation Repeating with CV Meta-level Test Set Transformation Prediction Results of Preduction ベースレベル 分類器 ベースレベル 分類器 Base-level classifiers Meta-level classifiers Learning of Base-level classifiers

32. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200332 How to transforme base-level attributes to meta-level attributes Classifier by Algorithm 1 Learning with Algorithm 1 1.Getting prediction probability to each class value with the classifier. 2. Adding base-level class value of each base-level instance A algorithm C class values #Meta-level Att. = AC

33. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200333 Meta-level classifiers (Ex.) The prediction probability of”0” With classifier by algorithm 1 The prediction probability of”1” With classifier by algorithm 2 The prediction probability of”1” With classifier by algorithm 1 The prediction probability of”0” With classifier by algorithm 3 Prediction “1” ．．．． ≦ 0.1 ＞ 0.1 ≦ 0.95 ＞ 0.95 ＞ 0.9 ≦ 0.9 Prediction “0” Prediction “1”

34. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200334 Accuracies of 8 base-level learning algorithms to Statlog data sets Stacking Base-level algorithms Accuracy （％） CAMLET Base-level algorithms Accuracy （％）

35. H.Abe & T. YamaguchiParallel and Distributed Computing for Machine Learning 200335 C4.5 Decision Tree without pruning (reproduction of CAMLET) Generating training and validation data sets with a void validation set Generating a classifier set (decision tree) with entropy + information ratio Evaluating a classifier set with set evaluation Evaluating classifier sets to test data set with single classifier set evaluation 1. Select just one control structure 2. Fill each method with specific methods 3. Instantiate this spec. 4. Compile the spec. 5. Execute its executable code 6. Refine the spec. (if needed)