1. Department of Computer Science, University of Waikato, New Zealand Geoffrey Holmes, Bernhard Pfahringer and Richard Kirkby Traditional machine learning model Data stream requirements Scaling up existing methods Network caching CHIC architecture Experimental Design Results Discussion Further Work Cache Hierarchy Inspired Compression: A Novel Architecture for Data Streams

2. 2 Traditional Machine Learning Create train/test splits of the data (possibly via cross-validation) Load ALL the training data into main memory Compute a model from the training data (may involve multiple passes) Load all the test data into main memory Compute accuracy measures from the test data

3. 3 Consequences Data is processed one instance at a time Very few incremental methods – none used seriously in practice. Many existing techniques don’t scale Machine Learning is perceived as a small to medium data set field of study. Larger data sets are tackled through sampling or building several models on separate portions and combining their predictions

4. 4 Data Streams Takes the “stream” view, source maybe finite or infinite Concept of train/test less well defined, could train for a while then test for a while – what is the definition of “a while”? What ever you do you can be sure that ALL the data will NOT fit in main memory

5. 5 Data Stream Constraints Cannot store instances (not all anyway) Cannot use more than available memory – no swapping to disk Cannot dwell too long over an instance - must keep up with the incoming rate of instances Cannot wait to make predictions – need to be ready to make predictions at any time

6. 6 Scaling up existing methods Could learn models using existing methods in batches and then merge models Could merge instances (meta-instances) Could use a cache model where we keep a set of models and update the cache in time – eg use least recently used, least frequently used type strategies Could do the above but use performance measures to decide the make-up of the cache.

7. 7 Caching in Data Communications Web proxy caches provide a good model for what we need to satisfy stream constraints Real caches are hierarchical (Squid) The hierarchy provides a mechanism for sharing the load and increasing the likelihood of a page hit When full a cache needs a replacement policy To replicate this system we need to design a hierarchy, fill it (with models) and implement a model replacement policy

8. 8 General CHIC Architecture Idea: Build a hierarchy of levels (N) as follows:  Level Zero: data buffer from stream  Level One: Build models from data at level zero  Level Two to N-1: Fill with “best” models from lower levels  Level N: Adopt models from level N-1 but also discard models so that new ones can be entered For prediction use all models in hierarchy and vote

9. 9 Features Can use any machine learning algorithm to build models Implements a form of batch-incremental learning Replacement policy can be performance based As with the web cache CHIC fills up initially and then keeps the best performing models If a variety of models is used at the lower levels then it is possible to adapt to the source of the data.

10. 10 Experimental Design Try to demonstrate adaptation to data source  Learn a mixture of models at levels one and two and let the performance based promotion mechanism take over Evaluate: two issues  Need performance measure (model promotion/deletion)  Overall performance of hierarchy (adopt a first test then train approach)

11. 11 Data Sources Random Naïve Bayes  Weight attribute labels and class values (here we use 5 classes, 5 nominal and 5 numeric attributes) Random Tree  Choose a depth and randomly assign splitter nodes, here 5 nodes deep, leaves starting from depth 3, as above number of attributes/classes. Random RBF  Random set of centers for each class, center is weighted and has a standard deviation – here 50 centers, 10 numeric attributes and 5 classes Real data  Forest covertype (UCI repository), 7 classes, 500K instances.

12. 12 Specific CHIC Architecture Six levels with 16 models per level Data buffer of size 1000 instances First level uses four algorithms to generate models  Naïve Bayes(N), C4.5(J), linear SVM(L), RBF Kernel SVM(R)

13. 13 Example Read 1000 instances into buffer, build 4 models, repeat on next 3 buffers – leads to level one full Read next 1000 and build 4 more models Using the buffer data as test data evaluate all models at level 1 – promote best 4 (groupwise) to level 2 and replace the worst 4 with the new ones. Continue in this manner (at 12,000 instances level 2 will be full) Note: Levels 1&2 always have 4 models from the 4 groups From level 2 ONLY promote the best (adapt to source)

14. 14 Example Contd Four models are promoted to level 3 the 4 worst deleted freeing 8 spaces. Levels 3, 4 & 5 work on the same basis. Level 6 simply deletes the worst 4 models to free up space. At prediction time all models have an equal vote and the classification decision is arrived at by majority.

15. 15 Results after 1000 instances:after 2000 instances: N---J---L---R---NN--JJ--LL--RR-- after 3000 instances:after 4000 instances: NNN-JJJ-LLL-RRR-NNNNJJJJLLLLRRRR after 5000 instances:after 6000 instances: N-NNJJ-JL-LLR-RRNNNNJJJJLLLLRRRRNJLR------------ after 7000 instances:after 8000 instances: NNN-J-JJL-LLR-RRNNNNJJJJLLLLRRRRNJLRNJLR--------

16. 16 Continued after 9000 instances:after 10000 instances: NN-N-JJJLL-L-RRRNNNNJJJJLLLLRRRRNJLRNJLRNJLR---- after 11000 instances:after 12000 instances: -NNN-JJJLLL-RR-RNNNNJJJJLLLLRRRR NJLRNJLRNJLRNJLR after 13000 instances: NN-NJJ-JLL-LRR-R NNLJNLLRN-L-N-L- JJJJ------------

17. 17 Model Adaptation to Source random tree random naive Bayes N-NN-JJJLL-LR-RR -NNNJJ-JLLL-R-RR NNNNLNNLJLRNNJLR NJJJLLLLRNJJJLLR JJJJJNNLJJ---JL- NNNNNJJNJJ-NN--- JJJJJJJJJJJJJJJJ NNNNNNNNNNNNNNNN JJJJ--JJ-JJJJJ-J NNNNNNNN---NNNN- JJJJJJJJJJJJJJJJ NNNNNNNNNNNNNNNN

18. 18 Continued random RBF covertype NNN--JJJLL-LRR-R N-NN-JJJ-LLLR-RR NNJJLRNJNNJJNJLR NNJJJRJLNRJ---R RRJRJJRRRRJRRJJJLJJLLJLJLJL--N-- RRRRRRRRRRRRRRRRLLLLLLLLLLLLJLLJ RRRRRRRRRRRRRRRR LLLLLLLLJL-J-L-- RRRRRRRRRRRRRRRR LLLLLLLLLLLLLLLL

19. 19 Learning Curve – Random Tree

20. 20 Learning Curve - CoverType

21. 21 Conclusion Novel architecture for data streams Operates much like a web cache (hierarchy and replacement policy) Provides scaling-up mechanism for arbitrary classifiers (batch-incremental) Can be adapted to clustering, regression, association rule learning Thousands of options still to explore!