1. Mining Data Streams Challenges, Techniques, and Future Work Ruoming Jin Joint work with Prof. Gagan Agrawal

2. August 10-17, 2003 Major Power outrage simultaneously hits a dozen of big cities in the east of America and Canada Suddenly, millions of people have to live without electricity Internet Worm Millions of computers were attacked In a single day, I received almost 100 emails generated by the worm Capable to collect and monitor the data from power grid and email server Unable to extract knowledge fast enough from the dynamic and huge amount of data!

3. Data Explosion The Challenge: Our ability to access, collect, generate, and store the data has been exceeding our ability to understand them Real Applications WALMART: 20M transactions per day AT&T: 300 M calls per day Earth Observing System from NASA: 50 GB per hour Amazon. COM: 4-5M sessions per day Power Grid/Sensor Network Internet/Intranet

4. Data Streams What is Data Streams? Continuous streams Hugh, Fast and Changing Why Data Streams? The arrival of streams and the volume of data are beyond our capability to store them Real-time processing Evolution of Data (Static/Dynamic) You can only have one look at the data!

5. Data Mining Extracting useful information or knowledge from large amounts of data Interesting patterns Regularity or Anomaly Typical Data Mining Tasks Association Rule Mining Classification Clustering Disk-resident or in-core datasets Traditional data mining needs multi-pass of the data!

6. Stream Data Mining How to run traditional mining tasks over data streams Single pass/multi-pass How to discover new information over data streams Changing How to perform data mining over dynamic data streams Concept drifting

7. Roadmap Thesis Statement Current Work Decision Tree Construction Frequent Itemsets Mining Future Work Mining Maximal/Closed/Approximate Frequent Itemsets Mining New Knowledge from Data Streams Mining Dynamic Data Streams Applications Conclusion

8. Motivation The need for efficient computation and low memory mining algorithms Real-time constraint Memory requirements The need to mine new information from data streams The need for having results with high accuracy and confidence Approximate results with high accuracy

9. Thesis Statement “Designing computation and memory efficient algorithms to provide approximate results with high accuracy and confidence helps mine useful information from data streams ”

10. Roadmap Thesis Statement Current Work Decision Tree Construction Frequent Itemsets Mining Future Work Mining Maximal/Closed/Approximate Frequent Itemsets Mining New Knowledge from Data Streams Mining Dynamic Data Streams Applications Conclusion

11. Decision Tree Construction 30 SalaryAgeEmploymentGroup 30KSelfC 40K35IndustryC 70K50AcademiaC 50K45SelfB 70K30AcademiaB 60K35IndustryA 60K35SelfA 70K30SelfA 40K45IndustryC Salary Age Employment Group C Group BGroup A <= 50K> 50K <= 40> 40 Academia, IndustrySelf Three predictor attributes: Numerical (salary, age), Categorical (employment) Class label attribute: group

12. The problem Basic algorithm (a greedy algorithm) Tree will be built in a top-down recursive way At start, all the training records are at the root; the records are partitioned recursively based on split criteria Split criteria are selected based on a heuristic or statistical measure (e.g., information gain or gini function) Analysis To find the split criteria, all the records falling into the node need to be scanned Scanning the entire datasets multiple times The difficulty to handle numerical attributes Streaming Data You can only have one look at the data Real-time constraint and memory requirement

13. Outline of Our Solution Motivation Three New Techniques Experimental results Conclusion

14. Very Fast Decision Tree (VFDT) ─ Domingo and Hulten (SIGKDD’00) Sampling based approach Given a desired confidence level (α), applying Hoeffding Inequality to test if enough samples has collected to find the best split criteria Accuracy Probabilistic bound on the different number of nodes between the tree built on samples and the one built on complete data Limitation Focus on processing categorical attributes Ideal environment

15. Our Contributions Efficient processing of numerical attributes High memory and computational overheads Numerical Interval Pruning (NIP) Determining exact split points in one pass Confidence interval ExactSplit algorithm Using smaller samples size for the same probabilistic bound Normal Test Efficient Decision Tree Construction on Streaming Data (R. Jin and G. Agrawal, SIGKDD’03) Accurate One Pass Mining of Streaming Data (R. Jin, A. Goswami and G. Agrawal, submitted to SDM’04)

16. Numerical Interval Pruning (NIP) – Efficiently Handling Numerical Attributes Existing methods Preprocessing Online sorting Full Class Histogram Basic Ideas of NIP Hierarchical Information –Concise class Histogram and Detailed Information Divide the range of numerical attributes into intervals Summarize class histogram for intervals Only visit intervals likely to have best split point Drop the detailed information for pruned intervals (Approximate)

17. Finding Best Split Point The data comes from a IBM Quest synthetic dataset for function 0 Best Split Point

18. Summarizing and Pruning Intervals Upper bound of gains for intervals

19. Visiting Detailed Information Best Split Point

20. Re-pruning and Verification Gain of Best Split Point False Pruning Additional intervals needs to be visited if false pruning happens

21. Least Upper Bound of Gain for an Interval Possible Best Configuration-1Possible Best Configuration-2 [ 50,54 ]

22. Finding Exact Split Points in a Single Pass Confidence Interval (CI) Build CI near the approximate split points If the exact split points after processing all data falls into the CI, we will be able to determine it, and correct the descendant nodes also ExactSplit algorithm Recursively find exact split points and correct the descendant nodes from the root Dynamic shrinking Reduce the length of CI as more data instances are processed

23. Sample Size Problem Let n be the sample size of S, N be the normal distribution. Then, for the entropy function g, we have where, Normal Test Normal Test is better than Hoeffding Bound because later one does not utilize the normal distribution property.

24. Performance Results 700MHz Intel Pentium III, with 1GB SDRAM and a 18GB disk with Ultra 160 SCI Drive Stop condition: >=95% accuracy, depth of nodes =1% fraction of instances Start processing the nodes where data instances >=10,000, re-evaluate each node every 5,000 data instances

25. Instances Utilization ClassHist-H: Hoeffding bound and full class histograms Sample-H: Hoeffding bound and samples to evaluate candidate split conditions NIP-H: Hoeffding bound and Numerical Interval Pruning NIP-N: Normal test and Numerical Interval Pruning

26. Adult Dataset Predicting whether income exceeds $50K/yr based on census data 48842 instances, 14 attributes (6 continuous and 8 nominal) AlgorithmConcept SizeInaccuracyRunning TimeTIRIAP ClassHist-H9213.8%308.65527.03.0 Sample-H10413.8%366.09527.22.9 NIP-H9213.8%262.36927.03.1 NIP-N10013.7%157.21719.51.7 Running Time in seconds, TIR and IAP in millions

27. Summary Three new techniques enable an average of 39% reduction in execution times a 37% reduction in the number of data instances required an average of 79% accuracy to determine the exact split condition for the non-leaf nodes on the top 5 levels. The techniques can be applied to other applications, such as K-mean clustering (ongoing work)

28. Roadmap Thesis Statement Current Work Decision Tree Construction Frequent Itemsets Mining Future Work Mining Maximal/Closed/Approximate Frequent Itemsets Mining New Knowledge from Data Streams Mining Dynamic Data Streams Applications Conclusion

29. Frequent Itemsets Mining TIDTransactions 100A,B,EA,B,E 200B,DB,D 300A, B, E 400A,CA,C 500B, C 600A, C 700A, B 800A, B,C,E 900A, B, C 1000A, C, E Desired frequency 50% {A},{B},{C},{A,B}, {A,C} Down-closure property If an itemset is frequent, all of its subset must also be frequent Multi-pass algorithms or in-core datasets Apriori, Eclat, FP-tree

30. The Problem Streaming data You can only have one look of the data Impossible to find all the frequent itemsets in one pass Proposed solutions (with θ,ε) A one-pass algorithm to find a superset of the frequent (θ) itemsets, and each itemset in the superset has to appear more than a desired frequency(θ(1-ε)) A two-pass algorithm will find the exact frequent itemsets (eliminate the false positive)

31. Outline of Our Solution A simplified problem and its solution StreamMining Implementing Issues Experimental Results Conclusion

32. A Simplified Problem Finding frequent items Given a sequence (x 1,…x N ) where x i ∈ [1,n], and a real number θ between zero and one. Looking for x i whose frequency > θ N>>n>>1/θ The number of frequent items ≤ 1/θ P*(Nθ) ≤ N

33. KRP algorithm ─ R. Karp, et. al (TODS’ 03) Θ=0.35 ⌈ 1/θ ⌉ = 3 N=30n=12 N/ ( ⌈ 1/θ ⌉ ) ≤ Nθ

34. Frequent Itemsets 2-itemset is the key! n=10K, Θ=0.1%, average length=10, n*n=100M, |frequent 2-itemsets| ≤ 50K

35. Frequent Items for Transactions with Fixed Length Θ=0.60

36. Frequent Items for Transactions with Varied Length Θ=0.60

37. Enhance the Accuracy Θ=0.35 ⌈ 1/θ ⌉ = 3 ε =0.5 Θ(1- ε )=0.175 ⌈ 1/(θ ε) ⌉ = 6 N=30 n=12

38. StreamMining Sketch Put a transaction into the buffer Update 1-itemset counts Update/insert 2-itemsets If the 2-itemsets is beyond a threshold Crossover Applying the transactions in the buffer to update 3-itemsets, 4- itemsets … Clear buffer Perform additional Crossover

39. Implementing Issues Data Structure TreeHash, a prefix tree encoded into a hash table Frequently insert/delete/increment the potential frequent itemsets Optimizations Online dataset trimming Reducing subset checking Online checking

40. Experimental Results T10.I4.N10K Dataset, 12M transactions

41. Experimental Results (Cont’) T10.I4.N10K Dataset, 0.1% support level

42. Results for very large number of distinctive items T25.I4.N100K Dataset, 12M transactions

43. Real Dataset BMS-WebView-1 Dataset

44. Related Work One-pass algorithm Manku and Motwani Two-pass algorithm Partition Sampling based CARMA Oracle FP-tree and FP-stream Multi-pass algorithm

45. Discussion The new algorithm StreamMining High accuracy ( even when ε=1, the accuracy is 94% or higher) Memory efficient Handle very large number of distinctive itemsets and low threshold using reasonable amounts of memory Observations Reducing passes can not directly contribute to the performance Computational Intensive instead of I/O Intensive In-core algorithm is the key

46. Roadmap Thesis Statement Current Work Decision Tree Construction Frequent Itemsets Mining Future Work Mining Maximal/Closed/Approximate Frequent Itemsets Mining New Knowledge from Data Streams Mining Dynamic Data Streams Applications Conclusion

47. Mining Frequent Itemsets The problem Computational Intensive Different Solutions Maximal Frequent Itemsets Closed Frequent Itemsets Approximate Frequent Itemsets StreamMax Contour sets

48. Stream* Common characteristics of one-pass and two-pass algorithms for streaming data and very large datasets Maintaining a superset of frequent itemsets Different methods to update the supersets Applying slightly different in-core algorithms A framework to efficiently incorporate different in-core algorithms for mining streams Apriori, Eclat and FP-tree TreeHash and StreamMining/MM

49. Frequent Itemsets Mining over Dynamic Data Streams Sliding Window Model Recent data New queries raised from sliding window Frequent itemsets for the current window The intersection and union of frequent itemsets over windows Itemsets with large frequency changes Two key issues How to forget/delete information obsolete Computing the new queries systematically

50. Learning over Dynamic Streaming Data Concept Drifting CVFDT Ensemble classifiers Clustering Mining changes Demon Burst detection Cluster Changes

51. Can a single classifier perform as good as or even better than ensemble classifiers? Advantage of a single classifier Simple Intuitive Sampling approach Incorporating time and change as factors to sample data streams Dynamic Sample Sets

52. How the knowledge of changes will help us to mine under concept drifting? What’s the appropriate model to describe the changes Kernel methods How to incorporate the knowledge of changes into classifiers?

53. Roadmap Thesis Statement Current Work Decision Tree Construction Frequent Itemsets Mining Future Work Mining Maximal/Closed/Approximate Frequent Itemsets Mining New Knowledge from Data Streams Mining Dynamic Data Streams Applications Conclusion

54. Network-based Network Intrusion Detection

55. Very large and Evolving Databases Some very large datasets are not streaming data Possibly recorded on storage No real time constraints But arrives in order and updated in a block fashion Typical Examples War-Mart AT&T EOS Stream mining which provides approximate results is helpful and even necessary to monitor and facilitate mining exact knowledge from such databases

56. Roadmap Thesis Statement Current Work Decision Tree Construction Frequent Itemsets Mining Future Work Mining Maximal/Closed/Approximate Frequent Itemsets Mining New Knowledge from Data Streams Mining Dynamic Data Streams Applications Conclusion

57. Rule of Thumbs Application Driven Network Security E-Commercial Sensor Network The key techniques Sampling Counting Approximation Data structure

58. Conclusion The model of data streams and our motivation to mine streams Decision tree construction and frequent Itemsets mining over data streams Future work Applications