1. DIMACS Summer School Applications of Sketch Based Techniques to Data Mining Problems Nikos Koudas AT&T Labs Research joint work with: G. Cormonde, P. Indyk, S. Muthukrishnan

2. DIMACS Summer School Taming Massive Data Sets Requirements of data mining algorithms –operate on very large data sets –scalability –incremental Most data mining algorithms have super-linear complexity Deploying mining algorithms on very large data sets, most likely will result in terrible performance

3. DIMACS Summer School Sketching Reduce the dimensionality of the data in a systematic way, constructing short “data summaries” Effectively reduce data volume. Deploy mining algorithms on the reduced data volume. Main issue: Preserve data properties, the mining algorithm is concerned with, (e.g., distances) in the reduced data volume.

4. DIMACS Summer School Applications Clustering time series data Clustering tabular data sets

5. DIMACS Summer School Introduction Time series data abound in many applications. –Financial, performance data, geographical and meteorological information, solar and space data etc. Various works deal with management and analysis aspects of time series data: –Indexing, storage and retrieval –Analysis and mining (forecasting, outlier and deviation detection, etc) Active research area in many research communities.

6. DIMACS Summer School Representative Trends Relaxed period Average trend

7. DIMACS Summer School Usage Examples Mining/Prediction –Identifying ‘periodic trends’ –Uncovering unexpected ‘periodic trends’ Performance management –Networking (routing, traffic engineering, bandwidth allocation) –System Tuning Financial databases –Cyclic behavior

8. DIMACS Summer School Definitions Given a time series V and an integer T define V(T) = {(V[iT+1], V[iT+2], …, V[iT+T])}, 0 <= i<= n/T-1 Define: –C i (V(T)) =  1<j<n/T-1 D(v i,u j ) Thus: –Relaxed Period: min C 0 (V(T)), for T in [l,u] –Average Trend: min C i (V(T)), for T in [l,u]

9. DIMACS Summer School Definitions T n/T vectors for each T, T in [l,u]n points in each time series Relaxed period T n/T vectors for each T, each of them a candidate avg. trend, T in [l,u] Average Trend

10. DIMACS Summer School Algorithms There exists a quadratic algorithm for identifying relaxed periods. There exists a cubic algorithm for identifying average trends. Simply evaluate the clustering for each T in [l,u], it takes linear time to evaluate relaxed periods for each T and quadratic time to evaluate for average trends.

11. DIMACS Summer School Algorithms But can we really run these? –Consider length of sessions in an AT&T service for each second for a year, it is more than 31M values and approximately 256MB. –Consider running the previous algorithms for say 10 years or on a finer time scale. Both brute force algorithms are impractical. Can we run faster on large datasets, too large to be in memory?

12. DIMACS Summer School Our Approach Identify representative trends faster but provide approximate answers: –General approach (expresses various notions of representative trends) –Provides guaranteed approximation performance, with high probability We present our approach in the following steps: –Define the sketch of a vector –Algorithms for finding the sketch of all sub-vectors of width T –Determine the sketch of all sub-vectors of width in a given range.

13. DIMACS Summer School Sketch of a vector Given a vector t of length l, we generate its sketch S(t) as follows: Pick a random vector u of length l, by picking each component u[i] from a normal distribution N(0,1) (normalized to 1). Define: –S(t)[i] = t. u =  j t[j]. u[j]

14. DIMACS Summer School Sketch Properties Theorem –For any given set L of vectors of length l, for a fixed  < 1/2, if k = 9 log|L|/  2, then for any pair of vectors u,w we have –(1  u-w|| 2 < ||S(u)-S(w)|| 2 <(1  u-w|| 2 with probability 1/2. By increasing k we can increase the probability of success This is the Johnson-Lindenstrauss (JL) lemma.

15. DIMACS Summer School Fixed window sketches Compute all sketches of sub-vectors of length l in a sequence of length n. There are n-l+1 such sub-vectors. Straightforward application of JL would require O(nlk) time since there are O(n) sub-vectors, each of length l and the sketch is of size k, not practical. l k

16. DIMACS Summer School Key Observation We can compute ALL such sketches fast by using the fast fourier transform. The problem of computing sketches of all sub- vectors of length l simultaneously is exactly the problem of computing the convolution of two vectors t and u Given two vectors A[1…a] and B[1…b] their convolution is C[1…a+b] where –C[k] =  1<i<b A[k-i]B[i] for 2 <= k <= a+b

17. DIMACS Summer School Example 2 1 3 1Convolution with (-0.97,-0.2) = (-0.4,-2.14,-1.57,-3.11,-0.97) (0.11,0.99) = (1.98,1.21,3.08,1.32,0.11) S1[0] S2[0] S3[0] S1[1] S2[1] S3[1]

18. DIMACS Summer School Computing all sketches of width in a given range Compute all sketches of all sub-vectors of length between l and u. Brute force is cubic and prohibitive Applying our observation would be quadratic and still prohibitive Can we compute all sketches of width in a given range faster?

19. DIMACS Summer School Approach We will construct a pool of sketches that we will pre-compute and store. Following this preprocessing we will be able to determine the sketch of any sub-vector in O(1) fairly accurately. Pick an l <= L <= u and construct all sketches of length L as before using convolutions, this is O(nlog 2 n) in the worst case. Assume for now that L a power of 2 actually construct two such pools S1 and S2

20. DIMACS Summer School Approach Consider any vector t[i,….i+j-1] we have two cases: –j = some power of 2 (L), in this case we have it in the pool and we can look it up in O(1) –2 r < j < 2 r+1 in this case we can compute the sketch as follows: S(t[i,….,i+j-1])[j] = S1(t[i,…,i+2 r -1])[j] + S2[t[i+j- 2 r,….,i+j-1])[j] both terms belong to the pool

21. DIMACS Summer School Example 2 1 3 1 2 3 2 1 S2 S1 S1 0 +S2 1 S(U) = U

22. DIMACS Summer School Why is this enough? Theorem –For any given set L of vector of length l, for fixed  < 1/2 if k = 9 log L/  2, then for any pair of vectors u,w in L (1  u-w|| 2 <= ||S(u)-S(w)|| 2 <= 2 (1  ||u-w|| 2 with probability 1/2

23. DIMACS Summer School Putting it all together Given V and [l,u] range –relaxed period: Compute sketches in time O(nlog(u-l)klogu) Consider every T in [l,u] and compute C 0 (V(T)) for every T. Choosing k as described will guarantee that we are at most 2+  away from the true relaxed period –Average Trends Proceed similarly by evaluating C i (V(T)) for every i

24. DIMACS Summer School Implementation Issues Computing Sketches –The pool of sketches can be computed with a single pass over the data set. We only need to keep a window worth of data across successive sketch computations. Retrieving Sketches –Required sketches are retrieved by performing random IO. However across successive evaluations for various values if T, required sketches are related. Random IO can be limited due to prefetching.

25. DIMACS Summer School Experimental Evaluation Real data from a service AT&T provides (utilization information). Size varying from 16MB (approx. 1 month) to 256MB (approx 1 year) worth of data. Evaluated: –Time to construct sketches –Scalability of sketch construction –Efficiency of the proposed convolution based technique –Time to compute relaxed period and average trends computing sketches from scratch and with pre-computed sketches Using brute force approaches –Accuracy of sketching –Comparison with other time series reduction techniques

26. DIMACS Summer School Time to construct sketches

27. DIMACS Summer School Time to construct sketches

28. DIMACS Summer School Time to construct without convolution

29. DIMACS Summer School Time to construct sketches without convolution

30. DIMACS Summer School Computing relaxed periods

31. DIMACS Summer School Computing Relaxed Periods

32. DIMACS Summer School Computing relaxed period with precomputed sketches

33. DIMACS Summer School Computing Relaxed Periods Without Precomputed Sketches

34. DIMACS Summer School Brute Force Algorithms

35. DIMACS Summer School Brute Force Algorithms

36. DIMACS Summer School Computing Average Trend

37. DIMACS Summer School Computing Average Trend

38. DIMACS Summer School Accuracy of Sketches

39. DIMACS Summer School Clustering Tabular Data Many applications produce data in two dimensional array form. Consider traditional telecommunication applications: –Data are collected from a variety of collection stations across the country, recording call volume at some temporal granularity. –2d call volume data set (spatial ordering of collection stations versus time) recording temporal call activity, approx. 18MB/day.

40. DIMACS Summer School

41. Clustering tabular data Data elements to be clustered are rectangular data regions. Clustering might reveal interesting similarities (in call volume and time) between geographical regions. One month ~ 600MB of data. Sketch rectangular regions: –extend sketches in 2d –sketching with respect to any Lp norm p in (0.2]

42. DIMACS Summer School

43. Summary of results Sketch construction scales nicely with respect to data volume and sketch size. Convolution based sketch computation is very effective. Sketch based approach is orders of magnitude better than brute force for computing relaxed periods and average trends. Performance benefits increase for larger data sets. If sketches are pre-computed, clustering can be performed in seconds even for very large data sets. In practice sketches of low dimensionality provide great accuracy. Compared with other dimensionality reduction techniques, the sketch based approach is more accurate and effective.

44. DIMACS Summer School Conclusions Scalability to large data volume requirement of the data mining process. Effectively reduce data volume using sketches. Preserve data properties required by mining algorithms (e.g., various distances). Core techniques, various algorithms could benefit from them. Very large performance benefits, small loss in accuracy.

45. DIMACS Summer School Contact koudas@research.att.com www.research.att.com/~koudas