Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Clustering of Streaming Time Series is Meaningless presentation by Rafal Ladysz after the original paper by Eamonn Keogh Jessica Lin Wagner Truppel Computer.

Similar presentations


Presentation on theme: "1 Clustering of Streaming Time Series is Meaningless presentation by Rafal Ladysz after the original paper by Eamonn Keogh Jessica Lin Wagner Truppel Computer."— Presentation transcript:

1 1 Clustering of Streaming Time Series is Meaningless presentation by Rafal Ladysz after the original paper by Eamonn Keogh Jessica Lin Wagner Truppel Computer Science & Engineering, University of California-Riverside

2 2 interesting and important topic foreward of the original paper reads: –“Clustering is perhaps the most frequently used data mining algorithm, being useful in it's own right as an exploratory technique, and also as a subroutine in more complex data mining algorithms such as rule discovery, indexing, summarization, anomaly detection, and classification” –“Time series data is perhaps the most frequently encountered type of data examined by the data mining community” thus, a lot of interest, works, papers, conferences on these two, nevertheless “it has never appeared in the literature” what the title claims

3 3 QUIZ questions (asked upfront) what are two main ways of clustering time series data? (name and describe each in one sentence) one can “convert” hierarchical clustering into k-means clustering: which of these two is deterministic (if any)? what method can help subclustering time series work?

4 4 time series (TS) mini-primer intuitive definition: sequence of real numbers (usually acquired in equal time intervals) examples of experimental time series –meteorological observations –EEG, EKG, patient’s temperature (medical) –laser light intensity measured –stock market indices –predator-prey population recorded possible division –periodic/non-periodic –stochastic (random)/chaotic (deterministic)

5 5 possible TS hierarchy tree the leaf nodes refer to the actual representation, and the internal nodes refer to the classification of the approach credit: Keogh et al.

6 6 TS: illustration S&P laser Lorenz earthquake chaotic

7 7 mining TS general examples –anomaly detection (deviation from some mean value, e.g. monitoring functioning of space shuttle) –classification/ forecasting –rule discovery (surprising/interesting patterns) particular example (of my current interest) –detecting chaos in dynamic TS data streams getting insight of the underlying system’s dynamics computing some crucial parameter(s) –possible applications of the above EEG stock market weather-related catastrophes (extremally complex)

8 8 TS – similarity issue in many (though not all) cases similarity is necessary to investigate TS data we need some measure of similarity to mine TS –classification, e.g. ECG patterns of new patients as indicator of heart deseases with known ECG pattern –clustering, e.g. groupping websites with similar traffic patterns –association, e.g. a plateau followed by a sudden decrease in EEG an epileptic seizure can happen we need it for searching particular pattern (once we can use techniques/tools to mine TS)

9 9 TS similarity – possible measures in general – there are many and what to use depends on the application an obvious similarity measure is one based on Euclidean distance (with its pros and cons): –each sequence as a point in n-dimensional Euclidean space, where n = length of TS points then similarity L p between TS sequences X, Y is L p = (  i=1 n |x i – y i | p ) 1/p old problem of dimensionality curse exists –thus scalability is desired and enforces –trade off between accuracy and efficiency

10 10 Euclidean distance for TS in action credit: A. K. Singh

11 11 similarity of TS – when we use it Indexing problem –find all lakes whose water level fluctuations are similar to X Subsequence Similarity problem –find out other days in which stock X had similar movements as today Clustering problem –group regions that have similar sales patterns Rule Discovery problem –find rules such as “if stock X goes up and Y remains the same, then Z will go down soon”

12 12 clustering algorithms: quick look at three of them well known k-means –choosing k: the number of clusters to generate –initializing k centers of clusters to be generated – keep re-estimating k clusters’ centers... greedy... converges but not (necessarily) to global minimum... depends on initialization is step 2 –stops when no changes (in cluster membership) hierarchical clustering density-based clustering

13 13 hierarchical clustering: step by step 1. distances between objects: compute and put into distance matrix 2. search through distance matrix to find two closest (i.e. most similar) objects (clusters in next iterations) 3. join the two to get cluster of at least two objects 4. update distance matrix (new clusters generated) 5. repeat step 2 until there is one cluster of all objects (from step 1) Q: is it bottom up (aglomerative) or top down?

14 14 hierarchical clustering: illustration TS being clustered hierarchically - starting with 10 sequences sliding either way along green line the “cut off” line determines k (clusters) - thus determines “bottom-up” or “top-down” way so we can “convert” hierarchical clustering to k-means cluster. averages

15 15 hierarchical clustering summary it produces the same results every time with a given set of data (unlike k-means clustering) cons: –splitting or merging “irreversible” in next iterations (i.e. no element redistribution among clusters) –poor scaling (quadratic in input size) pros: –no input parameters (like number of clasters k) –simplicity –can be integrated with other clustering methods

16 16 density-based clustering (DBC) based on density - local cluster criterion recognizes clusters as “dense regions” major features: – discover clusters of arbitrary shape – handle noise – one scan – need density parameters as termination condition sources and algorithms: – DBSCAN: Ester, et al. (KDD’96) – OPTICS: Ankerst, et al (SIGMOD’99). – DENCLUE: Hinneburg & D. Keim (KDD’98) – CLIQUE: Agrawal, et al. (SIGMOD’98)

17 17 TS and its subsequences formally, TS can be expressed as an ordered set of m variables or a point in m-dim space TS = t 1, t 2,..., t m this formality enables applying clustering to a set of TS sequences as if they were such points C p denotes a subsequence of length w of a TS, where w < m: C p = t p, t p+1,..., t p+w-1, 1  p  m-w+1 a technique of “sliding window” (of size w) is a useful concept here

18 18 subsequences via sliding window sliding window extracts all subsequences C p described earlier from a given TS a matrix S of all such subsequences can be built by moving the sliding window across a given TS and placing subsequence C p in the p th row of S whose size is (m-w+1) times w far left: first eight subsequences C p, each of length 16; middle: C 67 of the same length

19 19 sliding window and its matrix denoting all possible subclusters C p C 1 = {t 1, t 2, …, t 10 } C 2 = {t 2, t 3, …, t 11 } C m-w = {t m-9, t m-8, …t m } and their corresponding matrix S:

20 20 meaninglessness of STS clustering to demonstrate meaninglessness of STS clustering two algorithms have been used: –k-means –hierarchical clustering important remark: to minimize any “methodological” bias, the whole clustering (besides STS sliding window clustering) has been performed to provide control results for comparison

21 21 let A, B denote cluster centers derived from two different runs of k-means algorithm over the same data set (expect different results): the cluster_distance(A, B) defines the distance between two sets of clusters: A and B remark: the above definition enforces closest pairs from A, B variability of k-means: one data set

22 22 variability of k-means: two data sets applying this approach for different data sets experiment: performing 3 random restarts of k-means (applying sliding window) on a stock market dataset set X: the 3 resulting sets of cluster centers similarly with 3 random runs of k-means on a random walk dataset set Y: the resulting cluster centers

23 23 more definitions denote the avarage cluster_distance between each set of cluster centers in X to each other set of cluster centers in X (as it was for one data set) by within_set_X_distance denote the average cluster distance between each set of cluster centers in X to cluster centers in Y by between_set_X_and_Y_distance

24 24 a brief analysis of the claster meaningfullness(X, Y) numerator (within set distance X) measuring clustering algorithm’s sensitivity to initial conditions (seeds); briefly: it asumes zero for same results on the other hand: there is no reason for similarity of clustering results for two different (and unrelated) data sets: briefly: denominator (between set X and Y distance) should be (relatively) large overall tendency: claster meaningfullness(X, Y)  0 if X, Y differ

25 25 experiment: STS vs whole clustering to obtain control set of results (for comparison) the same experiment has been repeated by k-means –for the same data –using whole clustering method (i.e. randomly extracted subsequences) entire process has ben repeated 100 times for every combination of parameters k and w: k={3, 5, 7, 11} w={8, 16, 32} results: first surprise!!! comparison: whole (yellow) vs. STS Z-axis: meaningfulness value

26 26 same experiment: hierarchical clust. having proven meaningless of k-means clustering of STS, the experiment has been performed using hierarchical clustering new challenge: defining distance between two clusters: linkage method - applicable for bottom-up clustering cluster objects can be based on different methods: Single Linkage: the minimum distance between them (nearest neighbour rule) Complete Linkage: the maximum distance between them (furthest neighbour rule) Average Linkage: the average distance between all pairs of objects (one member of the pair must be from a different cluster) cluster meaningfullness comparison: whole clustering vs. STS clustering using hierarchical approach; data used: S&P 500; again, no significant difference!

27 27 why it is really surprising: dissimilarity of data sets the below two TS are very dissimilar neverteless, the experimental results obtained for buoy sensor and ocean TS (using k-means) continue showing meaningless of STS clust.

28 28 preliminary conclusions the authors reported similar results –using other clustering algorithms, e.g. EM, SOMs (self-organizing featire maps) –applied to more than 40 data sets –using Euclidean, L, Mahalanobis and “time warping distances” –and normalization techniques and for all of those combinations observed –whole clustering of TS usually is to be meaningful –sliding window clustering of STS never is meaningful

29 29 looking for explanation another comparison of both methods using cylinder, bell and funnel data sets 30 instances generated for each pattern (90 total) k-means applied (k = 3) all (three) clusters have been recognized close resemblance found

30 30 more results, more surprises the 90 TS data sets (generated) have been concatenated to one long TS sliding window: w =128, k-mean with k = 3 (as expected!) the above graph illustrates obtained result, i.e. cluster centers found by subsequence clustering (using sliding the window described above) a big surprise: the lines are sinusoids – no resemblence to any patterns in data sets used as it was for whole clustering summarizing: regardless clustering algorithm, number of clusters, datasets used: if w << m and STS then sinusoid

31 31 summarizing once again the authors conclude: –obtained approximate sinusoids with STS clustering regardless of the clustering algorithm, the number of clusters, or the dataset used –if sinusoids appear as cluster centers for every dataset, then clearly it will be impossible to distinguish one dataset’s clusters from another this is all the more true as the “joint phase” of the sinusoids is arbitrary – does not depend on any input-related parameters recall that independence on such parameters was defined as mininglessness

32 32 another concept: Hidden Constraint let’s agree with the following theorem: for any TS dataset, if TS is clustered using sliding windows with w<

33 33 (more) intuitive proof of the theorem consider a time series TS and a single datapoint t i, where w  i  m-w+1 as the sliding window pass by, t i goes on to appear exactly once in every possible location within it t i contribution to the overall shape is the same every where and must be a horizontal line the average of many horizontal lines is just another horizontal line titi

34 34 trivial match: the main idea consider TS subsequence C p being a member of a cluster searchng for similar subsequences, where one can expect them to be? in closest proximity! thus:..., C p-2, C p-1, C p+1, C p+2,...

35 35 trivial match: definition trivial match: C and M are subsequences beginning at p and q, respectively, while R is a distance M is a trivial match to C of order R: if either p = q or there does not exist a subsequence M’ beginning at q’ and such that D(C, M’) > R, and either q < q’ < p or p < q’ < q p

36 36 trivial match: observation smooth, slowly changing subsequences tend to have many trivial matches rapidly changing subsequences (i.e. their features) tend to have very few trivial matches the smooth pattern is surrounded by many trivial matches – sort of “compelling” as a cluster center highly featured, noisy pattern has few trivial matches, often ignored as a cluster center candidate illustration of the observation A: TS sequence with a cluster of 3 square waves; w = 64 B: number of trivial matches

37 37 tentative conclusions smooth patterns are surrounded by many trivial matches –extremely promising cluster center in clustering algorithms –D(C,M) < R in 1920’s, Evgeny Slutsky demonstrated that any noisy time series will converge to a sine wave after repeated applications of moving window smoothing –STS, though not exactly such, is closely related

38 38 sine qua non for STS cluster the weighted mean of the k patterns must sum to a horizontal line (constant line) rach of the k patterns must have approximately equal number of trivial matches the chances of both conditions being met is essentially zero…

39 39 a “tentative” solution proposed a method as an existence proof only that such an algorithm exists at all (conceptually) presented below is a motif-based clustering definition of K-motifs: –given TS, a subsequence of length n, distance range R –the most significant motif in TS called 1-Motif is the subsequence C 1 with the highest count of non-trivial matches –each subsequent K-motif in TS is the C K which differs from C 1 in that additionaly: D(C K, C i ) > 2R for all 1  i < K the motif (red) occurs 4 times; winding(4) dataset used

40 40 motif vs. cluster when mining motifs, we must specify an additional parameter R assuming the distance R is defined as Euclidean, motifs always define circular regions in space, whereas clusters may have arbitrary shapes motifs generally define a small subset of the data, and not the entire dataset the definition of motifs explicitly eliminates trivial matches

41 41 algorithm for motif-based clustering 1.decide on a value for k 2.discover the K-motifs in the data, for K=k  c (c is some constant about 2 to 30) 3.run k-means, or k partitional hierarchical clustering, or any other clustering algorithm on the subsequences covered by K-motifs

42 42 experimental results repeated experiment for searching cluster centers for cyllinder-bell-funnel trio the results obtained are “okey”, i.e. they resemble the original patterns (see right) of the three TS data sets (as well as results obtained using whole clustering approach)

43 43 side remark: another point of view by Anne Denton –needlessly to say her Ph.D. thesis was entitled “Fast kernel- density-based classification and clustering using P-trees”, a good motivation to defend meaningfullness of STS experimental setup: –data sets “halved” before clustering –comparing derived cluster centers from both halves using meaningfullness measure (“within/between”) and similar cluster distance measure claim: such a test is “stricter” than that reported so far (based on separate runs of k-means on same data) conclusion: kernel-based clustering shows meaningful results for subsequence clustering

44 44 references 1.Keogh, Lin, Truppel: “Clustering of Time Series Subsequences is Meaningless: Implications for Previous and Future Research” 2.Han, Kamber: “Data mining. Concepts and Techniques” 3.Lin, Keogh, Lonardim Chiu: “A symbolic representation of Time Series...” 4.Denton: “Density-based Clustering of Time Series Subsequences” 5.Sprott: “Chaos and Time-Series Analysis” references of the above ones and many pertaining web pages THANK YOU!


Download ppt "1 Clustering of Streaming Time Series is Meaningless presentation by Rafal Ladysz after the original paper by Eamonn Keogh Jessica Lin Wagner Truppel Computer."

Similar presentations


Ads by Google