1. High Performance Discovery from Time Series Streams
Dennis Shasha
Joint work with Yunyue Zhu
Courant Institute, New York University

2. Overall Outline Data mining – both classical and activist
Algorithmic tools for time series
Surprise.

3. Goal of this work
Time series are important in so many applications – biology, medicine, finance, music, physics, …
A few fundamental operations occur all the time: burst detection, correlation, pattern matching.
Do them fast to make data exploration faster, real time, and more fun.
Extend functionality for music and science.

4. StatStream (VLDB,2002): Example
Stock prices streams
The New York Stock Exchange (NYSE)
50,000 securities (streams); 100,000 ticks (trade and quote)
Pairs Trading, a.k.a. Correlation Trading
Query:“which pairs of stocks were correlated with a value of over 0.9 for the last three hours?”
XYZ and ABC have been correlated with a correlation of 0.95 for the last three hours.
Now XYZ and ABC become less correlated as XYZ goes up and ABC goes down.
They should converge back later.
I will sell XYZ and buy ABC …

5. Online Detection of High Correlation
Given tens of thousands of high speed time series data streams, to detect high-value correlation, including synchronized and time-lagged, over sliding windows in real time.
Real time
high update frequency of the data stream
fixed response time, online
Correlated!

6. Online Detection of High Correlation
Given tens of thousands of high speed time series data streams, to detect high-value correlation, including synchronized and time-lagged, over sliding windows in real time.
Real time
high update frequency of the data stream
fixed response time, online

7. Online Detection of High Correlation
Given tens of thousands of high speed time series data streams, to detect high-value correlation, including synchronized and time-lagged, over sliding windows in real time.
Real time
high update frequency of the data stream
fixed response time, online
Correlated!

8. StatStream: Algorithm
Naive algorithm
N : number of streams
w : size of sliding window
space O(N) and time O(N2w) VS space O(N2) and time O(N2) .
Suppose that the streams are updated every second.
With a Pentium 4 PC, the exact computing method can only monitor 700 streams with a delay of 2 minutes.
Our Approach
Use Discrete Fourier Transform to approximate correlation
Use grid structure to filter out unlikely pairs
Our approach can monitor 10,000 streams with a delay of 2 minutes.

9. StatStream: Stream synoptic data structure
Three level time interval hierarchy
Time point, Basic window, Sliding window
Basic window (the key to our technique)
The computation for basic window i must finish by the end of the basic window i+1
The basic window time is the system response time.
Digests
Basic window digests:
sum
DFT coefs
Sliding window
Basic window
Time point

10. StatStream: Stream synoptic data structure
Three level time interval hierarchy
Time point, Basic window, Sliding window
Basic window (the key to our technique)
The computation for basic window i must finish by the end of the basic window i+1
The basic window time is the system response time.
Digests
Basic window digests:
sum
DFT coefs
Sliding window
Basic window
Time point
Basic window digests:
sum
DFT coefs

11. StatStream: Stream synoptic data structure
Three level time interval hierarchy
Time point, Basic window, Sliding window
Basic window (the key to our technique)
The computation for basic window i must finish by the end of the basic window i+1
The basic window time is the system response time.
Digests
Basic window digests:
sum
DFT coefs
Sliding window
Basic window
Time point
Basic window digests:
sum
DFT coefs
Sliding window digests:
sum
DFT coefs

12. StatStream: Stream synoptic data structure
Three level time interval hierarchy
Time point, Basic window, Sliding window
Basic window (the key to our technique)
The computation for basic window i must finish by the end of the basic window i+1
The basic window time is the system response time.
Digests
Basic window digests:
sum
DFT coefs
Sliding window
Basic window
Time point
Basic window digests:
sum
DFT coefs
Sliding window digests:
sum
DFT coefs

13. StatStream: Stream synoptic data structure
Three level time interval hierarchy
Time point, Basic window, Sliding window
Basic window (the key to our technique)
The computation for basic window i must finish by the end of the basic window i+1
The basic window time is the system response time.
Digests
Basic window digests:
sum
DFT coefs
Basic window digests:
sum
DFT coefs
Basic window digests:
sum
DFT coefs
Time point
Basic window
Sliding window

14. Synchronized Correlation Uses Basic Windows
Inner-product of aligned basic windows
Stream x
Stream y
Basic window
Sliding window
Inner-product within a sliding window is the sum of the inner-products in all the basic windows in the sliding window.

15. Approximate Synchronized Correlation
Approximate with an orthogonal function family (e.g. DFT)
x x x x x x x x8
f1(1) f1(2) f1(3) f1(4) f1(5) f1(6) f1(7) f1(8)
f2(1) f2(2) f2(3) f2(4) f2(5) f2(6) f2(7) f2(8)
f3(1) f3(2) f3(3) f3(4) f3(5) f3(6) f3(7) f3(8)

16. Approximate Synchronized Correlation
Approximate with an orthogonal function family (e.g. DFT)
x x x x x x x x8

17. Approximate Synchronized Correlation
Approximate with an orthogonal function family (e.g. DFT)
x x x x x x x x8
y y y y y y y y8

18. Approximate Synchronized Correlation
Approximate with an orthogonal function family (e.g. DFT)
Inner product of the time series Inner product of the digests
The time and space complexity is reduced from O(b) to O(n).
b : size of basic window
n : size of the digests (n<<b)
e.g. 120 time points reduce to 4 digests
x x x x x x x x8
y y y y y y y y8

19. Approximate lagged Correlation
Inner-product with unaligned windows
sliding window
The time complexity is reduced from O(b) to O(n2) , as opposed to O(n) for synchronized correlation. Reason: terms for different frequencies are non-zero in the lagged case.

20. Grid Structure(to avoid checking all pairs)
The DFT coefficients yields a vector.
High correlation => closeness in the vector space
We can use a grid structure and look in the neighborhood, this will return a super set of highly correlated pairs. x

21. Empirical Study : Speed
Our algorithm is parallelizable.

22. Empirical Study: Precision
Approximation errors
Larger size of digests, larger size of sliding window and smaller size of basic window give better approximation
The approximation errors are small for the stock data.

23. Sketches : Random Projection
Correlation between time series of the returns of stock
Since most stock price time series are close to random walks, their return time series are close to white noise
DFT/DWT can’t capture approximate white noise series because there is no clear trend (too many frequency components).
Solution : Sketches (a form of random landmark)
Sketches pool: matrix of random variables drawn from stable distribution
Sketches : The random projection of all time series to lower dimensions by multiplication with the same matrix
The Euclidean distance (correlation) between time series is approximated by the distance between their sketches with a probabilistic guarantee.

24. Burst Detection

25. Burst Detection: Applications
Discovering intervals with unusually large numbers of events.
In astrophysics, the sky is constantly observed for high-energy particles. When a particular astrophysical event happens, a shower of high-energy particles arrives in addition to the background noise. Might last milliseconds or days…
In telecommunications, if the number of packages lost within a certain time period exceeds some threshold, it might indicate some network anomaly. Exact duration is unknown.
In finance, stocks with unusual high trading volumes should attract the notice of traders (or perhaps regulators).

26. Bursts across different window sizes in Gamma Rays
Challenge : to discover not only the time of the burst, but also the duration of the burst.

27. Elastic Burst Detection: Problem Statement
Problem: Given a time series of positive numbers x1, x2,..., xn, and a threshold function f(w), w=1,2,...,n, find the subsequences of any size such that their sums are above the thresholds:
all 0<w<n, 0<m<n-w, such that xm+ xm+1+…+ xm+w-1 ≥ f(w)
Brute force search : O(n^2) time
Our shifted wavelet tree (SWT): O(n+k) time.
k is the size of the output, i.e. the number of windows with bursts

28. Burst Detection: Data Structure and Algorithm
Define threshold for node for size 2k to be threshold for window of size 1+ 2k-1

29. Burst Detection: Example

30. Burst Detection: Example
False Alarm
True Alarm

31. False Alarms (requires work, but no errors)

32. Empirical Study : Gamma Ray Burst

33. Extension to other aggregates
SWT can be used for any aggregate that is monotonic
SUM, COUNT and MAX are monotonically increasing
the alarm threshold is aggregate<threshold
MIN is monotonically decreasing
Spread =MAX-MIN
Application in Finance
Stock with burst of trading or quote(bid/ask) volume (Hammer!)
Stock prices with high spread

34. Empirical Study : Stock Price Spread Burst

35. Extension to high dimensions

36. Elastic Burst in two dimensions
Population Distribution in the US

37. How to find the threshold for Elastic Burst?
Suppose that the moving sum of a time series is a random variable from a normal distribution.
Let the number of bursts in the time series within sliding window size w be So(w) and its expectation be Se(w).
Se(w) can be computed from the historical data.
Given a threshold probability p, we set the threshold of burst f(w) for window size w such that Pr[So(w) ≥ f(w)] ≤p.

38. Find threshold for Elastic Bursts
Φ(x) is the normal cdf, so symmetric around 0:
Therefore
Φ(x) p x
Φ-1(p)

39. Summary
Able to detect bursts of many different durations in essentially linear time.
Can be used both for time series and for spatial searching.
Can specify thresholds either with absolute numbers or with probability of hit.
Algorithm is simple to implement and has low constants (code is available).
Ok, it’s embarrassingly simple.

40. With a Little Help From My Warped Correlation
Karen’s humming Match:
Dennis’s humming Match:
“What would you do if I sang out of tune?"
Yunyue’s humming Match:

41. Related Work in Query by Humming
Traditional method: String Matching [Ghias et. al. 95, McNab et.al. 97,Uitdenbgerd and Zobel 99]
Music represented by string of pitch directions: U, D, S (degenerated interval)
Hum query is segmented to discrete notes, then string of pitch directions
Edit Distance between hum query and music score
Problem
Very hard to segment the hum query
Partial solution: users are asked to hum articulately
New Method : matching directly from audio [Mazzoni and Dannenberg 00]
slowed down by DTW

42. Time Series Representation of Query
Segment this!
An example hum query
Note segmentation is hard!

43. How to deal with poor hum queries?
No absolute pitch
Solution: the average pitch is subtracted
Incorrect tempo
Solution: Uniform Time Warping
Inaccurate pitch intervals
Solution: return the k-nearest neighbors
Local timing variations
Solution: Dynamic Time Warping

44. Dynamic Time Warping
Euclidean distance: sum of point-by-point distance
DTW distance: allowing stretching or squeezing the time axis locally

45. Envelope Transform using Piecewise Aggregate Approximation(PAA) [Keogh VLDB 02]

46. Envelope Transform using Piecewise Aggregate Approximation(PAA)
Advantage of tighter envelopes
Still no false negatives, and fewer false positives

47. Container Invariant Envelope Transform
Container-invariant A transformation T for envelope such that
Theorem: if a transformation is Container-invariant and Lower-bounding, then the distance between transformed times series x and transformed envelope of y lower bound their DTW distance.
Feature Space

48. The Vision
Ability to match time series quickly may open up entire new application areas, e.g. fast reaction to external events, music by humming and so on.
Main problems: accuracy, excessive specification.
Reference (advert): High Performance Discovery in Time Series (Springer 2004)