Presentation on theme: "A quick tour of the datasets for VLDB 2008 (does not include datasets already in the UCR archive)"— Presentation transcript:
A quick tour of the datasets for VLDB 2008 (does not include datasets already in the UCR archive)
Number of training objects80 Number of testing objects2320 Number of classes8 Length of time series1024 Euclidean Distance accuracy95.05% Some Name The dataset came from blah blah blah blah Why is difficult? Blah blah Formatting Note This is the one nearest neighbor, Euclidean distance accuracy for just the training set, measured using leaving-one-out. I measured the accuracy of 1NN-ED on the training set (only). This was to make sure we do not have any formatting misunderstandings You should test the 1NN-ED on the training set (only), and see if you get the same answers. Do this first, otherwise we may waste time.
Number of training objects55 Number of testing objects2345 Number of classes8 Length of time series1024 Euclidean Distance accuracy98.18% This figure is from [a]. The only change we made was to flip the data left to right, (and z-normalization) This dataset is described in Mallat, S. G. (1998), A Wavelet Tour of Signal Processing, San Diego: Academic Press. However the data we used was donated by Jeong [a]. The data was obtained by randomly choosing 55 objects for the training set and choosing the rest for the testing set. Each time series was also reversed. [a] M. K. Jeong, J. C. Lu, X. Huo, B. Vidakovic, and D. Chen (2006), "Wavelet- based Data Reduction Techniques for Process Fault Detection," Technometrics, 48(1), 26-40. http://web.utk.edu/~mjeong/http://web.utk.edu/~mjeong/ MALLAT TECHNOMETRICS Why is difficult? Many classes Some classes are globally similar, and have only local differences. Small training set (In [a], using 1024 instances for training, a decision tree got 96.87% accuracy. Since this was too easy, we reduced the size of the training set significantly).
Number of training objects67 Number of testing objects1029 Number of classes2 Length of time series24 Euclidean Distance accuracy95.522 See Keogh ICDM06 Eamonn Keogh, Li Wei, Xiaopeng Xi, Stefano Lonardi, Jin Shieh, Scott Sirowy (2006). Intelligent Icons: Integrating Lite-Weight Data Mining and Visualization into GUI Operating Systems. ICDM 2006. ItalyPowerDemand (3 years) Task Distinguish days from Oct to March (inclusive) from April to September Why is difficult? Borderline days (late Sep vs early Oct) Unusual days (soccer games etc) Under sampled data? August is radically different to the rest of the summer months. 1357911131517192123 From Keogh ICDM06
Number of training objects40 Number of testing objects1380 Number of classes4 Length of time series1639 Euclidean Distance accuracy85.00% CinC_ECG_torso Task Data is taken from ECG data for multiple torso-surface sites. There are 4 classes (4 different people) Why is difficult? See gray strip on figure. Depending on location on the body, the peak can be positive, neutral or negative. Similar remarks apply to all features. The figure shows aligned data, but the challenge data is slightly out of alignment.
Number of training objects155 Number of testing objects308 Number of classes5 Length of time series1092 Euclidean Distance accuracy51.61% Haptics Task Data is taken from 5 people entering their “passgraph” on a touchscreen. We only consider the X axis. Why is difficult? Small training set I think (but have not checked this) that the high variability at the beginning and end of the time series is just noise. We are just looking at the X-axis for simplicity, we should also be looking at Y- axis, pen pressure, pen acceleration… Novel Shoulder-Surfing Resistant Haptic-based Graphical Password Behzad Malek, Mauricio Orozco, Abdulmotaleb El Saddik 4 sample time series (before normalizing) 020040060080010001200 40 60 80 100 120 140 160 180 200
Number of training objects25 Number of testing objects995 Number of classes6 Length of time series398 Euclidean Distance accuracy84.0% Symbols Task Thirteen people participated in this experiment. They were asked to copy the randomly appearing symbol as best they could. There were 3 possible symbols, each person contributed about 30 attempts. Why is difficult? Individuality of the 13 individuals Each of the 6 classes looks only at the X or Y axis, we really should have 3 classes looking at the X and Y axis Two of the symbols are very very similar on the Y-axis Small training set This dataset was created for the contest by Jill Brady, a grad student at UCR. We gratefully acknowledge her. 050100150200250300350400 050100150200250300350400 X-axisY-axis
Number of training objects381 Number of testing objects760 Number of classes10 Length of time series99 Euclidean Distance accuracy72.178% MedicalImages Task The data are histograms of pixel intensity of medical images. The classes are “different human body regions.” Why is difficult? It is not clear that treating the raw data as time series is the best overall approach for this problems, but the original authors due report success with a “time warping” measure. Original time series are of different lengths, some are very short, making them all the same length may have introduced artifacts This dataset was donated by Joaquim C. Felipe, Agma J. M. Traina and Caetano Traina Jr. 0102030405060708090100
Number of training objects20 Number of testing objects601 Number of classes2 Length of time series70 Euclidean Distance accuracy90.0% SonyAIBORobotSurface Task The robot has roll/pitch/yaw accelerometers, here we looked at just X- axis. The task is to detect the surface being walked on. Why is difficult? Noisy data Small training set. See figure at left, with enough data it looks easy. This dataset was donated by Manuela Veloso and Douglas Vail of Carnegie Mellon University Red: Cement. Blue Carpet
Number of training objects27 Number of testing objects953 Number of classes2 Length of time series65 Euclidean Distance accuracy85.185% SonyAIBORobotSurfaceII Task The robot has roll/pitch/yaw accelerometers, here we looked at just Z- axis. The task is to detect the surface being walked on. Why is difficult? Noisy data Small training set. See figure at left, with enough data it looks easier. This dataset was donated by Manuela Veloso and Douglas Vail of Carnegie Mellon University Red: Cement. Blue Carpet or Field
Number of training objects23 Number of testing objects1139 Number of classes2 Length of time series82 Euclidean Distance accuracy78.261% TwoLeadECG Task Time series is taken from MIT-BIH Long- Term ECG Database (ltdb) Record ltdb/15814, begin at time 420, ending at 1019. The task is to distinguish between signal 0 and signal 1. Why is difficult? Subtle distinctions Small training set Beat extractor does not produce perfect alignment, but after using EM to align the signal (figure at left) it is clear that certain parts of the signal are more informative.
Number of training objects1000 Number of testing objects8236 Number of classes3 Length of time series1024 Euclidean Distance accuracy86.00% StarLightCurves Task Time series are star light curves falling into three classes. Why is difficult? Two of the three classes are quite similar. Large dataset (but the real datasets have billions of these!) Phase was aligned using standard astronomy tricks. However we tried circular shift invariant Euclidean distance (see [a]) our accuracy improved, suggesting the alignment is not perfect. 1 - CEPH 2 - EB 3 - RRL [a] Eamonn Keogh, Li Wei, Xiaopeng Xi, Sang-Hee Lee and Michail Vlachos (2006) LB_Keogh Supports Exact Indexing of Shapes under Rotation Invariance with Arbitrary Representations and Distance Measures. VLDB 2006.
Number of training objects16 Number of testing objects306 Number of classes4 Length of time series345 Euclidean Distance accuracy93.75% DiatomSizeReduction Task “Each successive generation of a clonaly reproducing diatom is slightly smaller than its forebears.”[a] Why is difficult? Small training set Possible errors caused by image processing step. Change in scale of diatoms shows up as “warping”. [a] http://rbg-web2.rbge.org.uk/DIADIST/index.htm?srseries.htm&mainhttp://rbg-web2.rbge.org.uk/DIADIST/index.htm?srseries.htm&main [b] Xiaopeng Xi, et al (2007). Finding Motifs in Database of Shapes. SDM'07 (many omitted) Eunotia tenella Gomphonema augur Fragilariforma bicapitata Stauroneis smithii 020040060080010001200 [b]
Number of training objects20 Number of testing objects1252 Number of classes2 Length of time series84 Euclidean Distance accuracy75.00% Motes Task Sensor data used in paper [b]. Here the task is to distinguish between sensor q8calibHumid and sensor q8calibHumTemp. The raw data has dropouts, which I left in. Why is difficult? Small training set. Lots of dropouts (however, when noise is removed, should be very easy). Here the dropouts had value zero. But after z-normalization these values changed. It would have been easier to do smart smoothing if the data was not normalized. [a] Raw data from Carlos Guestrin (CMU), Classification version by Keogh [b] Jimeng Sun, Spiros Papadimitriou, Christos Faloutsos: Online Latent Variable Detection in Sensor Networks. ICDE 2005: 1126-1127 050100150200250300350 0 5 10 15 20 25
Number of training objects487 Number of testing objects3840 Number of classes3 Length of time series166 Euclidean Distance accuracy63.383% ChlorineConcentration Task Sensor data used in paper [b]. Multiple sensors have spatial correlation, which I arbitrarily divided into 3 sets Why is difficult? The borderline cases are hard to classify. However with more data it would be easy. For example, when I randomly sample k items from the labeled test set, and do INN ED classification, I get… 1000 -> 76.5% accuracy 2000 -> 89.85% accuracy 3000 -> 96.8% accuracy [a] Stacia Thompson and Jeanne M. VanBriesen (CMU) Classification version by Keogh [b] Jimeng Sun, Spiros Papadimitriou, Christos Faloutsos: Online Latent Variable Detection in Sensor Networks. ICDE 2005: 1126-1127
Number of training objects23 Number of testing objects861 Number of classes2 Length of time series136 Euclidean Distance accuracy82.609% ECGFiveDays Task Data is from a 67 year old male. The two classes are simply 1)ECG date: 12/11/1990 2)ECG date: 17/11/1990 Why is difficult? Wandering baseline was not removed, this shows up as linear drift. Beat extractor does not produce perfect alignment, but after using EM to align the signal (figure at left) it is clear that certain parts of the signal are more informative. Wandering baseline Excerpt of Class 1
Number of training objects100 Number of testing objects550 Number of classes7 Length of time series1882 Euclidean Distance accuracy30.00% InlineSkate Task This data was been collected from experiments with inline speed skaters on a treadmill. Each time series represents an angular measurement of the ankle during one movement cycle. Cycles were of different lengths, we made them all the same length. Why is difficult? Lots of “warping” Long time series (for algorithms that scale poorly in dimensionality). The “cycle” extraction algorithm might not be perfect (this was done before we saw the data) The data was provided by Fabian Moerchen and Olaf Hoos.
Number of training objects200 Number of testing objects2050 Number of classes14 Length of time series131 Euclidean Distance accuracy75.50 FacesUCR Task This data consists of faces of grad students transformed into “time series” Why is difficult? Variation of head angle and expression. Some have glasses/no glasses versions All grad students look alike (well, some do). The transformation algorithm is a little brittle (we have since found more robust techniques). Photographs by Chotirat "Ann" Ratanamahatana, image conversion by Xiaopeng Xi and Eamonn Keogh
Number of training objects267 Number of testing objects638 Number of classes25 Length of time series270 Euclidean Distance accuracy58.80 WordsSynonyms Task This dataset consists of word profiles for George Washington's manuscripts. This dataset is the “50-words” dataset, remapped to 25 classes. The data was flipped left-right so that it would not be recognized. Why is difficult? There are two ways to be a member of each class. In this case, length normalization clearly does throw away useful info. Errors from the difficult task of OCR on old documents 050100150200250300350400450 0 0.5 1 The data was provided by Toni M. Rath and R. Manmatha. The time series representation of words is known to be very competitive with other representations [a]. Here the results might not be competitive because we are only using one (of four) time series per word, we are normalizing, and we have small training sets. [a] Word spotting for historical documents. Toni M. Rath and R. Manmatha International Journal on Document Analysis and Recognition. Volume 9, Numbers 2-4 / April, 2007