Presentation is loading. Please wait.

Presentation is loading. Please wait.

Data Mining - Massey University Exploratory Data Analysis and Data Visualization Chapter 2 credits: Hand, Mannila and Smyth Cook and Swayne ggobi Lecture.

Similar presentations


Presentation on theme: "Data Mining - Massey University Exploratory Data Analysis and Data Visualization Chapter 2 credits: Hand, Mannila and Smyth Cook and Swayne ggobi Lecture."— Presentation transcript:

1 Data Mining - Massey University Exploratory Data Analysis and Data Visualization Chapter 2 credits: Hand, Mannila and Smyth Cook and Swayne ggobi Lecture Notes: Padhraic Smyth’s UCI lecture notes R Graphics book

2 Data Mining - Massey University Outline EDA Visualization –One variable –Two variables –More than two variables –Other types of data –Dimension reduction

3 Data Mining - Massey University EDA and Visualization Exploratory Data Analysis (EDA) and Visualization are important (necessary?) steps in any analysis task. can be thought of as hypothesis generation get to know your data! –distributions (symmetric, normal, skewed) –data quality problems –outliers –correlations and inter-relationships –subsets of interest –suggest functional relationships Sometimes EDA or viz might be the goal! –but be careful of multiple comparisons

4 Data Mining - Massey University

5 EDA Good data analysis practice –You should always look at every variable - you will learn something! Deveaux example histogram? –Look at descriptive statistics Use means, medians, quantiles, boxplots R functions: summary(), hist(), table() –Visualization as part of EDA Humans are the best pattern recognition software Limitations : many dimensions, large data sets

6 Data Mining - Massey University Exploratory Data Analysis (EDA) get a general sense of the data interactive and visual –(cleverly/creatively) exploit human visual power to see patterns 1 to 5 dimensions (e.g. spatial, color, time, sound) –e.g. plot raw data/statistics, reduce dimensions as needed data-driven (model-free) especially useful in early stages of data mining –detect outliers (e.g. assess data quality) –test assumptions (e.g. normal distributions or skewed?) –identify useful raw data & transforms (e.g. log(x)) Bottom line: it is always well worth looking at your data!

7 Data Mining - Massey University Summary Statistics not visual sample statistics of data X –mean:  =  i X i / n {  minimizes  i (X i -  ) 2 } –mode: most common value in X –median: X=sort(X), median = X n/2 (half below, half above) –quartiles of sorted X: Q1 value = X 0.25n, Q3 value = X 0.75 n interquartile range: value(Q3) - value(Q1) range: max(X) - min(X) = X n - X 1 –variance:  2 =  i (X i -  ) 2 / n –skewness:  i (X i -  ) 3 / [ (  i (X i -  ) 2 ) 3/2 ] zero if symmetric; right-skewed more common (e.g. you v. Bill Gates) –number of distinct values for a variable (see unique() in R) –summary() very useful.

8 Data Mining - Massey University Single Variable Visualization Histogram: –Shows center, variability, skewness, modality, –outliers, or strange patterns. –Bins matter, use nclass option of hist –Beware of real zeros hist(DiastolicBP,col='orange',nclass=20)

9 Data Mining - Massey University Histograms number of weeks a credit card was used in a given year

10 Data Mining - Massey University Histograms small change to the “anchor point” can make a big difference:

11 Data Mining - Massey University Issues with Histograms For small data sets, histograms can be misleading. Small changes in the data or to the bucket boundaries can result in very different histograms. For large data sets, histograms can be quite effective at illustrating general properties of the distribution. Histograms effectively only work with 1 variable at a time –Difficult to extend to 2 dimensions, not possible for >2 –So histograms tell us nothing about the relationships among variables

12 Data Mining - Massey University Smoothed Histograms - Density Estimates Kernel estimates smooth out the contribution of each datapoint over a local neighborhood of that point. h is the kernel width Gaussian kernel is common: Formal procedures for optimal bandwidth choice R includes many options (? density )

13 Data Mining - Massey University

14 Boxplots Shows a lot of information about a variable in one plot –Median –IQR –Outliers –Range –Skewness Negatives –Overplotting –Hard to tell distributional shape –no standard implementation in software (many options)

15 Data Mining - Massey University Time Series Example 1 steady growth trend New Year bumps summer peaks summer bifurcations in air travel (favor early/late)

16 Data Mining - Massey University Time-Series Example 2 Scotland experiment on effects of milk on better health Unexpected “step effect” ??? mean weight vs mean age for 10k control group

17 Data Mining - Massey University Time Series Example 3 spatio-temporal data –growth of Wal-Mart in US –http://projects.flowingdata.com/walmart/

18 Data Mining - Massey University Displaying Two Variables For two numeric variables, the scatterplot is the obvious choice interesting?

19 Data Mining - Massey University 2D Scatterplots standard tool to display relation between 2 variables –e.g. y-axis = response, x-axis = suspected indicator useful to answer: –x,y related? no linearly nonlinearly –variance(y) depend on x? –outliers present? R: –plot(x,y,’.’);

20 Data Mining - Massey University Scatter Plot: No apparent relationship

21 Data Mining - Massey University Scatter Plot: Linear relationship

22 Data Mining - Massey University Scatter Plot: Quadratic relationship

23 Data Mining - Massey University Scatter plot: Homoscedastic Variation of Y Does Not Depend on X

24 Data Mining - Massey University Scatter plot: Heteroscedastic variation in Y differs depending on the value of X e.g., Y = annual tax paid, X = income

25 Data Mining - Massey University Two variables - continuous Scatterplots –But can be bad with lots of data

26 Data Mining - Massey University Transparent plotting plot( rnorm(1000), rnorm(1000), col="#0000ff22", pch=16,cex=3)

27 Data Mining - Massey University Alpha blending courtesy Simon Urbanek

28 Data Mining - Massey University Jittering Jittering points helps too plot(age, TimesPregnant) plot(jitter(age),jitter(TimesPregnant)

29 Data Mining - Massey University What to do for large data sets –Contour plots Two variables - continuous

30 Data Mining - Massey University Displaying Two Variables If one variable is categorical, use variations on single dimensional methods Library(‘trellis’) histogram(~DiastolicBP | TimesPregnant==0)

31 Data Mining - Massey University Two Variables - one categorical Side by side boxplots are very effective in showing differences in a quantitative variable across factor levels –tips data do men or women tip better –orchard sprays measuring potency of various orchard sprays in repelling honeybees

32 Data Mining - Massey University Barcharts and Spineplots stacked barcharts or histograms are useful but should be used with caution spineplots are nice, but can be hard to interpret

33 Data Mining - Massey University More than two variables Scatterplot matrices : pairs(x) somewhat ineffective for categorical data

34 Data Mining - Massey University More than two variables Get creative! Conditioning on variables –trellis or lattice plots –Cleveland models on human perception, all based on conditioning –all use the R formula model –a lot of control over the output –alternate versions of standard R plot functions plot => xyplot barplot => barchart boxplot =>bwplot Earthquake data: –locations of 1000 seismic events of MB > 4.0. The events occurred in a cube near Fiji since 1964

35 Data Mining - Massey University

36

37 Starplots

38 Data Mining - Massey University Using Icons to Encode Information, e.g., Star Plots Each star represents a single observation. Star plots are used to examine the relative values for a single data point The star plot consists of a sequence of equi-angular spokes, called radii, with each spoke representing one of the variables. Useful for small data sets with up to 10 or so variables Limitations? –Small data sets, small dimensions –Ordering of variables may affect perception 1 Price 2 Mileage (MPG) Repair Record (1 = Worst, 5 = Best) Repair Record (1 = Worst, 5 = Best) 5 Headroom 6 Rear Seat Room 7 Trunk Space 8 Weight 9 Length

39 Data Mining - Massey University Chernoff’s Faces described by ten facial characteristic parameters: head eccentricity, eye eccentricity, pupil size, eyebrow slant, nose size, mouth shape, eye spacing, eye size, mouth length and degree of mouth opening Chernoff faces applet more icon plots

40 Data Mining - Massey University Chernoff faces

41 Data Mining - Massey University Mosaic plots for categorical data

42 Data Mining - Massey University Mosaic Plots Good for plotting many categorical variables sensitive to the order which they are applied

43 Data Mining - Massey University Networks and Graphs creating networks where they might not obviously exist

44 Data Mining - Massey University Interactive Visualization Multi-dimensional viz is easiest using a tool that allows for variable selction –ggobi is such a tool. Brushing and linking of different plots demo –http://www.ggobi.org/book/chap-toolbox/toolbox-brushing-categorical.mov

45 Data Mining - Massey University What’s missing? pie charts –very popular –good for showing simple relations of proportions –hard to get a real sense of what is going on –barplots, histograms usually better (but less pretty) 3D –nice to be able to show three dimensions –hard to do well –often done poorly –3d best shown through “spinning” in 2D uses various types of projecting into 2D see video

46 Data Mining - Massey University

47 Dimension Reduction One way to visualize high dimensional data is to reduce it to 2 or 3 dimensions –Variable selection e.g. stepwise –Principle Components find linear projection onto p-space with maximal variance –Multi-dimensional scaling takes a matrix of (dis)similarities and embeds the points in p- dimensional space to retain those similarities

48 Data Mining - Massey University Lab #2 Explore graphics with demo(graphics) Download Di Cook’s music data set and create some simple graphics Use the USArrests data to plot scatterplots and do rudimentary interactive viz with identify().


Download ppt "Data Mining - Massey University Exploratory Data Analysis and Data Visualization Chapter 2 credits: Hand, Mannila and Smyth Cook and Swayne ggobi Lecture."

Similar presentations


Ads by Google