Presentation is loading. Please wait.

Presentation is loading. Please wait.

Object Orie’d Data Analysis, Last Time Cornea Data & Robust PCA –Elliptical PCA Big Picture PCA –Optimization View –Gaussian Likelihood View –Correlation.

Published byGeorgia Leonard Modified over 8 years ago

Similar presentations

Presentation on theme: "Object Orie’d Data Analysis, Last Time Cornea Data & Robust PCA –Elliptical PCA Big Picture PCA –Optimization View –Gaussian Likelihood View –Correlation."— Presentation transcript:

1 Object Orie’d Data Analysis, Last Time Cornea Data & Robust PCA –Elliptical PCA Big Picture PCA –Optimization View –Gaussian Likelihood View –Correlation PCA Finding Clusters with PCA –Can be useful –But can also miss some

2 Participant Presentations 36 participants ~ 15 minutes each 4 per class meeting Requires 9 class meetings … Have revised Schedule on Class Web PageClass Web Page Please sign up (email) for preferred time (1 st come, 1 st served) Please help for next week (if you can just pull something off the shelf)

3 PCA to find clusters PCA of Mass Flux Data:

4 PCA to find clusters Return to Investigation of PC1 Clusters: Can see 3 bumps in smooth histogram Main Question: Important structure or sampling variability? Approach: SiZer (SIgnificance of ZERo crossings of deriv.)

5 SiZer Background Two Major Settings: 2-d scatterplot smoothing 1-d histograms (continuous data, not discrete bar plots) Central Question: Which features are really there? Solution, Part 1: Scale space Solution, Part 2: SiZer

6 SiZer Background Bralower ’ s Fossil Data - Global Climate

7 SiZer Background Smooths - Suggest Structure - Real?

8 SiZer Background Smooths of Fossil Data (details given later) Dotted line: undersmoothed (feels sampling variability) Dashed line: oversmoothed (important features missed?) Solid line: smoothed about right? Central question: Which features are “ really there ” ?

9 SiZer Background Smoothing Setting 2: Histograms Family Income Data: British Family Expenditure Survey For the year 1975 Distribution of Family Incomes ~ 7000 families

10 SiZer Background Family Income Data, Histogram Analysis:Histogram Analysis Again under- and over- smoothing issues Perhaps 2 modes in data? Histogram Problem 1: Binwidth (well known) Central question: Which features are “ really there ” ? e.g. 2 modes? Same problem as existence of “ clusters ” in PCA

11 SiZer Background Why not use (conventional) histograms? Histogram Problem 2: Bin shiftBin shift (less well known) For same binwidth Get much different impression By only “ shifting grid location “ Get it right by chance?

12 SiZer Background Why not use (conventional) histograms? Solution to binshift problem: Average over all shifts 1st peak all in one bin: bimodal 1st peak split between bins: unimodal Smooth histo ’ m provides understanding, So should use for data analysis Another name: Kernel Density Estimate

13 SiZer Background Why not use (conventional) histograms? Solution to binshift problem: Average over all shifts 1st peak all in one bin: bimodal 1st peak split between bins: unimodal Smooth histo ’ m provides understanding, So should use for data analysis Another name: Kernel Density Estimate

14 SiZer Background Kernel density estimation Recommended Reference ( many books): Wand, M. P. and Jones, M. C. (1995) View 1: Smooth histogram View 2: Distribute probability mass, according to data E.g. Chondrite dataChondrite data (from how many sources?)

15 SiZer Background Kernel density estimation (cont.) Central Issue: width of window, i.e. “ bandwidth ”, E.g. Incomes data:Incomes data Controls critical amount of smoothing Old Approach: Data based bandwidth selection Recommended reference: Jones et al (1996) New Approach: "scale space" (look at all of them)

16 SiZer Background Scale Space – Idea from Computer Vision Conceptual basis: Oversmoothing = “ view from afar ” (macroscopic) Undersmoothing = “ zoomed in view ” (microscopic) Main idea: all smooths contain useful information, so study “ full spectrum ” (i. e. all smoothing levels) Recommended reference: Lindeberg (1994)

17 SiZer Background Fun Scale Spaces Views (of Family Incomes Data) Spectrum Movie

18 SiZer Background Fun Scale Spaces Views (Incomes Data) Spectrum Overlay

19 SiZer Background Fun Scale Spaces Views (Incomes Data) Surface View

20 SiZer Background Fun Scale Spaces Views (of Family Incomes Data) Note: The scale space viewpoint makes Data Dased Bandwidth Selection Much less important (than I once thought ….)

21 SiZer Background SiZer: Significance of Zero crossings, of the derivative, in scale space Combines: –needed statistical inference –novel visualization To get: a powerful exploratory data analysis method Main reference: Chaudhuri & Marron (1999)

22 SiZer Background Basic idea: a bump is characterized by: an increase followed by a decrease Generalization: Many features of interest captured by sign of the slope of the smooth Foundation of SiZer: Statistical inference on slopes, over scale space

23 SiZer Background SiZer Visual presentation: Color map over scale space: Blue: slope significantly upwards (derivative CI above 0) Red: slope significantly downwards (derivative CI below 0) Purple: slope insignificant (derivative CI contains 0)

24 SiZer Background SiZer analysis of Fossils data:Fossils data Upper Left: Scatterplot, family of smooths, 1 highlighted Upper Right: Scale space rep ’ n of family, with SiZer colors Lower Left: SiZer map, more easy to view Lower Right: SiCon map – replace slope by curvature Slider (in movie viewer) highlights different smoothing levels

25 SiZer Background SiZer analysis of Fossils data (cont.):Fossils data Oversmoothed (top of SiZer map): Decreases at left, not on right Medium smoothed (middle of SiZer map): Main valley significant, and left most increase Smaller valley not statistically significant Undersmoothed (bottom of SiZer map): “ noise wiggles ” not significant Additional SiZer color: gray - not enough data for inference

26 SiZer Background SiZer analysis of Fossils data (cont.):Fossils data Common Question: Which is right? Decreases on left, then flat (top of SiZer map) Up, then down, then up again (middle of SiZer map) No significant features (bottom of SiZer map) Answer: All are right Just different scales of view, i.e. levels of resolution of data

27 SiZer Background SiZer analysis of British Incomes data:British Incomes data Oversmoothed: Only one mode Medium smoothed: Two modes, statistically significant Confirmed by Schmitz & Marron, (1992) Undersmoothed: many noise wiggles, not significant Again: all are correct, just different scales

28 SiZer Background Historical Note & Acknowledgements: Scale Space: S. M. Pizer SiZer: Probal Chaudhuri Main Reference: Chaudhuri & Marron (1999)

29 SiZer Background Toy E.g. - Marron & Wand Trimodal #9 Increasing n Only 1 Signif ’ t Mode Now 2 Signif ’ t Modes Finally all 3 Modes

30 SiZer Background E.g. - Marron & Wand Discrete Comb #15 Increasing n Coarse Bumps Only Now Fine bumps too Someday: “ draw ” local Bandwidth on SiZer map

31 SiZer Background Finance "tick data":tick data (time, price) of single stock transactions Idea: "on line" version of SiZer for viewing and understanding trends Notes: trends depend heavily on scale double points and more background color transition (flop over at top)

32 SiZer Background Internet traffic data analysis: SiZer analysis of time series of packet times at internet hub (UNC) across very wide range of scales needs more pixels than screen allows thus do zooming view (zoom in over time) –zoom in to yellow bd ’ ry in next frame –readjust vertical axis

33 SiZer Background Internet traffic data analysis (cont.) Insights from SiZer analysis: Coarse scales: amazing amount of significant structure Evidence of self-similar fractal type process? Fewer significant features at small scales But they exist, so not Poisson process Poisson approximation OK at small scale??? Smooths (top part) stable at large scales?

34 Dependent SiZer Park, Marron, and Rondonotti (2004) SiZer compares data with white noise Inappropriate in time series Dependent SiZer compares data with an assumed model Visual Goodness of Fit test

35 Dep ’ ent SiZer : 2002 Apr 13 Sat 1 pm – 3 pm

36 Zoomed view (to red region, i.e. “ flat top ” )

37 Further Zoom: finds very periodic behavior!

38 Possible Physical Explanation IP “ Port Scan ” Common device of hackers Searching for “ break in points ” Send query to every possible (within UNC domain): –IP address –Port Number Replies can indicate system weaknesses Internet Traffic is hard to model

Download ppt "Object Orie’d Data Analysis, Last Time Cornea Data & Robust PCA –Elliptical PCA Big Picture PCA –Optimization View –Gaussian Likelihood View –Correlation."

Similar presentations

Sales Forecasting using Dynamic Bayesian Networks Steve Djajasaputra SNN Nijmegen The Netherlands.

Sales Forecasting using Dynamic Bayesian Networks Steve Djajasaputra SNN Nijmegen The Netherlands.
Very simple to create with each dot representing a data value. Best for non continuous data but can be made for and quantitative data 2004 US Womens Soccer.

Very simple to create with each dot representing a data value. Best for non continuous data but can be made for and quantitative data 2004 US Womens Soccer.
Introduction to Non Parametric Statistics Kernel Density Estimation.

Introduction to Non Parametric Statistics Kernel Density Estimation.
Object Orie’d Data Analysis, Last Time •Clustering –Quantify with Cluster Index –Simple 1-d examples –Local mininizers –Impact of outliers •SigClust –When.

Object Orie’d Data Analysis, Last Time •Clustering –Quantify with Cluster Index –Simple 1-d examples –Local mininizers –Impact of outliers •SigClust –When.
Toyota InfoTechnology Center U.S.A, Inc. 1 Mixture Models of End-host Network Traffic John Mark Agosta, Jaideep Chandrashekar, Mark Crovella, Nina Taft.

Toyota InfoTechnology Center U.S.A, Inc. 1 Mixture Models of End-host Network Traffic John Mark Agosta, Jaideep Chandrashekar, Mark Crovella, Nina Taft.
Further Maths Hello 2014 Univariate Data. What is data? Data is another word for information. By studying data, we are able to display the information.

Further Maths Hello 2014 Univariate Data. What is data? Data is another word for information. By studying data, we are able to display the information.
Mr F’s Maths Notes Statistics 8. Scatter Diagrams.

Mr F’s Maths Notes Statistics 8. Scatter Diagrams.
The Digital Image.

The Digital Image.
Introduction to Behavioral Statistics Measurement The assignment of numerals to objects or events according to a set of rules. The rules used define.

Introduction to Behavioral Statistics Measurement The assignment of numerals to objects or events according to a set of rules. The rules used define.
Chapter 3: Central Tendency. Central Tendency In general terms, central tendency is a statistical measure that determines a single value that accurately.

Chapter 3: Central Tendency. Central Tendency In general terms, central tendency is a statistical measure that determines a single value that accurately.
Mean-shift and its application for object tracking

Mean-shift and its application for object tracking
Quantitative Skills 1: Graphing

Quantitative Skills 1: Graphing
FUNCTIONS AND MODELS 1. In this section, we assume that you have access to a graphing calculator or a computer with graphing software. FUNCTIONS AND MODELS.

FUNCTIONS AND MODELS 1. In this section, we assume that you have access to a graphing calculator or a computer with graphing software. FUNCTIONS AND MODELS.
1 Introduction to Dijet Resonance Search Exercise John Paul Chou, Eva Halkiadakis, Robert Harris, Kalanand Mishra and Jason St. John CMS Data Analysis.

1 Introduction to Dijet Resonance Search Exercise John Paul Chou, Eva Halkiadakis, Robert Harris, Kalanand Mishra and Jason St. John CMS Data Analysis.
THE MANAGEMENT AND CONTROL OF QUALITY, 5e, © 2002 South-Western/Thomson Learning TM 1 Chapter 9 Statistical Thinking and Applications.

THE MANAGEMENT AND CONTROL OF QUALITY, 5e, © 2002 South-Western/Thomson Learning TM 1 Chapter 9 Statistical Thinking and Applications.
Object Orie’d Data Analysis, Last Time Statistical Smoothing –Histograms – Density Estimation –Scatterplot Smoothing – Nonpar. Regression SiZer Analysis.

Object Orie’d Data Analysis, Last Time Statistical Smoothing –Histograms – Density Estimation –Scatterplot Smoothing – Nonpar. Regression SiZer Analysis.
Object Orie’d Data Analysis, Last Time Cornea Data & Robust PCA –Elliptical PCA Big Picture PCA –Optimization View –Gaussian Likelihood View –Correlation.

Object Orie’d Data Analysis, Last Time Cornea Data & Robust PCA –Elliptical PCA Big Picture PCA –Optimization View –Gaussian Likelihood View –Correlation.
Robust PCA Robust PCA 3: Spherical PCA. Robust PCA.

Robust PCA Robust PCA 3: Spherical PCA. Robust PCA.
TYPES OF STATISTICAL METHODS USED IN PSYCHOLOGY Statistics.

TYPES OF STATISTICAL METHODS USED IN PSYCHOLOGY Statistics.
Visual Displays of Data Chapter 3. Uses of Graphs Positive and negative uses – Can accurately and succinctly present information – Can reveal/conceal.

Visual Displays of Data Chapter 3. Uses of Graphs Positive and negative uses – Can accurately and succinctly present information – Can reveal/conceal.

Similar presentations

Ads by Google