Presentation is loading. Please wait.

Presentation is loading. Please wait.

Detecting Novel Associations in Large Data Sets Sean Patrick Murphy A pragmatic discussion of by David N. Reshef, Yakir Reshef, Hilary.

Similar presentations


Presentation on theme: "Detecting Novel Associations in Large Data Sets Sean Patrick Murphy A pragmatic discussion of by David N. Reshef, Yakir Reshef, Hilary."— Presentation transcript:

1 Detecting Novel Associations in Large Data Sets Sean Patrick Murphy A pragmatic discussion of by David N. Reshef, Yakir Reshef, Hilary Finucane, Sharon Grossman, Gilean McVean, Peter Turnbaugh, Eric Lander, Michael Mitzenmacher, and Pardis Sabeti

2 Getting Started Blog overview - maximal-information-coefficient/ maximal-information-coefficient/ MINE code (Java-based with python and R wrappers) MINE homepage - Science article and supplemental information - act act mandelbrot-detecting-novel-associations-in-large-data- sets/ mandelbrot-detecting-novel-associations-in-large-data- sets/

3 So who actually read the paper?

4 Outline 1.Motivation 2.Explanation 3.Application

5 The Problem 10,000+ variables Hundreds, thousands, millions of observations Your boss wants you to find all possible relationships between all different variable pairs … Where do you start? Motivation

6 Scatter Plots? Motivation

7 50 Variables  1225 different scatter plots to examine! Motivation

8 Other Options? Correlation Matrix Factor Analysis/Principal Component Analysis Audience recommendations? Motivation

9 Possible Problems A large number of possible relationships Each has a different statistical test Need to have a hypothesis about the relationship that might be present in the data Motivation

10 Desired Properties Generality – the correlation coefficient should be sensitive to a wide range of possible dependencies, including superpositions of functions. Equitability – the score of the coefficient should be influenced by noise, but not by the form of the dependency between variables Motivation

11 Enter the Maximal Information Coefficient (MIC) Explanation

12 Algorithm Intuition Explanation

13 x y We have a dataset D Explanation

14

15 Definition of mutual information (for discrete random variables) Explanation

16 MI = 0.5 MI = 0.6 MI = 0.7 Maximum mutual information Explanation

17 Characteristic Matrix Explanation We have to normalize by min {log x, log y} to enable comparison across grids.

18 2x3 Explanation MI = 0.65 MI = 0.56 MI = 0.71

19 Characteristic Matrix Explanation

20 Characteristic Matrix Explanation

21 This highest value is the Maximal Information Coefficient (MIC) This surface is just a 3D representation of the characteristic matrix. 1.Every entry of the characteristic matrix is between 0 and 1, inclusive 2.MIC(X,Y) = MIC(Y,X) – symmetric 3.MIC is invariant under order preserving transformations of the axis Explanation

22 How Big is the Characteristic Matrix? Technically, infinite in size This is unwieldy So we set bounds on xy < B(n) = n 0.6 n = number of data points This is an empirically set value Explanation

23 How Do We Compute the Maximum Information for a Particular xy Grid? Heuristic-based, dynamic programming Pseudo-code in supplemental materials Only approximate solution, seems to work Authors acknowledge better algorithm should be found At the moment, mostly irrelevant as the authors have released a Java implementation of the algorithm Explanation

24 With probability approaching 1 as sample size grows (i)MIC assigns scores that tend to 1 for all never- constant noiseless functional relationships (ii)MIC assigns scores that tend to 1 for a larger class of noiseless relationships (including superpositions of noiseless functional relationships) (iii)MIC assigns scores that tend to 0 to statistically independent variables Useful Properties of the MIC Statistic Application

25 MIC Application

26

27 So what does the MIC mean? Uncorrected p-value tables are available to download for various sample sizes of data Null hypothesis is variables are statistically independent Value-Tables Application

28 MINE = Maximal Information-based Nonparametric Exploration Hopefully this part is self explanatory now Nonparametric vs parametric could be a session unto itself. Here, we do not rely on assumptions that the data in question are drawn from a specific probability distribution (such as the normal distribution). Application MINE statistics leverage the extra information captured by the characteristic matrix to offer more insight into the relationships between variables.

29 Minimum Cell Number (MCN) - measures the complexity of an association in terms of the number of cells required Application Maximum Edge Value (MEV <= MIC) – measures closeness to being a function (vertical line test ) Maximum Asymmetry Score (MAS<= MIC) – measures deviations from monotonicity

30 Application MAS – monotonicity MEV – vertical line test MCN – complexity

31 Application this takes too long … change it first R: MINE(“MLB2008.csv”,”one.pair”,var1.id=2,var2.id=12) Java: java -jar MINE.jar MLB2008.csv -onePair 2 12 Seeks relationships between salary and home runs, 338 pairs Usage

32 Notes Does not work on textual data (must be numeric) Long execution times Outputs MIC and other mentioned MINE statistics, not the Characteristic Matrix Output is.csv, a row per variable pair Application

33 Creative Commons Attribution-NonCommercial- NoDerivs 3.0 Unported License You are free to: to copy, distribute and transmit the work With the following conditions: Attribution — You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). Noncommercial — You may not use this work for commercial purposes. No Derivative Works — You may not alter, transform, or build upon this work. Application

34 Now What? Data Triage Pipeline Application Complex Data SetMIC Ranked list of variable relationships to examine in more depth with the tool(s) of your choice

35 Lingering Questions Can this be extended to higher-dimensional relationships? Just how approximate is the current MIC algorithm? Who wants to develop an open source implementation? What other MINE statistics are waiting for discovery? Execution time – the algorithm is embarrassingly parallel – easily HADOOPified Many tests reported by the paper only introduced vertical noise into the data? There is also some question as to its power vs Pearson and Dcor (http://www- stat.stanford.edu/~tibs/reshef/comment.pdf)

36 Comment by N. Simon and R. Tibshiran stat.stanford.e du/~tibs/reshe f/script.R Noise Level Power

37

38 Backup Slides

39


Download ppt "Detecting Novel Associations in Large Data Sets Sean Patrick Murphy A pragmatic discussion of by David N. Reshef, Yakir Reshef, Hilary."

Similar presentations


Ads by Google