2. Achim Tresch Computational Biology ‘Omics’ - Analysis of high dimensional Data

3. Microarray of Ms. Smith Expression profile of Ms. Smith Ms. Smith Classification

4. The expression profile... - a list of 30,000 numbers - some of them reflect her health problem (e.g., cancer) -the profile is an image of Ms. Smith‘s physiology How can these numbers tell us (predict) whether Ms. Smith has tumor type A or tumor type B ? The 30.000 properties of Mrs. Smith Classification

5. Mrs. Smith Compare her profile to profiles of people with tumor type A and to patients with tumor type B ? Looking for similarities Classification

6. Mrs. Smith There are patients of known class, the training samples There are patients of unknown class, the ”new“ samples Training and Prediction

7. Mrs. Smith Use the training samples...... to learn how to predict ”new“ samples Training and Prediction

8. Color coded expression levels of trainings samples Ms. Smith  type A Ms. Smith  type B Ms. Smith  borderline Which color shade is a good decision boundary? Prediction using one Gene A B

9. Use the cutoff with the fewest misclassifications on the trainings samples Smallest training error Distribution of expression values in type B Distribution of expression values in type A Decision boundary Training error Optimal decision rule A B

10. Test error The decision boundary was chosen to minimize the training error The two distributions of expression values for type A and B will be similar but not identical in a set of new cases We can not adjust the decision boundary because we do not know the class of the new samples Test errors are usually larger then training errors This phenomenon is called overfitting Optimal decision rule Test set Training set Training error

11. The top gene The average of the top 10 genes ALL vs. AML Golub et al. Combining information across genes Taking means across genes

12. Using a weighted average with “good weights” you get an improved separation Combining information across genes Expression values weights

13. The geometry of weighted averages Calculating a weighted average is identical to projecting (orthogonally) the expression profiles onto the line defined by the weights vector (of length 1). Combining information across genes

14. Hyperplanes A B Together with an offset β 0 the weight vector defines a hyperplane that cuts the data in two groups 2 genes 3 genes Linear decision rules

15. Linear Signatures Linear decision rules A B 2 genes If y≥0  Disease A If y<0  Disease B

16. Nearest Centroids

17. Diagonal Linear Discriminant Analysis (DLDA) Rescale axis according to the variances of genes Linear Discriminant Analysis

18. Discriminant Analysis The data often shows evidence of non identical covariances of genes in the two groups Hence using LDA, DLDA or NC introduces a model biad (=wrong model assumptions, here due to oversimplification) Linear Discriminant Analysis

19. Gene Filtering - Rank genes according to a score - Choose top n genes - Build a signature with these genes only Still 30.000 weights, but most of them are zero … Note that the data decides which are zero and which are not Limitation: You have no(?) chance to find these two genes among 30,000 non- informative genes Feature Reduction

20. How many genes? Is this a biological or a statistical question? Biology: How many genes carry diagnostic information? Statistics: How many genes should we use for classification ? The microarray offers 30.000 genes or more Feature Reduction

21. Finding the needle in the haystack A common myth: Classification information in gene expression signatures is restricted to a small number of genes, the challenge is to find them Feature Reduction

22. The Avalanche Verbundprojekt maligne Lymphome MYC-neg MYC-pos Aggressive lymphomas with and without a MYC-breakpoint Feature Reduction

23. Independent Validation The accuracy of a signature on the data it was learned from is biased because of the overfitting phenomenon Validation of a signature requires independent test data Cross Validation Test error Test set Training set Training error

24. ok mistake Generating Test Sets Split data randomly into … test … … and training data Cross Validation Learn the classifier on the training data, and apply it to the test data

25. Cross validation evaluates the algorithm with which the signature was build Gene selection and adjustment of all parameters must be repeated for every learning step within the training sample of the cross validation Train Eval Train Eval Cross Validation n-fold Cross-Validation (n=5) …

26. Estimators of performance have a variance … … which can be high. The chances of a meaningless signature to produce 100% accuracy on test data is high if the test data includes only few patients Nested 10-fold- CV Variance from 100 random partitions Cross Validation

27. The gap between training error and test error becomes wider There is a good statistical reason for not including hundreds of genes in a model even if they are biologically effected Bias & Overfitting

28. The shrunken centroid method and the PAM package Tibshirani et al 2002 genes Centroid Shrinkage

29. Shrinkage 

30. Train Select Train Select Compute the CV-Performance for several values of  Pick the  that gives you the smallest number of CV- Misclassifications Adaptive Model Selection PAM does this routinely How much shrinkage is good in PAM (partitioning around medoids? cross validation Centroid Shrinkage

31. The test data must not be used for gene selection or adaptive model selection, otherwise the observed (Cross Validation-based) accuracy is biased Selection bias Selection Bias

32. Small , many genes poor performance due to overfitting High , few genes, poor performance due to lack of information – underfitting - The optimal  is somewhere in the middle Cross Validation

33. Assume protein A binds to protein B and inhibits it The clinical phenotype is caused by active protein A Predictive information is in expression of A minus expression of B Naïve Idea: Don’t calculate weights based on single gene scores but optimize over all possible hyperplanes Predictive genes are not causal genes Calling signature genes markers for a certain disease is misleading!

34. Problem 1: No separating line Problem 2: Many separating lines Why is this a problem? Only one of these problems exists Optimal decision rules

35. This problem is related to overfitting... more soon Optimal decision rules

36. With the microarray we have more genes than patients Think about this in three dimensions There are three genes, two patients with known diagnosis (red and yellow) and Ms. Smith (green) There is always one plane separating red and yellow with Ms. Smith on the yellow side and a second separating plane with Ms. Smith on the red side OK! If all points fall onto one line it does not always work. However, for measured values this is very unlikely and never happens in practice. The p>N problem

37. From the data alone we can not decide which genes are important for the diagnosis, nor can we give a reliable diagnosis for a new patient This has little to do medicine. It is a geometrical problem. The overfitting disaster The p>N problem If you find a separating signature, it does not mean (yet) that you have a top publication...... in most cases it means nothing.

38. There always exist separating signatures caused by overfitting - meaningless signatures - Hopefully there is also a separating signature caused by a disease mechanism, or which at least are predictive for the disease - meaningful signatures – We need to learn how to find and validate meaningful signatures Finding meaningful signatures

39. Which hyperplane is the best? Separating hyperplanes

40. Fat planes: With an infinitely thin plane the data can always be separated correctly, but not necessarily with a fat one. Again if a large margin separation exists, chances are good that we found something relevant. Large Margin Classifiers Support Vector Machines (SVMs)

41. Maximal Margin Hyperplane There are theoretical results that the size of the margin correlates with the test (!) error (V. Vapnik) SVMs are not only optimized to fit to the training data but for predictive performance directly Support Vector Machines (SVMs)

42. No separable training set Penalty of error: distance to hyperplane multiplied by a parameter c Balance over- and underfitting Support Vector Machines (SVMs)

43. Documenting a signature is conceptually different from giving a list of genes, although is is what most publications give you In order to validate a signature on external data or apply it in practice: - All model parameters need to be specified - The scale of the normalized data to which the model refers needs to be specified External Validation and Documentation

44. Cross Validation: Split Data into Training and Test Data - select genes - find the optimal number of genes Training data only: Machine Learning - learn model parameters External Validation Establishing a signature

45. 1. Decide on your diagnosis model (PAM,SVM,etc...) and don‘t change your mind later on 2. Split your profiles randomly into a training set and a test set 3. Put the data in the test set away... far away! 4. Train your model only using the data in the training set (select genes, define centroids, calculate normal vectors for large margin separators, perform adaptive model selection...) don‘t even think of touching the test data at this time 5. Apply the model to the test data... don‘t even think of changing the model at this time 6. Do steps 1-5 only once and accept the result... don‘t even think of optimizing this procedure Cookbook for good classifiers

46. Rainer Spang, University of Regensburg Acknowledgements Florian Markowetz, Cancer Research UK, Cambridge