2. L15:Microarray analysis (Classification)

3. The Biological Problem Two conditions that need to be differentiated, (Have different treatments). EX: ALL (Acute Lymphocytic Leukemia) & AML (Acute Myelogenous Leukima) Possibly, the set of genes over-expressed are different in the two conditions

4. Geometric formulation Each sample is a vector with dimension equal to the number of genes. We have two classes of vectors (AML, ALL), and would like to separate them, if possible, with a hyperplane.

5. Basic geometry What is ||x|| 2 ? What is x/||x|| Dot product? x=(x 1,x 2 ) y

6. Dot Product Let  be a unit vector. –||  || = 1 Recall that –  T x = ||x|| cos  What is  T x if x is orthogonal (perpendicular) to  ?  x   T x = ||x|| cos 

7. Hyperplane How can we define a hyperplane L? Find the unit vector that is perpendicular (normal to the hyperplane)

8. Points on the hyperplane Consider a hyperplane L defined by unit vector , and distance  0 from the origin Notes; –For all x  L, x T  must be the same, x T  =  0 –For any two points x 1, x 2, (x 1 - x 2 ) T  =0 Therefore, given a vector , and an offset  0, the hyperplane is the set of all points –{x : x T  =  0 } x1x1 x2x2

9. Hyperplane properties Given an arbitrary point x, what is the distance from x to the plane L? D(x,L) = (  T x -  0 ) When are points x1 and x2 on different sides of the hyperplane? x 00

10. Hyperplane properties Given an arbitrary point x, what is the distance from x to the plane L? –D(x,L) = (  T x -  0 ) When are points x 1 and x 2 on different sides of the hyperplane? Ans: If D(x 1,L)* D(x 2,L) < 0 x 00

11. Separating by a hyperplane Input: A training set of +ve & -ve examples Recall that a hyperplane is represented by – {x:-  0 +  1 x 1 +  2 x 2 =0} or –(in higher dimensions) {x:  T x-  0 =0} Goal: Find a hyperplane that ‘separates’ the two classes. Classification: A new point x is +ve if it lies on the +ve side of the hyperplane (D(x,L)> 0), -ve otherwise. x2x2 x1x1 + -

12. Hyperplane separation What happens if we have many choices of a hyperplane? –We try to maximize the distance of the points from the hyperplane. What happens if the classes are not separable by a hyperplane? –We define a function based on the amount of mis-classification, and try to minimize it

13. Error in classification Sample Function: sum of distances of all misclassified points –Let y i =-1 for +ve example i, y i =+1 otherwise. The best hyperplane is one that minimizes D( ,  0 ) Other definitions are also possible. x2x2 x1x1 + - 

14. Restating Classification The (supervised) classification problem can now be reformulated as an optimization problem. Goal: Find the hyperplane ( ,  0 ), that optimizes the objective D( ,  0 ). No efficient algorithm is known for this problem, but a simple generic optimization can be applied. Start with a randomly chosen ( ,  0 ) Move to a neighboring (  ’,  ’ 0 ) if D(  ’,  ’ 0 )< D( ,  0 )

15. Gradient Descent The function D(  ) defines the error. We follow an iterative refinement. In each step, refine  so the error is reduced. Gradient descent is an approach to such iterative refinement. D(  )  D’(  )

16. Rosenblatt’s perceptron learning algorithm

17. Classification based on perceptron learning Use Rosenblatt’s algorithm to compute the hyperplane L=( ,  0 ). Assign x to class 1 if D(x,L)=  T x-  0 >= 0, and to class 2 otherwise. x  00

18. Perceptron learning If many solutions are possible, it does no choose between solutions If data is not linearly separable, it does not terminate, and it is hard to detect. Time of convergence is not well understood

19. Linear Discriminant analysis Provides an alternative approach to classification with a linear function. Project all points, including the means, onto vector . We want to choose  such that –Difference of projected means is large. –Variance within group is small x2x2 x1x1 + - 

20. Choosing the right  x2x2 x1x1 + - 22 x2x2 x1x1 + - 11  1 is a better choice than  2 as the variance within a group is small, and difference of means is large. How do we compute the best  ?

21. Linear Discriminant analysis Fisher Criterion

22. LDA cont’d x2x2 x1x1 + -  What is the projection of a point x onto  ? –Ans:  T x What is the distance between projected means? x

23. LDA Cont’d Fisher Criterion

24. LDA Therefore, a simple computation (Matrix inverse) is sufficient to compute the ‘best’ separating hyperplane

25. Maximum Likelihood discrimination Consider the simple case of single dimensional data. Compute a distribution of the values in each class. values Pr

26. Maximum Likelihood discrimination Suppose we knew the distribution of points in each class  i. –We can compute Pr(x|  i ) for all classes i, and take the maximum The true distribution is not known, so usually, we assume that it is Gaussian

27. ML discrimination Suppose all the points were in 1 dimension, and all classes were normally distributed. 11 22 x

28. ML discrimination (multi- dimensional case) Not part of the syllabus.

29. Supervised classification summary Most techniques for supervised classification are based on the notion of a separating hyperplane. The ‘optimal’ separation can be computed using various combinatorial (perceptron), algebraic (LDA), or statistical (ML) analyses.

30. Dimensionality reduction Many genes have highly correlated expression profiles. By discarding some of the genes, we can greatly reduce the dimensionality of the problem. There are other, more principled ways to do such dimensionality reduction.

31. Principle Components Analysis Consider the expression values of 2 genes over 6 samples. Clearly, the expression of the two genes is highly correlated. Projecting all the genes on a single line could explain most of the data. This is a generalization of “discarding the gene”.

32. PCA Suppose all of the data were to be reduced by projecting to a single line  from the mean. How do we select the line  ? m

33. PCA cont’d Let each point x k map to x’ k. We want to mimimize the error Observation 1: Each point x k maps to x’ k = m +  T (x k -m)  m  xkxk x’ k

34. PCA: motivating example Consider the expression values of 2 genes over 6 samples. Clearly, the expression of g 1 is not informative, and it suffices to look at g 2 values. Dimensionality can be reduced by discarding the gene g 1 g1g1 g2g2

35. PCA: Ex2 Consider the expression values of 2 genes over 6 samples. Clearly, the expression of the two genes is highly correlated. Projecting all the genes on a single line could explain most of the data.

36. PCA Suppose all of the data were to be reduced by projecting to a single line  from the mean. How do we select the line  ? m 

37. PCA cont’d Let each point x k map to x’ k =m+a k . We want to minimize the error Observation 1: Each point x k maps to x’ k = m +  T (x k -m)  –(a k =  T (x k -m)) m  xkxk x’ k

38. Proof of Observation 1 Differentiating w.r.t a k

39. Minimizing PCA Error To minimize error, we must maximize  T S  By definition, =  T S  implies that is an eigenvalue, and  the corresponding eigenvector. Therefore, we must choose the eigenvector corresponding to the largest eigenvalue.

40. PCA The single best dimension is given by the eigenvector of the largest eigenvalue of S The best k dimensions can be obtained by the eigenvectors {  1,  2, …,  k } corresponding to the k largest eigenvalues. To obtain the k dimensional surface, take B T M BTBT 1T1T M