Machine Learning Recitation 6 Sep 30, 2009 Oznur Tastan

Outline Multivariate Gaussians Logistic regression

Multivariate Gaussians (or "multinormal distribution“ or “multivariate normal distribution”) Multivariate case: Vector of observations x, vector of means  and covariance matrix  Univariate case: single mean  and variance  Dimension of xDeterminant

Multivariate Gaussians do not depend on x normalization constants Multivariate case Univariate case depends on x and positive

The mean vector

Covariance of two random variables Recall for two random variables x i, x j

The covariance matrix transpose operator Var(x m )=Cov(x m, x m )

An example: 2 variate case The pdf of the multivariate will be: Covariance matrix Determinant

An example: 2 variate case Recall in general case independence implies uncorrelation but uncorrelation does not necessarily implies independence. Multivariate Gaussians is a special case where uncorrelation implies independence as well. Factorized into two independent Gaussians! They are independent!

Diagonal covariance matrix Diagonal matrix: m matrix where off-diagonal terms are zero If all the variables are independent from each other, The covariance matrix will be an diagonal one. Reverse is also true: If the covariance matrix is a diagonal one they are independent

Gaussian Intuitions: Size of   = [0 0]  = [0 0]  = [0 0]  = I  = 0.6 I  = 2 I As  becomes larger, Gaussian becomes more spread out Identity matrix

Gaussian Intuitions: Off-diagonal As the off-diagonal entries increase, more correlation between value of x and value of y

Gaussian Intuitions: off-diagonal and diagonal Decreasing non-diagonal entries (#1-2) Increasing variance of one dimension in diagonal (#3)

Isocontours

Isocontours example We have showed Now let’s try to find for some constant c the isocontour

Isocontours continued

Define Equation of an ellipse Centered on μ 1, μ 2 and axis lengths 2r 1 and 2r 2

We had started with diaogonal matrix In the diagonal covariance matrix case the ellipses will be axis aligned.

Don’t confuse Multivariate Gaussians with Mixtures of Gaussians Mixture of Gaussians: Component Mixing coefficient K=3

Logistic regression Linear regression Outcome variable Y is continuous Logistic regression Outcome variable Y is binary

Logistic function (Logit function) z logit(z) Notice σ(z) is always bounded between [0,1] (a nice property) and as z increase σ(z) approaches 1, as z decreases σ(z) approaches to 0 This term is [0, infinity]

Logistic regression Learn a function to map X values to Y given data The function we try to learn is P(Y|X) X can be continuous or discrete Discrete

Logistic regression

Classification If this holds Y=0 is more probable Than Y=1 given X

Classification Take log both sides Classification rule if this holds Y=0

Logistic regression is a linear classifier Y=0 Y=1 Decision boundary

Classification X1 σ(z)= σ(w 0 +w 1 X 1 )) Notice σ(z) is 0.5 when X1=2 wo=+2, to check evaluate at X1=0 g(z)~0.1 σ(z) is 0.5 when X1=0 to see Classify as Y=0

Estimating the parameters Given data Objective: Train the model to get w that maximizes the conditional likelihood

Difference with Naïve Bayes of Logistic Regression Loss function! Optimize different functions → Obtain different solutions Naïve Bayes argmax P(X|Y) P(Y) Logistic Regression argmax P(Y|X)

Naïve Bayes and Logistic Regression Have a look at the Tom Mitchell’s book chapter Linked under Sep 23 Lecture Readings as well.

Some matlab tips for the last question in HW3 logical function might be useful for dividing into splits. An example of logical in use (please read the Matlab help) S=X(logical(X(:,1)==1),:) this will also work S=X((X(:1)==1,:)) This will subset the portion of the X matrix where the first column has value 1 and will put in matrix S (like Data>Filter in Excel) Matlab has functions for mean, std, sum, inv, log2 Scaling data to zero mean and unit variance: shifting the mean by the mean (subtracting the mean from every element of the vector) and scaling such that it has variance=1 ( dividing the every element of the vector by standard deviation) To be able to do that in matrices. You will need the repmat function, have a look at that otherwise the size of the matrices would not match..etc Elementwise multiplication use.*

References ogReg.pdf Carlos Guestrin lecture notes Andrew Ng lecture notes