ETHEM ALPAYDIN © The MIT Press, Lecture Slides for

Multivariate Data Multiple measurements (sensors) d inputs/features/attributes: d-variate N instances/observations/examples Data matrix: Typically these variables are correlated. 3 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Multivariate Parameters 4 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Parameter Estimation Given a multivariate sample, estimates for these parameters can be calculated: 5 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Estimation of Missing Values What to do if certain instances have missing attributes? Ignore those instances: not a good idea if the sample is small Use ‘missing’ as an attribute: may give information Imputation: Fill in the missing value Mean imputation: Use the most likely value (e.g., mean) Imputation by regression: Predict based on other attributes 6 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Multivariate Normal Distribution 7 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) Covariance Matrix

Multivariate Normal Distribution Mahalanobis distance: (x – μ) T ∑ –1 (x – μ) measures the distance from x to μ in terms of ∑ (x – μ) T ∑ –1 (x – μ)=C 2 the d-dimensional hyperellipsoid ( 超橢圓體 ) centered at μ, and its shape and orientation are defined by ∑. It is a useful way of determining similarity of an unknown sample set to a known one. It differs from Euclidean distance in that it takes into account the correlations of the data set and is scale-invariant. 8 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Multivariate Normal Distribution Mahalanobis distance: (x – μ) T ∑ –1 (x – μ) Example: Bivariate case (d = 2) 9 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Bivariate Normal 10 If X and Y are independent then Cov(X, Y)=0. However, if Cov(X, Y)=0 then X and Y may not be independent. Read more: Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

11 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Independent Inputs: Naive Bayes If x i are independent, off-diagonals of ∑ are 0, Mahalanobis distance reduces to weighted (by 1/σ i ) Euclidean distance: If variances are also equal, reduces to Euclidean distance. 12 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Parametric Classification When, if the class-conditional densities are taken as normal density, p (x | C i ) ~ N ( μ i, ∑ i ), we have Discriminant functions are 13 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Estimation of Parameters Given a training sample for K ≥ 2 classes, X = { x t, r t }, where r t i = 1 if x t ∈ C i, and 0 otherwise. 14 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) Compare with Eq (p. 71)

Different S i Quadratic discriminant (Fig. 5.3) This quadratic discriminant can also be written as where 15 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

likelihoods posterior for C 1 quadratic discriminant: P (C 1 |x ) = 0.5 C1C1 C2C2 16 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Common Covariance Matrix S Shared common sample covariance S Discriminant which is a linear discriminant Note: the term cancels since it is common in all discriminants. 17 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Common Covariance Matrix S C1C1 C2C2 18 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) linear discriminant: P (C 1 |x ) = 0.5

Diagonal S When x j, j = 1,…, d, are independent (Naive Bayes’ assumption), ∑ is diagonal all off-diagonals of the covariance matrix to be zero p (x j |C i ) are univariate Gaussians Classify based on weighted Euclidean distance (in s j units) to the nearest mean 19 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Diagonal S variances may be different C1C1 C2C2 When x j, j = 1,..d, are independent, ∑ is diagonal 20 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Diagonal S, equal variances Nearest mean classifier: Classify based on Euclidean distance to the nearest mean Each mean can be considered a prototype or template and this is template matching 21 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Diagonal S, equal variances C1C1 C2C2 22 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) Nearest mean classifier

Model Selection As we increase complexity (less restricted S), bias decreases and variance increases Assume simple models (allow some bias) to control variance (regularization) AssumptionCovariance matrixNo of parameters Shared, HypersphericSi=S=s2ISi=S=s2I1 Shared, Axis-alignedS i =S, with s ij =0, i != jd Shared, HyperellipsoidalSi=SSi=Sd (d+1)/2 Different, Hyperellipsoidal SiSi K (d (d+1)/2) 23 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

24 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) AssumptionCovariance matrix No of parameter s Shared, HypersphericSi=S=s2ISi=S=s2I1 Shared, Axis-alignedS i =S, with s ij =0, i != jd Shared, Hyperellipsoidal Si=SSi=Sd (d+1)/2 Different, Hyperellipsoidal SiSi K (d (d+1)/2)

Discrete Features Let x j (discrete attributes) be binary (Bernoulli): If x j are independent binary variables, the discriminant is linear Estimated parameters 25 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Discrete Features Multinomial (1-of-n j ) features: x j  {v 1, v 2,..., v n j } The probability that x j belonging to class C i takes value v k. If x j are independent The maximum likelihood estimator for p ijk is 26 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Multivariate Regression Multivariate linear model Taking the derivative with respect to the parameters, w j, j = 0,…,d, we get the normal equations. (see eq. 5.37, P104) Solution: 27 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Multivariate Regression Multivariate polynomial model: Define new higher-order variables z 1 =x 1, z 2 =x 2, z 3 =x 1 2, z 4 =x 2 2, z 5 =x 1 x 2 and use the linear model in this new z space (basis functions, kernel trick, SVM: Chapter 13) 28 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Exercise Let us say in two dimensions, we have two classes with exactly the same mean. What type of boundaries can be defined? 29 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)