III. Multi-Dimensional Random Variables and Application in Vector Quantization

Matrix Notation for Representing Vectors Assume X is a two-dimensional vector, then in matrix notation it is represented as: The norm of any vector X || X || could be computed using the dot product as follows: For any vector X the unit vector UX is defined as x1 x2 X UX

Projection and Dot Product We would like to evaluate the projection of vector Y over vector X using the matrix notation (i.e, we would like to compute the value m), For proof see lecture notes

Principle Component Analysis Suppose we have a number of samples in a two dimensional space and we would like to identify a unit vector U that crosses the origin for which these samples are the closest. This could be achieved by finding the vector for which the sum of squares of distances to such vector is minimized (i.e., find U that minimizes d12+d22+d32+d42+…)

Principle Component Analysis From the Pythagoras theorem, we could argue that the closest vector to the observed samples is also equivalently a problem of finding the vector over which the sum of projection of the sample points square on it is maximized. l d is equivalent to m For constant l

Principle Component Analysis Define

Principle Component Analysis Therefore the unit vector U (i.e., UUT=1) that is closest to sample points X1, X2, X3, X4 satisfies Suppose the maximum value is λ The equation above is an eigen vector problem for the matrix XTX which means that the maximum we are seeking λ must solve the equation above and it is therefore one of the eigen values (maximum eigen value) for the SQUARE matrix XTX

Two-Dimensional Random Variables Assume X is a two-dimensional random variable, then in matrix notation X=[X1 X2] What does it mean that X is a two-dimensional random variable? It means that there is this experiment/phenomenon that could be expressed in terms of 2 random variables X1 and X2 Example: Weather (W) could be categorized as a two-dimensional random variable if we characterize it by Temperature (T) and Humidity (H). (i.e., W = [T, H])

Parameters of Two-Dimensional R.V. Mean of any of the Random Variables Variance of any of the Random Variables

Parameters for Relation Between R.V.s Correlation Covariance

Covariance Matrix

2-D R.V. & Principle Component Analysis Assume X is a two-dimensional random variable, then in matrix notation X=[X1 X2] Assume X’ is a two-dimensional random variable, then in matrix notation X’=[X’1 X’2] where X1’=X1 - E[X’1], X2’=X2 - E[X’2] The vector U that is closest to sample points of the two-dimensional random variable X is the eigen vector that corresponds to the maximum eigen value for the Covariance Matrix Rx’. i.e., U solves (Proof in lecture notes)