1. III. Multi-Dimensional Random Variables and Application in Vector Quantization

2. Matrix Notation for Representing Vectors
Assume X is a two-dimensional vector, then in matrix notation it is represented as:
The norm of any vector X
|| X || could be computed
using the dot product as follows:
For any vector X the unit vector UX is defined as x1 x2 X UX

3. Projection and Dot Product
We would like to evaluate the projection of vector Y over vector X using the matrix notation (i.e, we would like to compute the value m),
For proof see lecture notes

4. Principle Component Analysis
Suppose we have a number of samples in a two dimensional space and we would like to identify a unit vector U that crosses the origin for which these samples are the closest.
This could be achieved by finding the vector for which the sum of squares of distances to such vector is minimized (i.e., find U that minimizes d12+d22+d32+d42+…)

5. Principle Component Analysis
From the Pythagoras theorem, we could argue that the closest vector to the observed samples is also equivalently a problem of finding the vector over which the sum of projection of the sample points square on it is maximized. l d
is equivalent to m
For constant l

6. Principle Component Analysis
Define

7. Principle Component Analysis
Therefore the unit vector U (i.e., UUT=1) that is closest to sample points X1, X2, X3, X4 satisfies
Suppose the maximum value is λ
The equation above is an eigen vector problem for the matrix XTX which means that the maximum we are seeking λ must solve the equation above and it is therefore one of the eigen values (maximum eigen value) for the SQUARE matrix XTX

8. Two-Dimensional Random Variables
Assume X is a two-dimensional random variable, then in matrix notation X=[X1 X2]
What does it mean that X is a two-dimensional random variable?
It means that there is this experiment/phenomenon that could be expressed in terms of 2 random variables X1 and X2
Example: Weather (W) could be categorized as a two-dimensional random variable if we characterize it by Temperature (T) and Humidity (H). (i.e., W = [T, H])

9. Parameters of Two-Dimensional R.V.
Mean of any of the Random Variables
Variance of any of the Random Variables

10. Parameters for Relation Between R.V.s
Correlation
Covariance

11. Covariance Matrix

12. 2-D R.V. & Principle Component Analysis
Assume X is a two-dimensional random variable, then in matrix notation X=[X1 X2]
Assume X’ is a two-dimensional random variable, then in matrix notation X’=[X’1 X’2]
where X1’=X1 - E[X’1], X2’=X2 - E[X’2]
The vector U that is closest to sample points of the two-dimensional random variable X is the eigen vector that corresponds to the maximum eigen value for the Covariance Matrix Rx’. i.e., U solves
(Proof in lecture notes)