1. III. Multi-Dimensional Random Variables and Application in Vector Quantization

2. © Tallal Elshabrawy 2

3. 3

4. 4

5. 5

6. 6

7. 7 Karhunen-Loeve Decomposition v1v1 u2u2 X=[X 1, X 2 ] X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let u 1, u 2 (unit vectors) be a new co-ordinate system such that u 1, u 2 are the eigen vectors of the covariance matrix R X u1u1 v2v2 u 1, u 2 are orthogonal to each other For distinct eigen vectors, values

8. © Tallal Elshabrawy 8 Karhunen-Loeve Decomposition v1v1 u2u2 X=[X 1, X 2 ] X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let u 1, u 2 (unit vectors) be a new co-ordinate system such that u 1, u 2 are the eigen vectors of the covariance matrix R X u1u1 v2v2 Representation of X over u 1, u 2 Projection of X over u 1 unit vector u 1 Projection of X over u 2 unit vector u 2

9. © Tallal Elshabrawy 9 Karhunen-Loeve Decomposition v1v1 u2u2 X=[X 1, X 2 ] X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let u 1, u 2 (unit vectors) be a new co-ordinate system such that u 1, u 2 are the eigen vectors of the covariance matrix R X u1u1 v2v2 Diagonalization of R X

10. © Tallal Elshabrawy 10 Karhunen-Loeve Decomposition v1v1 u2u2 X=[X 1, X 2 ] X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let u 1, u 2 (unit vectors) be a new co-ordinate system such that u 1, u 2 are the eigen vectors of the covariance matrix R X u1u1 v2v2 Diagonalization of R X

11. © Tallal Elshabrawy 11 Karhunen-Loeve Decomposition v1v1 u2u2 X=[X 1, X 2 ] X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let u 1, u 2 (unit vectors) be a new co-ordinate system such that u 1, u 2 are the eigen vectors of the covariance matrix R X u1u1 v2v2 Diagonalization of R X

12. © Tallal Elshabrawy 12 K-L Transformation of 2-D Random Variable v1v1 u2u2 X X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let Y = [Y 1, Y 2 ] be the transformation of the random variable X over the co- ordinate system u 1, u 2 u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2

13. © Tallal Elshabrawy 13 K-L Transformation of 2-D Random Variable v1v1 u2u2 X X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let Y = [Y 1, Y 2 ] be the transformation of the random variable X over the co- ordinate system u 1, u 2 u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2 Define E[X 1 ] E[X 2 ] E[Y 1 ] E[Y 2 ]

14. © Tallal Elshabrawy 14 K-L Transformation of 2-D Random Variable v1v1 u2u2 X X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let Y = [Y 1, Y 2 ] be the transformation of the random variable X over the co- ordinate system u 1, u 2 u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2 Covariance Matrix R X

15. © Tallal Elshabrawy 15 K-L Transformation of 2-D Random Variable v1v1 u2u2 X X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let Y = [Y 1, Y 2 ] be the transformation of the random variable X over the co- ordinate system u 1, u 2 u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2 Covariance Matrix R Y

16. © Tallal Elshabrawy 16 K-L Transformation of 2-D Random Variable v1v1 u2u2 X X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let Y = [Y 1, Y 2 ] be the transformation of the random variable X over the co- ordinate system u 1, u 2 u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2 Covariance Matrix R Y

17. © Tallal Elshabrawy 17 K-L Transformation of 2-D Random Variable v1v1 u2u2 X X = [X 1, X 2 ] is a sample vector from a two-dimensional random variable defined over v 1, v 2 (unit vectors) co-ordinate system Let Y = [Y 1, Y 2 ] be the transformation of the random variable X over the co- ordinate system u 1, u 2 u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2 Y 1 and Y 2 are UNCORRELATED Covariance Matrix R Y

18. © Tallal Elshabrawy 18 K-L Transformation of 2-D Random Variable Therefore using principle component analysis, it is possible to transform a random variable X with components X 1 and X 2 that are correlated into another random variable Y whose components Y 1 and Y 2 are uncorrelated v1v1 u2u2 X u1u1 v2v2 X1X1 X2X2 Y Y1Y1 Y2Y2 u 1, u 2 are eigen vectors of R X λ 1, λ 2 are eigen values of R X

19. © Tallal Elshabrawy 19 Two-Dimensional Gaussian Distribution In the previous slides, we have talked about the transformed random variable Y whose components are uncorrelated and have mean m Y and covariance matrix R Y. What would be the distribution of Y. Well this depends on what is the distribution of X Generally X and Y do not follow a Gaussian distribution. However, if X is a two-dimensional Gaussian distribution then Y as well would be a two-dimensional Gaussian distribution

20. © Tallal Elshabrawy 20 Two-Dimensional Gaussian Distribution Suppose we have a two-dimensional random variable Y whose components Y 1 and Y 2 are independent and Gaussian

21. © Tallal Elshabrawy 21 Two-Dimensional Gaussian Distribution Suppose we have a two-dimensional random variable Y whose components Y 1 and Y 2 are independent and Gaussian

22. © Tallal Elshabrawy 22 Two-Dimensional Gaussian Distribution Suppose we have a two-dimensional random variable Y whose components Y 1 and Y 2 are independent and Gaussian

23. © Tallal Elshabrawy 23 Two-Dimensional Gaussian Distribution Suppose we have a two-dimensional random variable Y whose components Y 1 and Y 2 are independent and Gaussian This formula is valid for any multi- dimensional Gaussian random variable whether its components are correlated or not

24. © Tallal Elshabrawy 24 Two-Dimensional Gaussian Distribution

25. © Tallal Elshabrawy 25 Two-Dimensional Gaussian Distribution

26. © Tallal Elshabrawy 26 Two-Dimensional Gaussian Distribution

27. © Tallal Elshabrawy 27 Two-Dimensional Gaussian Distribution

28. © Tallal Elshabrawy 28 Two-Dimensional Gaussian Distribution

29. © Tallal Elshabrawy 29 Two-Dimensional Gaussian Distribution

30. © Tallal Elshabrawy 30 Two-Dimensional Gaussian Distribution

31. © Tallal Elshabrawy 31 Two-Dimensional Gaussian Distribution

32. © Tallal Elshabrawy 32 Two-Dimensional Gaussian Distribution

33. © Tallal Elshabrawy 33 Two-Dimensional Gaussian Distribution

34. © Tallal Elshabrawy 34 Two-Dimensional Gaussian Distribution

35. © Tallal Elshabrawy 35 Two-Dimensional Gaussian Distribution More Energy Less Energy Y 1 and Y 2 Correlated

36. © Tallal Elshabrawy 36 Two-Dimensional Gaussian Distribution Equal Energy By rotating the axis, the resultant transformed random variables are uncorrelated