1. Unitary Transforms

2. Image Transform: A Revisit With A Coding Perspective

3. Why Do Transforms? Fast computation –E.g., convolution vs. multiplication for filter with wide support Conceptual insights for various image processing –E.g., spatial frequency info. (smooth, moderate change, fast change, etc.) Obtain transformed data as measurement –E.g., blurred images, radiology images (medical and astrophysics) –Often need inverse transform –May need to get assistance from other transforms For efficient storage and transmission –Pick a few “representatives” (basis) –Just store/send the “contribution” from each basis

4. Basic Process of Transform Coding Figure is from slides at Gonzalez/ Woods DIP book website (Chapter 8)

5. 1-D DFT and Vector Form { z(n) }  { Z(k) } n, k = 0, 1, …, N-1 W N = exp{ - j2  / N } ~ complex conjugate of primitive N th root of unity Vector form and interpretation for inverse transform z =  k Z(k) a k a k = [ 1 W N -k W N -2k … W N -(N-1)k ] T /  N Basis vectors   –a k H = a k * T = [ 1 W N k W N 2k … W N (N-1)k ] /  N –Use a k H as row vectors to construct a matrix F –Z = F z  z = F *T Z = F * Z –F is symmetric (F T =F) and unitary (F -1 = F H where F H = F *T )  

6. Basis Vectors and Basis Images A basis for a vector space ~ a set of vectors –Linearly independent ~  a i v i = 0 if and only if all a i =0 –Uniquely represent every vector in the space by their linear combination ~  b i v i ( “spanning set” {v i } ) Orthonormal basis –Orthogonality ~ inner product = y *T x= 0 –Normalized length ~ || x || 2 = = x *T x= 1 Inner product for 2-D arrays – =  m  n f(m,n) g * (m,n) = G 1 *T F 1 (rewrite matrix into vector) u !! Don’t do FG ~ may not even be a valid operation for MxN matrices! 2D Basis Matrices (Basis Images)   –Represent any images of the same size as a linear combination of basis images A vector space consists of a set of vectors, a field of scalars, a vector addition operation, and a scalar multiplication operation.

7. 1-D Unitary Transform Linear invertible transform –1-D sequence { x(0), x(1), …, x(N-1) } as a vector – y = A x and A is invertible Unitary matrix ~ A -1 = A *T –Denote A *T as A H ~ “Hermitian” –x = A -1 y = A *T y =  a i *T y(i) –Hermitian of row vectors of A form a set of orthonormal basis vectors a i *T = [a * (i,0), …, a * (i,N-1)] T Orthogonal matrix ~ A -1 = A T –Real-valued unitary matrix is also an orthogonal matrix –Row vectors of real orthogonal matrix A form orthonormal basis vectors

8. Exercise – Question for Today: Which is unitary or orthogonal? 2-D DFT –Write the forward and inverse transform –Express in terms of matrix-vector form –Find basis images inv(A1) = [2, –3; -1, 2]

9. Properties of 1-D Unitary Transform y = A x Energy Conservation – || y || 2 = || x || 2 u || y || 2 = || Ax || 2 = (Ax) *T (Ax)= x *T A *T A x = x *T x = || x || 2 Rotation –The angles between vectors are preserved –A unitary transformation is a rotation of a vector in an N-dimension space, i.e., a rotation of basis coordinates

10. Properties of 1-D Unitary Transform (cont’d) Energy Compaction –Many common unitary transforms tend to pack a large fraction of signal energy into just a few transform coefficients Decorrelation –Highly correlated input elements  quite uncorrelated output coefficients –Covariance matrix E[ ( y – E(y) ) ( y – E(y) ) *T ] u small correlation implies small off-diagonal terms Example: recall the effect of DFT Question: What unitary transform gives the best compaction and decorrelation? => Will revisit this issue in a few lectures

11. Review: 1-D Discrete Cosine Transform (DCT) Transform matrix C –c(k,n) =  (0) for k=0 –c(k,n) =  (k) cos[  (2n+1)/2N] for k>0 C is real and orthogonal – rows of C form orthonormal basis – C is not symmetric! – DCT is not the real part of unitary DFT!  See Assignment#3 u related to DFT of a symmetrically extended signal

12. Periodicity Implied by DFT and DCT Figure is from slides at Gonzalez/ Woods DIP book website (Chapter 8)

13. Example of 1-D DCT 100 50 0 -50 -100 01234567n01234567n z(n) 100 50 0 -50 -100 01234567k01234567k Z(k) DCT From Ken Lam’s DCT talk 2001 (HK Polytech)

14. UMCP ENEE631 Slides (created by M.Wu © 2001) Example of 1-D DCT (cont’d) k Z (k) Transform coeff. 1.0 0.0 1.0 0.0 1.0 0.0 1.0 0.0 1.0 0.0 1.0 0.0 1.0 0.0 1.0 0.0 Basis vectors 100 0 -100 100 0 -100 100 0 -100 100 0 -100 100 0 -100 100 0 -100 100 0 -100 100 0 -100 u=0 u=0 to 1 u=0 to 4 u=0 to 5 u=0 to 2 u=0 to 3 u=0 to 6 u=0 to 7 Reconstructions n z (n) Original signal From Ken Lam’s DCT talk 2001 (HK Polytech)

15. Fast Transform via FFT Define new sequence – reorder odd and even elements Split DCT sum into odd and even terms Other real-value fast algorithms

16. 2-D DCT Separable orthogonal transform –Apply 1-D DCT to each row, then to each column Y = C X C T  X = C T Y C =  m  n y(m,n) B m,n DCT basis images: –Equivalent to represent an NxN image with a set of orthonormal NxN “basis images” –Each DCT coefficient indicates the contribution from (or similarity to) the corresponding basis image

17. 2-D Transform: General Case A general 2-D linear transform {a k,l (m,n)} –y(k,l) is a transform coefficient for Image {x(m,n)} –{y(k,l)} is “Transformed Image” –Equiv to rewriting all from 2-D to 1-D and applying 1-D transform Computational complexity –N 2 values to compute –N 2 terms in summation per output coefficient –O(N 4 ) for transforming an NxN image!

18. 2-D Linear Transforms Image transform {a k,l (m,n)} –y(k,l) is a transform coefficient for Image {x(m,n)} –{y(k,l)} is “Transformed Image” –Equiv to rewriting all from 2-D to 1-D and applying 1-D transform May generalize the transf. (series expansion) from NxN to NxM a1 a2 a3 orth. proj. gives min. distance Orthonomality condition – – Assure any truncated expansion of the form will minimize sum of squared errors when y(k,l) take values as above Completeness condition – – assure zero error when taking full basis

19. 2-D Separable Unitary Transforms Restrict to separable transform – a k,l (m,n) = a k (m) b l (n), denote this as a(k,m) b(l,n) Use 1-D unitary transform as building block – {a k (m)} k and {b l (n)} l are 1-D complete orthonormal sets of basis vectors u use as row vectors to obtain unitary matrices A={a(k,m)} & B={b(l,n)} –Apply to columns and rows Y = A X B T u often choose same unitary matrix as A and B (e.g., 2-D DFT) For square NxN image A: Y = A X A T  X = A H Y A * –For rectangular MxN image A: Y = A M X A N T  X = A M H Y A N * Complexity ~ O(N 3 ) –May further reduce complexity if unitary transf. has fast implementation

20. Basis Images X = A H Y A * => x(m,n) =  k  l a*(k,m)a*(l,n) y(k,l) –Represent X with NxN basis images weighted by coeff. Y –Obtain basis image by setting Y={  (k-k 0, l-l 0 )} & getting X  { a*(k 0,m)a*(l 0,n) } m,n  in matrix form A* k,l = a* k a l *T ~ a* k is k th column vector of A H  trasnf. coeff. y(k,l) is the inner product of A* k,l with the image Exercise –Unitary transform or not? –Find basis images –Represent an image X with basis images (Jain’s e.g.5.1, pp137)

21. Common Unitary Transforms and Basis Images DFTDCT Haar transform K-L transform See also: Jain’s Fig.5.2 pp136

22. Exercise (1) Unitary or not? –And find basis for the unitary matrix Find basis images and represent image X with basis images –X = A H Y A * (separable) => x(m,n) =  k  l a*(k,m)a*(l,n) y(k,l) u Represent X with NxN basis images weighted by coeff. Y  Obtain basis image { a*(k 0,m)a*(l 0,n) } m,n by setting Y={  (k-k 0, l-l 0 )} & getting X  In matrix form A* k,l = a* k a l *T ~ a* k is k th column vector of A *T ( a k T is k th row vector of A )  Trasnform coeff. y(k,l) is the inner product of A* k,l with the image Jain’s e.g.5.1, pp137 A’ [5 –1;-2 0] A : [1,1]’[1 1]/2, … (1.n; 2. y.)

23. 2-D DFT 2-D DFT is Separable –Y = F X F  X = F * Y F * –Basis images B k,l = (a k ) (a l ) T where a k = [ 1 W N -k W N -2k … W N -(N-1)k ] T /  N

24. Summary and Review (1) 1-D transform of a vector –Represent an N-sample sequence as a vector in N-dimension vector space –Transform u Different representation of this vector in the space via different basis u e.g., 1-D DFT from time domain to frequency domain –Forward transform u In the form of inner product u Project a vector onto a new set of basis to obtain N “coefficients” –Inverse transform u Use linear combination of basis vectors weighted by transform coeff. to represent the original signal 2-D transform of a matrix –Rewrite the matrix into a vector and apply 1-D transform –Separable transform allows applying transform to rows then columns

25. Summary and Review (1) cont’d Vector/matrix representation of 1-D & 2-D sampled signal –Representing an image as a matrix or sometimes as a long vector Basis functions/vectors and orthonomal basis –Used for representing the space via their linear combinations –Many possible sets of basis and orthonomal basis Unitary transform on input x ~ A -1 = A *T –y = A x  x = A -1 y = A *T y =  a i *T y(i) ~ represented by basis vectors {a i *T } –Rows (and columns) of a unitary matrix form an orthonormal basis General 2-D transform and separable unitary 2-D transform –2-D transform involves O(N 4 ) computation –Separable: Y = A X A T = (A X) A T ~ O(N 3 ) computation u Apply 1-D transform to all columns, then apply 1-D transform to rows

26. Summary of Today’s Lecture Unitary transform and properties –Basis vectors and Basis images DCT transform Next time: putting together … –transform coding and JPEG compression standard Readings –Jain’s book 5.1-5.7 –Gonzalez’s book 8.5.2 –Wang’s book 9.1

27. Clarifications “Dimension” –Dimension of a signal ~ # of index variables u audio and speech is 1-D signal, image is 2-D, video is 3-D –Dimension of a vector space ~ # of basis vectors in it Eigenvalues of unitary transform –All eigenvalues have unit magnitude (could be complex valued)  By definition of eigenvalues ~ A x = x  By energy perservation of unitary ~ || A x || = ||x|| –Eigenvalues here are different from the eigenvalues in K-L transform u K-L concerns the eigen of covariance matrix of random vector –Eigenvectors ~ we generally consider the orthonormalized ones

28. Optimal Transform

29. Recall: Why use transform in coding/compression? –Decorrelate the correlated data –Pack energy into a small number of coefficients –Interested in unitary/orthogonal or approximate orthogonal transforms u Energy preservation s.t. quantization effects can be better understood and controlled Unitary transforms we’ve dealt so far are data independent –Transform basis/filters are not depending on the signals we are processing What unitary transform gives the best energy compaction and decorrelation? –“Optimal” in a statistical sense to allow the codec works well with many images u Signal statistics would play an important role

30. Review: Correlation After a Linear Transform Consider an Nx1 zero-mean random vector x – Covariance (autocorrelation) matrix R x = E[ x x H ] u give ideas of correlation between elements u R x is a diagonal matrix for if all N r.v.’s are uncorrelated Apply a linear transform to x: y = A x What is the correlation matrix for y ? R y = E[ y y H ] = E[ (Ax) (Ax) H ] = E[ A x x H A H ] = A E[ x x H ] A H = A R x A H Decorrelation: try to search for A that can produce a decorrelated y (equiv. a diagonal correlation matrix R y )

31. K-L Transform (Principal Component Analysis) Eigen decomposition of R x : R x u k = k u k –Recall the properties of R x u Hermitian (conjugate symmetric R H = R); u Nonnegative definite (real non-negative eigen values) Karhunen-Loeve Transform (KLT) y = U H x  x = U y with U = [ u 1, … u N ] –KLT is a unitary transform with basis vectors in U being the orthonormalized eigenvectors of R x –U H R x U = diag{ 1, 2, …, N } i.e. KLT performs decorrelation –Often order {u i } so that 1  2  …  N –Also known as the Hotelling transform or the Principle Component Analysis (PCA)

32. Properties of K-L Transform Decorrelation –E[ y y H ]= E[ (U H x) (U H x) H ]= U H E[ x x H ] U = diag{ 1, 2, …, N } –Note: Other matrices (unitary or nonunitary) may also decorrelate the transformed sequence [ Jain’s e.g.5.7 pp166] Minimizing MSE under basis restriction –If only allow to keep m coefficients for any 1  m  N, what’s the best way to minimize reconstruction error?  Keep the coefficients w.r.t. the eigenvectors of the first m largest eigenvalues u Theorem 5.1 and Proof in Jain’s Book (pp166)

33. KLT Basis Restriction Basis restriction –Keep only a subset of m transform coefficients and then perform inverse transform (1  m  N) –Basis restriction error: MSE between original & new sequences Goal: to find the forward and backward transform matrices to minimize the restriction error for each and every m –The minimum is achieved by KLT arranged according to the decreasing order of the eigenvalues of R

34. K-L Transform for Images Work with 2-D autocorrelation function –R(m,n; m’,n’)= E[ x(m, n) x(m’, n’) ] for all 0  m, m’, n, n’  N-1 –K-L Basis images is the orthonormalized eigenfunctions of R Rewrite images into vector form (N 2 x1) –Need solve the eigen problem for N 2 xN 2 matrix! ~ O(N 6 ) Reduced computation for separable R –R(m,n; m’,n’)= r 1 (m,m’) r 2 (n,n’) –Only need solve the eigen problem for two NxN matrices ~ O(N 3 ) –KLT can now be performed separably on rows and columns u Reducing the transform complexity from O(N 4 ) to O(N 3 )

35. Pros and Cons of K-L Transform Optimality –Decorrelation and MMSE for the same# of partial coeff. Data dependent –Have to estimate the 2 nd -order statistics to determine the transform –Can we get data-independent transform with similar performance? u DCT Applications – (non-universal) compression – pattern recognition: e.g., eigen faces – analyze the principal (“dominating”) components

36. Energy Compaction of DCT vs. KLT DCT has excellent energy compaction for highly correlated data DCT is a good replacement for K-L –Close to optimal for highly correlated data –Not depend on specific data like K-L does –Fast algorithm available [ref and statistics: Jain’s pp153, 168-175]

37. Energy Compaction of DCT vs. KLT (cont’d) Preliminaries –The matrices R, R -1, and  R -1 share the same eigen vectors –DCT basis vectors are eigenvectors of a symmetric tri-diagonal matrix Q c –Covariance matrix R of 1 st -order stationary Markov sequence with has an inverse in the form of symmetric tri-diagonal matrix DCT is close to KLT on 1 st -order stationary Markov –For highly correlated sequence, a scaled version of R -1 approx. Q c

38. Energy Compaction of DCT vs. KLT (cont’d) Excellent energy compaction of DCT –for highly correlated data DCT is close to K-L transf. of 1 st -order stationary Markov –DCT basis vectors are eigenvectors of a symmetric tridiagonal matrix Q c –Covariance matrix R of 1 st -order stationary Markov sequence has an inverse in the form of symmetric tridiagonal matrix –For highly correlated sequence, the scaled version of R -1 approx. Q c  See Jain’s pp183 for details. DCT is a good replacement for K-L –Close to optimal for highly correlated data –Not depend on specific data like K-L does –Fast algorithm available

39. SVD – Singular Value Decomposition u Jain’s book Section 5.13

40. Summary and Review on Unitary Transform Representation with orthonormal basis  Unitary transform –Preserve energy Common unitary transforms –DFT, DCT, Haar, KLT Which transform to choose? –Depend on need in particular task/application –DFT ~ reflect physical meaning of frequency or spatial frequency –KLT ~ optimal in energy compaction –DCT ~ real-to-real, and close to KLT’s energy compaction => A comparison table in Jain’s book Table 5.3

41. Summary of Today’s Lecture Optimal bit allocation Optimal transform - KLT Next Lecture: –Video coding Readings on Wavelet –Results on rate-distortion theory: Jain’s book Section 2.13 u Further exploration – Tutorial on rate-distortion by Ortega-Ramchandran in IEEE Sig. Proc. Magazine, Nov. 98 –KLT: Jain’s book 2.9, 5.6, 5.11, 5.14, (further exploration – 5.12, 5.13)

43. Slides from Spring ’04 Instructor: Min Wu ECE Department, Univ. of Maryland, College Park   www.ajconline.umd.edu (select ENEE631 S’04)   minwu@eng.umd.edu