1. Machine Vision and Dig. Image Analysis 1 Prof. Heikki Kälviäinen CT50A6100 Lectures 3: Image Transforms Professor Heikki Kälviäinen Machine Vision and Pattern Recognition Laboratory Department of Information Technology Faculty of Technology Management Lappeenranta University of Technology (LUT) Heikki.Kalviainen@lut.fi http://www.lut.fi/~kalviai http://www.it.lut.fi/ip/research/mvpr/

2. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 2 Content Fourier Transform. –Discrete Fourier Transform. –Properties of 2-D Fourier Transform. Separability, periodicity, translation, rotation, scaling, convolution, correlation. –Fast Fourier Transform. –Fourier Transform for image processing and analysis. Gabor filtering. –2-D Gabor filter. –Applications. Other transforms.

3. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 3 Image Transforms Fourier Transform Cosine Sine Hadamard Haar Slant Karhunen-Loeve Fast KL Sinusoidal SDV etc.

4. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 4 Fourier Transform Jean Baptiste Joseph Fourier. In 1807: Any periodic function can be expressed as the sum of sines and/or cosines of different frequencies, each multiplied by a different coefficient. The sum is nowadays called a Fourier series. An image is a 2D signal. From: R.C. Gonzalez and R.E. Woods: Digital Image Processing, 3rd edition, 2008.

5. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 5

6. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 6

7. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 7 Images and Their Fourier Spectra i = sqrt(-1) F(k) also denoted F(u) F(x,y) also denoted F(u,v)

8. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 8

9. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 9 Discrete Fourier Transform (DFT)

10. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 10 Properties of 2-D Fourier Transform Separability. –2-D can be computed as series of 1-D. Periodicity. Translation, rotation, and scaling. –Invariant features. Convolution and correlation. –Spatial filtering in the frequency domain. Etc.

11. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 11

12. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 12

13. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 13

14. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 14

15. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 15 Fast Fourier Transform (FFT) Brute-force implementation of Fourier Transform requires on the order of (MN)^2 summations and additions. 1024 x 1024 pixels => the order of a trillion multiplications and summations for one DFT => computationally too heavy. FFT => the order of MN log_2 MN. Based on the successive doubling method.

16. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 16

17. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 17

18. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 18

19. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 19 Fourier Transform for Image Processing and Analysis Properties of Fourier Transform offer for example the following use: Feature extraction. –Frequency domain features. Image compression. Image filtering. –Image enhancement in the frequency domain. –Preprocessing.

20. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 20 Fourier Transform: Feature Extraction Fourier transforms of texture. Regular patterns. Feature extraction.

21. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 21 Fourier Transform: Image Compression Radius (pixels) % Image power 8 95 16 97 32 98 64 99.4 128 99.8 Not so many coefficients needed => image compression. Lossy/lossless compression?

22. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 22 Fourier Transform: Image Filtering Enhancement in the frequency domain. Convolution: f(x)*g(x)  F(u) G(u). Ideal low pass filter: Original (left) and filtered image (right).

23. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 23 Enhancement in the Frequency Domain Ideal high pass filter: –Original (left) and filtered image (right).

24. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 24 Image Processing Using Gabor Filtering For local and global feature extraction. Filtering in time (spatial) space and frequency space. For image processing and analysis two important parameters: –Frequency f. –Orientation theta. Application example: –Face detection and recognition. –FACEDETECT project (http://www.it.lut.fi/project/facedetect/):http://www.it.lut.fi/project/facedetect/ Machine Vision and Pattern Recognition Laboratory (MVPR), Department of Information Technology, LUT, Finland. Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK.

25. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 25 Feature Detector: 2-D Gabor Filter

26. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 26 Gabor Features Maximal joint localization in the spatial and frequency domain. Smooth and noise tolerant. Parameters for invariance manipulation: Frequency Envelope sharpness Orientation

27. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 27 Constructing Response Matrix Filter response r  (x,y; f,  ) can be calculated for various frequencies f and orientations  to construct a response matrix. columns represent orientations rows represent frequencies image rotation appears as a circular shift of the columns image scaling appears as a shift of the rows (high frequencies may vanish ) A SCALE AND ROTATION INVARIANT TREATMENT OF THE RESPONSE MATRIX CAN BE ESTABLISHED, AND THUS, WE CAN CONCENTRATE ONLY HOW TO CLASSIFY THEM IN THE STANDARD POSE

28. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 28 2-D Gabor Features What do they ”see”?

29. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 29 Face verification (authentication) Validating a claimed identity based on the image of a face: are you Mr./Ms. X? Face recognition (identification) Identifying a person based on an image of his/her face: who are you? Face detection/localization Location of human faces in images at different positions, scales, orientations, and lighting conditions. Face Detection and Recognition

30. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 30 Proposed Algorithm Avoiding a scanning window. Using feature detectors. Shape-free texture model for the final decision.

31. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 31 Evidence Extraction Requirements Scale invariant extraction. Rotation invariant extraction. Provides sufficiently small amount of correct candidate points. (n best points from each class; needs confidence measure). Preferred Estimation of evidence scale and orientation. Fast extraction (scalability).

32. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 32 Classifier Construction eye nostril eye Gaussian mixture model densities (EM estimation) Stability property guarantees approximately the Gaussian form of classes in the feature space. One class may still consist of several sub-clusters (open eye, closed eye, etc.). Bayesian classification of features

33. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 33 Affine Learned Correspondences Aligned images of objects and manually selected features Variability and correspondences 1 2 3 4 56

34. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 34 Affine Hypothesis Search 2 2 3 1 1 1 2 4 5 2 2 1 1.Evidence extraction. 2. Affine search and match to correspon- dence model. Instance approved False alarms occur and hypothesis verification is needed

35. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 35 Face Space Normalization of space where shape variations and capture effects are removed from patterns. Based on three points on the face -> affine registration. Optimal with regard to the photometric variance over a big set of faces.

36. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 36 Features & Feature Detectors Features = salient parts of face. Small localization variance and frequent occurrence over population. Illumination, scale, rotation, and translation invariance. Automatic analysis using the face space desirable.

37. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 37

38. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 38

39. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 39

40. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 40

41. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 41

42. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 42 Other transforms DFT, Cosine, Sine, Hadamar, Haar, Slant, Karhunen-Loeve, Fast KL, Sinusoidal transform, SVD transform. Basis images: Haar (wavelets) (left), Hadamard (right). –Haar values: positive-black, negative-white, zero-gray –Hadamard values: one – black, minus one – white.

43. Machine Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 43 Summary Image transforms from the spatial domain into the frequency domain. Fourier Transform. –For overall image processing and analysis. –Feature extraction, image compression, and image filtering (enhancement). –Discrete Fourier Transform (DFT) and Fast Fourier Transform (FFT). –Properties of 2-D Fourier Transform. Separability, periodicity, translation, rotation, scaling, convolution, correlation. Gabor filtering. –For local and global feature extraction. –Orientation and frequency. –2-D Gabor filter. Other transforms. –For example, Cosine Transform for image compression. –More detailed information later in the course.