1. CT50A6100 Machine Vision and DIA 1 Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov Lectures 5: Image Compression Professor Heikki Kälviäinen Machine Vision and Pattern Recognition Laboratory Department of Information Technology Faculty of Technology Management Lappeenranta University of Technology (LUT) Heikki.Kalviainen@lut.fi http://www.lut.fi/~kalviai http://www.it.lut.fi/ip/research/mvpr/

2. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 2 Content Introduction. Fundamentals in data compression. Binary Image Compression. Continuous tone images. Video image compression. The material at the following site are used: http://cs.joensuu.fi/pages/franti/imagecomp/ Special thanks to the authors of the material Prof. Pasi Fränti and Dr. Alexander Kolesnikov from University of Joensuu, Finland.

3. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 3 Introduction Why do we need to compress images? Image types. Parameters of digital images. Lossless vs. lossy compression. Measures: rate, distortion, etc.

4. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 4 What is data and image compression? Data compression is the art and science of representing information in a compact form. Data is a sequence of symbols taken from a discrete alphabet. Still image data, that is a collection of 2-D arrays (one for each color plane) of values representing intensity (color) of the point in corresponding spatial location (pixel).

5. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 5 Why do we need image compression? Still image:  One page of A4 format at 600 dpi is > 100 MB.  One color image in a digital camera generates 10-30 MB.  Scanned 3”  7” photograph at 300 dpi is 30 MB. Digital cinema:  4K  2K  3  12 bits/pel = 48 MB/frame or 1 GB/sec or 70 GB/min.

6. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 6 Why do we need image compression? (cont.) 1) Storage. 2) Transmission. 3) Data access. 1990-2000 Disc capacities: 100MB -> 20 GB (200 times!) but seek time: 15 milliseconds  10 milliseconds and transfer rate: 1MB/sec ->2 MB/sec. Compression improves overall response time in some applications.

7. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 7 Source of images Image scanner. Digital camera. Video camera. Ultra-sound (US), Computer Tomography (CT), Magnetic resonance image (MRI), digital X-ray (XR), Infrared. Etc.

8. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 8 Image types Why do we need special algorithms for images?

9. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 9 Binary image: 1 bit/pixel

10. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 10 Grayscale image: 8 bits/pixel Intensity = 0-255

11. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 11 Parameters of digital images

12. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 12 True color image: 3*8 bits/pixel

13. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 13 RGB color space RedGreenBlue

14. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 14 YUV color space Y U V

15. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 15 RGB  YUV R, G, B -- red, green, blue Y -- the luminance U,V -- the chrominance components Most of the information is collected to the Y component, while the information content in the U and V is less.

16. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 16 Palette color image [ R,G,B] = LUT[Index] Example: [64,64,0] = LUT[98] Look-up-table

17. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 17 Multicomponent image: n*8 bits/pixel Spectral image: n components according to wavelengths. Three components R, G, B => “usual” color image.

18. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 18 Multicomponent image: n*8 bits/pixel (cont.) Spectral components and spatial components. For example, remote sensing (satellite images).

19. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 19 Why we can compress image? Statistical redundancy: 1) Spatial correlation a) Local: pixels at neighboring locations have similar intensities. b) Global: reoccurring patterns. 2) Spectral correlation – between color planes. 3) Temporal correlation – between consecutive frames. Tolerance to fidelity: (toistotarkkuus) 1) Perceptual redundancy. 2) Limitation of rendering hardware.

20. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 20 Lossy vs. lossless compression Lossless compression: reversible, information preserving text compression algorithms, binary images, palette images. Lossy compression: irreversible grayscale, color, video. Near-lossless compression: medical imaging, remote sensing. 1) Why do we need lossy compression? 2) When we can use lossy compession?

21. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 21 Lossy vs. lossless compression (cont.)

22. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 22 What measures? Bit rate: How much per pixel? Compression ratio: How much smaller? Computation time: How fast? Distortion: How much error in the presentation?

23. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 23 Rate measures Bit rate: Compression ratio : bits/pixel k = the number of bits per pixel in the original image C/N = the bit rate of the compressed image

24. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 24 Distortion measures Mean average error (MAE): Mean square error (MSE): Signal-to-noise ratio (SNR): Pulse-signal-to-noise ratio (PSNR): (decibels) A is amplitude of the signal: A = 2 8 -1=255 for 8-bits signal.

25. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 25 Other issues Coder and decoder computation complexity. Memory requirements. Fixed rate or variable rate. Error resilience (sensitivity). Symmetric or asymmetric. Decompress at multiple resolutions. Decompress at various bit rates. Standard or proprietary (application based).

26. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 26 Fundamentals in data compression Modeling and coding: –How, and in what order the image is processed? –What are the symbols (pixels, blocks) to be coded? –What is the statistical model of these symbols? Requirement: –Uniquely decodable: different input => different output. –Instantaneously decodable: the symbol can be recognized after its last bit has been received.

27. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 27 Modeling: Segmentation and order of processing Segmentation: –Local (pixels) or global (fractal compression). –Compromise: block coding. Order of processing: –In what order the blocks (or the pixels) are processed? –In what order the pixels inside the block are processed?

28. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 28 Modeling: Order of processing Order of processing: –Row-major order: top-to-down, left-to-right. –Zigzag scanning: Pixel-wise processing (a). DCT-transformed block (Discrete Cosine Transform) (b). –Progressive modeling: The quality of an image quality increases gradually as data are received. For example in pyramid coding: first the low resolution version, then increasing the resolution.

29. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 29 Modeling: Order of processing

30. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 30 Modeling: Statistical modeling Set of symbols (alphabet) S={s 1, s 2, …, s N }, N is number of symbols in the alphabet. Probability distribution of the symbols: P={p 1, p 2, …, p N } According to Shannon, the entropy H of an information source S is defined as follows:

31. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 31 Modeling: Statistical modeling The amount of information in symbol s i, i.e., the number of bits to code or code length for the symbol s i : The average number of bits for the source S:

32. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 32 Modeling: Statistical modeling Modeling schemes: –Static modeling: Static model (code table). One-pass method: encoding. ASCII data: p(‘e’)= 10 %, p(‘t’)= 8 %. –Semi-adaptive modeling: Two-pass method: (1) analysis, (2) encoding. –Adaptive (or dynamic) modeling: Symbol by symbol on-line adaptation during coding/encoding. One-pass method: analysis and encoding.

33. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 33 Modeling: Statistical modeling Modeling schemes: – Context modeling: Spatial dependencies between the pixels. For example, what is the most probable symbol after a known sequence of symbols? –Predictive modeling (for coding prediction errors): Prediction of the current pixel value. Calculating the prediction error. Modeling the error distribution. Differential pulse code modulation (DPCM).

34. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 34 Coding: Huffman coding INIT: Put all nodes in an OPEN list and keep it sorted all times according to their probabilities. REPEAT a) From OPEN pick two nodes having the lowest probabilities, create a parent node of them. b) Assign the sum of the children’s probabilities to the parent node and inset it into OPEN. c) Assign code 0 and 1 to the two branches of the tree, and delete the children from OPEN.

35. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 35 Huffman Coding: Example Symbol p i -log 2 (p i ) CodeSubtotal A 15/39 1.38 0 2*15 B 7/39 2.48 100 3*7 C 6/39 2.70 101 3*6 D 6/39 2.70 110 3*6 E 5/39 2.96 111 3*5 Total: 87 bits 0 1 10 1 0 A C D E 1 B H = 2.19 bits L = 87/39=2.23 bits Binary tree

36. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 36 Huffman Coding: Decoding A - 0 B - 100 C - 101 D - 110 E - 111 0 1 10 1 0 A C D E 1 B Binary tree Bit stream: 1000100010101010110111 (22 bits) Codes: 100 0 100 0 101 0 101 0 110 111 Message: B A B A C A C A D E

37. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 37 Properties of Huffman coding Optimum code for a given data set requires two passes. Code construction complexity O(N log N). Fast lookup table based implementation. Requires at least one bit per symbol. Average codeword length is within one bit of zero-order entropy (Tighter bounds are known): H  R  H+1 bit Susceptible to bit errors.

38. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 38 Coding: Arithmetic coding Alphabet extension (blocking symbols) can lead to coding efficiency. How about treating entire sequence as one symbol! Not practical with Huffman coding. Arithmetic coding allows you to do precisely this. Basic idea: map data sequences to sub-intervals in [0,1) with lengths equal to the probability of corresponding sequence. QM-coder is an arithmetic coding tailored for binary data. 1) Huffman coder: H  R  H + 1 bit/pel 2) Block coder: H n  R n  H n + 1/n bit/pel 3) Arithmetic coder: H  R   H + 1 bit/message (!)

39. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 39 Arithmetic coding: Example 0.70

40. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 40 Binary image compression Binary images consist only of two colors, black and white. The probability distribution of the alphabet is often very skew: p(white)=0.98, and p(black)=0.02. Moreover, the images usually have large homogenous areas of the same color.

41. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 41 Binary image compression: Methods Run-length encoding. Predictive encoding. READ code. CCITT group 3 and group 4 standards. Block coding. JBIG, JBIG2 (Joint Bilevel Image Experts Group). Standard by CCITT and ISO. Context-based compression pixel by pixel. QM-coder (arithmetic coder).

42. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 42 Run-length coding: Idea Pre-processing method, good when one symbol occurs with high probability or when symbols are dependent. Count how many repeated symbol occur. Source ’symbol’ = length of run. Example: …, 4b, 9w, 2b, 2w, 6b, 6w, 2b,...

43. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 43 Run-length encoding: CCITT standard Huffman code table

44. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 44 JBIG Graphic (line art) Halftone Bilevel (binary) documents. Both graphics and pictures (halftone).

45. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 45 Comparison of algorithms

46. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 46 Continuous tone images: lossless compression Lossless and near-lossless compression. –Bit-plane coding: to bit-planes of a grayscale image. –Lossless JPEG (Joint Photographic Experts Group). Pixel by pixel by predicting the current pixel on the basis of the neighboring pixels. Prediction errors coded by Huffman or arithmetic coding (QM-coder).

47. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 47 Continuous tone images: lossy compression Vector quantization: codebooks. JPEG (Joint Photographic Experts Group). Lossy coding of continuous tone still images (color and grayscale). Based on Discrete Cosine Transform (DCT): 0) Image is divided into block N  N. 1) The blocks are transformed with 2-D DCT. 2) DCT coefficients are quantized. 3) The quantized coefficients are encoded.

48. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 48 JPEG: Encoding and Decoding

49. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 49 Divide image into NN blocks 8x8 block Input image

50. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 50 2-D DCT basis functions: N=8 Low High Low High Low 8x8 block

51. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 51 2-D Transform Coding +... y 00 y 01 y 10 y 12 y 23

52. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 52 Zig-zag ordering of DCT coefficients Converting a 2-D matrix into a 1-D array, so that the frequency (horizontal and vertical) increases in this order and the coefficents variance are decreasing in this order. AC: Alternating current DC: Direct current

53. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 53 Example of DCT for image block Matlab: y=dct(x)

54. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 54 Performance of the JPEG algorithm 8 bpp0.6 bpp 0.37 bpp 0.22 bpp

55. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 55 Continuous tone images: more methods Pyramid coding. Fractal coding. Wavelet transform. –JPEG 2000.

56. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 56 JPEG JPEG2000 JPEG: 0.25 bpp JPEG2000: 0.25 bpp

57. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 57 Video image compression Video images are three-dimensional generalization of still images (spatial correlation) where the third dimension is time (spatial and temporal correlation). Each frame of a video sequence can be compressed by any image compression algorithm. Motion JPEG (M-JPEG). –Images separately JPEG coded. MPEG (Moving Pictures Expert Group). –Temporal correlations used. –Two basic techniques: Block based motion compensation. DCT based compression.

58. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 58 Video images: compression ratios NTSC TV HDTV Film quality Channel Bit rate 168 Mb/s 933 Mb/s 2300 Mb/s PC LAN 30 kb/s 5,600:1 31,000:1 76,000:1 Modems 56 kb/s 3,000:1 17,000:1 41,000:1 ISDN 64 - 144 kb/s 1,166:1 6,400:1 16,000:1 T-1, DSL 1.5 Mb/s 112:1 622:1 1,500:1 Ethernet 10 Mb/s 17:1 93:1 230:1 T-3 42 Mb/s 4:1 22:1 54:1 Fiber optic 200 Mb/s 1:1 5:1 11:1

59. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 59 MPEGs MPEG-1 (1992): VideoCD. MPEG-2 (1994): DVD, digital TV, SVCD. * about 50:1 compression, typically 3-10 Mbps. MPEG-3: was abandoned. MPEG-4 (1999 + ): DivX (starting from Version 5). * designed specially for low-bandwidth. MPEG-7 (>1998): * searching and indexing of a/v data, using Description Tools.

60. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 60 MPEG-1: Blocks The pictures are divided into 16x16 macroblocks, each consisting of four 8x8 elementary blocks. The choice of the prediction method is chosen for each macroblock separately. The intra-coded blocks are quantized differently from the predicted blocks: * Intra-coded blocks contain information in all frequencies and are quantized differently from the predicted blocks * The predicted blocks, contain mostly high frequencies and can be quantized with more coarse quantization tables.

61. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 61 MPEG-1: Inter-block Prediction Bidirectional prediction. Forward prediction. Backward prediction. Intra coding. II PP

62. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 62 MPEG-1: Predictions schemes I: Intra pictures are coded as still images by DCT. P: Predicted pictures are coded with reference to a past picture. The difference between the prediction and the original picture is then compressed by DCT. B: Bidirectional pictures, the prediction can be made both to a past and a future frame. Bidirectional pictures are never used as reference.

63. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 63 Motion estimation and compensation The prediction block in the reference frame is not necessarily in the same coordinates than the block in the current frame. Because of motion in the image sequence, the most suitable predictor for the current block may exist anywhere in the reference frame. The motion estimation specifies where the best prediction (best match) is found. Motion compensation consists of calculating the difference between the reference and the current block.

64. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 64 Motion estimation: 1 Exhaustive search block matching. Slow!

65. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 65 Motion estimation : 2 Hierarchical block matching.

66. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 66 Multicomponent (spectral) image compression

67. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 67 Compression: Spectral reduction and clustering

68. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 68 Spectral video: MPEG for multicomponent images

69. CT50A6100 Machine Vision and DIA Prof. Heikki Kälviäinen, Prof. P. Franti, Dr. A. Kolesnikov 69 Summary: Image compression Why do we need special image compression algorithms? MPEG JPEG JBIG M-JPEG Huffman coding Arithmetic coding Multicomponent images Fractal coding, pyramid coding, bit-plane coding, vector quantization DCT RLE Wavelet transform