1. ENEE631 Digital Image Processing (Spring'04) Basics on Image Compression Spring ’04 Instructor: Min Wu ECE Department, Univ. of Maryland, College Park   www.ajconline.umd.edu (select ENEE631 S’04)   minwu@eng.umd.edu Based on ENEE631 Spring’04 Section 8

2. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [2] Overview Last Time: image restoration –A few commonly seen linear distortions in imaging system –Deconvolution: inverse filtering, pseudo-inverse filtering –Wiener filtering: balance between inverse filtering & noise removal Today: –Wrap up with image restoration –Basics compression techniques UMCP ENEE631 Slides (created by M.Wu © 2004)

3. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [3] Overview Last Time: –Further discussion on image restoration –Filtering through frequency-domain operations –Basic concept on source compression: codec, source vs channel coding –Basic coding techniques: PCM, Entropy coding Today: –Continue on source coding/compression techniques => Lossless coding vs. Lossy coding UMCP ENEE631 Slides (created by M.Wu © 2004)

4. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [4] Image Compression Part-1. Basics UMCP ENEE631 Slides (created by M.Wu © 2004)

5. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [5] Why Need Compression? Savings in storage and transmission –multimedia data (esp. image and video) have large data volume –difficult to send real-time uncompressed video over current network Accommodate relatively slow storage devices –they do not allow playing back uncompressed multimedia data in real time u 1x CD-ROM transfer rate ~ 150 kB/s u 320 x 240 x 24 fps color video bit rate ~ 5.5MB/s => 36 seconds needed to transfer 1-sec uncompressed video from CD UMCP ENEE631 Slides (created by M.Wu © 2001)

6. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [6] Example: Storing An Encyclopedia –500,000 pages of text (2kB/page) ~ 1GB => 2:1 compress –3,000 color pictures (640  480  24bits) ~ 3GB => 15:1 –500 maps (640  480  16bits=0.6MB/map) ~ 0.3GB => 10:1 –60 minutes of stereo sound (176kB/s) ~ 0.6GB => 6:1 –30 animations with average 2 minutes long (640  320  16bits  16frames/s=6.5MB/s) ~ 23.4GB => 50:1 –50 digitized movies with average 1 minute long (640  480  24bits  30frames/s = 27.6MB/s) ~ 82.8GB => 50:1  Require a total of 111.1GB storage capacity if without compression  Reduce to 2.96GB if with compression From Ken Lam’s DCT talk 2001 (HK Polytech) UMCP ENEE631 Slides (created by M.Wu © 2001)

7. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [7] Outline Lossless encoding tools –Entropy coding: Huffman, Lemple-Ziv, and others (Arithmetic coding) –Run-length coding Lossy tools for reducing redundancy –Quantization and truncations Predictive coding Transform coding – Zonal coding (truncation) – Rate-distortion and bit allocation – Tree structure Vector quantization UMCP ENEE631 Slides (created by M.Wu © 2001)

8. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [8] PCM coding How to encode a digital image into bits? –Sampling and perform uniform quantization u “Pulse Coded Modulation” (PCM) u 8 bits per pixel ~ good for grayscale image/video u 10-12 bpp ~ needed for medical images Reduce # of bpp for reasonable quality via quantization –Quantization reduces # of possible levels to encode –Visual quantization: dithering, companding, etc. u Halftone use 1bpp but usually upsampling ~ saving less than 2:1 Encoder-Decoder pair  codec I(x,y) Input image SamplerQuantizer Encoder transmit image capturing device UMCP ENEE631 Slides (created by M.Wu © 2001)

9. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [9] Discussions on Improving PCM Quantized PCM values may not be equally likely –Can we do better than encode each value using same # bits? Example –P(“0” ) = 0.5, P(“1”) = 0.25, P(“2”) = 0.125, P(“3”) = 0.125 –If to use same # bits for all values u Need 2 bits to represent the four possibilities if treat –If to use less bits for likely values “0” ~ Variable Length Codes (VLC) u “0” => [0], “1” => [10], “2” => [110], “3” => [111] u Use  i p i l i =1.75 bits on average ~ saves 0.25 bpp! Bring probability into the picture –Use prob. distribution to reduce average # bits per quantized sample UMCP ENEE631 Slides (created by M.Wu © 2001)

10. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [10] Entropy Coding Idea: use fewer bits for commonly seen values At least how many # bits needed? –Limit of compression => “Entropy” u Measures the uncertainty or avg. amount of information of a source u Definition: H =  i p i log 2 (1 / p i ) bits l e.g., entropy of previous example is 1.75 u Can’t represent a source perfectly with less than avg. H bits per sample u Can represent a source perfectly with avg. H+  bits per sample ( Shannon Lossless Coding Theorem ) –“Compressability” depends on the sources Important to design a codebook to decode coded stream efficiently and without ambiguity  See info. theory course (EE721) for more theoretical details UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

11. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [11] E.g. of Entropy Coding: Huffman Coding Variable length code –assigning about log 2 (1 / p i ) bits for the i th value u has to be integer# of bits per symbol Step-1 –Arrange p i in decreasing order and consider them as tree leaves Step-2 –Merge two nodes with smallest probabilities to a new node and sum up probabilities –Arbitrarily assign 1 and 0 to each pair of merging branch Step-3 –Repeat until no more than one node left. –Read out codeword sequentially from root to leaf UMCP ENEE631 Slides (created by M.Wu © 2001)

12. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [12] Huffman Coding (cont’d) S0 S1 S2 S3 S4 S5 S6 S7 000 001 010 011 100 101 110 111 PCM 00 10 010 011 1100 1101 1110 1111 Huffman (trace from root) 0.25 0.21 0.15 0.14 0.0625 0.125 1 0 1 0 0.25 1 0 0.29 1 0 0.54 1 0 0.46 1 0 1.0 1 0 UMCP ENEE631 Slides (created by M.Wu © 2001)

13. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [13] Huffman Coding: Pros & Cons Pro –Simplicity in implementation (table lookup) –Given symbol size, Huffman coding gives best coding efficiency Con –Need to obtain source statistics –The codelength has to be integer => lead to gaps between its average codelength and entropy Improvement –Code a group of symbols as a whole: allow fractional # bits/symbol –Arithmetic coding: fractional # bits/symbol –Lempel-Ziv coding or LZW algorithm u “universal”, no need to estimate source statistics u fix-length codeword for variable-length source symbols UMCP ENEE631 Slides (created by M.Wu © 2001)

14. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [14] Run-Length Coding How to efficiently encode it? e.g. a row in a binary doc image: “ 0 0 0 0 0 0 0 1 1 0 0 0 1 0 1 0 0 0 0 0 0 1 1 1 …” Run-length coding (RLC) –Code length of runs of “0” between successive “1” u run-length of “0” ~ # of “0” between “1” u good if often getting frequent large runs of “0” and sparse “1” –E.g., => (7) (0) (3) (1) (6) (0) (0) … … –Assign fixed-length codeword to run-length in a range (e.g. 0~7) –Or use variable-length code like Huffman to further improve RLC also applicable to general a data sequence with many consecutive “0” (or long runs of other values) UMCP ENEE631 Slides (created by M.Wu © 2001)

15. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [15] Probability Model of Run-Length Coding Assume “0” occurs independently w.p. p: (p is close to 1) Prob. of getting an L runs of “0”: possible runs L=0,1, …, M –P( L = l ) = p L (1-p) for 0  l  M-1 (geometric distribution) –P( L = M ) = p M (when having M or more “0”) Avg. # binary symbols per zero-run –S avg =  L (L+1) p L (1-p) + M p M = (1 – p M ) / ( 1 – p ) Compression ratio C = S avg / log 2 (M+1) = (1 – p M ) / [( 1–p ) log 2 (M+1)] Example –p = 0.9, M=15  S avg = 7.94, C = 1.985, H = 0.469 bpp –Avg. run-length coding rate B avg = 4 bit / 7.94 ~ 0.516 bpp –Coding efficiency = H / B avg ~ 91% UMCP ENEE631 Slides (created by M.Wu © 2001)

16. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [16] Overview and Logistics Last Time: –Lossless compression techniques: Entropy Coding; Run-length Coding –Lossy technique: (scalar) Uniform Quantization Today: –MMSE Quantizer –Quantizer for uniformly distributed source –Companding: Quantizer with pre- and post- nonlinear transformation –Quantizer in predictive coding –Vector vs. Scalar Quantizer –Revisit DFT Transform: A basis perspective => Unitary transform UMCP ENEE631 Slides (created by M.Wu © 2004)

17. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [17] Quantization UMCP ENEE631 Slides (created by M.Wu © 2004)

18. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [18] Review of Quantization L-level Quantization –Minimize errors for this lossy process –What L values to use? –Map what range of continous values to each of L values? t min t max Uniform partition – –Maximum errors = ( t max - t min ) / 2L = A / 2L u u dynamic range A – –Best solution? u u Consider minimizing maximum absolute error (min-max) vs. MSE u u what if the value between [a, b] is more likely than other intervals? t min t max tktk t k+1 (t max— t max )/2L quantization error UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

19. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [19] Bring in Probability Distribution Minimize error in a probability sense –MMSE (minimum mean square error) u assign high penalty to large error and to likely occurring values u squared error gives convenience in math.: differential, etc. An optimization problem –What {t k } and {r k } to use? –Necessary conditions: by setting partial differentials to zero t1t1 t L+1 p.d.f p u (x) r1r1 rLrL Allocate more reconstruct. values in more probable ranges UMCP ENEE631 Slides (created by M.Wu © 2001)

20. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [20] MMSE Quantizer (Lloyd-Max) Reconstruction and decision levels need to satisfy –A nonlinear problem to solve Solve iteratively –Choose initial values of {t k } (0), compute {r k } (0) –Compute new values {t k } (1), and {r k } (1) …… For large number of quantization levels –Approx. constant pdf within t  [t k, t k+1 ), i.e. p(t) = p(t k ’) for t k ’=(t k +t k+1 )/2 Reference: S.P. Lloyd: “Least Squares Quantization in PCM”, IEEE Trans. Info. Theory, vol.IT-28, March 1982, pp.129-137 UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

21. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [21] Quantizer for Piecewise Constant p.d.f. (Jain’s Fig.4.17) UMCP ENEE631 Slides (created by M.Wu © 2001)

22. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [22] MMSE Quantizer for Uniform Distribution Uniform quantizer –Optimal for uniform distributed r.v. in MMSE sense –MSE = q 2 / 12 with q = A / L SNR of uniform quantizer –Variance of uniform distributed r.v. = A 2 / 12 –SNR = 10 log 10 (A 2 / q 2 ) = 20 log 10 L (dB) –If L = 2 B, SNR = (20 log 10 2)*B = 6B (dB) u “1 bit is worth 6 dB.” Rate-Distortion tradeoff t1t1 t L+1 A 1/A p.d.f. of uniform distribution t1t1 t L+1 UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

23. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [23] Quantization – A “Lossy Step” in Source Coding Quantizer achieves compression in a lossy way –Lloyd-Max quantizer minimizes MSE distortion with a given rate Need at least how many # bits for certain amount of error? –(information-theoretic) Rate-Distortion theory Rate distortion function of a r.v. –Minimum average rate R D bits/sample required to represent this r.v. while allowing a fixed distortion D –R(D) = min I(X;X*) u minimize over p(X*|X) given a source p(X) –For Gaussian r.v. and MSE u 1bit more cuts down distortion to ¼ => 6dB D RDRD 22 See Info. Theory course/books for detailed proof of R-D theorem UMCP ENEE739M Slides (created by M.Wu © 2002)

24. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [24] Shannon’s Channel Capacity Entropy ~ measure the amount of uncertainty –H =  i p i log 2 (1 / p i ) bits (see also differential entropy for cont’s r.v.) u Can’t represent a source perfectly with less than avg. H bits per sample but can do so with avg. H+  bits per sample Mutual Information I(X; Y) = E[ log p(X,Y) / (p(X)p(Y)) ] –Measure the amount of info. that one r.v. contains about another r.v. Communicating through imperfect channel –What’s the maximum amount of info. one can accurately convey per channel use? ~ how many diff. msg one can distinguish with n ch. use? –“Capacity” and “Accuracy” are in asymptotical sense –C = max I(X; Y) u maximize over p(X) given p(Y|X) u for discrete channel, I(X; Y) = H(Y) - H(Y|X) u for Binary Symmetric Channel: C = 1 – h(p) where h(p) is bin. entropy u for AWGN channel, C = ½ * log( 1 +  2 x /  2 n ) See Information Theory course and books for more details (ENEE721, Cover-Thomas’ book ) UMCP ENEE739M Slides (created by M.Wu © 2002)

25. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [25] Think More About Uniform Quantizer Uniformly quantizing [0, 255] to 16 levels Compare relative changes –x1 = 100  [96, 112)  quantize to “104” ~ 4% change –x2 = 12  [0, 16)  quantize to “8” ~ 33% change –Large relative changes could be easily noticeable by eyes! UMCP ENEE631 Slides (created by M.Wu © 2001)

26. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [26] Compandor Compressor-Expander –Uniform quantizer proceeded and succeeded by nonlinear transformations u nonlinear function amplifies or suppresses particular ranges Why use compandor? –Inputs of smaller values suffer higher percentage of distortion under uniform quantizer –Nonlinearity in perceived luminance u small difference in low luminance is more visible –Overall companding system may approximate Lloyd-Max quantizer f(.) compressor g(.) expander Uniform quantizer u w yu’ UMCP ENEE631 Slides (created by M.Wu © 2001)

27. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [27] Compandor (cont’d) Nonlinear transformation functions –To make overall system approximate Lloyd-Max quantizer –Without iterative process to determine parameters (Jain’s Fig.4.19) UMCP ENEE631 Slides (created by M.Wu © 2001)

28. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [28] Quantization in Coding Correlated Sequence Consider: high correlation between successive samples Predictive coding –Basic principle: Remove redundancy between successive pixels and only encode residual between actual and predicted –Residue usually has much smaller dynamic range u Allow fewer quantization levels for the same MSE => get compression –Compression efficiency depends on intersample redundancy First try: Any problem with this codec? u Q (n) Predictor + e Q (n) u P (n) = f[u Q (n-1)] Decoder u(n) Predictor Quantizer _ e(n) e Q (n)Encoder u’ P (n) = f[u(n-1)] UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

29. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [29] Predictive Coding (cont’d) Problem with 1 st try –Input to predictor are different at encoder and decoder u decoder doesn’t know u(n)! –Mismatch error could propagate to future reconstructed samples Solution: Differential PCM (DPCM) –Use quantized sequence u Q (n) for prediction at both encoder and decoder –Simple predictor f[ x ] = x –Prediction error e(n) –Quantized prediction error e Q (n) –Distortion d(n) = e(n) – e Q (n) u Q (n) Predictor + e Q (n) u P (n) = f[u Q (n-1)] Decoder Encoder u(n) Predictor Quantizer _ e(n)e Q (n) + u P (n)=f[u Q (n-1)] u Q (n) UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002) Note: “Predictor” contains one-step buffer as input to the prediction

30. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [30] Predictive Coding (cont’d) Problem with 1 st try –Input to predictor are different at encoder and decoder u decoder doesn’t know u(n)! –Mismatch error could propagate to future reconstructed samples Solution: Differential PCM (DPCM) –Use quantized sequence u q (n) for prediction at both encoder and decoder u u q ~ (n)= f( u q (n-1), u q (n-2), …, u q (n-m) ) –Prediction error e(n) –Quantized prediction error e q (n) –Distortion d(n) = e(n) – e q (n) u q (n) Predictor + e q (n) u q ~ (n) Decoder Encoder u(n) Predictor Quantizer _ e(n)e q (n) + + u q ~ (n) u q (n) What predictor to use? UMCP ENEE631 Slides (created by M.Wu © 2001)

31. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [31] More on Predictor Causality required for coding purpose –Can’t use the samples that decoder hasn’t got as reference Use last sample u q (n-1) –Equiv. to coding the difference p th –order auto-regressive (AR) model –Use a linear predictor from past samples –Determining the predictor coeff. (discuss more later) Line-by-line DPCM – – predict from the past samples in the same line 2-D DPCM – – predict from past samples in the same line and from previous lines UMCP ENEE631 Slides (created by M.Wu © 2001)

32. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [32] Comparison and Further Improvement Adaptive technique (brief) –Adapting the quantizer and predictor to variations in local image statistics –Improve subjective image quality especially on boundary of different regions –Ref. Jain’s pp495 –See more in ENEE724 Adaptive Signal Processing From Jain’s Fig.11.12 UMCP ENEE631 Slides (created by M.Wu © 2001)

33. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [33] Vector Quantization Encode a set of values together –Find the representative combinations –Encode the indices of combinations Stages – Codebook design – Encoder – Decoder Scalar vs. Vector quantization –VQ allows flexible partition of coding cells –VQ could naturally explore the correlation between elements –SQ is simpler in implementation From Bovik’s Handbook Sec.5.3 scalar quantization of 2 elements vector quantization of 2 elements UMCP ENEE631 Slides (created by M.Wu © 2001)

34. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [34] Outline of Core Parts in VQ Design codebook –Optimization formulation is similar to MMSE scalar quantizer –Given a set of representative points u “Nearest neighbor” rule to determine partition boundaries –Given a set of partition boundaries u “Probability centroid” rule to determine representative points that minimizes mean distortion in each cell Search for codeword at encoder –Tedious exhaustive search –Design codebook with special structures to speed up encoding u E.g., tree-structured VQ Reference: Wang’s book Section 8.6 A. Gersho and R. M. Gray, Vector Quantization and Signal Compression, Kluwer Publisher. R. M. Gray, ``Vector Quantization,'' IEEE ASSP Magazine, pp. 4--29, April 1984. vector quantization of 2 elements UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

35. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [35] Summary: List of Compression Tools Lossless encoding tools –Entropy coding: Huffman, Lemple-Ziv, and others (Arithmetic coding) –Run-length coding Lossy tools for reducing bit rate –Quantization: scalar quantizer vs. vector quantizer –Truncations: discard unimportant parts of data Facilitating compression via Prediction –Convert the full signal to prediction residue with smaller dynamic range –Encode prediction parameters and residues with less bits Facilitating compression via Transforms –Transform into a domain with improved energy compaction UMCP ENEE631 Slides (created by M.Wu © 2004)

36. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [36] Summary of Today’s Lecture Image restoration – Further discussions Basics compression techniques –PCM coding –Entropy coding –Run-length coding Next time –Cont’d on image compression Readings –Jain’s book 8.1-8.4; 11.1-11.2, 11.9, 4.5-4.6 (or) Gonzalez’s book 5.5-5.8; 8.1-8.2, 8.3.1, 8.3.4, 8.4 –Wang’s book 8.1, 8.4, 8.5 UMCP ENEE631 Slides (created by M.Wu © 2004)

37. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [37] Summary of Today’s Lecture Entropy coding and Run-length coding Quantization –Uniform vs non-uniform quantization (Lloyd-Max, companding) –Quantization in predictive coding (careful in designing decoder) –Scalar vs Vector quantization Next time –Compacting signal energy via transforms Readings –Jain’s book 11.1-11.2; 4.5-4.7 (or) Gonzalez’s book 8.3.1; 8.4; 8.5.1 –Wang’s book 8.4; 8.5 UMCP ENEE631 Slides (created by M.Wu © 2004)

38. ENEE631 Digital Image Processing (Spring'04) Lec9 – Basics on Compression [38] Reference [Copy] Compandor design of Waveform coding book [Copy] Transform coding article in Sig.Proc. Magzine [Copy] Bovik’s book Chapt.5.1 (Lossless coding) … huffman, lempel-ziv, etc. [links of E-copy] Wallace’s JPEG paper [VQ] –A. Gersho and R. M. Gray, Vector Quantization and Signal Compression.Vector Quantization and Signal Compression –R. M. Gray, ``Vector Quantization,'' IEEE ASSP Magazine, pp. 4--29, April 1984. –Y. Linde, A. Buzo, and R. M. Gray, ``An Algorithm for Vector Quantizer Design,'' IEEE Transactions on Communications, pp. 702--710, January 1980. Figure is from slides at Gonzalez/ Woods DIP book website (Chapter 3)