Lossy Image Compression From lossless to lossy Subjective and Objective Image Quality Quantization basics Uniform Quantization and 6dB/bit Rule Lossy predictive coding Differential Pulse Coded Modulation (DPCM) Lossy transform coding Image Transforms (Discrete Cosine Transform) Joint Photographic Expert Group (JPEG) EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Why Lossy? In most applications related to consumer electronics, lossless compression is not necessary What we care is the subjective quality of the decoded image, not those intensity values With the relaxation, it is possible to achieve a higher compression ratio (CR) For photographic images, CR is usually below 2 for lossless, but can reach over 10 for lossy EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing A Simple Experiment Bit-plane representation A=a0+a12+a222+ … … +a727 Least Significant Bit (LSB) Most Significant Bit (MSB) Example A=129  a0a1a2 …a7=10000001 a0a1a2 …a7=00110011  A=4+8+64+128=204 EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing A Simple Experiment (Con’t) How will the reduction of gray-level resolution affect the image quality? Test 1: make all pixels even numbers (i.e., knock down a0 to be zero) Test 2: make all pixels multiples of 4 (i.e., knock down a0,a1 to be zeros) Test 3: make all pixels multiples of 4 (i.e., knock down a0,a1,a2 to be zeros) EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Experiment Results original Test 1 Test 3 Test 2 EE465: Introduction to Digital Image Processing

How to Measure Image Quality? Subjective Evaluated by human observers Do not require the original copy as a reference Reliable, accurate yet impractical Objective Easy to operate (automatic) Often requires the original copy as the reference (measures fidelity rather than quality) Works better if taking HVS model into account EE465: Introduction to Digital Image Processing

Objective Quality Measures Mean Square Error (MSE) Peak Signal-to-Noise-Ratio (PSNR) original decoded Question: Can you think of a counter-example to prove objective measure is not consistent with subjective evaluation? EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Answer Shifted (MSE=337.8) Original cameraman image By shifting the last row of the image to become the first row, we affect little on subjective quality but the measured MSE is large EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Lossy Image Compression From lossless to lossy Subjective and Objective Image Quality Quantization basics Uniform Quantization and 6dB/bit Rule Lossy predictive coding Differential Pulse Coded Modulation (DPCM) Lossy transform coding Image Transforms (Discrete Cosine Transform) Joint Photographic Expert Group (JPEG) EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing What is Quantization? In Physics To limit the possible values of a magnitude or quantity to a discrete set of values by quantum mechanical rules In image compression To limit the possible values of a pixel value or a transform coefficient to a discrete set of values by information theoretic rules EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Examples Unlike entropy, we encounter it everyday (so it is not a monster) Continuous to discrete a quarter of milk, two gallons of gas, normal temperature is 98.6F, my height is 5 foot 9 inches Discrete to discrete Round your tax return to integers The mileage of my car is about 55K. EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing A Game Played with Bits Precision is finite: the more precise, the more bits you need (to resolve the uncertainty) Keep a card in secret and ask your partner to guess. He/she can only ask Yes/No questions: is it bigger than 7? Is it less than 4? ... However, not every bit has the same impact How much did you pay for your car? (two thousands vs. $2016.78) EE465: Introduction to Digital Image Processing

Scalar vs. Vector Quantization We only consider the scalar quantization (SQ) in this course Even for a sequence of values, we will process (quantize) each sample independently Vector quantization (VQ) is the extension of SQ into high-dimensional space Widely used in speech compression, but not for image compression EE465: Introduction to Digital Image Processing

Definition of (Scalar) Quantization original value quantization index quantized value f f -1 x ^ x sS Quantizer Q x ^ x f finds the closest (in terms of Euclidean distance) approximation of x from a codebook C (a collection of codewords) and assign its index to s; f-1 operates like a table look-up to return the corresponding codeword EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Numerical Example (I) xR C={-1,0,1} S={1,2,3} 1 2 3 -1 1 red: codewords green: index in C Three codewords x ^ 1 -0.5 x 0.5 -1 EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Numerical Example (II) x{0,1,…,255} C={8,24,40,56,72,…,248} S={1,2,3,…,16} 1 2 3 ... 16 8 248 24 40 ... red: codewords green: index in C sixteen codewords x ^ (note that the scales are different) 40 24 8 x floor operation 16 32 EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Numerical Example (III) 232 216 232 232 216 200 216 40 56 56 72 40 136 104 88 24 225 222 235 228 220 206 209 44 49 56 64 42 128 106 94 27 Q x x ^ Notes  For scalar quantization, each sample is quantized independently  Quantization is irreversible EE465: Introduction to Digital Image Processing

Lossy Data Compression discrete source X S entropy coding (e.g., Huffman) binary bit stream f super-channel S entropy decoding ^ X f-1 EE465: Introduction to Digital Image Processing

Quantization Noise (Distortion) x ^ x Quantization noise: For a continuous random variable, distortion is defined by probability distribution function For a discrete random variable, distortion is defined by probabilities EE465: Introduction to Digital Image Processing

Revisit the Game with Bits Recall the goal of communication is to transmit bits for the purpose of resolving uncertainty Not all bits are equal in the sense of resolving uncertainty (e.g., MSB vs. LSB) The game rule now changes to Given a bit budget, how to most effectively resolve the uncertainty (reduce the distortion)? Given a distortion target, how to use the fewest bits to achieve it? EE465: Introduction to Digital Image Processing

Shannon’s Rate-Distortion Theory Q1 Q2 Rate (bps) Question: which quantization scheme wins, 1 or 2? EE465: Introduction to Digital Image Processing

The Game of Quantizer Design Assume P(X) or Pi is given, design the mapping scheme such that it achieves the smallest distortion for a given rate or vice versa Usually this game is complicated; however, we do know how to play this game for certain class of source (e.g., uniform distribution) EE465: Introduction to Digital Image Processing

Uniform Quantization for Uniform Distribution A scalar quantization is called uniform quantization (UQ) if all its codewords are uniformly distributed (equally-distanced) Example (quantization stepsize =16)     8 40 ... 248 24 Uniform Distribution denoted by U[-A,A] f(x) 1/2A x -A A EE465: Introduction to Digital Image Processing

Quantization Noise of UQ     A -A f(e) 1/  e - /2 /2 Quantization noise of UQ on uniform distribution is also uniformly distributed Recall Variance of U[- /2, /2] is EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing 6dB/bit Rule Signal: X ~ U[-A,A] Noise: e ~ U[- /2, /2] Choose N=2n (n-bit) codewords for X (quantization stepsize) N=2n EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing How Much is 6dB? 6dB increases in SNR implies the reduction of MSE by 75% To cut MSE by half, we approximately need 0.5bps EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Lossy Image Compression From lossless to lossy Subjective and Objective Image Quality Quantization basics Uniform Quantization and 6dB/bit Rule Lossy predictive coding Differential Pulse Coded Modulation (DPCM) Lossy transform coding Image Transforms (Discrete Cosine Transform) Joint Photographic Expert Group (JPEG) EE465: Introduction to Digital Image Processing

Predictive Coding Revisited - Encoding x1 x2 … … xN y1 y2 … … yN y1=x1 yn=xn-xn-1, n=2,…,N - Decoding y1 y2 … … yN x1 x2 … … xN x1=y1 xn=yn+xn-1, n=2,…,N xn yn yn xn _ + xn-1 xn-1 xn D + D xn-1 Decoder DPCM Loop Encoder EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Open-loop DPCM ^ ^ xn yn yn xn _ Q + ^ xn-1 xn-1 xn D + D xn-1 Decoder Encoder Notes:  Prediction is based on the past unquantized sample  Quantization is located outside the DPCM loop EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Catastrophic Error Propagation with Open-loop DPCM -1 error caused by quantization original samples 90 92 91 93 93 95 … a b prediction residues a-b 90 2 2 2 a b decoded samples 90 92 92 94 94 96 … a+b EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Closed-loop DPCM ^ ^ ^ xn yn yn yn xn _ Q + xn-1 ^ ^ ^ xn-1 xn D + D ^ xn-1 Decoder Encoder Xn,yn: unquantized samples and prediction residues ^ ^ Xn,yn: decoded samples and quantized prediction residues Notes:  Prediction is based on the past decoded sample  Quantization is located inside the DPCM loop EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Numerical Example xn 90 92 91 93 93 95 … a yn 90 2 -2 3 2 a-b Q b 90 3 -3 3 3 yn ^ a b a+b xn ^ 90 93 90 93 93 96 … EE465: Introduction to Digital Image Processing

Closed-loop DPCM Analysis ^ xn yn yn _ Q A ^ ^ B xn-1 xn D + ^ xn-1 A: B: The distortion introduced to prediction residue yn is identical to that introduced to the original sample xn EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing DPCM Summary Open-loop DPCM Prediction is based on original unquantized samples Catastrophic error propagation problems (decoder does not have access to the original but quantized samples to do prediction) Closed-loop DPCM Both encoder and decoder employ decoded samples to do prediction Quantization noise affects the accuracy of prediction DPCM is only suitable for lossy image coding at high bit rate (small quantization noise) EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Lossy Image Compression From lossless to lossy Subjective and Objective Image Quality Quantization basics Uniform Quantization and 6dB/bit Rule Lossy predictive coding Differential Pulse Coded Modulation (DPCM) Lossy transform coding Image Transforms (Discrete Cosine Transform) Joint Photographic Expert Group (JPEG) EE465: Introduction to Digital Image Processing

An Example of 1D Transform with Two Variables y2 y1 (1,1) (1.414,0) x1 Transform matrix EE465: Introduction to Digital Image Processing

Generalization into N Variables forward transform basis vectors (column vectors of transform matrix) EE465: Introduction to Digital Image Processing

Decorrelating Property of Transform x2 y2 y1 x1 x1 and x2 are highly correlated y1 and y2 are less correlated p(x1x2)  p(x1)p(x2) p(y1y2)  p(y1)p(y2) EE465: Introduction to Digital Image Processing

Transform=Change of Coordinates Intuitively speaking, transform plays the role of facilitating the source modeling Due to the decorrelating property of transform, it is easier to model transform coefficients (Y) instead of pixel values (X) An appropriate choice of transform (transform matrix A) depends on the source statistics P(X) We will only consider the class of transforms corresponding to unitary matrices EE465: Introduction to Digital Image Processing

Unitary Matrix and 1D Unitary Transform Definition conjugate transpose A matrix A is called unitary if A-1=A*T When the transform matrix A is unitary, the defined transform is called unitary transform Example For a real matrix A, it is unitary if A-1=AT EE465: Introduction to Digital Image Processing

Inverse of Unitary Transform For a unitary transform, its inverse is defined by Inverse Transform basis vectors corresponding to inverse transform EE465: Introduction to Digital Image Processing

Properties of Unitary Transform Energy compaction: only few transform coefficients have large magnitude Such property is related to the decorrelating role of unitary transform Energy conservation: unitary transform preserves the 2-norm of input vectors Such property essentially comes from the fact that rotating coordinates does not affect Euclidean distance EE465: Introduction to Digital Image Processing

Energy Compaction Example Hadamard matrix significant insignificant EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Energy Conservation A is unitary Proof EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Numerical Example Check: EE465: Introduction to Digital Image Processing

Implication of Energy Conservation Q T T-1 A is unitary EE465: Introduction to Digital Image Processing

Summary of 1D Unitary Transform Unitary matrix: A-1=A*T Unitary transform: A unitary Properties of 1D unitary transform Energy compaction: most of transform coefficients yi are small Energy conservation: quantization can be directly performed to transform coefficients EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing From 1D to 2D Do N 1D transforms in parallel EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Definition of 2D Transform 2D forward transform 1D column transform 1D row transform EE465: Introduction to Digital Image Processing

2D Transform=Two Sequential 1D Transforms (left matrix multiplication first) column transform row transform row transform (right matrix multiplication first) column transform Conclusion:  2D separable transform can be decomposed into two sequential  The ordering of 1D transforms does not matter EE465: Introduction to Digital Image Processing

From Basis Vectors to Basis Images 1D transform matrix A consists of basis vectors (column vectors) 2D transform corresponds to a collection of N-by-N basis images basis image EE465: Introduction to Digital Image Processing

Example of Basis Images Hadamard matrix: j i Bij EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing 2D Unitary Transform Suppose A is a unitary matrix, forward transform inverse transform Proof Since A is a unitary matrix, we have EE465: Introduction to Digital Image Processing

Energy Compaction Property of 2D Unitary Transform  Example A coefficient is called significant if its magnitude is above a pre-selected threshold th insignificant coefficients (th=64) EE465: Introduction to Digital Image Processing

Energy Conservation Property of 2D Unitary Transform 2-norm of a matrix X A unitary Example: You are asked to prove such property in your homework EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Implication of Energy Conservation Q T T-1 Similar to 1D case, quantization noise in the transform domain has the same energy as that in the spatial domain EE465: Introduction to Digital Image Processing

Important 2D Unitary Transforms Discrete Fourier Transform Widely used in non-coding applications (frequency-domain approaches) Discrete Cosine Transform Used in JPEG standard Hadamard Transform All entries are 1 N=2: Haar Transform (simplest wavelet transform for multi-resolution analysis) EE465: Introduction to Digital Image Processing

Discrete Cosine Transform EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing 1D DCT forward transform inverse transform Properties  Real and orthogonal excellent energy compaction property  DCT is NOT the real part of DFT Fact The real and imaginary parts of DFT are generally not orthogonal matrices  fast implementation available: O(Nlog2N) EE465: Introduction to Digital Image Processing

2D DCT Notice the excellent energy compaction property of DCT Its DCT coefficients Y (2451 significant coefficients, th=64) Original cameraman image X Notice the excellent energy compaction property of DCT EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Summary of 2D Unitary Transform Unitary matrix: A-1=A*T Unitary transform: A unitary Properties of 2D unitary transform Energy compaction: most of transform coefficients yi are small Energy conservation: quantization can be directly performed to transform coefficients EE465: Introduction to Digital Image Processing

A Tour of JPEG Coding Algorithm Q C Flow-chart diagram of DCT-based coding algorithm specified by Joint Photographic Expert Group (JPEG) EE465: Introduction to Digital Image Processing

Transform Coding of Images Why not transform the whole image together? Require a large memory to store transform matrix It is not a good idea for compression due to spatially varying statistics within an image Idea of partitioning an image into blocks Each block is viewed as a smaller-image and processed independently It is not a magic, but a compromise EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing 8-by-8 DCT Basis Images EE465: Introduction to Digital Image Processing

Block Processing under MATLAB Type “help blkproc” to learn the usage of this function B = BLKPROC(A,[M N],FUN) processes the image A by applying the function FUN to each distinct M-by-N block of A, padding A with zeros if necessary. Example I = imread('cameraman.tif'); fun = @dct2; J = blkproc(I,[8 8],fun); EE465: Introduction to Digital Image Processing

Block-based DCT Example I J note that white lines are artificially added to the border of each 8-by-8 block to denote that each block is processed independently EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Boundary Padding Example 12 13 14 15 16 17 18 19 12 13 14 15 16 17 18 19 12 13 14 15 16 17 18 19 12 13 14 15 16 17 18 19 padded regions When the width/height of an image is not the multiple of 8, the boundary is artificially padded with repeated columns/rows to make them multiple of 8 EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Work with a Toy Example Any 8-by-8 block in an image is processed in a similar fashion EE465: Introduction to Digital Image Processing

Encoding Stage I: Transform • Step 1: DC level shifting 128 (DC level) _ EE465: Introduction to Digital Image Processing

Encoding Step 1: Transform (Con’t) • Step 2: 8-by-8 DCT 88 DCT EE465: Introduction to Digital Image Processing

Encoding Stage II: Quantization Q-table : specifies quantization stepsize (see slide #28) Notes: Q-table can be specified by customer Q-table is scaled up/down by a chosen quality factor Quantization stepsize Qij is dependent on the coordinates (i,j) within the 8-by-8 block Quantization stepsize Qij increases from top-left to bottom-right EE465: Introduction to Digital Image Processing

Encoding Stage II: Quantization (Con’t) Example xij sij f EE465: Introduction to Digital Image Processing

Encoding Stage III: Entropy Coding zigzag scan (20,5,-3,-1,-2,-3,1,1,-1,-1, 0,0,1,2,3,-2,1,1,0,0,0,0,0, 0,1,1,0,1,EOB) End Of the Block: All following coefficients are zero Zigzag Scan EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Run-length Coding (20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB) DC coefficient AC coefficient - DC coefficient : DPCM coding encoded bit stream - AC coefficient : run-length coding (run, level) (5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB) (0,5),(0,-3),(0,-1),(0,-2),(0,-3),(0,1),(0,1),(0,-1),(0,-1),(2,0),(0,1), (0,2),(0,3),(0,-2),(0,1),(0,1),(6,0),(0,1),(0,1),(1,0),(0,1),EOB Huffman coding encoded bit stream EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing JPEG Decoding Stage I: Entropy Decoding encoded bit stream Huffman decoding (0,5),(0,-3),(0,-1),(0,-2),(0,-3),(0,1),(0,1),(0,-1),(0,-1),(2,0),(0,1), (0,2),(0,3),(0,-2),(0,1),(0,1),(6,0),(0,1),(0,1),(1,0),(0,1),EOB encoded bit stream DPCM decoding (20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB) DC coefficient AC coefficients EE465: Introduction to Digital Image Processing

JPEG Decoding Stage II: Inverse Quantization (20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB) zigzag f-1 EE465: Introduction to Digital Image Processing

JPEG Decoding Stage III: Inverse Transform 88 IDCT 128 (DC level) + EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Quantization Noise ^ X X ^ Distortion calculation: MSE=||X-X||2 Rate calculation: Rate=length of encoded bit stream/number of pixels (bps) EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing JPEG Examples 10 (8k bytes) 50 (21k bytes) 90 (58k bytes) worst quality, highest compression best quality, lowest compression 100 EE465: Introduction to Digital Image Processing

JPEG Coding Algorithm Summary EE465: Introduction to Digital Image Processing

Lossy Compression of Color Images Color-space transformation luminance chrominance forward transform inverse transform Luminance channel can be treated just like a gray-scale image EE465: Introduction to Digital Image Processing

Compression of Chrominance Channels Down-sampling Q-Table 2 Cr/Cb Human eye is relatively insensitive to the high-frequency content of the chrominance channels. EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Image Example raw image data JPEG compressed data 768×512 ×24=1.18M bytes 26K bytes Compression ratio=44.6 EE465: Introduction to Digital Image Processing

A Glimpse into Wavelet Transform* The idea of multi-resolution analysis (MRA) Low-resolution (LR) High-resolution (HR) EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing MRA by Haar Transform LP HP Example EE465: Introduction to Digital Image Processing

Reordering Coefficients into Subbands approximation horizontal details Apply Haar transform to each 2-by-2 block vertical details diagonal details é 200 200 ù ê ú 100 100 regrouping Y = ê ú ê ú ê ú ë û EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Multi-level Wavelet Transform Apply one-level Haar transform ú û ù ê ë é = 100 200 Y Apply Haar transform to the LL band only EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Image Example EE465: Introduction to Digital Image Processing

EE465: Introduction to Digital Image Processing Wavelet vs. DCT discrete cosine transform based wavelet transform based JPEG (CR=64) JPEG2000 (CR=64) EE465: Introduction to Digital Image Processing

Lossy Image Compression Summary Quantization introduces irreversible information loss Lossy predictive coding: open-loop DPCM vs. closed-loop DPCM Lossy transform coding: energy compaction an preservation properties of unitary transforms Objective measure for image distortion MSE or PSNR are widely used for their convenience Does not necessarily match subjective quality EE465: Introduction to Digital Image Processing