Lecture 7 Source Coding and Compression Lossy Compression Concepts Dr.-Ing. Khaled Shawky Hassan Room: C3-222, ext: 1204, Email: khaled.shawky@guc.edu.eg 1 1

Lossy Compression Block Diagram 2

Lossy Compression If the original source is discrete: Lossless coding: bit rate >= entropy rate One can further quantize source samples to reach a lower rate If the original source is continuous: Lossless coding will require an infinite bit rate! One must quantize source samples to reach a finite bit rate Lossy coding rate is bounded between the mutual information between the original source (Channel rate) and the quantized source that satisfy a distortion criterion Quantization methods: Scalar quantization (Previously studied – revisited with advanced terminologies) Vector quantization (New method!) 3

Layering of Source Coding of Analog Signals Source coding includes Formatting (input data) Sampling Quantization Symbols to bits (Encoding) Lossless compression Decoding includes Lossless decompression Formatting (output) Bits to symbols (Decoding) Symbols to sequence of numbers (opposite of Quantization) Sequence to waveform (Reconstruction, opposite of sampling) Mirror 4

Layering of Source Coding of Analog Signals Lookup Table: Changes the symbols to a sequence of numbers (which is the opposite of the Quantizer) Analog filter: Interpolate between the sequence (output number) to generate waveform 5

Formatting of Analog Data To transform an analog waveform into a form that is compatible with a digital communication, the following steps are taken: Sampling Quantization and Encoding Base-band transmission pulse-coded modulation (PCM) {send the quantization levels as pulses} 6

First: Sampling of the continuous Signal Sampling: limits the analog/ continuous signal to only limited samples separated by Ts Sampling rate: fs= 1/Ts What will be the frequency response after sampling??? 7

Sampling in Frequency Domain VERY IMPORTANT: Sampling in time domain is equivalent to a repetition in the frequency domain at fs and its multiples 8

Sampling in Frequency Domain To cut the band of the required signal without overlapping of the higher frequency component of the repeated band to the original band (around “0”) 9

High Compression: Under-sampled Signal If we need to sample the analog signal with fewer number of samples, we have to reduce the sampling rate fs= 1/Ts, i.e., increase Ts. What will be the frequency response after sampling??? Ans: Aliasing will happen as in the figure on the left-hand. 10

Aliasing Phenomenon The phenomenon of a high-frequency component in the spectrum of the signal seemingly taking on the identify of a lower frequency in the spectrum of its sampled version To combat the effects of aliasing in practices: Before sampling : a low-pass “anti-aliasing filter” is used to attenuate those high-frequency components of a message signal that are not essential to the information being conveyed by the signal (filter-BW=fa) The filtered signal is sampled at a rate slightly higher than the Nyquist rate of the new filter bandwidth (fs >= 2fa < 2W, where W is the bandwidth of the original signal) Physically realizable reconstruction filter The reconstruction filter is of a low-pass kind with a passband extending from –fa to fa (where fa is the bandwidth) Thus use Nyquist formula

Low Compression: Over-sampled Signal If we need to sample the analog signal with higher number of samples, we have to increase the sampling rate fs= 1/Ts, i.e., decrease the duration Ts. This will introduce more data to store, HENCE, LESS COMRISSION! However, this will be easier to design receiving filters(i.e., not sharp ones) 12

Sampling Theorem The sampling theorem for strictly band-limited signals of finite energy in two equivalent parts Analysis : A band-limited signal of finite energy that has no frequency components higher than B hertz is completely described by specifying the values of the signal at instants of time separated by 1/(2B) seconds. Synthesis : A band-limited signal of finite energy that has no frequency components higher than B hertz is completely recovered form knowledge of its samples taken at the rate of 2B samples per second. (using a low pass filter of cutoff freq. B) Nyquist rate (fs) The sampling rate of 2B samples per second for a signal bandwidth of B hertz Nyquist interval (Ts) 1/(2B) (measured in seconds) 13

Type of Sampling Ideal Practical Natural Sampling Sample and Hold (Flat-top) 14

Ideal Sampling (or Impulse Sampling) x(t)x(t) x(t) Ts Is accomplished by the multiplication of the signal x(t) by the uniform train of impulses Consider the instantaneous sampling of the analog signal x(t) Train of impulse functions select sample values at regular intervals 15

Ideal Sampling (or Impulse Sampling) Exactly as if we multiply the continuous signal with a train of impulses separated by distance Ts The width of each impulse is ALMOST Zero 16

Practical Sampling In practice we cannot perform ideal sampling It is not practically possible to create a train of impulses Thus a non-ideal approach to sampling must be used We can approximate a train of impulses using a train of very thin rectangular pulses: 17

Natural Sampling If we multiply x(t) by a train of rectangular pulses xp(t), we obtain a gated waveform that approximates the ideal sampled waveform, known as natural sampling or “gating” 18

Natural Sampling Each pulse in xp(t) has width Ts and amplitude 1/Ts The top of each pulse follows the variation of the signal being sampled Xs (f) is the replication of X(f) periodically every fs Hz Xs (f) is weighted by Cn  Fourier Series Coefficient The problem with a natural sampled waveform is that the tops of the sample pulses are not flat It is not compatible with a digital system since the amplitude of each sample has infinite number of possible values Another technique known as flat top sampling is used to alleviate this problem; here, the pulse is held to a constant height for the whole sample period This technique is used to realize Sample-and-Hold (S/H) operation In S/H, input signal is continuously sampled and then the value is held for as long as it takes to for the A/D to acquire its value 19

Flat-Top Sampling (sample and Hold) Time Domain Frequency Domain 20

Pulse-Amplitude Modulation (PAM) Output of Sampling is known as PAM Pulse-Amplitude Modulation (PAM) The amplitude of regularly spaced pulses are varied in proportion to the corresponding sample values of a continuous message signal. The levels of the proposed quantizer may be the length M of the M-PAM 21

Coding example: PCM Transmitted sequence (110,111,100, …) discrete signal 22

Dr.-Ing. Khaled Shawky Hassan Email: khaled.shawky@guc.edu.eg Quantization and Distortion Dr.-Ing. Khaled Shawky Hassan Room: C3-222, ext: 1204, Email: khaled.shawky@guc.edu.eg 23 23

Revisiting Quantizer Quantization: 4/16/2017 Revisiting Quantizer Quantization: • Reduces the number of distinct output values to a much smaller set. • Main source of the “loss” in lossy compression. • Three different forms of quantization: 1– Uniform: midrise and midtread quantizers. 2– Nonuniform: companded quantizer. 3– Vector Quantization. From the R-D function: dR/dD = -1/2 * (1/D) * log_2(e), i.e. as D is increasing, |dR/dD| decreasing 24

Recall: Express everything in bits 0 and 1 Discrete finite ensemble: a,b,c,d  00, 01, 10, 11 in general: k binary digits specify 2k messages M messages need log2M bits Analoge signal: Example, ADC with 2 bits only! 1) sample at every Ts 2) represent sample value binary v 11 10 01 00 t Output 00, 10, 01, 01, 11 25

Uniform Quantization Uniform Quantization where i=1, 2, . . . , 9 and Q (x) is the output of the quantizer with respect to the input x 26

Quantization Amplitude quantizing: Mapping samples of a continuous amplitude waveform to a finite set of amplitudes. In Out Quantized values Average quantization noise power Signal peak power Signal power to average quantization noise power 27

Uniform Scalar Quantization • A uniform scalar quantizer partitions the domain of input values into equally spaced intervals, except possibly at the two outer intervals. – The output or reconstruction value corresponding to each interval is taken to be the midpoint of the interval. – The length of each interval is referred to as the step size, denoted by the symbol Δ=(2*Xmax/M)=(2*Xmax/2R). • Two types of uniform scalar quantizers: – Midrise quantizers have even number of output levels. – Midtread quantizers have odd number of output levels, including zero as one of them 28

4/16/2017 Revisiting Quantizer From the R-D function: dR/dD = -1/2 * (1/D) * log_2(e), i.e. as D is increasing, |dR/dD| decreasing Uniform Scalar Quantizers: (a) Midrise, (b) Midtread. 29

4/16/2017 Revisiting Quantizer • For the special case where Δ (what is it again?) = 1, we can simply compute the output values for these quantizers as: • Performance of an M level quantizer. Let B = {b0, b1, . . . , bM} be the set of decision boundaries and Y = {y1, y2, . . . , yM} be the set of reconstruction or output values. • Suppose the input is uniformly distributed in the interval [−Xmax,Xmax]. The rate (in bits!!) of the quantizer is: From the R-D function: dR/dD = -1/2 * (1/D) * log_2(e), i.e. as D is increasing, |dR/dD| decreasing 30

Quantization Error of Uniformly Distributed Source 4/16/2017 Quantization Error of Uniformly Distributed Source • Granular distortion: quantization error caused by the quantizer for bounded input. – To get an overall figure for granular distortion, notice that decision boundaries bi for a midrise quantizer are [(i − 1)Δ, iΔ], i = 1..M/2, covering positive data X (and another half for negative X values). – Output values yi are the midpoints iΔ−Δ/2, i = 1..M/2 • Since the reconstruction values yi are the midpoints of each interval, the quantization error must lie within the values [− Δ/2, Δ/2]. For a uniformly distributed source, the graph of the quantization error is shown in the next Fig From the R-D function: dR/dD = -1/2 * (1/D) * log_2(e), i.e. as D is increasing, |dR/dD| decreasing 31

Quantization error of a uniformly distributed source. 4/16/2017 Quantization Error of Uniformly Distributed Source Zoom on one-level error! From the R-D function: dR/dD = -1/2 * (1/D) * log_2(e), i.e. as D is increasing, |dR/dD| decreasing Quantization error of a uniformly distributed source. 32

Quantization example x(nTs): sampled values xq(nTs): quantized values amplitude x(t) x(nTs): sampled values xq(nTs): quantized values boundaries Quant. levels Red-Green = Quantization error 111 3.1867 Quant. levels 110 2.2762 101 1.3657 100 0.4552 011 -0.4552 010 -1.3657 001 -2.2762 000 -3.1867 Ts: sampling time t PCM codeword 110 110 111 110 100 010 011 100 100 011 PCM TX sequence 33

Uniform Quantization of a Non-uniformly Distributed Source 4/16/2017 Uniform Quantization of a Non-uniformly Distributed Source
Many data are non-uniformly distributed and even unbounded From the R-D function: dR/dD = -1/2 * (1/D) * log_2(e), i.e. as D is increasing, |dR/dD| decreasing This results in two kinds of errors: Granular noise: Quantization error in inner bins.
Overload noise: Quantization error in the outer-most bins. 34

Uniform Quantization of a Non-uniformly Distributed Source 4/16/2017 Uniform Quantization of a Non-uniformly Distributed Source
Many data are non-uniformly distributed and even unbounded From the R-D function: dR/dD = -1/2 * (1/D) * log_2(e), i.e. as D is increasing, |dR/dD| decreasing 35

Example 2: Gaussian PDF Quantization One bit quantization (using the conditional mean of the half): 34% PDF With more bits, the solution is less obvious. Let X ~ N(0, ) 36

How to minimize the Quantization Error? Let us denote the distortion as MSE; then it will be for a certain thresholds: Given M (the levels), the optimal bi and yi that minimize MSE satisfy: Lagrangian condition : yi is the centroid of interval [b(i-1), b(i)]; (conditional mean)

How to minimize the Quantization Error? Now, differentiating w.r.t. bi This gives: Summary of Lloyd-Max conditions:

A special case of Lloyd-Max Quantizer If f(x) = c (uniform), Lloyd-Max quantizer reduces to uniform quantizer This gives: Then, the solution will be an iterative method !!

How to minimize the Quantization Error? Scalar Quantizer: Lloyd algorithm: pdf-optimized quantizer assuming that distribution is known The Lloyd algorithm functions (SQ) Start with an initial set of reconstruction values Set k = 0, D(0) = 0 (distortion). Select threshold e. Find decision boundaries Compute distortion (MSE) D(k) , and if e > D(k) –D(k-1) , stop; otherwise k = k+1 (new iteration), then compute new reconstruction values from the PDF fX(x)

Non-uniform Quantization 41

Non-uniform Quantization Non-uniform quantizers have unequally spaced levels The spacing can be chosen to optimize the Signal-to-Noise Ratio for a particular type of signal It is characterized by: Variable step size Quantizer size depend on signal size; i.e., if the signal amplitude is reduced, it will find more levels in the middle to be will represented. This does not require automatic scaling of the signal as in the automatic- gain control (AGC)! 42

Non-uniform Quantization (Companding) Compressor Uniform Quantizer Expander 43

Non-uniform Quantization (Companding) Compressor Uniform Quantizer Expander The 3 stages combine to give the characteristics of a Non-uniform quantizer, which is stile a 45o line with more levels near to the orgin. 44

Companded quantization. 4/16/2017 Non-Uniform Scalar Quantization Non-Uniformly Distributed Source Companded quantization. • Companded quantization is nonlinear. • As shown above, a compander consists of a compressor function G, a uniform quantizer, and an expander function G−1. • The two commonly used companders are the μ-law and A-law companders. From the R-D function: dR/dD = -1/2 * (1/D) * log_2(e), i.e. as D is increasing, |dR/dD| decreasing 45

Non-Uniform Scalar Quantization 4/16/2017 Non-Uniform Scalar Quantization Non-Uniformly Distributed Source – Compressor function “G” From the R-D function: dR/dD = -1/2 * (1/D) * log_2(e), i.e. as D is increasing, |dR/dD| decreasing 46

Example of a Compressor Function Fig-a Signal at Compressor Input Fig-b Signal at Compressor Output 47

Types of Companding Basically, companding introduces a nonlinearity into the signal This maps a nonuniform distribution into something that more closely resembles a uniform distribution A standard ADC with uniform spacing between levels can be used after the compressor The companding operation is inverted at the receiver There are in fact two standard logarithm based companding techniques US standard called µ-law companding European standard called A-law companding 48

x and y represent the input and output voltages  -Law Companding Standard (North & South America, and Japan) function (compression y, expander y-1) where x and y represent the input and output voltages  is a constant number determined by experiment In the U.S., telephone lines uses companding with  = 255 Samples 4 kHz speech waveform at 8,000 sample/sec Encodes each sample with 8 bits, L = 256 quantizer levels Hence data rate R = 64 kbit/sec  = 0 corresponds to uniform quantization 49

A-Law Companding Standard (Europe, China, Russia, Asia, Africa) where x and y represent the input and output voltages A is a constant number determined by experiment 50

Example on Companding 51

Example on Companding G G−1 52

Example on Companding Example 1: Suppose we had an input of 0.9: If we quantize directly with the uniform quantizer, we get an output of 0.5, resulting in a quantization error of 0.4. If we use the companded quantizer, we first use the compressor mapping, mapping the input value of 0.9 to 1.8. Quantizing this with the same uniform quantizer results in an output of 1.5, with an apparent error of 0.3. The expander then maps this to the final reconstruction value of 0.75, which is 0.15 away from the input (<< 0.4) For all values in the interval [−1 1] (in this case), we will get a decrease in the quantization error Remember: Q_midr = ceil(x)-0.5 Q_midt = floor(x+0.5) 53

Example on Companding Example 2: Suppose we have an input of 2.7: If we quantized this directly with the uniform quantizer, we would get an output of 2.5, with a corresponding error of 0.2 Applying the compressor mapping, the value of 2.7 would be mapped to 3.13, resulting in a quantized value of 3.5. Mapping this back through the expander, we get a reconstructed value of 3.25, which differs from the input by 0.55 (>> 0.2) For all values outside the interval [−1 1] (in this case), we will get an increase in the quantization error!! Remember: Q_midr = ceil(x)-0.5 Q_midt = floor(x+0.5) 54

Example on Companding Note: What is the effective input-output map of this quantizer? All inputs in the interval [0, 0.5] get mapped into the interval [0, 1], For which the quantizer output is 0.5. Which in turn corresponds to the reconstruction value (after C-1 ) of 0.25 Similarly, the interval [0.5, 1] are represented by the values [1 to 2] (represented by 1.5), then after C-1 it is 0.75 55

Lossy Compression Example with Lena Original 8 bits/pixel 3 bits/pixel 56 2 bits/pixel 1 bit/pixel

The Quantization Problem/Distortion Consider first one the simplest cases: quantization of a continuous valued random variable (analogue signal) to a discrete random value (RV) (digital signal). The quantized version of the RV X is denoted as X’(X), With R bits available, the quantized representation can get 2R different values. The problem: find the optimum set of values (reproduction points, code-points) for the quantized version and the regions associated to those. Optimum is measured by distortion measure, which Is often the mean squared error: E[X–X’]2 57

(Communication System-Simon Haykin, Question in Lecture !!) The basic elements of a PCM system (Communication System-Simon Haykin, Question in Lecture !!) 58

Questions 1- What are the two compressors in a basic continuous transmitter and how do we evaluate their errors? 2-What is the function of the low-pass filter before the sampling at the receiver? 3- What is the output format from each block in the communication blocks (analog, disc. binary? 4- How can sampler reduce the overall rate? State what will be problematic in this case? How to select the low-pass filter in this case (before the sampler) 5- What will happen in the Quantizer is uniform and the signal dynamic range decreased dramatically? 6- When will be the repeater (Regenerators, Amplifiers) be safe to use without Automatic Gain Control? 7- What will be the optimum Quantizer? What is the more simple solution? 8- Describe what will be the regenerators signal if a DSP with memory is used. 9- Assume a Gaussian source with ~N(0,sigma^2), how can you use 2-bit (4-level) Quantizer safely and what will be the probability of the possible quantization error in your case? 10- Draw the basic lossy compression block-diagram showing all compression (lossy / lossless) steps and define the output of each block.