1. Introduction to JPEG and MPEG Ingemar J. Cox University College London

2. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox2 Outline  Elementary information theory  Lossless compression  Quantization  Fundamentals of images  Discrete Cosine Transform (DCT)  JPEG  MPEG-1, MPEG-2

3. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox3 Bibliography  D. MacKay, “Information Theory, Inference and learning Algorithms”, Cambridge University Press, 2003. http://www.inference.phy.cam.ac.uk/itprnn/book.html  W. B. Pennebaker and J. L. Mitchell, “JPEG Still Image Data Compression Standard”, Chapman Hall, 1993 (ISBN 0-442-01272-1).  G. K. Wallace, “The JPEG Still-Picture Compression Standard”, IEEE Trans. On Consumer Electronics, 38, 1, 18-34, 1992.  http://en.wikipedia.org/wiki/JPEG

4. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox4 Bibliography  http://en.wikipedia.org/wiki/MPEG-2 http://en.wikipedia.org/wiki/MPEG-2  T. Sikora, “MPEG Digital Video-Coding Standards”, IEEE Signal Processing Magazine, 82-100, September 1997

5. Elementary Information Theory

6. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox6 Elementary Information Theory  How much information does a symbol convey?  Intuitively, the more unpredictable or surprising it is, the more information is conveyed.  Conversely, if we strongly expected something, and it occurs, we have not learnt very much

7. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox7 Elementary Information Theory  If p is the probability that a symbol will occur  Then the amount of information, I, conveyed is:  The information, I, is measured in bits  It is the optimum code length for the symbol

8. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox8 Elementary Information Theory  The entropy, H, is the average information per symbol  Provides a lower bound on the compression that can be achieved

9. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox9 Elementary Information theory  A simple example. Suppose we need to transmit four possible weather conditions: 1.Sunny 2.Cloudy 3.Rainy 4.Snowy  If all conditions are equally likely, p(s)=0.25, and H=2  i.e. we need a minimum of 2 bits per symbol

10. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox10 Elementary information theory  Suppose instead that it is: 1.Sunny 0.5 of the time 2.Cloudy 0.25 of the time 3.Rainy 0.125 of the time, and 4.Snowy 0.125 of the time  Then the entropy is

11. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox11 Elementary Information Theory  Variable length codewords  Huffman code – integer code lengths  Arithmetic codes – non-integer code lengths

12. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox12 Elementary Information Theory  Huffman code WeatherProbabilityInformationInteger code Sunny0.510 Cloudy0.25210 Rainy0.1253110 Snowy0.1253111

13. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox13 Elementary Information Theory  Previous illustration is an example of a lossless code  I.e. we are able to recover the information exactly

14. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox14 Elementary Information Theory  Note that we have assumed that each symbol is independent of the other symbols  I.e. the current symbol provides no information regarding the next symbol

15. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox15 Quantization  Quantization is the process of approximating a continuous (or range of values) by a (much) smaller range of values  Where Round(y) rounds y to the nearest integer   is the quantization stepsize

16. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox16 Quantization  Example:  =2 0 1-3-22345-5-4 0 12-2 0 24-4

17. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox17 Quantization  Quantization plays an important role in lossy compression  This is where the loss happens

18. Fundamentals of Images

19. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox19 Fundamentals of images  An image consists of pixels (picture elements)  Each pixel represents luminance (and colour)  Typically, 8-bits per pixel

20. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox20 Fundamentals of images  Colour  Colour spaces (representations)  RGB (red-green-blue)  CMY (cyan-magenta-yellow)  YUV Y = 0.3R+0.6G+0.1B (luminance) U=R-Y V=B-Y  Greyscale  Binary

21. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox21 Fundamentals of images  A TV frame is about 640x480 pixels  If each pixels is represented by 8-bits for each colour, then the total image size is  640×480*3=921,600 bytes or  7.4Mbits  At 30 frames per second, this would be   220Mbits/second

22. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox22 Fundamentals of images  Do we need all these bits?

23. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox23 Fundamentals of images  Here is an image represented with 8-bits per pixel

24. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox24 Fundamentals of images  Here is the same image at 7-bits per pixel

25. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox25 Fundamentals of images  And at 6-bits per pixel

26. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox26 Fundamentals of images  And at 5-bits per pixel

27. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox27 Fundamentals of images  And at 4-bits per pixel

28. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox28 Fundamentals of images  Do we need all these bits?  No!  The previous example illustrated the eye’s sensitivity to luminance  We can build a perceptual model  Only code what is important to the human visual system (HVS)  Usually a function of spatial frequency

29. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox29 Fundamentals of Images  Just as audio has temporal frequencies  Images have spatial frequencies  Transforms  Fourier transform  Discrete cosine transform  Wavelet transform  Hadamard transform

30. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox30 Discrete cosine transform  Forward DCT  Inverse DCT

31. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox31 Basis functions  DC term

32. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox32 Basis functions  First term

33. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox33 Basis functions  Second term

34. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox34 Basis functions  Third term

35. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox35 Basis functions  Fourth term

36. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox36 Basis functions  Fifth term

37. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox37 Basis functions  Sixth term

38. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox38 Basis functions  Seventh term

39. DCT Example

40. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox40 Example  Signal

41. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox41 Example  DCT coefficients are:  4.2426  0  -3.1543  0  -0.2242  0

42. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox42 Example: DCT decomposition  DC term

43. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox43 Example: DCT decomposition  2 nd AC term

44. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox44 Example: DCT decomposition  6 th AC term

45. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox45 Example: summation of DCT terms  First two non-zero coefficients

46. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox46 Example: summation of DCT terms  All 3 non-zero coefficients

47. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox47 Example  What if we quantize DCT coefficients?   =1  Quantized DCT coefficients are:  4  0  -3  0

48. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox48 Example  Approximate reconstruction

49. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox49 Example  Exact reconstruction

50. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox50 2-D DCT Transform  Let i(x,y) represent an image with N rows and M columns  Its DCT I(u,v) is given by  where

51. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox51 Fundamentals of images  Discrete cosine transform  Coefficients are approximately uncorrelated  Except DC term  C.f. original 8×8 pixel block  Concentrates more power in the low frequency coefficients  Computationally efficient  Block-based DCT  Compute DCT on 8×8 blocks of pixels

52. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox52 Fundamentals of images  Basis functions for the 8×8 DCT (courtesy Wikipedia)

53. Fundamentals of JPEG

54. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox54 Fundamentals of JPEG DCTQuantizerEntropy coder IDCTDequantizer Entropy decoder Compressed image data Encoder Decoder

55. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox55 Fundamentals of JPEG  JPEG works on 8×8 blocks  Extract 8×8 block of pixels  Convert to DCT domain  Quantize each coefficient  Different stepsize for each coefficient  Based on sensitivity of human visual system  Order coefficients in zig-zag order  Entropy code the quantized values

56. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox56 Fundamentals of JPEG  A common quantization table is 1611101624405161 12 141926586055 1413162440576956 1417222951878062 182237566810910377 243555648110411392 49647887103121120101 7292959811210010399

57. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox57 Fundamentals of JPEG  Zig-zag ordering 015614152728 2471316262942 38121725304143 911182431404453 1019233239455254 2022333846515560 2134374750565961 3536484957586263

58. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox58 Fundamentals of JPEG  Entropy coding  Run length encoding followed by  Huffman  Arithmetic  DC term treated separately  Differential Pulse Code Modulation (DPCM)  2-step process 1.Convert zig-zag sequence to a symbol sequence 2.Convert symbols to a data stream

59. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox59 Fundamentals of JPEG  Modes  Sequential  Progressive  Spectral selection Send lower frequency coefficients first  Successive approximation Send lower precision first, and subsequently refine  Lossless  Hierarchical  Send low resolution image first

60. Fundamentals of MPEG-1/2

61. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox61 Fundamentals of MPEG  A sequence of 2D images  Temporal correlation as well as spatial correlation  TV broadcast  Frame-based  Field-based

62. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox62 MPEG  Moving Picture Experts Group  Standard for video compression  Similarities with JPEG

63. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox63 MPEG  Design is a compromise between  Bit rate  Encoder/decoder complexity  Random access capability

64. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox64 MPEG  Images  Spatial redundancy  Perceptual redundancy  Video  Spatial redundancy  Intraframe coding  Temporal redundancy  Interframe coding  Perceptual redundancy

65. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox65 MPEG  Consider a sequence of n frames of video.  It consists of:  I-frames  P-frames  B-frames  A sequence of one I-frame followed by P- and B-frames is known as a GOP  Group of Pictures  E.g. IBBPBBPBBPBBP

66. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox66 MPEG  I-frames  Intraframe coded  No motion compensation  P-frames  Interframe coded  Motion compensation Based on past frames only  B-frames  Interframe coded  Motion compensation Based on past and future frames

67. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox67 MPEG  Motion-compensated prediction  Divide current frame, i, into disjoint 16×16 macroblocks  Search a window in previous frame, i-1, for closest match  Calculate the prediction error  For each of the four 8×8 blocks in the macroblock, perform DCT-based coding  Transmit motion vector + entropy coded prediction error (lossy coding)

68. UCL Adastral Park Postgraduate Campus Nov 27th 2006Ingemar J. Cox68 MPEG  Like JPEG, the DC term is treated separately  DPCM  B-frame compression high  Need buffer and delay