1. JPEG 2000 Image Coding System Coding of Still Pictures JPEG 2000 Part I Final Committee Draft Version 1.0 April, 2000

2. Why another JPEG? JPEG2000 can provide higher compression efficiency  for low bit-rate application larger images (up to 64k by 64k) superior performance for natural images (e.g. photo) computer-generated images (e.g. graphics) compound documents (e.g. bi-level text) five progression orders progressive by quality (SNR) or by resolution  lossless and lossy compression in a single stream ROI (Region of Interest) coding

3. JPEG v.s. JPEG2000 (Main Differences) DCT v.s. DWT (Discrete Wavelet Transform) Way of coding coefficients JPEG Zigzag scan Huffman coding, run-length coding JPEG2000 Coefficient bit modeling Scan algorithm Probability estimation Arithmetic entropy coding

4. Main Ideas of JPEG2000 DWT Way of coding coefficients Coefficient bit modeling Scan algorithm: bitplane and sub-bitplane (coding pass) Probability estimation Arithmetic entropy coding Tag tree to save tree information in header For progressive transmission ROI coding Five progression orders

5. Image -> Tiles Apply DWT for each tile Tile

6. Basic Coding Scheme of JPEG2000 DWT Scan Algorithm Probability Estimation Entropy Coding Tile of a Image Bit Stream DC level shift (Component Transform) Coefficient bit modeling Quantization

7. DC Level Shift e.g. The ith component is unsigned. The bit depth of the ith component is 8. (  Each sample value is in (0, 255))  I(x, y) = I(x, y) – 2 (  Each sample value is in (-128, 127)) (8 – 1)

8. Component Transform To decorrelate redundancy between components  to improve compression efficiency Component transformation is applied to the three first components. ICT (Irreversible Component Transform) For lossy compression RCT (Reversible Component Transform) For lossy and lossless compression

9. ICT (Irreversible Component Transform) The Forward ICT: Given I0, I1, I2 in the first three components: Y0 = 0.299 I0 + 0.587 I1 + 0.144 I2 Y1 = -0.16875 I0 + 0.33126 I1 + 0.5 I2 Y2 = 0.5 I0 + 0.41869 I1 + 0.08131 I2 If the first three components are R, G, and B, then the Forward ICT can be seen as a approximation of a YCbCr transformation. The Inverse ICT: I0 = Y0 + 1.402 Y2 I1 = Y0 – 0.34413 Y1 – 0.71414 Y2 I2 = Y0 + 1.772 Y1

10. RCT (Reversible Component Transform) The Forward RCT: Given I0, I1, I2 in the first three components: Y0 = floor( (I0 + 2 I1 + I2) / 4 ) Y1 = I2 – I1 Y2 = I0 – I1 The Inverse RCT I1 = Y0 – floor( (Y2 + Y1) / 4 ) I0 = Y2 + I1 I2 = Y1 + I1

11. High Pass: Y(2n+1) = X(2n+1) - floor( (X(2n) + X(2n+2)) / 2 ) Example of DWT: Lifting-based (5, 3) filter 171172164165142996355691711645563 012345678 3 174 12-3-113 168 144 60 64 Low Pass: Y(2n) = X(2n) + floor( (X(2n-1) + X(2n+1) + 2) / 4 ) Extension Run … 2 2 1414 1414 1 2 2 8 1 4 3 4 1 4 8

12. Image “ Lena ” (256x256) Resolution levels (each consists of 1 or 3 sub-bands) LL0 HL1, LH1, HH1 HL2, LH2, HH2 Precincts (  positions) (packet partitions) e.g. 8 precincts(16x32) Code-blocks e.g. 2 code-blocks(16x16) HL3, LH3, HH3 Example of DWT (con ’ t) HL3 HH3 LH3

13. Example of DWT (con ’ t) (100,100,100,100,100,100, 0, 0, 100,100,100,100,100,100) (100, 100, 113, 0, 88, 100, 100)(0, 0, 50, -50, 0, 0, 0) 100 0000000000 113 88 100 0 0 0 0 50 -50 100 100 100 0 1D DWT (for row) HighLow 1D DWT

14. Example of DWT (con ’ t) DWT

15. Quantization Each of the transform coefficients f(u, v) is quantized to the value q(u, v). q = sign(f) Quantization step-size = 2 (1+ ) Pair (Exponent, mantissa ) for sub-band b is obtained either explicitly or implicitly. Nominal dynamic range is the bit depth for sub- band b to represent the original image. |f|  b  b R -  b b 2 11  b  b  b R b

16. Quantization (con ’ t) Explicit quantization (, ) are signaled in the bit stream for every sub-band. Implicit quantization (, ) is signaled in the bit stream for the LL sub-band. (, ) are derived implicitly from (, ).  b  b  0  0  b  b  0  0

17. Quantization (con ’ t) RR+1 R+2 R+1 R+2 R is the bit depth of the original image 12 24 2 2 4 Two level analysis gain R b for each sub-band b R 

18. 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 LSB Bitplane 0 10 0 0 0 0 7 0 0 0 0 3 0 0 0 0 – 2 0 0 0 0 0 0 0 0 10 -> + 1 0 1 0 7 -> + 0 1 1 1 3 -> + 0 0 1 1 -2 -> - 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 MSBLSB 0 1 0 0 0 0 0 0 0 0 MSB Code-block

19. Code-block Scan Pattern 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 LSB 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 MSB Bitplane0 Bitplane1 Bitplane2 Bitplane3

20. Sub-bitplane (Coding Pass) Each bitplane is coded in 3 sub-bitplanes (coding passes) Significance propagation (Magnitude) Refinement Clean-up Each coefficient bit in a bitplane is coded only one of the 3 sub-bitplane Depend on the “ significance state ” of that coefficient and its 8 neighboring coefficients

21. Significance State State of a coefficient at a particular bitplane If a coefficient has the first “ 1 ” bit at or before the given bitplane is considered “ significant ”,  significance state = 1 otherwise it is considered “ insignificant ”.  significance state = 0

22. Example of Sub-bitplane Coding Order 0 10 1+ 0 1 0 7 0 1+ 1 1 3 0 0 1+ 1 -2 0 0 1- 0 0 … Bitplane3 Bitplane2 Bitplane1 Bitplane0 (Significance state = 1)  The coefficient is significant S R C C SR C S RC

23. (Neighbor) Context: Function of Neighbors States 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 LSB 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 MSBBitplane3 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 Bitplane2 Significance state (after coding bitplane3) 0 1 0 0 0 0 0 0 0 0 V0D0D1 H0H1 D2V1D3 8 neighbors states = 0  context = 0

24. Bits in a Sub-bitplane significance state = 1 context ≠ 0 Significance propagationRefinementClean-up Significance propagationRefinementClean-up true false more important (significant)(insignificant) (most likely to be significant)

25. Bits in a Sub-bitplane (con ’ t) Bits coded in each sub-bitplane (coding pass) Significance propagation Bits from “ most likely to become significant ” coefficients  (significance state = 0) and (context ≠ 0) Refinement Bits from already significant coefficients  (significance state = 1) Clean-up Bits from all remaining coefficients  (significance state = 0) and (context = 0)

26. Coding Operations 4 coding operations Significance coding (9 context bins) Sign coding (5 context bins) Magnitude refinement coding (3 context bins) Clean-up coding (run-length coding and 9 context bins) Coding operations in each sub-bitplane Significance propagation Significance coding and sign coding Refinement Magnitude refinement coding Clean-up Clean-up coding and sign coding

27. 9 Context Bins for Significance and Clean-up Codings V0D0D1 H0H1 D2V1D3 LL and LHHLHHContext (label) ΣHΣVΣDΣDΣHΣVΣDΣDΣ(H+V)ΣDΣD 22 ≧3≧3 8 1 ≧1≧1 ≧1≧1 1 ≧1≧1 27 10 ≧1≧1 01 ≧1≧1 026 100010 ≧2≧2 15 0220114 0110013 00 ≧2≧2 00 ≧2≧2 ≧2≧2 02 001001101 000000000 100 00 000 e.g. ΣH = 0 ΣV = 1 ΣD = 0 For HL 5

28. 5 Context Bins for Sign Coding HVContext (label)XOR bit ++13 0  expected value = 0 (expected sign = +) +012 +-11 0+10 009 0- 1  expected value = 1 (expected sign = -) -+11 -012 --13 V0 H0H1 V1 1+ 00 0 e.g. H = 0 V = + 10 1- 1+ 0 H = 0 V = - 10 HVHV Code (sign bit) XOR (XOR bit) bit

29. 3 Context Bins for Magnitude Refinement Coding First refinement for this coefficient Context label 16 Σ(H+V+D) = 0 Context label 15 Context label 14 false true V0D0D1 H0H1 D2V1D3

30. If the four contiguous coefficient bits currently satisfy Run-length Coding for Cleanup Coding Code 0 (using context label 17) Code the four bits (using context label 18) Four bits are all 0 true false Code 1 (using context label 17) significance state = 0 context = 0 UNIFORM context Run-length context 0 1 0 0 0 0 0 0 0 0

31. Huffman Coding 設法讓 “ 較常出現 ” 的 symbol ，使用較短 的 code 。 Symbolabcd 出現機率 3/41/81/16 Fixed-Length Code00011011 Huffman Code010110111 資訊量 (= log(1/p) ) 0.415344 平均資訊量 (Entropy) (= Σ p*log(1/p) ) 0.415(3/4) + 3(1/8) + 4(1/16) + 4(1/16) = 1.18625 平均長度 2(3/4) + 2(1/8) + 2(1/16) + 2(1/16) = 2 1(3/4) + 2(1/8) + 3(1/16) + 3(1/16) = 1.375

32. AC: Arithmetic (Entropy) Coding 3/4 1/8 1/16 d c b a 0 1 0.25 0.125 0.0625 0.25 0.16796875 0.1796875 0.16815856 0.168701172 a 0.125 0.25 b b d a Encode babda 0.16815856 0.15625

33. Example of AC for Two Symbols 0.9 0.1 0 1 0 A = 1 0 A = 0.9 Encode 0  A = A – 0.1 C = C + 0.1 C = 0.1C = 0 0.09 A: the interval C: the code 0 A = 0.81 C = 0.19 0.081 0.810.729 Encode 0  A = A – 0.09 C = C + 0.09 0 A = 0.729 C = 0.271 Encode 0  A = A – 0.081 C = C + 0.081

34. Example of AC for Two Symbols (con ’ t) 0.9 0.1 0 1 0 A = 1 0 A = 0.1 Encode 1  A = 0.1 C = C C = 0 0.01 A: the interval C: the code 0 A = 0.01 C = 0 0.001 0.090.009 Encode 1  A = 0.01 C = C …

35. AC for Two Symbols Two symbols (0 and 1) LPS (Less Probable Symbol) Probability Qe MPS (More Probable Symbol) Probability (1-Qe) E.g. Pr(0) = 0.9 Pr(1) = 0.1 MPS = 0 LPS = 1 Qe = 0.1

36. The MQ-Coder 1-Qe Qe MPS LPS 0 A = 1 0 A – Qe Encode MPS  A = A – Qe C = C + Qe C = Qe 0.75 ≦ A ＜ 1.5 C = 0 1-Qe Qe MPS LPS 0 A = 1 0 Qe Encode LPS  A = Qe C = 0 A ～ 1

37. The MQ-Coder (con ’ t) Keep the interval A in the range 0.75 ≦ A ＜ 1.5 (by using renormalization, if needed), and save code in C Code MPS A = A – Qe (i.e. sub-interval for the LPS) C = C + Qe Code LPS A = Qe (i.e. sub-interval for the MPS)

38. Renormalization of The MQ-Coder Out one bit (most significant bit) of C Double A and C  A<<=1, C<<=1 Obtain a new probability estimate (state) for the context

39. Finite State Machine of The MQ-Coder Pr(0) = 0.7 Pr(1) = 0.7 Pr(0) = 0.86 Pr(0) = 0.94 Pr(0) = 0.5 Pr(0) = 0.67Pr(0) = 0.51 Pr(0) = 0.5 0 0 0 1 1 1 0 1 Slowly change Pr(0) Fast change Pr(0) MPS = 0 …… … …

40. Initialization and Termination of The MQ-Coder Initialization Start at the initial state  Pr(LPS) = Qe Termination Terminate coding operations Generate the required terminate marker

41. Usage of The MQ-Coder Each context bin invokes a MQ-Coder. The initial state of each MQ-Coder: UNIFORM context (label 18) the state with Pr(0) = 0.5, and no state-transaction Run-length context (label 17) the state with Pr(0) = 0.936988 All-zero-neighbors context (label 0) the state with Pr(0) = 0.969947 Other contexts (labels 1-16) the state with Pr(0) = 0.5

42. Layer A layer consists of some number of consecutive sub-bitplanes (coding passes) e.g. layer0 = {the first 2 sub-bitplanes} layer1 = {the following 3 sub-bit-planes} … layer0layer1 higher sub-bitplaneslower sub-bitplanes …layer2

43. Packet Header 2 sub-bitplanes from code-block0 1 sub-bitplane from code-block1 5 sub-bitplanes from code-block2 … e.g.

44. Tag Tree A way of representing a 2D array in a hierarchical way To save information in a packet header Only the information needed for the current code-block is stored in the packet header Ability to progressive transmit

45. Example of Tag Tree 643322343422332242 6 4 3 3 2 2 3 4 3422 3 3 2 2 42 0011 (A>0, A=B=2) 011 (C=n1=3) 0001 (n2>C=3, n2=C+3) 1 (D=B) 01 (n3>D, n3=D+1) A B C n1n2 D n3 A, B, C, n1 n2 D, n3 Minimum of the sub-tree 22 2 2 3 2 322

46. Information in Packet Header Bit for zero ( “ 0 ” ) or non-zero ( “ 1 ” ) length packet For each precinct of each sub-band For all code-blocks in the precinct in raster order Inclusion information (  the layer ID where the code-block first appears in the precinct of the sub-band) If included at the 1 st time  tag tree else one bit (the code-block is included(1)/not included(0)) If the code-block is included If included at the 1 st time Zero bit-planes information  tag tree # of sub-bitplanes Length for the code-block …

47. ROI (Region of Interest) ROI is a part of an image coded earlier in the codestream than the rest part (background) A tile can be either Outside the ROI Inside the ROI Partly inside the ROI ROI coding Arbitrary region shapes Any number of regions Maxshift method

48. ROI Coding 4 steps: Generate ROI mask M(x, y) Select the scaling value s Maxshift method s ≧ (bit depth of max(BG)) Scale down all BG by s Write s into codestream 1(x, y) is needed for ROI 0otherwise (  (x, y)  BG (background))  M(x,y) =

49. ROI Mask Generation e.g. 17117216416514299635569 012345678 3 174 12-3-11 168 144 60 64 2 2 1414 1414 …… Sample in ROI High Low

50. ROI Maxshift Method ROI coefficients Coefficient values ROI coefficients max(BG) Maxshift e.g. max(BG) = 10 s = 4 Shift-left 4 bits for each coefficient in ROI Mask

51. Advantages of ROI Maxshift Method Support for arbitrary region shapes and any number of regions with minimal complexity No need for sending shape information shape codec ROI mask at decoder side Decoder as simple as non-ROI capable decoder Support progression by accuracy and by quality

52. Error Resilience Error resilience tools Packet level Entropy coding level

53. Error Resilience Tools at Packet Level Packets Re-synch marker SOP (Start of Packet) at packet boundaries Locate errors in a packet Short packet format (packed packet) Move the packet headers to either Tile-part header (PPT) Main header (PPM) of the codestream

54. Error Resilience Tools at Entropy Coding Level Code-blocks Re-synch marker at code-block boundaries Locate errors in a code-block Bit-planes Put segmentation symbol at the end of each bit- plane for error detection Sub-bitplanes (coding passes) The AC terminate on each sub-bitplane some sub-bitplanes (  selective AC bypass) last sub-bitplane Reset context probabilities on each sub-bitplane

55. Selective AC bypass SPAC, terminateAC MRAC, terminateAC CAC, terminateACAC, terminate SP (significance propagation) AC, terminateAC MR (magnitude refinement) AC, terminateAC C (cleanup)AC, terminateAC C (cleanup)AC, terminateAC 1 2 4 … SPAC, terminateACRaw MRAC, terminateACRaw, terminate CAC, terminateACAC, terminate 5 … SPAC, terminateACRaw MRAC, terminateACRaw, terminate CAC, terminate Final AC terminate on each sub-bitplane AC terminate on last sub-bitplane ………… …………

56. Progression Order The interleaving of the packets can progress along 4 axes: Component e.g. Y, Cb, Cr Resolution e.g. LL0, HL1,LH1,HH1, HL2,LH2,HH2, … Position Layer

57. Layer-resolution-component- position Progressive (1) layer0layer1 higher sub-bitplaneslower sub-bitplanes LL0HL1 LH1 HH1HL2 LH2 HH2… YCbCr position0posotion1posotion2 … …

58. Resolution-layer-component- position Progressive (2) layer0layer1 LL0HL1 LH1 HH1HL2 LH2 HH2… YCbCr position0position1position2 … …

59. Resolution-position-component- layer Progressive (3) layer0layer1 LL0HL1 LH1 HH1HL2 LH2 HH2… YCbCr position0position1position2 … …

60. Position-component-resolution- layer Progressive (4) layer0layer1 LL0HL1 LH1 HH1HL2 LH2 HH2… YCbCr position0position1position2 … …

61. Component-position-resolution- layer Progressive (5) layer0layer1 LL0HL1 LH1 HH1HL2 LH2 HH2… YCbCr position0position1position2 … …

62. What we have learned? Operations of DWT Intelligent scan order More important coefficient bits are coded earlier. MQ coder (Arithmetic coder) for two symbols Tag tree ROI coding

63. Other Topics Rate control (Annex 10) Visual frequency weighting (Annex J.8) TCQ (Trellis coded quantization) Postprocessing