1. Lecture 4: Data Compression Techniques TSBK01 Image Coding and Data Compression Jörgen Ahlberg Div. of Sensor Technology Swedish Defence Research Agency (FOI)

2. Outline Huffman coding Huffman coding Arithmetic coding Arithmetic coding Application: JBIG Application: JBIG Universal coding Universal coding LZ-coding LZ-coding LZ77, LZ78, LZW LZ77, LZ78, LZW Applications: GIF and PNG Applications: GIF and PNG

3. Repetition Coding: Assigning binary codewords to (blocks of) source symbols. Coding: Assigning binary codewords to (blocks of) source symbols. Variable-length codes (VLC) and fixed- length codes. Variable-length codes (VLC) and fixed- length codes. Instantaneous codes ½ Uniqely decodable codes ½ Non-singular codes ½ All codes Instantaneous codes ½ Uniqely decodable codes ½ Non-singular codes ½ All codes Tree codes are instantaneous. Tree codes are instantaneous. Tree code, Krafts Inequality. Tree code, Krafts Inequality.

4. Creating a Code: The Data Compression Problem Assume a source with an alphabet A and known symbol probabilities {p i }. Assume a source with an alphabet A and known symbol probabilities {p i }. Goal: Chose the codeword lengths as to minimize the bitrate, i.e., the average number of bits per symbol l i ¢ p i. Goal: Chose the codeword lengths as to minimize the bitrate, i.e., the average number of bits per symbol l i ¢ p i. Trivial solution: l i = 0 8 i. Trivial solution: l i = 0 8 i. Restriction: We want an instantaneous code, so 2 -l i · 1 (KI) must be valid. Restriction: We want an instantaneous code, so 2 -l i · 1 (KI) must be valid. Solution (at least in theory): l i = – log p i Solution (at least in theory): l i = – log p i

5. In practice… Use some nice algorithm to find the code tree Use some nice algorithm to find the code tree –Huffman coding –Tunnstall coding

6. Huffman Coding Two-step algorithm: Two-step algorithm: 1.Iterate: –Merge the least probable symbols. –Sort. 2.Assign bits. a d b c 0.5 0.25 0.125 0.5 0.25 0.5 0.25 0.5 Merge Sort Assign 0 1 0 10 11 0 10 110 111 Get code

7. Coding of the BMS Trick: Code blocks of symbols (extended source). Trick: Code blocks of symbols (extended source). Example: p 1 = ¼, p 2 = ¾. Example: p 1 = ¼, p 2 = ¾. Applying the Huffman algorithm directly: 1 bit/symbol. Applying the Huffman algorithm directly: 1 bit/symbol. Block P (block) Code 009/160) 013/1610 approx 0.85 103/16110bits/symbol 111/16111

8. Huffman Coding: Pros and Cons +Fast implementations. + Error resilient: resynchronizes in ~ l 2 steps. -The code tree grows exponentially when the source is extended. -The symbol probabilities are built-in in the code. Hard to use Huffman coding for extended sources / large alphabets or when the symbol probabilities are varying by time.

9. Arithmetic Coding Shannon-Fano-Elias Shannon-Fano-Elias Basic idea: Split the interval [0,1] according to the symbol probabilities. Basic idea: Split the interval [0,1] according to the symbol probabilities. Example: A = {a,b,c,d}, P = {½, ¼, 1/8, 1/8}. Example: A = {a,b,c,d}, P = {½, ¼, 1/8, 1/8}.

10. b c a 0.6 0.5 0.2 0.8 0.2 Start in b. Code the sequence c c a. cb a 0.9 Code the sequence c c a. ) Code the interval [0.9, 0.96] Bit IntervalDecoder 1c0.5-1 10.75-1 10.875-1 0 -0.9375 1c a0.90624-0.9375 bc 0.510 cba 0.910.960.98

11. An Image Coding Application Consider the image content in a local environment of a pixel as a state in a Markov model. Consider the image content in a local environment of a pixel as a state in a Markov model. Example (binary image): Example (binary image): Such an environment is called a context. Such an environment is called a context. A probability distribution for X can be estimated for each state. Then arithmetic coding is used. A probability distribution for X can be estimated for each state. Then arithmetic coding is used. This is the basic idea behind the JBIG algorithm for binary images and data. This is the basic idea behind the JBIG algorithm for binary images and data. X 0 0 1 1 0

12. Flushing the Coder The coding process is ended (restarted) and the coder flushed The coding process is ended (restarted) and the coder flushed –after a given number of symbols (FIVO) or –When the interval is too small for a fixed number of output bits (VIFO).

13. Universal Coding A universal coder doesnt need to know the statistics in advance. Instead, estimate from data. A universal coder doesnt need to know the statistics in advance. Instead, estimate from data. Forward estimation: Estimate statistics in a first pass and transmit to the decoder. Forward estimation: Estimate statistics in a first pass and transmit to the decoder. Backward estimation: Estimate from already transmitted (received) symbols. Backward estimation: Estimate from already transmitted (received) symbols.

14. Universal Coding: Examples 1.An adaptive arithmetic coder 2.An adaptive dictionary technique –The LZ coders [Sayood 5] 3.An adaptive Huffman coder [Sayood 3.4] Arithmetic coder Statistics estimation

15. Ziv-Lempel Coding (ZL or LZ) Named after J. Ziv and A. Lempel (1977). Named after J. Ziv and A. Lempel (1977). Adaptive dictionary technique. Adaptive dictionary technique. –Store previously coded symbols in a buffer. –Search for the current sequence of symbols to code. –If found, transmit buffer offset and length.

16. LZ77 abcabd acab d eee f c Search bufferLook-ahead buffer Output triplet 12345678 3 8013d0e2f 2 Transmitted to decoder: If the size of the search buffer is N and the size of the alphabet is M we need bits to code a triplet. Variation: Variation: Use a VLC to code the triplets! PKZip, Zip, Lharc, PNG, gzip, ARJ

17. Drawback with LZ77 Repetetive patterns with a period longer than the search buffer size are not found. Repetetive patterns with a period longer than the search buffer size are not found. If the search buffer size is 4, the sequence a b c d e a b c d e a b c d e a b c d e … will be expanded, not compressed. If the search buffer size is 4, the sequence a b c d e a b c d e a b c d e a b c d e … will be expanded, not compressed.

18. LZ78 Store patterns in a dictionary Store patterns in a dictionary Transmit a tuple Transmit a tuple

19. LZ78 Output tuple Dictionary: 1 a 2 b 3 c 4 a b 5 a b c abcaba bc 0a Transmitted to decoder: 0b0c1b4c Decoded:a b c a a b b c Strategy needed for limiting dictionary size!

20. LZW Modification to LZ78 by Terry Welch, 1984. Modification to LZ78 by Terry Welch, 1984. Applications: GIF, v42bis Applications: GIF, v42bis Patented by UniSys Corp. Patented by UniSys Corp. Transmit only the dictionary index. Transmit only the dictionary index. The alphabet is stored in the dictionary in advance. The alphabet is stored in the dictionary in advance.

21. LZW Output: dictionary index Encoder dictionary: 1 a 2 b 3 c 4 d 5 a b abcaba bc 1 Transmitted: 2355 Decoded: abca b 6 bc 7 ca 8 aba 9 abc Decoder dictionary: 1 a 2 b 3 c 4 d 5 a b 6 bc 7 ca 8 aba Input sequence:

22. And now for some applications: GIF & PNG

23. GIF CompuServe Graphics Interchange Format (1987, 89). CompuServe Graphics Interchange Format (1987, 89). Features: Features: –Designed for up/downloading images to/from BBSes via PSTN. –1-, 4-, or 8-bit colour palettes. –Interlace for progressive decoding (four passes, starts with every 8th row). –Transparent colour for non-rectangular images. –Supports multiple images in one file (animated GIFs).

24. GIF: Method Compression by LZW. Compression by LZW. Dictionary size 2 b+1 8-bit symbols Dictionary size 2 b+1 8-bit symbols –b is the number of bits in the palette. Dictionary size doubled if filled (max 4096). Dictionary size doubled if filled (max 4096). Works well on computer generated images. Works well on computer generated images.

25. GIF: Problems Unsuitable for natural images (photos): Unsuitable for natural images (photos): –Maximum 256 colors ( ) bad quality). –Repetetive patterns uncommon ( ) bad compression). LZW patented by UniSys Corp. LZW patented by UniSys Corp. Alternative: PNG Alternative: PNG

26. PNG: Portable Network Graphics Designed to replace GIF. Designed to replace GIF. Some features: Some features: –Indexed or true-colour images ( · 16 bits per plane). –Alpha channel. –Gamma information. –Error detection. No support for multiple images in one file. No support for multiple images in one file. –Use MNG for that. Method: Method: –Compression by LZ77 using a 32KB search buffer. –The LZ77 triplets are Huffman coded. More information: www.w3.org/TR/REC-png.html More information: www.w3.org/TR/REC-png.html

27. Summary Huffman coding Huffman coding –Simple, easy, fast –Complexity grows exponentially with the block length –Statistics built-in in the code Arithmetic coding Arithmetic coding –Complexity grows linearly with the block size –Easily adapted to variable statistics ) used for coding of Markov sources Universal coding Universal coding –Adaptive Huffman or arithmetic coder –LZ77: Buffer with previously sent sequences –LZ77: Buffer with previously sent sequences –LZ78: Dictionary instead of buffer –LZ78: Dictionary instead of buffer –LZW: Modification to LZ78 –LZW: Modification to LZ78

28. Summary, cont Where are the algorithms used? Where are the algorithms used? –Huffman coding: JPEG, MPEG, PNG, … –Arithmetic coding: JPEG, JBIG, MPEG-4, … –LZ77: PNG, PKZip, Zip, gzip, … –LZW: compress, GIF, v42bis, …

29. Finally These methods work best if the source alphabet is small and the distribution skewed. These methods work best if the source alphabet is small and the distribution skewed. –Text –Graphics Analog sources (images, sound) require other methods Analog sources (images, sound) require other methods –complex dependencies –accepted distortion