Lecture 4 Source Coding and Compression Dr.-Ing. Khaled Shawky Hassan

Slides:



Advertisements
Similar presentations
Lecture 4 (week 2) Source Coding and Compression
Advertisements

Michael Alves, Patrick Dugan, Robert Daniels, Carlos Vicuna
Greedy Algorithms (Huffman Coding)
Data Compressor---Huffman Encoding and Decoding. Huffman Encoding Compression Typically, in files and messages, Each character requires 1 byte or 8 bits.
Lecture 6 Source Coding and Compression Dr.-Ing. Khaled Shawky Hassan
Image Compression, Transform Coding & the Haar Transform 4c8 – Dr. David Corrigan.
Data Compression Michael J. Watts
Lecture04 Data Compression.
Compression & Huffman Codes
School of Computing Science Simon Fraser University
Adaptive Huffman coding The trees are automatically generated by a Generator Designed at SUCO.
Lecture 6: Huffman Code Thinh Nguyen Oregon State University.
Spatial and Temporal Data Mining
A Data Compression Algorithm: Huffman Compression
DL Compression – Beeri/Feitelson1 Compression דחיסה Introduction Information theory Text compression IL compression.
Data Structures – LECTURE 10 Huffman coding
Chapter 9: Huffman Codes
Fundamentals of Multimedia Chapter 7 Lossless Compression Algorithms Ze-Nian Li and Mark S. Drew 건국대학교 인터넷미디어공학부 임 창 훈.
Lossless Data Compression Using run-length and Huffman Compression pages
Huffman Coding. Main properties : –Use variable-length code for encoding a source symbol. –Shorter codes are assigned to the most frequently used symbols,
Data Compression Basics & Huffman Coding
1 Lossless Compression Multimedia Systems (Module 2) r Lesson 1: m Minimum Redundancy Coding based on Information Theory: Shannon-Fano Coding Huffman Coding.
Data Compression Gabriel Laden CS146 – Dr. Sin-Min Lee Spring 2004.
Data Structures and Algorithms Huffman compression: An Application of Binary Trees and Priority Queues.
Huffman Coding Vida Movahedi October Contents A simple example Definitions Huffman Coding Algorithm Image Compression.
1 Lossless Compression Multimedia Systems (Module 2 Lesson 2) Summary:  Adaptive Coding  Adaptive Huffman Coding Sibling Property Update Algorithm 
Dr.-Ing. Khaled Shawky Hassan
CS Spring 2011 CS 414 – Multimedia Systems Design Lecture 7 – Basics of Compression (Part 2) Klara Nahrstedt Spring 2011.
Page 110/6/2015 CSE 40373/60373: Multimedia Systems So far  Audio (scalar values with time), image (2-D data) and video (2-D with time)  Higher fidelity.
Huffman Encoding Veronica Morales.
1 Analysis of Algorithms Chapter - 08 Data Compression.
Fundamental Structures of Computer Science Feb. 24, 2005 Ananda Guna Lempel-Ziv Compression.
Multimedia Data Introduction to Lossless Data Compression Dr Sandra I. Woolley Electronic, Electrical.
Compression.  Compression ratio: how much is the size reduced?  Symmetric/asymmetric: time difference to compress, decompress?  Lossless; lossy: any.
1 Source Coding and Compression Dr.-Ing. Khaled Shawky Hassan Room: C3-222, ext: 1204, Lecture 5.
Lossless Compression CIS 465 Multimedia. Compression Compression: the process of coding that will effectively reduce the total number of bits needed to.
1 Source Coding and Compression Dr.-Ing. Khaled Shawky Hassan Room: C3-217, ext: 1204, Lecture 4 (Week 2)
COMPRESSION. Compression in General: Why Compress? So Many Bits, So Little Time (Space) CD audio rate: 2 * 2 * 8 * = 1,411,200 bps CD audio storage:
Huffman coding Content 1 Encoding and decoding messages Fixed-length coding Variable-length coding 2 Huffman coding.
Huffman Code and Data Decomposition Pranav Shah CS157B.
Adaptive Huffman Coding. Gabriele Monfardini - Corso di Basi di Dati Multimediali a.a Why Adaptive Huffman Coding? Huffman coding suffers.
Abdullah Aldahami ( ) April 6,  Huffman Coding is a simple algorithm that generates a set of variable sized codes with the minimum average.
CS Spring 2011 CS 414 – Multimedia Systems Design Lecture 6 – Basics of Compression (Part 1) Klara Nahrstedt Spring 2011.
Lecture 4: Lossless Compression(1) Hongli Luo Fall 2011.
CS654: Digital Image Analysis Lecture 34: Different Coding Techniques.
Bahareh Sarrafzadeh 6111 Fall 2009
Lecture 7 Source Coding and Compression Dr.-Ing. Khaled Shawky Hassan
1 Source Coding and Compression Dr.-Ing. Khaled Shawky Hassan Room: C3-222, ext: 1204, Lecture 7 (W5)
Lossless Decomposition and Huffman Codes Sophia Soohoo CS 157B.
Multi-media Data compression
1 Data Compression Hae-sun Jung CS146 Dr. Sin-Min Lee Spring 2004.
Chapter 7 Lossless Compression Algorithms 7.1 Introduction 7.2 Basics of Information Theory 7.3 Run-Length Coding 7.4 Variable-Length Coding (VLC) 7.5.
Lecture 12 Huffman Algorithm. In computer science and information theory, a Huffman code is a particular type of optimal prefix code that is commonly.
بسم الله الرحمن الرحيم My Project Huffman Code. Introduction Introduction Encoding And Decoding Encoding And Decoding Applications Applications Advantages.
CS Spring 2012 CS 414 – Multimedia Systems Design Lecture 7 – Basics of Compression (Part 2) Klara Nahrstedt Spring 2012.
ECE 101 An Introduction to Information Technology Information Coding.
Huffman code and Lossless Decomposition Prof. Sin-Min Lee Department of Computer Science.
Chapter 11. Chapter Summary  Introduction to trees (11.1)  Application of trees (11.2)  Tree traversal (11.3)  Spanning trees (11.4)
BAB 3 HUFFMAN CODE Universitas Telkom INFORMATION THEORY.
Lossless Compression-Statistical Model Lossless Compression One important to note about entropy is that, unlike the thermodynamic measure of entropy,
Data Coding Run Length Coding
Huffman Coding, Arithmetic Coding, and JBIG2
Chapter 8 – Binary Search Tree
Chapter 9: Huffman Codes
Chapter 11 Data Compression
Huffman Encoding Huffman code is method for the compression for standard text documents. It makes use of a binary tree to develop codes of varying lengths.
CSE 589 Applied Algorithms Spring 1999
Algorithms CSCI 235, Spring 2019 Lecture 31 Huffman Codes
Presentation transcript:

Lecture 4 Source Coding and Compression Dr.-Ing. Khaled Shawky Hassan Room: C3-222, ext: 1204, Email: khaled.shawky@guc.edu.eg 1 1

Static vs. Adaptive Coding Static (Two-Pass Model) Encoder Initialize the data model based on a first pass over the data (i.e., perform the probabilities analysis) Transmit the data model (encoder). Send data and while there is more data to send: -- Encode the next symbol using the existing data model and send it. Decoder Receive the data model (decoder). Receive the data and while there is more data to receive -- Decode the next symbol using the data model and output it. Summary about the Two-Pass procedure: 1. Collect statistics, generate codewords (1st pass round) 2. Perform actual encoding/compression (2nd pass round) 3. Not practical in many situations (e.g., compressing network transmissions) 2

Static vs. Adaptive Coding Adaptive (One-Pass Model) Encoder Initialize the data model as fixed probability and fixed code length. Send data first and while there is more data to send Encode the next symbol using the data model (if we have) and send it. Modify the existing data model based on the last symbol. Decoder Initialize the data model as per agreement. While there is more data to receive Decode the next symbol using the data model and output it. Modify the data model based on the decoded symbol. What Do We Find ? 3 No Encoder map to send!

Huffman Coding (e.g.: Lossless JPEG) Properties: Huffman codes are built from the bottom up, starting with the leaves of the tree and working progressively closer to the root Huffman coding will always at least work more efficient than Shannon-Fano coding, so it has become the predominate entropy coding method It was shown that Huffman coding cannot be improved or with any other integral bit-width coding stream Sibling Property: Defined by Gallager [Gallager 1978]: “A binary code tree has the sibling property if each node (except the root) has a sibling and if the nodes can be listed in order of nonincreasing (decreasing) weight with each node adjacent to its sibling.” Thus: 1- If A is the parent node of B (left) and C (right) is a child of B, then W(A) > W(B) > W(C) Thus if A is the parent node of B (left) and C (right), then W(B) < W(C) 4

W(#1) ≤ W(#3) ≤ W(#3) ≤ … ≤ W(#7) ≤ W(#8) ≤ W(#9) Huffman Coding Properties A binary tree is a Huffman tree if and only if it obeys the sibling property, i.e., W(#1) ≤ W(#3) ≤ W(#3) ≤ … ≤ W(#7) ≤ W(#8) ≤ W(#9) Non-Decreasing Order 5

Adaptive Huffman Coding Algorithm: Given: alphabet S = {s1, …, sn} (NO Probabilities !!!) Pick a fixed default binary codes for all symbols (block/quadratic code) Start with an empty “Huffman” tree (I said and I mean it – Empty ) Read symbol s from source If NYT(s) %% (//) Not Yet Transmitted Send NYT, default(s) (except for the first symbol) Update the tree (and keep it Huffman) Else Send codeword for s Update tree Repeat until done with all symbols in the source 6

Example (Adaptive Huffman) Assume we are encoding the message [a a r d v a r k] The total number of nodes in this tree will be (at most) 2*n – 1 + 2 = 2*26 -1 +2 = 53 where n is the number of usable alphabets and +2 is only for the “NYT” and its “node” The first letter to be transmitted is “a” As a does not yet exist in the tree, we send a binary code 00000 for a and then add a to the tree The NYT node gives birth to a new NYT node and a terminal node corresponding to “a” In this example, we will consider only 51 nodes and leaves (instead of 53!!). However, the correct is 53. The weight of the terminal node will be higher than the NYT node, so we assign the number 49 to the NYT node and 50 to the terminal node “a” The next symbol is a, and the transmitted code is 1 now (as a = 1 only now!) Lest see an example … (we first starts with a fixed code!) 7

Example: Adaptive Huffman Coding Input: aardvark To keep the rest of the slides as is, we started as the book with 51; however, the correct thing is to start with 53! Output: Symbol Code NYT a r d v k 8

Example: Adaptive Huffman Coding Input: aardvark Output: 00000 Symbol Code NYT a r d v k 9

Example: Adaptive Huffman Coding Input: aardvark 1 Output: 00000 Symbol Code NYT a 1 r d v k 1 10

Example: Adaptive Huffman Coding Input: aardvark Output: 000001 Symbol Code NYT a 1 r d v k 11

Example: Adaptive Huffman Coding Input: aardvark Output: 000001010001 Symbol Code NYT 00 a 1 r 01 d v k 12

Example: Adaptive Huffman Coding Input: aardvark Output:0000010100010000011 Symbol Code NYT 000 a 1 r 01 d 001 v k 13

Example: Adaptive Huffman Coding Input: aardvark Output:0000010100010000011000 Symbol Code NYT 0000 a 1 r 01 d 001 v 0001?? k 14

Example: Adaptive Huffman Coding Input: aardvark Output:0000010100010000011000 Symbol Code NYT 0000 a 1 r 01 d 001 v 0001?? k 15

Example: Adaptive Huffman Coding Input: aardvark Output:0000010100010000011000 Symbol Code NYT 0000 a 1 r 01 d 001 v 0101 ?? k 16

Example: Adaptive Huffman Coding Input: aardvark Output:0000010100010000011000 Symbol Code NYT 000 a 1 r 01 d 001 v ?? k 17

Example: Adaptive Huffman Coding Input: aardvark Output:000001010001000001100010101 Symbol Code NYT 1100 a r 10 d 111 v 1101 k 18

Example: Adaptive Huffman Coding Input: aardvark Output:0000010100010000011000101010 Symbol Code NYT 1100 a r 10 d 111 v 1101 k 19

Example: Adaptive Huffman Coding Input: aardvark Output:000001010001000001100010101010 Symbol Code NYT 1100 a r 10 d 111 v 1101 k 20

Example: Adaptive Huffman Coding Input: aardvark Output:000001010001000001100010101010110001010 Symbol Code NYT 11000? a r 10 d 111 v 1101 k 11001?? 21

Example: Adaptive Huffman Coding Input: aardvark Output:000001010001000001100010101010110001010 Symbol Code NYT 11000? a r 10 d 111 v 1101 k 11001 ?? 22

Example: Adaptive Huffman Coding Input: aardvark Output:000001010001000001100010101010110001010 Symbol Code NYT 11100 a r 10 d 110 v 1111 k 11101 23

Adaptive Huffman Decoding Input: a Output:000001010001000001100010101010110001010 Symbol Code NYT a r d v k 1 1 24

Adaptive Huffman Decoding Input: aa Output:000001010001000001100010101010110001010 Symbol Code NYT a 1 r d v k 25

Adaptive Huffman Decoding Input: aar Output:000001010001000001100010101010110001010 Symbol Code NYT 00 a 1 r 01 d v k 26

Adaptive Huffman Decoding Input: aard Output:000001010001000001100010101010110001010 Symbol Code NYT 000 a 1 r 01 d 001 v k 27

Adaptive Huffman Decoding Input: aardv Output:000001010001000001100010101010110001010 Symbol Code NYT 0000 ? a 1 r 01 d 001 v 0001?? k 28

Adaptive Huffman Decoding Input: aardv Output:000001010001000001100010101010110001010 Symbol Code NYT 0000 a 1 r 01 d 001 v 0001?? k 29

Adaptive Huffman Decoding Input: aardv Output:000001010001000001100010101010110001010 Symbol Code NYT 0000 a 1 r 01 d 001 v 0101 ?? k 30

Adaptive Huffman Decoding Input: aardv Output:000001010001000001100010101010110001010 Symbol Code NYT 000 a 1 r 01 d 001 v ?? k 31

Adaptive Huffman Decoding Input: aardv Output:000001010001000001100010101010110001010 Symbol Code NYT 1100 a r 10 d 111 v 1101 k 32

Adaptive Huffman Decoding Input: aardva Output:000001010001000001100010101010110001010 Symbol Code NYT 1100 a r 10 d 111 v 1101 k 33

Adaptive Huffman Decoding Input: aardvar Output:000001010001000001100010101010110001010 Symbol Code NYT 1100 a r 10 d 111 v 1101 k 34

Adaptive Huffman Decoding Input: aardvar Output:000001010001000001100010101010110001010 Symbol Code NYT 1100 a r 10 d 111 v 1101 k 35

Adaptive Huffman Decoding Input: aardvark Output:000001010001000001100010101010110001010 Symbol Code NYT 11000 a r 10 d 111 v 1101 k ?? 36

Adaptive Huffman Decoding Input: aardvark Output:000001010001000001100010101010110001010 ? Symbol Code NYT 11100 a r 10 d 110 v 1111 k 11101 37

Adaptive Huffman Exercise Try to solve the following! Find the adaptive Huffman encoder (compressor) for the following text: raaaabcbaacvkl Assuming 26 alphabet set! 38

Adaptive Huffman Notes To Follow the Text Book example: If the source has an alphabet {a1,a2, …, am} of size m , then pick e and r such that m = 2e+r and 0 ≤ r <2e . The letter ak is encoded as the ﴾e+1﴿-bit corresponds to k−1, iff 1≤ k ≤2r; else, ak is encoded as (only) the e-bit binary representation of k−r−1. Example: suppose m = 26, then e = 4, and r=10. Then symbol a1 is encoded as 00000, (“a” in English) the symbol a2 is encoded as 00001, (“b” in English) and the symbol a22 is encoded as 1011 (“b” in English) 39

Adaptive Huffman Applications Lossless Image Compression Steps to have lossless image compression: Generate a Huffman code for each uncompressed image (but already quantized and compressed with lossy methods) Encode the image using the Huffman code Save it in a file again !!! The original (uncompressed) image representation uses 8 bits/pixel. The image consists of 256 rows of 256 pixels, so the uncompressed representation uses 65,536 bytes. Compression ratio → number of bytes (uncompressed): number of bytes compressed 40 40

Adaptive Huffman Applications Lossless Image Compression 41

Adaptive Huffman Applications Lossless Image Compression Image Name Bits/Pixel Total Size (B) Compression Ratio Sena 7.01 57,504 1.14 Sensin 7.49 61,430 1.07 Earth 4.94 40,534 1.62 Omaha 7.12 58,374 1.12 Huffman (Lossless JPEG) Compression Based on Pixel value 42

Adaptive Huffman Applications Lossless Image Compression Image Name Bits/Pixel Total Size (B) Compression Ratio Sena 4.02 32,968 1.99 Sensin 4.70 38,541 1.70 Earth 4.13 33,880 1.93 Omaha 6.42 52,643 1.24 Huffman Compression Based on Pixel Difference value and Two-Pass Model 43

Adaptive Huffman Applications Lossless Image Compression Image Name Bits/Pixel Total Size (B) Compression Ratio Sena 3.93 32,261 2.03 Sensin 4.63 37,896 1.73 Earth 4.82 39,504 1.66 Omaha 6.39 52,321 1.25 Huffman Compression Based on Pixel Difference Value and One-Pass Adaptive Model 44

Adaptive Huffman Applications Lossless Image Compression Image Name Bits/Pixel Total Size (B) Compression Ratio Sena 3.93 32,261 2.03 Sensin 4.63 37,896 1.73 Earth 4.82 39,504 1.66 Omaha 6.39 52,321 1.25 Huffman Compression Based on Pixel Difference Value and One-Pass Adaptive Model 45

Optimality of Huffman Codes! The necessary conditions for an optimal variable-length binary code: Condition 1: Given any two letters aj and ak, if P(aj) ≥ P(ak), then lj ≤ lk, where lj is the number of bits in the codeword for aj. Condition 2: The two least probable letters have codewords with the same maximum length lm. Condition 3: In the tree corresponding to the optimum code, there must be two branches stemming from each intermediate node. Condition 4: Suppose we change an intermediate node into a leaf node by combining all the leaves descending from it into a composite word of a reduced alphabet. Then, if the original tree was optimal for the original alphabet, the reduced tree is optimal for the reduced alphabet. 46

Minimum Variance Huffman Codes By performing the sorting procedure in a slightly different manner, we could have found a different Huffman code. 47

Huffman Coding: Self Study! 3.2.1 Minimum Variance Huffman Codes (pp. 46 – 47 {redo the examples}) 3.2.3 Length of Huffman Codes (pp. 49 ~ 51 and the example 3.2.2) 3.2.3 Huffman Codes optimality condition!! 48