1. ELG5126 Source Coding and Data Compression
Eric Dubois

2. Context: signal transmission
binary stream
Information
Source
Encoder
Channel
Information
Receiver
Decoder
signal
binary stream

3. Context: signal transmission
binary stream
Information
Source
Encoder
Channel
aka ‘data’
Information
Receiver
Decoder
signal
binary stream

4. Context: signal transmission
binary stream
Information
Source
Encoder
Channel
error
measure
aka ‘data’
Information
Receiver
Decoder
signal
binary stream

5. Examples of information sources
Speech
Image
Video
Text file
Music
Radiograph
Binary executable computer program
Computer graphics primitives
Weather radar map

6. Examples of channels Airwaves (EM radiation) Cable Telephone line
Hard disk
CD, DVD
Flash memory device
Optical path
Internet

7. Examples of information receivers
TV screen and viewer
Audio system and listener
Computer file
Image printer and viewer
Compute engine

8. Possible error measures
No errors permitted (lossless coding)
Numerical measures of error, e.g. mean- squared error (MSE), signal-to-noise ratio (SNR)
Numerical measures of perceptual difference
Mean opinion scores from human users

9. Measures of ‘channel rate’
Data rate (bits per second)
Transmission time (seconds)
File size (bytes)
Average number of bits per source symbol

10. What is compression?
There is usually a ‘natural’ representation for the source data at a given level of fidelity and sampling rate. Examples:
8 bits per character in ASCII data
24 bits per RGB color pixel
16 bits audio signal sample
This natural representation leads to a certain raw channel rate (which is generally too high).
Compression involves reducing the channel rate for a given level of distortion (which may be zero for lossless coding).

11. Compression ratio Example: HDTV, 1080I Raw channel rate: 1493 Mbit/s
(1920*1080*30*24)
Compressed channel rate: ~20 Mbit/s
Compression ratio: ~75

12. Sources Categories of sources continuous time or domain: x(t), x(h,v)
discrete time or domain: x[n], x[m,n]
continuous amplitude or value: xR
discrete amplitude or value: x A = {a1, a2, … aM}
We will only consider discrete domain sources. We assume that continuous domain signals can be sampled with negligible loss. This is not considered in this course.
We will mainly concentrate on one-dimensional signals such as text, speech, audio, etc. Extensions to images are covered in ELG5378.
A source signal is a sequence of values drawn from a source alphabet A: x[1], x[2], … , x[n]  A

13. Source Coder
A source coder transforms a source sequence into a coded sequence whose values are drawn from a code alphabet G : u[1], u[2], …, u[i]  G
Normally G = {0,1}, and we will limit ourselves to this case.
Note that the time indexes for the source sequence x[n] and the coded sequence u[i] do not correspond.
The decoder must estimate the source signal on the basis of the received coded sequence û[i]. This may be different from u[i] if there are transmission errors. We will generally assume that there are no transmission errors.

14. Categories of compression coders
Lossless coding: The source sequence has discrete values, and these must be reproduced without error. Examples where this is required is text, data, executables, and some quantized signals such as X-rays.
Lossy coding: The source sequence may be either continuous or discrete valued. There exists a distortion criterion. The decoded sequence may be mathematically different from the source sequence, but the distortion should be kept sufficiently small. Examples are speech and images. Often a perceptual distortion criterion is desired.
Lossless coding methods are often a component of a lossy coding system.

15. The compression problem
There are two variants of the compression problem
For a given source and distortion measure, minimize the channel rate for a given level of distortion D0 (which can be zero).
For a given source and distortion measure, minimize the distortion (or maximize the quality) for a given channel rate R0.

16. Rate versus distortion performance
In a coding system, there is typically a tradeoff between rate and distortion R D

17. Rate versus distortion performance
In a coding system, there is typically a tradeoff between rate and distortion R D D0

18. Rate versus distortion performance
In a coding system, there is typically a tradeoff between rate and distortion R R0 D

19. When can we compress? When there is statistical redundancy.
For example, for a sequence of outcomes of a fair 16-sided die, we need 4 bits to represent each outcome. No compression is possible.
In English text, some letters occur far more often than others. We can assign shorter codes to the common ones and longer codes to the uncommon ones and achieve compression (e.g., Morse code).

20. Statistical redundancy
There are many types of statistical redundancy.
For example, in English text, we are pretty sure that the next letter after a Q will be a U, so we can exploit it.
The key to successful compression will be to formulate models that capture the statistical redundancy in the source.

21. When can we compress? (2) When there is irrelevancy.
In many cases, the data is specified more precisely than it needs to be for the intended purpose.
The data may be oversampled, or quantized more finely than it needs to be, either everywhere, or in some parts of the signal.
This particularly applies to data meant only for consumption and not further processing.

22. Exploiting irrelevancy
To exploit irrelevancy, we need a good model of the requirements of the receiver, e.g., human vision, hearing, etc.
We also need a suitable representation of the data, e.g., transform or wavelet representations.
Again, the key to success will be the formulation of appropriate models.

23. The elements of a source coder
Change of representation
Quantization (not for lossless coding)
Binary code assignment
All will depend on good models of the source and the receiver.

24. The Course Outline

25. Professor Eric Dubois CBY A-512 Tel: 562-5800 X 6400

26. Textbook
Textbook: K. Sayood, Introduction to Data Compression, third edition, Morgan Kaufmann Publishers, 2006.

27. Prerequisite
Basic probability and signal processing as typically obtained in an undergraduate Electrical Engineering program
(e.g., at uOttawa,
ELG3125 Signal and System Analysis,
ELG3126 Random Signals and Systems

28. Objective
The objective of this course is to present the fundamental principles underlying data and waveform compression.
The course begins with the study of lossless compression of discrete sources. These techniques are applicable to compression of text, data, programs and any other type of information where no loss is tolerable. They also form an integral part of schemes for lossy compression of waveforms such as audio and video signals, which is the topic of the second part of the course.

29. Objective
The main goal of the course is to provide an understanding of the basic techniques and theories underlying popular compression systems and standards such as ZIP, FAX, MP3, JPEG, MPEG and so on, as well as the principles underlying future systems.
Some of the applications will be addressed in student projects.

30. Course Outline
Lossless coding: Discrete sources, binary codes, entropy, Huffman and related codes, Markov models, adaptive coding.
Arithmetic coding: Principles, coding and decoding techniques, implementation issues.
Dictionary techniques: Principles, static dictionary, adaptive dictionary.
Waveform coding: Distortion measures, rate- distortion theory and bounds, models.

31. Course Outline (2)
Quantization: Formulation, performance, uniform and non-uniform quantizers, quantizer optimization, vector quantization.
Predictive coding: Prediction theory, differential coding (DPCM), adaptive coding.
Transform and subband coding: Change of basis, block transforms and filter banks, bit allocation and quantization.
Applications (student projects)

32. Grading
20% Assignments: Several assignments, to be handed in during class on the due-date specified. There will be a 5% penalty for each day late, and no assignment will be accepted after one week.
30% Project: An individual project on an application of data compression involving some experimental work. A project report and presentation at the end of the course will be required. More details will follow early in the course.
20% Midterm exam: Closed-book exam, 80 minutes in length.
30% Final exam: Closed-book exam, 3 hours in length, covering the whole course.

33. Enjoy the course