1. Image/Video Compression September 28, 1999
Lawrence A. Rowe
University of California, Berkeley
URL:
L.A. Rowe

2. Outline Background Block Transform Coding Other Coding Algorithms
Software/Hardware CODEC’s
Pragmatic Issues
Multimedia Systems and Applications

3. size of uncompressed video in gigabytes
Video Data Size
size of uncompressed video in gigabytes
image size of video
1280x720 (1.77)
640x480 (1.33)
320x240
160x120
Multimedia Systems and Applications

4. Video Bit Rate Calculation
width * height * depth * fps
= bits/sec
compression factor
width ~ pixels (160, 320, 640, 720, 1280, 1920, …)
height ~ pixels (120, 240, 480, 485, 720, 1080, …)
depth ~ bits (1, 4, 8, 15, 16, 24, …)
fps ~ frames per second (5, 15, 20, 24, 30, …)
compression factor (1, 6, 24, …)
Multimedia Systems and Applications

5. Effects of Compression
storage for 1 hour of compressed video in megabytes
3 bytes/pixel, 30 frames/sec
Multimedia Systems and Applications

6. Be Careful... mpeg 200:1, jpeg 24:1 compressed analog digital
source
digital
representation
compressed
representation vs
Multimedia Systems and Applications

7. Another View Data Rate Size/Hour 128 Kbs 60 MB 384 Kbs 170 MB
1.5 Mbs 680 MB
3.0 Mbs GB
6.0 Mbs GB
25 Mbs 11.0 GB
Multimedia Systems and Applications

8. Perceptual Coding Encode source signal using lossy compression
Lossless algorithms typically reduce signal by 3:1
Must use lossy algorithm to get adequate compression
Hide errors where humans will not see or hear it
Study hearing and vision system to understand how we see/hear
Masking refers to one signal overwhelming/hiding another (e.g., loud siren or bright flash)
Audio perception is kHz but most sounds in low frequencies (e.g., 2 kHz to 4 kHz)
Visual perception influenced by edges and low frequencies
Multimedia Systems and Applications

9. What is… JPEG - Joint Photographic Experts Group
Still image compression, intraframe picture technology
MJPEG is sequence of images coded with JPEG
MPEG - Moving Picture Experts Group
Many standards MPEG1, MPEG2, and MPEG4
Very sophisticated technology involving intra- and interframe picture coding and many other optimizations => high quality and cost in time/computation
H.261/H.263/H Video Conferencing
Low to medium bit rate, quality, and computational cost Used in H.320 and H.323 video conferencing standards
Multimedia Systems and Applications

10. Coding Overview Digitize Intraframe compression Interframe compression
Subsample to reduce data
Intraframe compression
Remove redundancy within frame (spatial compression)
Interframe compression
Remove redundancy between frames (temporal compression)
Symbol coding
Efficient coding of sequence of symbols
Multimedia Systems and Applications

11. Digitizing Modify color space Encode only 1 field Reduce frame rate
24 bit RGB => 15 or 16 bit RGB
24 bit RGB => YUV (8 bit Y, 4 bit U, 4 bit V)
24 bit RGB => 8 bit color map
Encode only 1 field
Reduce frame rate
Film is 24 fps so why encode 30 frames of video
Is 20 fps good enough? 18? 15? 12? 8? 4? ...
Variable frame rate?
Multimedia Systems and Applications

12. Block Transform Encoding
DCT
Zig-zag
Quantize
Huffman Code
Run-length Code
Multimedia Systems and Applications

13. Block Encoding DCT Quantize original image zigzag run-length Huffman
DC component
DCT
Quantize
original image
AC components
zigzag
run-length
code
Huffman
code
coded bitstream < 10 bits (0.55 bits/pixel)
Multimedia Systems and Applications

14. Result of Coding/Decoding
original block
reconstructed block
errors
Multimedia Systems and Applications

15. Examples Uncompressed (262 KB) Compressed (50) (22 KB, 12:1)
Multimedia Systems and Applications

16. Discrete Cosine Transform
4C(u)C(v)
n-1 n-1
(2j+1)up
(2k+1)vp
å å
f(j,k) cos
F[u,v] =
cos n2 2n 2n
j=0 k=0
where 1
for w=0
Ö 2
C(w) =
for w=1,2,…,n-1 1
Inverse is very similar
DCT better at reducing redundancy than Discrete Fourier Transform but computationally expensive
Multimedia Systems and Applications

17. DCT vs DFT · · · · · original signal · · recovered from DCT
recovered from DFT
Multimedia Systems and Applications

18. Inter-frame Compression
Pixel difference with previous frame
If previous pixel very similar, skip it
Send sequence of blocks rather than frames
If previous block similar, skip it or send difference
Motion compensation
Search around block in previous frame for a better matching block and encode position and error difference
Search in future frame
Average block in previous and future frame
Multimedia Systems and Applications

19. Background JPEG still image (1 bpp) MPEG H.26x Symmetric codec
Interleaved audio and video
Low cost decoder at expense of high cost encoder (asymmetric)
H.26x
Video conferencing standard (QCIF, CIF, 4CIF, and 16CIF)
Variable bit rates and coding flexibility
Multimedia Systems and Applications

20. MPEG Standards MPEG1 - vhs quality (1992)
CIF images, 4:2:0 sampling, 1.5 Mbs
Frame encoding
MPEG2 - broadcast quality (1994)
CCIR 601 images, 4:2:2 sampling, 15 Mbs
Interlaced and progressive scanning
Frame and field encoding
Multimedia Systems and Applications

21. MPEG Technology Picture coding types Motion compensation
Bit rate control
Picture order
Multimedia Systems and Applications

22. Do MPEG slides HERE Frame types Jargon Motion vectors
I, P, B, and Bi frames
Jargon
GOP, macroblock, slice
Motion vectors
Predicted and interpolated vectors
Multimedia Systems and Applications

23. Compression Formats Cinepack (2 bpp) Motion JPEG (1-4 bpp)
Inter-frame, RGB15, software playback
Motion JPEG (1-4 bpp)
Intra-frame, DCT+Quantization, good for editing
MPEG (0.5-2 bpp)
Inter-frame, DCT+Quantization+Motion Compensation, excellent for playback
H.261/H.263
Inter-frame, DCT+Quantization+Motion Compensation, block stream, excellent for conferencing
Multimedia Systems and Applications

24. Quantitative Measures of Quality
Mean Squared Error
n-1 n-1
MSE = å å [ f(i,j) - f¢(i,j) ] 2
i=0 i=0
where f(i,j) is original image and f’(i,j) is image
after compression and decompression
Perceptual Measures
Perceptual Distortion Measure (Heeger)
Picture Quality Scale (Miyahara, et.al.)
Spatial/Temporal Measure (Webster, et.al.) 1 n2
Multimedia Systems and Applications

25. Discussion Where are the lossy steps?
Quantization and subsampling before coding
How do you choose quantization matrix?
Standard proscribes matrix based on psychophysics
Vary quantization by scaling Q matrix (i.e. MQUANT)
Custom designed Q matrix
Can improve compressioin by using motion compensation (MC)
Comparison of standards…
JPEG - still image uses fixed Q matrix
H video conferencing uses MC and variable quantization
MPEG - video playback uses MC and variable quantization
Multimedia Systems and Applications

26. Discussion How to do very low bit rate (28.8 Kbs)?
Small image, low quality/bit rate, and better coding
MPEG4 will address this problem
Gazillions of other proposals…
Fine, but who needs them
Scalable proposals will be useful
Must be prepared to deliver several formats
Multimedia Systems and Applications

27. H.261 Coding Example 1 2 5 4 3 6 7 1 2 3 4 5 6 7 1 2 5 4 3 6 1 2 3 4 5 6
Multimedia Systems and Applications

28. Other Techniques Vector Quantization Fractal Coding Wavelet Coding
Use small codebook of values
Slow encode, fast decode
Fractal Coding
Fit curves to signal
Slow encode, fast decode?
Wavelet Coding
Better transform than DCT – incorporates both spatial and frequency in transform
Used in most research work and scalable codecs
Multimedia Systems and Applications

29. Scalable Algorithms
Some applications want to control bandwidth and performance
Send low quality, low bit rate image on high priority channel and quality improvements on low priority channel(s)
May be dictated by network bandwidth (wireless), end station (PDA), or application need (video gallery)
Scalability parameters
Image size
Frame rate
Reconstructed image quality
Multimedia Systems and Applications

30. Pyramid Codes CodedImage-0 Low Quality CodedImage-1
Multimedia Systems and Applications

31. MPEG4 Standards Work Original goal was very low-bit rate coding
Can do remarkably well at rates as low as 10 bits/sec but coding/decoding time is large
Hierarchical VQ is popular because of low cost to decode, but trade-off is less coding efficiency
Recent activity has moved towards programmable decoders
Idea is to allow code to be downloaded to the decoding chip
Difficult trade-off between power consumption, quality and coding algorithm
Multimedia Systems and Applications

32. Understanding CODEC Performance
CODEC Compress Decompress
JPEG 314 inst/pixel 283 inst/pixel
MPEG inst/pixel inst/pixel
Problem is motion vector search in MPEG
Low bit-rate coding (1.5 Mbs) requires excellent coding
Multimedia Systems and Applications

33. MPEG Coding Performance
Decoding is easy
MPEG1 decoding in software on most platforms
Hardware decoders are widely available ($150/board)
Windows graphics accelerators with MPEG decoding now entering market (e.g., Matrox, Diamond, …)
Encoding is expensive
Sequential software encoders are 20:1 real-time
Real-time encoders use parallel processing
Real-time hardware encoders are expensive (e.g., $12K-$50K for MPEG1 and $100K-$500K for MPEG2)
Hardware-assisted off-line MPEG1 encoders (3:1) used for multimedia authoring at reasonable cost ($5k)
Multimedia Systems and Applications

34. Put decoding/encoding performance slides here
Multimedia Systems and Applications

35. Other Boards MJPEG Proprietary Algorithms Programmable Boards
DEC J300 (Turbochannel, CL560) costs $3K
Parallax (Sun, HP, IBM R6K, CL560) costs $4K-$7.5K
SGI Cosmo (CL560) cost (?)
Radius VVS (Mac, CL560) costs $3K (?)
Radius Truevision (PC, (?)) costs (?)
PC boards (ASL, RealMagic MPEG Editor, IIT, …)
Proprietary Algorithms
Intel Smart Video Recorder Pro ($3K, I860)
Programmable Boards
Sun VideoPix (CL4K) supports MJPEG, MPEG, CELLB costs $1.5K
Chromatics MPACT chip board (mid ‘96)
Multimedia Systems and Applications