A Brief Overview of the MPEG2 Standard Dr. David Corrigan

MPEG-2 the Basics  We have already covered a lot of the background of how video compression (or coding) works inside MPEG2.  Intra-Coding (I-frames) v Prediction Coding (P-frames) v Bi-Directional Prediction Coding (B-frames) & the Group of Pictures (GOP)  Most of the subsequent coding architecture is inherited from JPEG  DCT.  Quantisation.  Variable Length Coding (VLC) – ie Huffman + Run Length Coding.  But it is not that simple  motion vectors need to be coded, we may need to include direction prediction information  must deal with interlaced video  plus what about audio & subtitles (either in text or lossless image format)?  and what about streaming  so….

MPEG-2 the Basics  MPEG 2 is about more than video coding  Part 1 – Systems (describes how audio and video are plugged together)  Part 2 – Video  Part 3 – Audio (an extension of the MPEG 1 audio standards)  Part 4 – conformance testing  Part 5 – software simulation  Part 6 – Digital Storage Media Command and Control – (eg. rewind forward etc etc)  Part 7 – Advanced Audio Coding (AAC) – a 2 nd audio standard  there are even more parts

Challenges in MPEG2 (besides compression) 1. Multiplexing  How to combine audio video and text?  They must appear at the same time. 2. Media  Streams can be stored on a hard drive or DVD  Data can be broadcast or streamed on the internet 3. Sequencing  how to send data so that it will be received in the correct order?  how to sychronise the decoder and encoder?

Challenges in MPEG2 (besides compression) 4. Error Resilience  like synchronisation in JPEG.  but temporal propagation of errors is a problem in video. 5. Rate Control/Bandwidth  need to be able to specify a bit rate given the bandwidth available  need to be able to adaptive adjust the quantisation step size 6. Scalability/Multiplatform  adaptive quality based on the decoder hardware  can we have one stream for both low and high quality video?

Scalability in MPEG 2 SNR Scalability Spatial Scalability

Profiles and Levels  MPEG 2 supports a wide variety of scenarios  eg. high quality tv broadcast, low bit rate internet streaming etc  decoders can have varying degrees of complexity + plus a decoder for internet streaming should not have to support decoding of digital tv signals.  MPEG 2 defines Profiles and Level for streams  Profiles define the required decoder complexity (feature set) to decode the stream  Levels define the maximum allowed resolution frame rate and bit rate.

Levels in MPEG 2

Allowed Profile/Level Combinations 4:2:2 profile extends on the main profile but does not support scalabilty

Profile/Level Combinations  Standard Definition TV uses the Main Profile and the Main Level  allows bi-directional prediction but not scalability and stream must use 4:2:0 YC b C r chroma downsampling  Streams have a max resolution of 720x576, max frame rate of 30 fps and max bit rate of 15 mbits/second  also used on DVDs  HDTV uses the Main Profile and the High Level  The Main Profile defines the core set of algorithms in MPEG 2.

MPEG 2 Main Profile (Layers)  MPEG Sequence is organised into a hierarchy of layers, like an onion  The Sequence Layer – the entire video sequence  The GOP Layer – delineating exactly one Group of Pictures (PAL max 15, NTSC max 18 frames)  The Picture Layer – referring to a single I- P- or B-frame.  The Slice Layer – represents a horizontal group of macroblocks that do not span multiple rows.  The Macrobock Layer – represents unit of data for motion estimation (16x16). Conists of blocks for luminance and chrominance.  The Block Layer – contains the DCT coefficients for 1 8x8 block of pixels (can be either a luminance of chrominance block).

The GOP Layer (Frame Ordering)  When using IBBP…. prediction mode we have to reorder frames so that all prediction is “backward” (ie. causal)  so if a B-frame requires a subsequent P-frame for forward prediction the p-frame is placed first in the stream. B-frames from previous GOP come after I frame P-frame 4 is sent before B-frames 2 and 3

The Picture Layer (Interlacing)  The odd and even fields can be coded together as if it were a frame or the can be coded independently  if there is no motion then we can combine the two fields into a single image called a “frame-picture.” Better for compression efficiency.  if there is motion then the two fields are coded separately as if they were two pictures called “field-pictures”. Odd Field-Picture Even Field-Picture Frame Picture

The Slice Layer (Synchronisation)  Slices can be of arbitrary length but can not extend onto a new line.  They are the MPEG-2 solution to the problem of spatial synchronisation (errors can not propagate spatially between slices).  Slice length set depending on the error conditions  ie. shorter when the error rate is high.  Can get temporal propagation of errors too but they can extend longer than 1 GOP because of the prediction strategy.

The Macroblock Layer  Each macroblock contains 4 luminance blocks and 2 chrominance blocks if 4:2:0 (4 chrominance if 4:2:2)  I-frame macroblocks contain no vectors, 1 in P-frames and 2 in B-frames. If interlaced then the number of vectors doubles.  Macroblocks for P- and B-frames can be intra-coded if the prediction error (DFD) is too large.  Motion estimator not specified but the vectors are limited in range and are quantised to 0.5 pixel accuracy.

Coding of Motion Vectors

Coding Δ x and Δ y

 A table of how the choice of Size effects the range of difference that can be coded.  Size is set once at the start of each Picture Layer. (ie. it is the same over the entire picture).  It is common to choose larger size for P-frames cause motion is bigger.

Coding Δ x and Δ y

Huffman Codes for motion_code

Example

The Block Layer (Quantisation) Quantisation step sizes for intra-coded blocks Quantisation step size for prediction-coded blocks similar to the matrix used in JPEG a fixed Q step = 16 for all coefficients

The Block Layer (Quantisation)

The Block Layer (Scan Order) Scan Order for Progressive VideoScan Order for Interlaced Video

The Block Layer (Scan Order)  Idea is to maximise length of runs of zeros in the block.  So progressive frames use the zig-zag scan like JPEG  Interlaced Frames use an alternative scan because there are likely to be non-zero DCT coefficients toward the bottom left corner of the block.

Sequencing, Media and Multiplexing  We could have multiple elementary streams (ie. video, audio, text etc.). They have to combined into a single non-elementary stream and have to be both decoded and displayed in a certain order in the receiver.  The MPEG 2 Part 1 (Systems) standard specifies two different ways of doing this  Program Stream (PS) – used for reliable media such as DVDs  Transport Stream (TS) – used for Digital TV Transmission over noisy channels.  Note there are other ways of doing this that exist outside of the standard  eg. the avi and mov file formats can be used with compressed MPEG2 data.  To do this the notions of time and packets are introduced.  each elementary stream is divided into packets. They can be of fixed or variable length.  these packets are interleaved by the encoder.  each packet carries a timestamp which tells the decoder the correct order.

MPEG 2 Program Stream (PS)  Consists of Packetised Elementary Stream (PES) packets.  PES packets contain 2 timestamps 1. Decoding Time Stamp (DTS) – this tells the decoder when the packet should be decoded. The data is then decoded into the bit stream. 2. Presentation Time Stamp (PTS) – this tells the decoder when the data should be displayed.  The systems part specifies that the decoder must contain a Systems Clock called the STC  when a decoder’s STC is equal to a packet’s DTS the data in the packet is decoded  when the STC is equal to a packet’s PTS the decoded data is sent to the display device (eg. graphics card or sound card)  the state of the encoders clock is placed in the stream at regular intervals. This synchronises the decoder with the encoder.

MPEG-2 Transport Stream (TS)  The transport stream uses a fixed packet length (188 bytes)  this allows easy decoder/encoder synchronisation.  it also allows error correction codes to be inserted.  Transport Streams can contain packets from a number of Programs  These can be different TV channels or maybe an EPG.  Each program has a unique Packet ID placed in the packet header.  Decoder can discard packets of other programs by checking the PID.