MPEG Moving Picture Experts Group

What defines good video quality? Size of pictures Bitrate of channel medium (especially in real-time applications) Resolution of the original images and/or frames Frame rate of source AND frame rate of reproduction medium (24 frames per second is standard movie quality) If any one of these factors is inferior, it can bottleneck the overall system and cause reduction in video quality: GIGO  garbage in = garbage out !!!

MPEG Types MPEG 1 : For lower resolution images Bit streams upto 1.5Mbps MPEG 2: Support higher resolution images Bit streams of upto 15Mbps MPEG 4: Support interactivity Composed of tools, algorithms, profiles, MSDL (MPEG Syntax Description Language)‏

Requirements Random Access, Reverse, Fast Forward, Search At any point in the stream (within ½ second) Can reduce quality somewhat during this task, if needed Audio/Video Synchronization Robustness to errors Not catastrophic if some bits are lost Lends itself to Internet streaming Coding/Decoding delay under 150 ms For interactive applications Ability to Edit Modify/Replace frames

Basics of MPEG Many “moving” images or image sequences consist of a static background with one or more moving foreground objects. We can get coding advantage from this. we code the first frame by baseline JPEG and use this frame as reference image. Treat the second image block by block and compare each block with the same block in the reference image. For blocks that have identical block in reference image, we only send a special code instead of whole code. For other blocks, we just encode them as usual.

Basics of MPEG Motion Vectors static background is a very special case, we should consider the displacement of the block. Motion vector is used to inform decoder exactly where in the previous image to get the data. Motion vector would be zero for a static background.

Basics of MPEG Block Matching--how to find the matching block? Matching criteria: In practice we couldn’t expect to find the exactly identical matching block, instead we look for close match. Most motion estimation schemes look for minimum mean square error(MMSE) between block. Matching block size: How large the matching block will affect coding efficiency block size MPEG used: 16×16

Basics of MPEG Search range: It’s reasonable to consider an displacement of 360 pixles/s or about 60pixels/image in standard- definition television. In real-world scenes there is usually more or faster motion horizontally than vertically, generally the width of search area should be twice the height. Suggested search range: ±60 pixles × 30 pixles

MPEG Compression Compression through Spatial Temporal

Spatial Redundancy Take advantage of similarity among most neighboring pixels

Loss of Resolution Original (63 kb) Low (7kb) Very Low (4 kb)

Question When may spatial redundancy reduction be ineffective? What kinds of images/movies?

Answer When may spatial redundancy elimination be ineffective? High-resolution images and displays May appear ‘coarse’ What kinds of images/movies? A varied image or ‘busy’ scene Many colors, few adjacent

Temporal Redundancy Take advantage of similarity between successive frames

Temporal Redundancy Reduction

Motion estimation for different frames XZ Y Available from earlier frame (X) Available from later frame (Z)

Question When may temporal redundancy reduction be ineffective?

Answer When may temporal redundancy reduction be ineffective? Many scene changes High motion

Typical Compression Techniques Within-the-frame coding techniques Between-the-frames coding techniques Block match motion compensation Image stitching Background sprite coding

Basics of MPEG Types of pictures I (intra) frame compressed using only intraframe coding Moderate compression but faster random access P (predicted) frame Coded with motion compression using past I frames or P frames Can be used as reference pictures for additional motion compensation B (bidirectional) frame Coded by motion compensation by either past or future I or P frames D (DC) frame Limited use: encodes only DC components of intraframe coding

I I BP BPB B P I Bi-directional prediction Unidirectional prediction

MPEG Compression Block Diagram Frame being coded DCTQuantize Variable length code Reference frame Inverse DCT De- quantize To transmit buffer

A typical group of pictures in display order A typical group of pictures in coding order I P B B B P B B B P B B B I B B B P B B B P B B B P

MPEG Compression Translate the RGB system into a YUV system Human perception is less sensitive to chrominance than to brightness Translate brightness into chrominance and then the resolution does not have to be as good  lower necessary bitrate Coloring Scheme: Normal JPEG Coloring Blocks Red Blue Green Yellow Cr Cb Cg Translation formulas: Y = Wr*R + Wb*B + Wg*G Cr = Wr* (R - Y) Cb = Wb* (B - Y) Cg =Wg* (G - Y)

Image Sources Intra coding of macroblocks – just as what JPEG does – MPEG has two default quantization tables, one for intra coding, another one for non-intra coding of residuals Macroblock : composed of six blocks (4:2:0 or 4:1:1 format) Four blocks of yellow (luminance) One block of Cb (chrominance) One block of Cr (chrominance)

Coding of Macroblock Non-intra coding of macroblocks The first step is to intra code the macroblock--just in case if we fail to find a reasonable match in motion estimation. Then we use motion estimation to find the nearest match and get the motion vector. Only luminance samples are used in motion estimation. Then each DCT block in macroblock will be treated separately. The residuals will be encode by DCT and quantization (use flat table) as in intra coding. DC along with AC This process is applied to all six blocks in the macroblock Motion vectors are coded predictively

Coding of Macroblock P-frames If the block can be skipped, we just send a “skip” code otherwise, we compare the number of total bits of inter and intra coding, choose the more efficient one. Mark this block accordingly. B-frames comparison among three methods of encoding

Coding of Macroblock Non-intra coding of macroblocks The first step is to intra code the macroblock--just in case if we fail to find a reasonable match in motion estimation. Then we use motion estimation to find the nearest match and get the motion vector. Only luminance samples are used in motion estimation. Then each DCT block in macroblock will be treated separately. The residuals will be encode by DCT and quantization (use flat table) as in intra coding. DC along with AC This process is applied to all six blocks in the macroblock Motion vectors are coded predictively

Coding of Macroblock Frame recorder DCT Quantize Variable- length coder Transmit buffer Prediction encoder De- quantize Inverse DCT Motion predictor Reference frame Rate controller IN OUT Scale factor Buffer fullness Prediction Motion vectors DC

MPEG: Structure of the Coded Bit-Stream Sequence layer : picture dimensions, pixel aspect ratio, picture rate, minimum buffer size, DCT quantization matrices GOP layer : will have one I picture, start with I or B picture, end with I or P picture, has closed GOP flag, timing info, user data Picture layer : temporal ref number, picture type, synchronization info, resolution, range of motion vectors Slices : position of slice in picture, quantization scale factor Macroblock : position, H and V motion vectors, which blocks are coded and transmitted GOP-1GOP-2GOP-n I B B B P B B.. Slice-1 Slice-2 … Slice-N mb-1mb-2mb-n Sequence layer GOP layer Picture layer Slice layer Macroblock layer 8x8 block

MPEG1 Audio Encoding Process 1. Break the signal into frames (a frame is usually several milliseconds in length) 2. Determine spectral energy distribution by taking the DFT, then break this distribubtion into sub-bands via filtering 3. Consider the specified bit rate (this will determine the number of bits per frame that can be used) 4. Compare the DFT signal to the human psychoacoutic model, perform masking 5. Use a modified Huffman code to achieve compression 6. After each frame has been compressed, place a header on each frame 7. Reassemble the new frames with headers back into a continuous bit stream

Image Sources Lots other. Sorry I forgot to save the sources.