Presentation on theme: "TSBK01 Image Coding and Data Compression Lecture 10 Jörgen Ahlberg"— Presentation transcript:
1TSBK01 Image Coding and Data Compression Lecture 10 Jörgen Ahlberg Video CodingTSBK01 Image Coding and Data CompressionLecture 10Jörgen Ahlberg
2Outline Colour coding Moving images: From 2D to 3D? Hybrid coding Video coding standards
3Three base colours enough to synthesize any visible colour! Part I: Colour CodingThe base colours of colour television areRed: 700 nmGreen: 546 nmBlue: 435 nmThree base colours enough to synthesize any visible colour!
4The Colour VectorBIn this plane, the luminance Y = R+G+B = 1RG
5The PAL colours R Y G Matrix R-Y B B-Y Y = 0.30B + 0.59G + 0.11B Cr = 0.70R G BCb = R G BY luminance; Cr, Cb chrominance
6Digital Colour Coding Y Y U V 4:2:0 U V 4:2:2 Change basis to YUV (almost the same as YCrCb).For more info on color spaces, see colour FAQ atThe Human Visual System perceives the luminance in higher resolution than the chrominance!Subsample the colour components.YUV4:2:0YUV4:2:2
7Part II: Coding of Moving Images Principle I - Extend known methods to 3DCoding MethodPrestanda (bpp)ComplexityDecoding complexityPCM6 – 8LowVQ0.5 – 2Very highPredictive2 – 5Transform0.5 – 1.5HighSubband/ Wavelet0.1 – 1.0Fractal
8Extending 2D Methods Predictive coding Transform coding Subband coding 3D predictorsMotion compensated predictorsTransform coding3D transformsSubband coding3D subband filtersBUT! The properties of the image signal are different in the temporal and the spatial domain!
17Motion CompensationTypically one motion vector per macroblock (4 transform blocks)Motion estimation is a time consuming processHierarchical motion estimationMaximum length of motion vectorsClever search strategiesMotion vector accuracy:Integer, half or quarter pixelBilinear interpolation
18Part IV: Video Coding Standards Mobile videophoneVideophone over PSTNISDN videophoneVideo CDDigital TVHDTV816643841.5520kbit/sMbit/sVery low bitrateLow bitrateMedium bitrateHigh bitrateMPEG-4H.263H.261MPEG-1MPEG-2
19StandardsH.26xStandards for real time communication like video telephony and video conferencing.Standardized by ITU.MPEGStandards for stored video data like movies on CDs, DVDs, etc.Standardized by ISO.
20H.261 Standard for ISDN picture phones in 1990. Motion compensation: One motion vector per macroblock.One macroblock = four 8£8 luminance blocks + two chrominance blocks (one U and one V).Motion vectors max 15 pixels long in each direction.Format:CIF (352£288) or QCIF (176£144)7.5 – 30 frames/s.Bitrate: Multiple of 64 kbit/s (=ISDN) including audio.Quality: Acceptable for small motion at 128 kbit/s.
21H.263Standard for picture telephones over analog subscriber lines in 1995.Format:CIF, QCIF or Sub-QCIF.Usually less than 10 frames/s.Bitrate: Typically 20 – 30 kbit/s.Quality: With new options as good as H.261 (at half the bitrate).
22MPEG Moving Pictures Expert Group – a committee under ISO and IEC. Original plan:MPEG-1 for 1.5 Mbit/s (VideoCD)MPEG-2 for 10 Mbit/s (Digital TV)MPEG-3 for 40 Mbit/s (HDTV)What happened:MPEG-1 for 1.5 Mbit/s (Video CD)MPEG-2 for 2 – 60 Mbit/s (TV and HDTV)MPEG-4, -7 and -21 for other things.
23MPEG-1 ISO/IEC standard in 1991. Target bitrate around 1.5 Mbit/s (Video CD).Properties:Bi-directionally predictively coded frames (”B-frames”, see next slide).More flexible than H.261.Almost JPEG for intra frames.Format:CIFNo interlace.24 – 30 frames/s.
25MPEG-coding of I-frames Intracoded8£8 DCTArbitrary weighting matrix for coefficientsPredictive coding of DC-coefficientsUniform quantizationZig-zag, run-level, entropy coding
26MPEG-coding of P-frames Motion compensated prediction from I- or P-frame.Half-pixel accuracy of motion vectors, bilinear interpolation.Predictive coding of motion vectors.Prediction error coded as I-frame.
27MPEG-coding of B-frames Motion compensated prediction from two consecutive I- or P-frames.Forward prediction only (1 vector/macroblock).Backward prediction only (1 vector/macroblock).Average of fwd and bwd (2 vectors/macroblock).Otherwise as P-frames.
28MPEG-2 ISO/IEC standard in 1994. Properties: Format: Bitrate: Handles interlace (optimized for TV)Even more flexible than MPEG-1Format:352£288704£576 (25 frames/s) or 720£480 (30 frames/s)1440£1152 or 1920£1080 (HDTV)Bitrate:2 – 60 Mbit/s~4 Mbits/s: Image quality similar to PAL / NTSC / SECAM.18 – 20 Mbit/s: HDTV.
29MPEG-2 (cont.) Profiles: Experience tells that: Simple profile without B-frames.Scaleable profiles.Experience tells that:At 1.5 – 2 Mbit/s MPEG-2 is not better than MPEG-1.With manual interaction at the coding, good quality can be achieved at 3 – 4 Mbit/s.Problems with implementing the full standard has caused compatibility problems.Buffering and rate control hard problems.
30MPEG-4 ISO/IEC standard in 1998, version 2 in 1999 Instead of frames as coding units, MPEG-4 use audio-visual objectsFocus is not primarily on compression, but on content-based functionalityContains definitions of:Media object types (video, audio, text, graphics, ...)Parameters for describing the objectsBitstream syntax for the (compressed) parametersScene description, file format, streaming, synchronization, ...Allows mixing of media objects.
31Parts of the MPEG-4 standard Part 1, Systems, containsThe bitstream syntax and the the binary ”language” for scene descriptionComputer graphics object descriptionsMultiplexing, transport, ...Part 2, Visual, containsVideo codingStill image codingTexture coding, ...Part 3, Audio, contains a toolbox of audio coders for different applications...
32Structure of an MPEG-4 Decoder A/V objectDecoderA/V objectDecoderBitstreamAudio/Video sceneMUXCompositorA/V objectDecoder
33MPEG-4 (Natural) Video Instead of frames: Video Object Planes Coded with Shape Adaptive DCTAlpha mapA video frameBackground VOPVOPSA DCT
35Synthetic/Natural Hybrid Coding Mix traditional video with 2D/3D graphicsCompose virtual environmentsEasy to add text, graphs, images, etcHigh compressionReceive object from separate sourcesUse predefined or locally defined objectsScaleabilityProgressive decodingBetter terminal gives better quality.
36Synthetic Objects 2D/3D graphics VRML scenes and objects Lines, polygonsStill imagesImage/video mapping on polygon meshesVRML scenes and objectsAnimated peopleMore on animation and virtual characters in Lecture 12!Synthetic audioMore on natural and synthetic audio in Lecture 11!
37All mixed in the decoder!!! Natural video object mapped on 2D meshNatural video objectStill image or natural video object mapped on animated 3D meshComputer graphics generated virtual environmentAll mixed in the decoder!!!
38Virtual Environments Downloaded virtual environment Different environments for different usersSimple change between environmentsSynthetic environments are cheaper than real ones
39Tools for Synthetic Objects Wavelet-based still image compressionScaleable quality and resolutionProgressive decodingCan be mapped on 2D or 3D meshesCompression of 2D and 3D meshesMesh geometry and animationTransmit vertex coordinates and let the receiving terminal calculate the polygonsA moving or still image can be mapped on the mesh (texture mapping).
40More Tools for Synthetic Objects Face and Body AnimationText-to-speech (TTS) interfaceView-dependent scaleable textureInformation about the users view position in a 3D scene is transmitted on a back-channelOnly the necessary texture information is transmitted to the user
41View-dependent Scaleable Texture The texture is mapped on a surfaceOriginal textureWhat the user sees
42Other formats Microsoft, RealVideo, QuickTime, ... All are variations of the hybrid coder used in MPEG-coders, with some extra features.
43ITU and ISO in cooperation: New StuffITU and ISO in cooperation:H.264 = MPEG-4 part 10Finished in 2003.
44H.264 / MPEG-4 part 10 4£4 integer transform (approximating DCT). Prediction of blocks of sizes up to 16£16.Motion vectors for blocks of sizes 4£4 up to 16£16.Up to 5 reference images for prediction.Non-uniform qunatization.Arithmetic coding of run-level pairs.
45More on audio coding in Lecture 11. What about the sound?MPEG-1Audio layer I, II and III (mp3).MPEG-2Four channels, same codec as in MPEG-1.AAC (Advanced Audio Codec) added later.MPEG-4AACTwo speech codersStructured audioAnd more...More on audio coding in Lecture 11.
46Conclusion Color coding Moving image coding Change basis from RGB to YUVColour components are compressed harder than the luminanceMoving image codingHybrid coding: Motion compensated predictive coding and transform coding of the prediction errorI-, P-, and B-framesObject-based coding (MPEG-4) mixing synthetic and natural audio & video
47Conclusion (cont) Standards MPEG-1: Video CD MPEG-2: Digital TV MPEG-4: MultimediaH.261: ISDN videophoneH.263: PSTN videophoneH.264 / MPEG-4 part 10: Universal video