1. School of Computer Science & Information Technology G6DPMM - Lecture 12 Digital Video

2. Sourcing Digital Video Film as Digital Film as Digital Becoming more common as costs fall Becoming more common as costs fall May be compressed in the camera or uncompressed May be compressed in the camera or uncompressed Stored on magnetic tape Stored on magnetic tape D1, D2, D3, D5, D9 (Digital-S) D1, D2, D3, D5, D9 (Digital-S) Digital Betacam, BetacamSX, BetacamIMX Digital Betacam, BetacamSX, BetacamIMX DVCAM DVCPRO DVCAM DVCPRO DV / MiniDV (consumer) DV / MiniDV (consumer) Digital-8 (uses Hi-8 tapes) Digital-8 (uses Hi-8 tapes) Compressed vs Uncompressed Compressed vs Uncompressed Film as Analogue Film as Analogue Digitise with capture board (frame grabber) Digitise with capture board (frame grabber) Wildly variable in price and quality Wildly variable in price and quality

3. Digital Video Playback Playback Software / Multimedia Framework Playback Software / Multimedia Framework Windows Media Player (Microsoft) Windows Media Player (Microsoft) Bundled with Windows (3.1 - XP) Bundled with Windows (3.1 - XP) Free versions for Mac OSX and Solaris Free versions for Mac OSX and Solaris Media framework – includes video infrastructure Media framework – includes video infrastructure File formats (e.g. AVI / WMV) File formats (e.g. AVI / WMV) Quicktime Quicktime Bundled with Macintosh (OS 7 – X) Bundled with Macintosh (OS 7 – X) Free version for Windows Free version for Windows Media framework – includes video infrastructure Media framework – includes video infrastructure MOV file format MOV file format RealNetworks RealNetworks RealPlayer (Windows / Macintosh / Linux / Solaris RealPlayer (Windows / Macintosh / Linux / Solaris Originally a streaming technology, now a full blown media framework Originally a streaming technology, now a full blown media framework

4. Digital Video Compression Video consists of a time-series of frames Video consists of a time-series of frames Frames are still images Frames are still images NB still image compression techniques NB still image compression techniques Sequences of frames may be compressed by: Sequences of frames may be compressed by: Motion estimation Motion estimation Only updating the parts of the frame that have moved Only updating the parts of the frame that have moved E.g. consider a talking head! E.g. consider a talking head! Motion compensation Motion compensation Finding static parts of the frame and moving them around Finding static parts of the frame and moving them around E.g. consider a pan or zoom E.g. consider a pan or zoom

5. Compression Codecs Playback software (media framework) does not usually contain compression/decompression software Playback software (media framework) does not usually contain compression/decompression software Compression software implementing a specific algorithm is called a codec Compression software implementing a specific algorithm is called a codec Codecs are required by the framework – many different codecs can be installed into a given framework Codecs are required by the framework – many different codecs can be installed into a given framework NB compatibility and distribution (legality) issues NB compatibility and distribution (legality) issues

6. Common Codecs Indeo Indeo Cinepak Cinepak P*64 P*64 Sorenson Sorenson ClearVideo ClearVideo VDOwave VDOwave RealVideo RealVideo MPEG MPEG NB this is not a single codec NB this is not a single codec MPEG also has many spin-offs – such as Microsoft WMV or DivX MPEG also has many spin-offs – such as Microsoft WMV or DivX

7. Video Compression Uncompressed video is expensive Uncompressed video is expensive Hard to store – very large volumes of data Hard to store – very large volumes of data Expensive equipment required to manipulate Expensive equipment required to manipulate Compression is always lossy Compression is always lossy Multi-way trade-off Multi-way trade-off File size / transfer rate (bandwidth) File size / transfer rate (bandwidth) Frame rate (typically 15fps for multimedia) Frame rate (typically 15fps for multimedia) Image quality Image quality Frame size Frame size Different codecs have differing strengths and weaknesses (e.g. Indeo vs Cinepak) Different codecs have differing strengths and weaknesses (e.g. Indeo vs Cinepak)

8. Indeo Algorithms developed for DVI – Intel i750 chipset Algorithms developed for DVI – Intel i750 chipset Indeo – implemented in software Indeo – implemented in software Proprietary intel algorithms Proprietary intel algorithms Two levels of compression Two levels of compression PLV (Production Level Video) – broadcast quality PLV (Production Level Video) – broadcast quality RTV (Real Time Video) RTV (Real Time Video) Typical compression ratio of 160:1 Typical compression ratio of 160:1 Beware versions (5.1 safe on modern machines, 3.2 fairly ubiquitous) Beware versions (5.1 safe on modern machines, 3.2 fairly ubiquitous)

9. MPEG Moving Picture Expert Group Moving Picture Expert Group Established by ISO in 1988 Established by ISO in 1988 Remit – define standards for movie with soundtracks Remit – define standards for movie with soundtracks Digital television Digital television Video phones Video phones Consumer electronics Consumer electronics Digital cinema Digital cinema Video games Video games Family of standards Family of standards Always lossy Always lossy Discrete Cosine Transform (DCT) algorithm Discrete Cosine Transform (DCT) algorithm

10. MPEG-1 (1993) The original MPEG standard The original MPEG standard Objective: deliver video and audio from CD- ROM with comparable quality to VHS tape and CD-Audio Objective: deliver video and audio from CD- ROM with comparable quality to VHS tape and CD-Audio Encodes an audiovisual signal in about 1.5 Mbps Encodes an audiovisual signal in about 1.5 Mbps Images are progressively scanned Images are progressively scanned This fundamentally limits quality This fundamentally limits quality Motion compensation Motion compensation I-Frames (reference) I-Frames (reference) P-Frames (predictive) P-Frames (predictive) B-Frames (bidirectional predictive) B-Frames (bidirectional predictive)

11. MPEG Frame Types I-Frames I-Frames Reference frame Reference frame Complete image – no missing information Complete image – no missing information Subsequent P-frames are based upon an I-frame Subsequent P-frames are based upon an I-frame P-Frames P-Frames Only contain the difference between themselves and the previous P-frame or I-frame Only contain the difference between themselves and the previous P-frame or I-frame Anything that does not change is not stored Anything that does not change is not stored B-Frames B-Frames Contain the difference between themselves and both frames on either side Contain the difference between themselves and both frames on either side B-frames are not always implemented because of memory requirements B-frames are not always implemented because of memory requirements

12. The MPEG GOP GOP Structure GOP Structure Group of Pictures Group of Pictures Specifies I / P frame structure (e.g. 1 I frame per 11 P frames. IPPPPPPPPPPPIPPPPPPPPPPPIPPPPPPPPPPPI Specifies I / P frame structure (e.g. 1 I frame per 11 P frames. IPPPPPPPPPPPIPPPPPPPPPPPIPPPPPPPPPPPI The fewer I frames the better the compression… The fewer I frames the better the compression… but frame to frame calculations are less accurate – leading to artefacts but frame to frame calculations are less accurate – leading to artefacts PEL (picture elements) PEL (picture elements) Block of pixels, used as a basis for compression Block of pixels, used as a basis for compression Motion of PEL is predicted based on I, P or B frames Motion of PEL is predicted based on I, P or B frames

13. The MPEG-1 standard Five parts (separate ISO standards) Five parts (separate ISO standards) 1. Systems 1. Systems 2. Video 2. Video 3. Audio (NB MPEG 1, level 3 – MP3) 3. Audio (NB MPEG 1, level 3 – MP3) 4. Conformance testing 4. Conformance testing 5. C implementation (encoding and decoding) 5. C implementation (encoding and decoding)

14. MPEG Audio MP3 is MPEG 1, audio layer 3 MP3 is MPEG 1, audio layer 3 Layer 1: 384, 4:1 compression Layer 1: 384, 4:1 compression Layer 2: 192 – 256kbit/s, 6:1..8:1 compression Layer 2: 192 – 256kbit/s, 6:1..8:1 compression Layer 3: 112 – 128 kbit/s, 10:1..12:1 compression Layer 3: 112 – 128 kbit/s, 10:1..12:1 compression Compression Compression Modified discrete cosine transfom (MDCT) Modified discrete cosine transfom (MDCT) 32 band polyphase quadrature filter (PQF) 32 band polyphase quadrature filter (PQF) Alias reduction postprocessing Alias reduction postprocessing Severe design limitations of MP3 Severe design limitations of MP3 Bitrate, time resolution, lack of scaleability – especially for high frequencies Bitrate, time resolution, lack of scaleability – especially for high frequencies

15. Applications of MPEG-1 MPEG-1 compression is cheap and rapid MPEG-1 compression is cheap and rapid Consumer Devices Consumer Devices Multimedia / CD-ROM Multimedia / CD-ROM Internet Video Internet Video Video CD (production is extremely cheap) Video CD (production is extremely cheap) Internet audio (MP2) Internet audio (MP2) Audio only devices (MP3 players) Audio only devices (MP3 players)

16. MPEG-2 (1994) A completely different standard from MPEG 1. A completely different standard from MPEG 1. Objective: Objective: Deliver video and audio at comparable quality to broadcast video at roughly 4 times the typical MPEG- 1 bit rate Deliver video and audio at comparable quality to broadcast video at roughly 4 times the typical MPEG- 1 bit rate Requires between 3 and 15 Mbps Requires between 3 and 15 Mbps Improvements over MPEG-1 Improvements over MPEG-1 Higher resolution Higher resolution Higher image quality Higher image quality Supports interlaced video formats Supports interlaced video formats Multiresolution scalability Multiresolution scalability Multiple sound track features Multiple sound track features Scaleable Scaleable

17. The MPEG-2 Standard ISO 13818 – 10 parts ISO 13818 – 10 parts 640 patents, held by 24 corporations and one university 640 patents, held by 24 corporations and one university Video Encoding (grossly simplified) Video Encoding (grossly simplified) I, P and B frames – similar to MPEG-1 I, P and B frames – similar to MPEG-1 Video is separated into: Video is separated into: 1 luminance and 2 chrominance channels, 1 luminance and 2 chrominance channels, 4 luminance macroblocks 4 luminance macroblocks variable number of chrominance macroblocks variable number of chrominance macroblocks Macroblocks of P and B frames are compared, and a motion vector calculated Macroblocks of P and B frames are compared, and a motion vector calculated Each block is treated with DCT, the result is run-length encoded and a huffman compression is applied Each block is treated with DCT, the result is run-length encoded and a huffman compression is applied

18. Scaleable MPEG MPEG-2 is scaleable (layered coding) MPEG-2 is scaleable (layered coding) Single coded bitstream is scaleable to various bitrates Single coded bitstream is scaleable to various bitrates Layers can be incrementally decoded Layers can be incrementally decoded Basic layer – low quality video Basic layer – low quality video Subsequent layers – higher quality video Subsequent layers – higher quality video If the bandwidth is low, only the bitstream from the base layer is sent If the bandwidth is low, only the bitstream from the base layer is sent If the bandwidth changes, more or fewer layers may be used If the bandwidth changes, more or fewer layers may be used The same principle can be used on noisy connections The same principle can be used on noisy connections

19. Applications of MPEG-2 DVD Video DVD Video HDTV HDTV Ditgital satelite / cable TV Ditgital satelite / cable TV Digital video recorders Digital video recorders Video conferencing Video conferencing Some games Some games Some multimedia (usually where very high quality is essential) Some multimedia (usually where very high quality is essential) NB encoding is processor intensive – thus more expensive than MPEG-1 NB encoding is processor intensive – thus more expensive than MPEG-1

20. Editing MPEG 1 & 2 High performance compression formats High performance compression formats Not designed for editing Not designed for editing Only I-frames are stored in their entirety Only I-frames are stored in their entirety This makes editing, rewinding or even bookmarking extremely difficult This makes editing, rewinding or even bookmarking extremely difficult MPEG editors tend to be crude MPEG editors tend to be crude Serious editing involves decmopressing and recompressing Serious editing involves decmopressing and recompressing Quality loss Quality loss Moral – don’t edit MPEG if at all possible Moral – don’t edit MPEG if at all possible Therefore don’t store digital video as MPEG! Therefore don’t store digital video as MPEG!

21. MPEG-4 (1998) A different approach to video A different approach to video Designed primarily for streaming video (e.g. on WWW), 3G phones, and other multimedia. Designed primarily for streaming video (e.g. on WWW), 3G phones, and other multimedia. MPEG-4 absorbs most of MPEG-1 and MPEG-2 MPEG-4 absorbs most of MPEG-1 and MPEG-2 Adds: Adds: VRML (3D) support VRML (3D) support Object-based coding Object-based coding User interactivity User interactivity Hypermedia capability Hypermedia capability Streams from different sources can be combined Streams from different sources can be combined Supports uploading and downloading Supports uploading and downloading Rights management protocol Rights management protocol

22. Implementation of MPEG-4 Designed to be modular Designed to be modular Standard in 20 parts (levels) Standard in 20 parts (levels) Each level contains profiles Each level contains profiles Developers implement components of MPEG-4 Developers implement components of MPEG-4 Probably no full implementation Probably no full implementation Various partial implementations, such as: Various partial implementations, such as: DivX DivX Microsoft ASF Microsoft ASF Quicktime 6 Quicktime 6

23. Object-based coding Media objects Media objects Audio and visual objects Audio and visual objects Natural objects (captured by camera) Natural objects (captured by camera) Synthetic objects (created by computer) Synthetic objects (created by computer) Visual objects can be any shape (not just rectangles) Visual objects can be any shape (not just rectangles) Video manipulation (insertion, deletion, rotation etc.) Video manipulation (insertion, deletion, rotation etc.) Video indexing and retrieval Video indexing and retrieval Texture maps Texture maps Sprites Sprites 3-d models and animation 3-d models and animation Probably the future of multimedia! Probably the future of multimedia!

24. MPEG-7 Multimedia content description interface Multimedia content description interface Aimed at addressing issues of content retrieval Aimed at addressing issues of content retrieval Intended primarily for digital libraries Intended primarily for digital libraries Object based (as MPEG-4) Object based (as MPEG-4) Supported data: Supported data: Still images Still images 3D models 3D models Sampled audio Sampled audio Speech Speech Video Video Composition information (combinations of the above) Composition information (combinations of the above) Data representation Data representation Binary objects, XML or both Binary objects, XML or both

25. MPEG-21 Designed to manage digital media resources Designed to manage digital media resources Digital item declaration Digital item declaration Digital item identification and description Digital item identification and description Content management and usage Content management and usage Intellectual property management and protection Intellectual property management and protection Terminals and networks Terminals and networks Content representation Content representation Event reporting Event reporting