MPEG-4. MPEG-4, or ISO/IEC 14496 is an international standard describing coding of audio-video objects the 1 st version of MPEG-4 became an international.

Slides:



Advertisements
Similar presentations
Implementing Streaming Media Standards with RealSystem Rob Lanphier Program Manager - Open Standards RealNetworks.
Advertisements

OSI Model OSI MODEL.
29.1 Chapter 29 Multimedia Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
EE442—Multimedia Networking Jane Dong California State University, Los Angeles.
Delivering Object- Based Audio Visual Services Hari Kalva, Javier Zamora, and Alexandros Eleftheriadis Columbia University, Packet Video 99.
1 MPEG-21 : Goals and Achievements Ian Burnett, Rik Van de Walle, Keith Hill, Jan Bormans and Fernando Pereira IEEE Multimedia, October-November 2003.
Protocols and the TCP/IP Suite
Howell Istance School of Computing De Montfort University
Philips Research France Delivery Context in MPEG-21 Sylvain Devillers Philips Research France Anthony Vetro Mitsubishi Electric Research Laboratories.
ATSC Digital Television
MPEG-4 Applications Interactive TV (broadcast) Streaming media on the web (internet) Multimedia titles (CD-ROM) Network games Mobile multimedia (picture.
TCP/IP Protocol Suite 1 Chapter 25 Upon completion you will be able to: Multimedia Know the characteristics of the 3 types of services Understand the methods.
Background MPEG : Moving Picture Experts Group: a working group of ISO/TEC “Compactly representing digital video and audio signal for consumer distribution”
The OSI Model A layered framework for the design of network systems that allows communication across all types of computer systems regardless of their.
MPEG-2 Transport streams tMyn1 MPEG-2 Transport streams The MPEG-2 Systems Standard specifies two methods for multiplexing the audio, video and other data.
MPEG-4 Cedar Wingate MUMT 621 Slide Presentation I Professor Ichiro Fujinaga September 24, 2009.
MPEG-4 By Christian Smith MUMT-621 Feb. 5 th 2012.
Data Communications Network Models.
HYPERTEXT MARKUP LANGUAGE (HTML)
Protocols and the TCP/IP Suite Chapter 4. Multilayer communication. A series of layers, each built upon the one below it. The purpose of each layer is.
Moving PicturestMyn1 Moving Pictures MPEG, Motion Picture Experts Group MPEG is a set of standards designed to support ”Coding of Moving Pictures and Associated.
MPEG-4 RTP transport Philippe Gentric Philips Digital Networks 49th IETF Conference San Diego, 14 December 2000.
Lecture 1 The OSI Model Reference: TCP/IP Protocol Suite, 4 th Edition (chapter 2) 1.
Presentation on Osi & TCP/IP MODEL
What is a Protocol A set of definitions and rules defining the method by which data is transferred between two or more entities or systems. The key elements.
CS Spring 2012 CS 414 – Multimedia Systems Design Lecture 12 – MPEG-2/ MPEG-4 (Part 6) Klara Nahrstedt Spring 2012.
1 Seminar Presentation Multimedia Audio / Video Communication Standards Instructor: Dr. Imran Ahmad By: Ju Wang November 7, 2003.
MPEG-4 Technology Strategy Analysis Sonja Kangas, Mihai Burlacu T Research Seminar on Telecommunications Business II Telecommunications Software.
Protocols and the TCP/IP Suite
Profiles and levelstMyn1 Profiles and levels MPEG-2 is intended to be generic, supporting a diverse range of applications Different algorithmic elements.
MPEG-21 : Overview MUMT 611 Doug Van Nort. Introduction Rather than audiovisual content, purpose is set of standards to deliver multimedia in secure environment.
CHAPTER FOUR COMPUTER SOFTWARE.
Networks – Network Architecture Network architecture is specification of design principles (including data formats and procedures) for creating a network.
Philips Softworks CONFIDENTIAL Opportunities in MHP & MPEG-4 Presentation at ITU MediaCOM 14 th March 2002 Paul Bristow Strategic Technologist, Philips.
The OSI Model.
Multimedia is a combination of text, art, sound, animation, and video.
CHAPTER TEN AUTHORING.
By NIST/ITL/IAD, Mike Rubinfeld, January 16, 2002 Page 1 L3 Overview L3 Standards Overview By Mike Rubinfeld Chairman, INCITS/L3 (MPEG & JPEG) NIST, Gaithersburg,
DMB 서비스 기술 임 영 권 Characteristics of T-DMB Backward Compatibility Efficient use of bandwidth Convergence between broadcasting & communication.
1 Mpeg-4 Overview Gerhard Roth. 2 Overview Much more general than all previous mpegs –standard finished in the last two years standardized ways to support:
MULTIMEDIA OVER WIRELESS BROADBAND NETWORKS BY: NEELIMA PUNJALA.
RUS Projects Communication Systems BeWü Development 1 Payload for MPEG-4 with Flexible Error Resiliency RTP Payload for MPEG-4 with Flexible Error Resiliency.
MPEG-4 Coding of audio-visual objects Presentation By: Ihab Ilyas.
William Stallings Data and Computer Communications
MPEG-4 standard MPEG-4 Multimedia Standard Olivier Dechazal.
Prepared by Engr.Jawad Ali BSc(Hons)Computer Systems Engineering University of Engineering and Technology Peshawar.
CHAPTER 4 PROTOCOLS AND THE TCP/IP SUITE Acknowledgement: The Slides Were Provided By Cory Beard, William Stallings For Their Textbook “Wireless Communication.
MPEG-4 streams Comp- ress Comp- ress Comp- ress Comp- ress D E L I V E R Y I N T E R F A C E (DAI) Comp- ress Scene Des. decomp- ress decomp- ress decomp-
The ISO/MPEG standardization process Requirements Call for proposals Evaluation Core experiments Draft specification National bodies agree.
MPEG-4: Multimedia Coding Standard Supporting Mobile Multimedia System Lian Mo, Alan Jiang, Junhua Ding April, 2001.
MPEG-7 Audio Overview Ichiro Fujinaga MUMT 611 McGill University.
Video Compression and Standards
Multimedia Communications Introduction ECE 591 Pro. Honggang Wang Notes: Some slides including figures are referred from textbooks and relevant materials.
Network Models. The OSI Model Open Systems Interconnection (OSI). Developed by the International Organization for Standardization (ISO). Model for understanding.
Report on MPEG activities (WP4) Schema 5 th Technical Committee Meeting Ipswich, February 2004 Josep R. Casas, UPC.
Introduction to MPEG  Moving Pictures Experts Group,  Geneva based working group under the ISO/IEC standards.  In charge of developing standards for.
Networks Standardisation & Protocols. Learning Objectives Explain the advantages of standardisation and describe some areas of standardisation such as.
Computer Engineering and Networks, College of Engineering, Majmaah University Protocols OSI reference MODEL TCp /ip model Mohammed Saleem Bhat
OSI Model OSI MODEL. Communication Architecture Strategy for connecting host computers and other communicating equipment. Defines necessary elements for.
OSI Model OSI MODEL.
What is a Protocol A set of definitions and rules defining the method by which data is transferred between two or more entities or systems. The key elements.
MPEG-4 Binary Information for Scenes (BIFS)
DEPARTMENT OF COMPUTER SCIENCE
Protocols and the TCP/IP Suite
MPEG-4 & Wireless Multimedia Streaming
Overview What is Multimedia? Characteristics of multimedia
Multimedia Content Description Interface
OSI Model OSI MODEL.
Protocols and the TCP/IP Suite
Presentation transcript:

MPEG-4

MPEG-4, or ISO/IEC is an international standard describing coding of audio-video objects the 1 st version of MPEG-4 became an international standard in 1999 and the 2 nd version in 2000 (6 parts); since then many parts were added and some are under development today MPEG-4 included object-based audio-video coding for Internet streaming, television broadcasting, but also digital storage MPEG-4 included interactivity and VRML support for 3D rendering has profiles and levels like MPEG-2 has 27 parts

MPEG-4 parts Part 1, Systems – synchronizing and multiplexing audio and video Part 2, Visual – coding visual data Part 3, Audio – coding audio data, enhancements to Advanced Audio Coding and new techniques Part 4, Conformance testing Part 5, Reference software Part 6, DMIF (Delivery Multimedia Integration Framework) Part 7, optimized reference software for coding audio- video objects Part 8, carry MPEG-4 content on IP networks

MPEG-4 parts (2) Part 9, reference hardware implementation Part 10, Advanced Video Coding (AVC) Part 11, Scene description and application engine; BIFS (Binary Format for Scene) and XMT (Extensible MPEG-4 Textual format) Part 12, ISO base media file format Part 13, IPMP extensions Part 14, MP4 file format, version 2 Part 15, AVC (advanced Video Coding) file format Part 16, Animation Framework eXtension (AFX) Part 17, timed text subtitle format Part 18, font compression and streaming Part 19, synthesized texture stream

MPEG-4 parts (3) Part 20, Lightweight Application Scene Representation (LASeR) and Simple Aggregation Format (SAF) Part 21, MPEG-J Graphics Framework eXtension (GFX) Part 22, Open Font Format Part 23, Symbolic Music Representation Part 24, audio and systems interaction Part 25, 3D Graphics Compression Model Part 26, audio conformance Part 27, 3D graphics conformance

Motivations for MPEG-4 Broad support for MM facilities are available 2D and 3D graphics, audio and video – but Incompatible content formats 3D graphics formats as VRML are badly integrated to 2D formats as FLASH or HTML Broadcast formats (MHEG) are not well suited for the Internet Some formats have a binary representation – not all SMIL, HTML+, etc. solve only a part of the problems Both authoring and delivery are cumbersome Bad support for multiple formats

MPEG-4: Audio/Visual (A/V) Objects Simple video coding (MPEG-1 and –2) A/V information is represented as a sequence of rectangular frames: Television paradigm Future: Web paradigm, Game paradigm … ? Object-based video coding (MPEG-4) A/V information: set of related stream objects Individual objects are encoded as needed Temporal and spatial composition to complex scenes Integration of text, “natural” and synthetic A/V A step towards semantic representation of A/V Communication + Computing + Film (TV…)

Main parts of MPEG-4 1. Systems – Scene description, multiplexing, synchronization, buffer management, intellectual property and protection management 2. Visual – Coded representation of natural and synthetic visual objects 3. Audio – Coded representation of natural and synthetic audio objects 4. Conformance Testing – Conformance conditions for bit streams and devices 5. Reference Software – Normative and non-normative tools to validate the standard 6. Delivery Multimedia Integration Framework (DMIF) – Generic session protocol for multimedia streaming

Main objectives – rich data Efficient representation for many data types Video from very low bit rates to very high quality 24 Kbs.. several Mbps (HDTV) Music and speech data for a very wide bit rate range Very low bit rate speech (1.2 – 2 Kbps).. Music (6 – 64 Kbps).. Stereo broadcast quality (128 Kbps) Synthetic objects Generic dynamic 2D and 3D objects Specific 2D and 3D objects e.g. human faces and bodies Speech and music can be synthesized by the decoder Text Graphics

Main objectives – robust + pervasive Resilience to residual errors Provided by the encoding layer Even under difficult channel conditions – e.g. mobile Platform independence Transport independence MPEG-2 Transport Stream for digital TV RTP for Internet applications DAB (Digital Audio Broadcast)... However, tight synchronization of media Intellectual property management + protection For both A/V contents and algorithms

Main objectives - scalability Scalability Enables partial decoding Audio - Scalable sound rendering quality Video - Progressive transmission of different quality levels - Spatial and temporal resolution Profiling Enables partial decoding Solutions for different settings Applications may use a small portion of the standard “Specify minimum for maximum usability”

Main objectives - genericity Independent representation of objects in a scene Independent access for their manipulation and re-use Composition of natural and synthetic A/V objects into one audiovisual scene Description of the objects and the events in a scene Capabilities for interaction and hyper linking Delivery media independent representation format Transparent communication between different delivery environments

Object-based architecture

MPEG-4 as a tool box MPEG-4 is a tool box (no monolithic standard) Main issue is not a better compression No “killer” application (as DTV for MPEG-2) Many new, different applications are possible Enriched broadcasting, remote surveillance, games, mobile multimedia, virtual environments etc. Profiles Binary Interchange Format for Scenes (BIFS) Based on VRML 2.0 for 3D objects “Programmable” scenes Efficient communication format

MPEG-4 Systems part

MPEG-4 scene, VRML-like model

Logical scene structure

MPEG-4 Terminal Components

Digital Terminal Architecture

BIFS tools – scene features 3D, 2D scene graph (hierarchical structure) 3D, 2D objects (meshes, spheres, cones etc.) 3D and 2D Composition, mixing 2D and 3D Sound composition – e.g. mixing, “new instruments”, special effects Scalability and scene control Terminal capabilities (TermCab) MPEG-J for terminal control Face and body animation XMT - Textual format; a bridge to the Web world

BIFS tools – command protocol Replace a scene with this new scene A replace command is an entry point like an I-frame The whole context is set to the new value Insert node in a grouping node Instead of replacing a whole scene, just adds a node Enables progressive downloads of a scene Delete node - deletion of an element costs a few bytes Change a field value; e.g. color, position, switch on/off an object

BIFS tools – animation protocol The BIFS Command Protocol is a synchronized, but non streaming media Anim is for continuous animation of scenes Modification of any value in the scene – Viewpoints, transforms, colors, lights The animation stream only contains the animation values Differential coding – extremely efficient

Elementary stream management Object description Relations between streams and to the scene Auxiliary streams: IPMP – Intellectual Property Management and Protection OCI – Object Content Information Synchronization + packetization – Time stamps, access unit identification, … System Decoder Model File format - a way to exchange MPEG-4 presentations

An example MPEG-4 scene

Object-based compression and delivery

Linking streams into the scene (1)

Linking streams into the scene (2)

Linking streams into the scene (3)

Linking streams into the scene (4)

Linking streams into the scene (5)

Linking streams into the scene (6) An object descriptor contains ES descriptors pointing to: Scalable coded content streams Alternate quality content streams Object content information IPMP information ES descriptors have subdescriptors to: Decoder configuration (stream type, header) Sync layer configuration (for flexible SL syntax) Quality of service information (for heterogeneous nets) Future / private extensions terminal may select suitable streams

Describing scalable content

Describing alternate content versions

Decoder configuration info in older standards cfg = configuration information (“stream headers”)

Decoder configuration information in MPEG-4 the OD (ESD) must be retrieved first for broadcast ODs must be repeated periodically

The Initial Object Descriptor Derived from the generic object descriptor – Contains additional elements to signal profile and level (P&L) P&L indications are the default way of content selection – The terminal reads the P&L indications and knows whether it has the capability to process the presentation Profiles are signaled in multiple separate dimensions Scene description Graphics Object descriptors Audio Visual The “first” object descriptor for an MPEG-4 presentation is always an initial object descriptor

Transport of object descriptors Object descriptors are encapsulated in OD commands – ObjectDescriptorUpdate / ObjectDescriptorRemove – ES_DescriptorUpdate / ES_DescriptorRemove OD commands are conveyed in their own object descriptor stream in a synchronized manner with time stamps – Objects / streams may be announced during a presentation There may be multiple OD & scene description streams – A partitioning of a large scene becomes possible Name scopes for identifiers (OD_ID, ES_ID) are defined – Resource management for sub scenes can be distributed Resource management aspect - If the location of streams is changed, only the ODs need modification. Not the scene description

Initial OD pointing to scene and OD stream

Initial OD pointing to a scalable scene

Auxiliary streams IPMP streams Information for Intellectual Property Management and Protection Structured in (time stamped) messages Content is defined by proprietary IPMP systems Complemented by IPMP descriptors OCI (Object Content Information) streams Meta data for an object (“Poor man’s MPEG-7”) Structured descriptors conveyed in (time stamped) messages Content author, date, keywords, description, language,... Some OCI descriptors may be directly in ODs or ESDs ES_Descriptors pointing to such streams may be attached to any object descriptor – scopes the IPMP or OCI stream An IPMP stream attached to the object descriptor stream is valid for all streams

Adding an OCI stream to an audio stream

Adding OCI descriptors to audio streams

Linking streams to a scene – including “upstreams”

MPEG-4 streams

Synchronization of multiple elementary streams Based on two well known concepts Clock references – Convey the speed of the encoder clock Time stamps – Convey the time at which an event should happen Time stamps and clock references are defined in the system decoder model conveyed on the sync layer

System Decoder Model (1)

System Decoder Model (2) Ideal model of the decoder behavior – Instantaneous decoding – delay is implementation’s problem Incorporates the timing model – Decoding & composition time Manages decoder buffer resources Useful for the encoder Ignores delivery jitter Designed for a rate-controlled “push” scenario – Applicable also to flow-controlled “pull” scenario Defines composition memory (CM) behavior A random access memory to the current composition unit CM resource management not implemented

Synchronization of elementary streams with time events in the scene description How are time events handled in the scene description? How is this related to time in the elementary streams? Which time base is valid for the scene description?

Cooperating entities in synchronization Time line (“object time base”) for the scene Scene description stream with time stamped BIFS access units Object descriptor stream with pointers to all other streams Video stream with (decoding & composition) time stamps Audio stream with (decoding & composition) time stamp Alternate time line for audio and video

A/V scene with time bases and stamps

Hide the video at time T1

Hide the video on frame boundary

The Synchronization Layer (SL) Synchronization layer (short: sync layer or SL) SL packet = one packet of data Consists of header and payload Defines a “wrapper syntax” for the atomic data: access unit Indicates boundaries of access units AccessUnitStartFlag, AccessUnitEndFlag, AULength Provides consistency checking for lost packets Carries object clock reference (OCR) stamps Carries decoding and composition time stamps (DTS, CTS)

Elementary Stream Interface (1)

Elementary Stream Interface (2)

Elementary Stream Interface (3)

Elementary Stream Interface (4)

The sync layer design Access units are conveyed in SL packets Access units may use more than one SL packet SL packets have a header to encode the information conveyed through the ESI SL packets that don’t start an AU have a smaller header

How is the sync layer designed ? As flexible as possible to be suitable for a wide range of data rates a wide range of different media streams Time stamps have variable length variable resolution Same for clock reference (OCR) values OCR may come via another stream Alternative to time stamps exists for lower bit rate Indication of start time and duration of units (accessUnitDuration,compositionUnitDuration)

SLConfigDescriptor syntax example class SLConfigDescriptor { uint (8) predefined; if (predefined==0) { bit(1) useAccessUnitStartFlag; bit(1) useAccessUnitEndFlag; bit(1) useRandomAccessPointFlag; bit(1) usePaddingFlag; bit(1) useTimeStampsFlag; uint(32) timeStampResolution; uint(32) OCRResolution; uint(6) timeStampLength; uint(6) OCRLength; if (!useTimeStamps) { SDL- Syntax Description Language

Wrapping SL packets in a suitable layer

MPEG-4 Delivery Framework (DMIF)

The MPEG-4 Layers and DMIF DMIF hides the delivery technology Adopts QoS metrics Compression Layer Media aware Delivery unaware Sync Layer Media unaware Delivery unaware Delivery Layer Media unaware Delivery aware

DMIF communication architecture

Multiplex of elementary streams Not a core MPEG task Just respond to specific needs for MPEG-4 content transmission Low delay Low overhead Low complexity This prompted the design of the “FlexMux” tool One single file format desirable This lead to the design of the MPEG-4 file format

Modes of FlexMux

How to configure MuxCode mode ?

A multiplex example

Multiplexing audio channels in FlexMux

Multiplexing all channels to MPEG-2 TS

MPEG-2 Transport Stream

MPEG-4 content access procedure Locate an MPEG-4 content item (e.g. by URL) and connect to it – Via the DMIF Application Interface (DAI) Retrieve the Initial Object Descriptor This Object Descriptor points to an BIFS + OD stream – Open these streams via DAI Scene Description points to other streams through Object Descriptors - Open the required streams via DAI Start playing!

MPEG-4 content access example