2. RUS Projects Communication Systems BeWü Development 1 Payload for MPEG-4 with Flexible Error Resiliency RTP Payload for MPEG-4 with Flexible Error Resiliency draft-ietf-avt-mpeg4streams-00.txt proposed experimental RFC formerly draft-guillemot-avt-genrtp-03.txt Christine Guillemot, Paul Christ, Stefan Wesner [, Anders Klemets]

3. RUS Projects Communication Systems BeWü Development 2 l draft-guillemot-avt-genrtp-03.txt => new title l Flexmux section added in discussion with authors - (see draft-rgcc-flexmuxmpeg4-00.txt); l Payload Type (PT): Different payload types should be assigned for MPEG4 ES, MPEG4 SL-PDU MPEG-4 FlexMux streams. A payload type in the dynamic range should be chosen. n Same format, same interface ES, SL, Flexmux l TSOFFSET removed, Ebits included l draft-guillemot-avt-genrtp-03.txt => new title l Flexmux section added in discussion with authors - (see draft-rgcc-flexmuxmpeg4-00.txt); l Payload Type (PT): Different payload types should be assigned for MPEG4 ES, MPEG4 SL-PDU MPEG-4 FlexMux streams. A payload type in the dynamic range should be chosen. n Same format, same interface ES, SL, Flexmux l TSOFFSET removed, Ebits included Modifications since Washington

4. RUS Projects Communication Systems BeWü Development 3 Multiple Dimensions of MPEG-4/RTP l Systems (OD) or non-systems (non-OD) framework l Various types of streams with different QoS requirements (MPEG-4 A/V, OD, BIFS, IPMP, other compressed AV formats, eg. H.263 streams) l Mixed live and streamed (pre-encoded) content l Session Types n Client - Server (unicast), “MU” n (Multi) Peer-to-Peer - (multicast) l Systems (OD) or non-systems (non-OD) framework l Various types of streams with different QoS requirements (MPEG-4 A/V, OD, BIFS, IPMP, other compressed AV formats, eg. H.263 streams) l Mixed live and streamed (pre-encoded) content l Session Types n Client - Server (unicast), “MU” n (Multi) Peer-to-Peer - (multicast)... in applications relying on

5. RUS Projects Communication Systems BeWü Development 4 l Unified solution for the transport of MPEG-4 n MPEG-4 SL packet streams - and for n MPEG-4 ES l Common solution for n both live and pre- encoded content l Applicable to the transport of FlexMux PD ( submitted draft-rgcc-avt-mpeg4flexmux-00.txt) l Unified solution for the transport of MPEG-4 n MPEG-4 SL packet streams - and for n MPEG-4 ES l Common solution for n both live and pre- encoded content l Applicable to the transport of FlexMux PD ( submitted draft-rgcc-avt-mpeg4flexmux-00.txt) Summary

6. RUS Projects Communication Systems BeWü Development 5 Summary l Abstraction of QoS monitoring and Congestion Contro “intelligence” n unique interface n architectural simplicity and consistent stream “adaptation” in high number of streams applications (mixing real-time & pre-encoded content) l Packet Loss flexible protection mechanisms l Full and partial AUs Segments with types or priority n Abstract media idiosyncrasies n Assuming a media and network aware adaptation layer n Towards UEP and/or Diffserv marking based on AUs l Abstraction of QoS monitoring and Congestion Contro “intelligence” n unique interface n architectural simplicity and consistent stream “adaptation” in high number of streams applications (mixing real-time & pre-encoded content) l Packet Loss flexible protection mechanisms l Full and partial AUs Segments with types or priority n Abstract media idiosyncrasies n Assuming a media and network aware adaptation layer n Towards UEP and/or Diffserv marking based on AUs

7. RUS Projects Communication Systems BeWü Development 6 Sample RTP packet for MPEG-4 ESs 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0|0| Res | | +-+-+-+-+-+-+-+-+ | |.. AU or partial AU. +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ If no additional data-based protection needed - One byte difference with draft-ietf-avt-mpeg4-es-00.txt, Byte offering additional flexibility in terms of protection - The packet would be of this form if protection supported by compression layer in the case of real-time

8. RUS Projects Communication Systems BeWü Development 7 Sample RTP packet for MPEG-4 ESs If additional data-based protection needed (eg. duplicated headers), (no fragmentation used). 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |G|E| XT | LENGTH |EBITS| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | |. + Extension data (e.g. VOP header) +-+-+-+-+-+-+-+-+. |G|E|0| res | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |.. Media Payload | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

9. RUS Projects Communication Systems BeWü Development 8 Sample RTP packet for MPEG-4 SL-PDUs 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |G|E|F| res | optional SL header paramaters as indicated by. +-+-+-+-+-+-+-+-+ the SLConfigDescriptor. |.. Media payload | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ If no additional data-based protection needed

10. RUS Projects Communication Systems BeWü Development 9 Sample RTP packet for MPEG-4 SL-PDUs If additional protection needed 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |G|E| XT | LENGTH |EBITS| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | |.. Extension data. | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |G|E|F| res | optional SL header paramaters as indicated by | +-+-+-+-+-+-+-+-+ the SLConfigDescriptor. |.. Media payload | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

11. RUS Projects Communication Systems BeWü Development 10 Usage of the format for MPEG-4 Flexmux- PDUs in draft-rgcc-avt-mpeg4flexmux- 00.txt « What is suggested here is to define objects in the RTP payload, as it is proposed in draft-ietf-avt-mpeg4streams-01.txt. The RTP payload being a succession of ob-jects. Each object is byte aligned. Each object starting with its length. » « A payload object is either a protection data object, or a complete FlexMux packet. Each payload object follows an identification byte, its object type byte. The RTP payload starts with one object type byte. » « The object type byte syntax: 0 1 2 3 4 5 6 7 +-+-+-+-+-+-+-+ |G|E| XT | +-+-+-+-+-+-+-+ »

12. RUS Projects Communication Systems BeWü Development 11 MPEG-4/RTP recent history l 12/98 Orlando, n draft-ietf-avt-rtp-mpeg4-00.txt, SL-packetized streams n draft-guillemot-avt-genrtp-00.txt, ES streams + flexible additional protection (if needed, eg. for streamed content) l 04/99 NY + 07/99 Oslo joint IETF/MPEG (phone) meetings n “No non-SL- and non-Systems ES RTP-mapping needed”: n “2 experimental RFCs...” l 10/99 ISO Melbourne n JNB non-System AV ES-mapping proposed l 12/99 Washington draft-jnb-mpeg4av-rtp-00.txt „ the normative way on how MPEG-4 Audio/Visual streams... mapped... to RTP“ l 12/98 Orlando, n draft-ietf-avt-rtp-mpeg4-00.txt, SL-packetized streams n draft-guillemot-avt-genrtp-00.txt, ES streams + flexible additional protection (if needed, eg. for streamed content) l 04/99 NY + 07/99 Oslo joint IETF/MPEG (phone) meetings n “No non-SL- and non-Systems ES RTP-mapping needed”: n “2 experimental RFCs...” l 10/99 ISO Melbourne n JNB non-System AV ES-mapping proposed l 12/99 Washington draft-jnb-mpeg4av-rtp-00.txt „ the normative way on how MPEG-4 Audio/Visual streams... mapped... to RTP“

13. RUS Projects Communication Systems BeWü Development 12 MPEG-4/RTP recent history l 03/2000 Adelaide, n draft-rgcc-avt-flexmuxmpeg4-00.txt, flexmux-packetized streams - –withdrawn last week in the Netherlands ?? n.... more to come? l 03/2000 Adelaide, n draft-rgcc-avt-flexmuxmpeg4-00.txt, flexmux-packetized streams - –withdrawn last week in the Netherlands ?? n.... more to come?

14. RUS Projects Communication Systems BeWü Development 13 Implementation status l Mapping/de-mapping completed l Integrating into IM1 as a DMIF plug-in (with CSELT) is ongoing l FEC implemented: duplicated headers, parity codes l Adaptivity for QoS monitoring under development l R-S under development l Mapping/de-mapping completed l Integrating into IM1 as a DMIF plug-in (with CSELT) is ongoing l FEC implemented: duplicated headers, parity codes l Adaptivity for QoS monitoring under development l R-S under development

15. RUS Projects Communication Systems BeWü Development 14 Information - Distribution l current code documentation n http://www-ks.rus.uni-stuttgart.de/PROJ/GP l ACTS COMIQS project ended february 2000 n http://www.ccett.fr/comiqs/welcome.htm l Used in 1 National Project (F, RNRT-VISI) l Continue in 2 European Projects (IST-SONG, IST-OPENISE) l Distributed to 4 Companies outside the above projects l current code documentation n http://www-ks.rus.uni-stuttgart.de/PROJ/GP l ACTS COMIQS project ended february 2000 n http://www.ccett.fr/comiqs/welcome.htm l Used in 1 National Project (F, RNRT-VISI) l Continue in 2 European Projects (IST-SONG, IST-OPENISE) l Distributed to 4 Companies outside the above projects

16. RUS Projects Communication Systems BeWü Development 15 Conclusion l Complexity of (future) MPEG-4 usage n Multiple streams (live and pre-encoded) with different QoS requirements n so far, multiple ways for handling protection in the different ESs (HEC specific to visual) l Stresses the need, for the sake of architectural « simplicity », n simple, single interface, avoiding parsers of the ESs l draft-ietf-avt-mpeg4streams-01.txt has the potential for an homogeneous way of handling efficient transport of the different types of MPEG-4 streams l Complexity of (future) MPEG-4 usage n Multiple streams (live and pre-encoded) with different QoS requirements n so far, multiple ways for handling protection in the different ESs (HEC specific to visual) l Stresses the need, for the sake of architectural « simplicity », n simple, single interface, avoiding parsers of the ESs l draft-ietf-avt-mpeg4streams-01.txt has the potential for an homogeneous way of handling efficient transport of the different types of MPEG-4 streams

17. RUS Projects Communication Systems BeWü Development 16 Application Case of draft-ietf-avt-mpeg4streams-01.txt to MPEG-4 Visual streams in draft-gc-avt-mpeg4visual-00.txt Christine Guillemot, Paul Christ, Stefan Wesner

18. RUS Projects Communication Systems BeWü Development 17 l Application of drat-ietf-avt-mpeg4streams- 01.txt to the transport of MPEG-4 visual streams l Common solution for n both live and pre- encoded content l Common solution for the different profiles l Application of drat-ietf-avt-mpeg4streams- 01.txt to the transport of MPEG-4 visual streams l Common solution for n both live and pre- encoded content l Common solution for the different profiles Summary

19. RUS Projects Communication Systems BeWü Development 18 l Simple Visual Profile: efficient, error resilient coding of rectangular video objects, suitable for applications on mobile networks. l Simple Scalable Profile: adds support for coding of temporal and spatial scalable objects, useful for applications which provide services at more than one level of quality due to bit-rate or decoder resource limitations. l Core Visual Profile : adds support for coding of arbitrary-shaped and temporally scalable objects, useful for applications such as those providing content-interactivity (Internet multimedia applications). l Simple Visual Profile: efficient, error resilient coding of rectangular video objects, suitable for applications on mobile networks. l Simple Scalable Profile: adds support for coding of temporal and spatial scalable objects, useful for applications which provide services at more than one level of quality due to bit-rate or decoder resource limitations. l Core Visual Profile : adds support for coding of arbitrary-shaped and temporally scalable objects, useful for applications such as those providing content-interactivity (Internet multimedia applications). Visual Profiles

20. RUS Projects Communication Systems BeWü Development 19 l Main Visual Profile : adds support for coding of interlaced, semi-transparent, and sprite objects to the Core Visual Profile; useful for interactive and entertainment-quality broadcast and DVD applications. l N-Bit Visual Profile: adds support for coding video objects having pixel-depths ranging from 4 to 12 bits to the Core Visual Profile. l …. More for synthetic and for natural/synthetic hybrid visual content l Main Visual Profile : adds support for coding of interlaced, semi-transparent, and sprite objects to the Core Visual Profile; useful for interactive and entertainment-quality broadcast and DVD applications. l N-Bit Visual Profile: adds support for coding video objects having pixel-depths ranging from 4 to 12 bits to the Core Visual Profile. l …. More for synthetic and for natural/synthetic hybrid visual content Visual Profiles (continued)

21. RUS Projects Communication Systems BeWü Development 20 l Very elaborate and powerfull syntax. l HEC mechanism useable initially for protecting header parameters needed in the simple profile l Lately corrigendum (DCOR1) in order to cover header parameters for arbitrary shape objects l Not (yet?) for enhancement layers when scalability is used in simple scalable, core, main l Very elaborate and powerfull syntax. l HEC mechanism useable initially for protecting header parameters needed in the simple profile l Lately corrigendum (DCOR1) in order to cover header parameters for arbitrary shape objects l Not (yet?) for enhancement layers when scalability is used in simple scalable, core, main Impact of Profiles on Syntax

22. RUS Projects Communication Systems BeWü Development 21 l Configuration information n global configuration information refering to the whole group of visual objects (visualobjectsequence()), n object configuration information refering to a single visual object (visualobject()) n object layer configuration information (visualobjectlayer()) n video object plane header (videoObjectPlane()) (for video) l Configuration information n global configuration information refering to the whole group of visual objects (visualobjectsequence()), n object configuration information refering to a single visual object (visualobject()) n object layer configuration information (visualobjectlayer()) n video object plane header (videoObjectPlane()) (for video) Elements of syntax

23. RUS Projects Communication Systems BeWü Development 22 l Two modes of transmission for visual object sequence layer, visual object layer and video object layer: n separate mode (in containers provided by MPEG-4 system (Ods) n combined mode (as part of the ES) l video object plane header as part of ES l Two modes of transmission for visual object sequence layer, visual object layer and video object layer: n separate mode (in containers provided by MPEG-4 system (Ods) n combined mode (as part of the ES) l video object plane header as part of ES Transmission of key segments

24. RUS Projects Communication Systems BeWü Development 23 l To avoid to have to support parsers in streaming applications l Same way of handling protection for real-time and pre- encoded content l simple and same interface Esdata dataLength number of padding bits at the end (if needed) degradationPriority and/or segment type (videoobjectlayer(), GOV header, video object plane header, video packet) l To avoid to have to support parsers in streaming applications l Same way of handling protection for real-time and pre- encoded content l simple and same interface Esdata dataLength number of padding bits at the end (if needed) degradationPriority and/or segment type (videoobjectlayer(), GOV header, video object plane header, video packet) Uniform way of handling protection of key segments

25. RUS Projects Communication Systems BeWü Development 24 Sample RTP packet for MPEG-4 visual streams 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |G|E| XT | LENGTH |EBITS| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | |. + eg. Video Object Plane header +-+-+-+-+-+-+-+-+. |G|E|0| res | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |.. Media Payload | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+