1. WWW9 Amsterdam Streaming Multimedia Metadata Frank Nack & Jane Hunter CWI, Amsterdam DSTC, Uni. Of Qld Frank.Nack@cwi.nl Jane.Hunter@dstc.edu.au

2. WWW9 Amsterdam Human Or System MPEG-7 System Architecture XML Schema Parser

3. WWW9 Amsterdam MPEG-7 Applications Indexing and retrieval of audiovisual archives (pull) Broadcast media selection (push media filtering e.g. news, radio, tv) Journalism, Entertainment Education Surveillance and remote sensing Telemedicine, Bio-medical applications

4. WWW9 Amsterdam When Is Metadata Streaming Required Applications performing in real time e.g. sports, news Interactive Applications e.g. education, VR games, etc.

5. WWW9 Amsterdam ? MPEG-7 Video Camera

6. WWW9 Amsterdam Types of Metadata Required Content Description Rights Formatting Administrative (financial, usage, rating) Delivery Security EPGs

7. Metadata and Content Referencing Provider Metadata Actors Service Provider Network Provider PDRConsumer Input Output Target Actors Descriptive information Applications EPG Copyright mgmt Service mgmt Content Referencing Security Copyright mgmt PDR Storage management Payment models Descriptive information Content Format Content mgmt Payment Models Usage History Content Reference Security CapturingLocal storingConsuming Content Creation & Packaging DeliveryStoringViewing Copying Related Processes Original descriptive information Creator IPR Creator Content Service Mgt in PDR Location and Access Criteria Payment Models Billing Security Content mgt (bookmarking etc) Descriptive information Payment User Preferences Usage History Descriptive information Content Format Content referencing Copyright protections Payment models Advanced Applications, EPG Content management in PDR Copyright management Storage management Billing METADATA FLOW IN THE TV-ANYTIME SYSTEM ( TV037.DOC )

8. WWW9 Amsterdam Descriptors: (Syntax & semantic of feature representation) D7 D2 D5 D6 D4 D1 D9 D8 D10  extension Description Definition  extension Language Definition 10101 1 0 Encoding & Delivery Tags <scene id=1> <time>.... <camera>.. <annotation Instantiation D3 Description Schemes D1 D3D2 D5D4D6 DS2 DS3 DS1 DS4 Structuring MPEG-7 Components

9. WWW9 Amsterdam Definition: The Description Definition Language (DDL) is a language that allows the creation of new Description Schemes and, possibly, Descriptors. It also allows the extension and modification of existing Description Schemes. (The MPEG-7 DDL is XML Schema Language with some extensions.) Description Definition Language (DDL)

10. WWW9 Amsterdam UML of MediaInformationDS

12. WWW9 Amsterdam Definition: An MPEG-7 Description consists of a DS (structure) and the set of Descriptor Values (instantiations) that describe the Data. MPEG-7 descriptions are XML but are there any non- XML metadata standards? Description

13. mpeg7_content:news1 file://D:/Mpeg7_17/news1.mpg MPEG-1 PAL CD color mono 666.478.608 38 18 1 352 288 25 MPEG-1

14. WWW9 Amsterdam Definition: A Coded Description is a Description that has been encoded to fulfil relevant requirements such as compression efficiency, error resilience, random access, etc. Coded Description

15. WWW9 Amsterdam Streaming MPEG-7 Two possible approaches 1Multiplex descriptive metadata on top of video stream - packets interleaved only one stream to parse don’t know what is lost - metadata or essence? 2Separate streams - one for video content, another for the XML metadata description(s) can prioritize streams, reserve bandwidth, specify QoS

16. Delivery-ready Description Description Ref. To Schemas + Instantiation (Text or binary) Transport Sync. & encapsulation data for storage or streaming (MPEG-2, MPEG-4) Delivery-ready Media MPEG-1 MPEG-2 MPEG-4 RealMedia, QT, AVI?? Transport Sync. & encapsulation data for storage or streaming (MPEG-2, MPEG-4) Stand-alone Elementary Streams (ESs) Stand-alone Media Stream Delivery-ready Described Media Metadata Media Data Transport Multiplex

17. WWW9 Amsterdam Metadata Categories realtime vs mediatime Specific metadata is application-dependent, however 4 relevant categories: Temporal delivery - metadata needs to be delivered at a certain real time - before or synchronized with the audiovisual delivery (content descriptions for filtering) Atemporal delivery - metadata can be delivered anytime (before, during, after - the billing information?) Temporal meaning - metadata linked by time stamp to video content e.g. keyframe. Atemporal meaning - e.g. title, director

18. WWW9 Amsterdam Elementary Streams Separate metadata into separate streams according to type, error sensitivity, temporal aspects Do we need to send schema definitions with the descriptions? Store schema defns. on set top box - update periodically by downloading from URLs Access Units - MPEG-7 data packets Synchronization layer - defines buffer model for MPEG-7 data plus interface to enable synchronous consumption MPEG-7 Systems Approach

19. Description Layer MPEG-7 Elementary Stream Flattened and compressed metadata description MPEG-7 AU ES segmented into packets with timing Des_ES_ID Media_ES_ID Media Elementary Stream Media AU ES segmented into packets with timing Description AU Link Media Layer

20. WWW9 Amsterdam Encoding Metadata for Streaming Encoder has to: Flatten nested descriptions; Determine optimum packet size; Resend header/contextual info periodically to cover errors, packet loss and mid-way entry; Add time stamps to support real time delivery and synchronizn; Binary encode/compress packets;

21. WWW9 Amsterdam Linking Need to be able to link from MPEG-7 descriptions to media or essence: MPEG-2 systems layer MPEG-4 MPEG-4 file format on top of MPEG-2 or MPEG-4

22. WWW9 Amsterdam Effect of Lossiness Built-in error checking but... What happens to timestamps/links to essence in the description when essence packets are lost? Link from metadata points to nothing. What happens if description packets are lost? Users miss lost ‘highlights’

23. WWW9 Amsterdam Streaming Choices Define a transport-independent MPEG-7 content stream and then define the carriage on transport mechanism MPEG-7 on MPEG-2 (transport stream private data) MPEG-7 on MPEG-4 MPEG-7 on file (extension of MP4 file format) Or Use WAP/WML to define the transport-independent MPEG-7 content stream

24. WWW9 Amsterdam MPEG 1/2 Internal Metadata Image Size Frame Rate Motion Vectors User Data Private Data Streams Copyright Markers Color Separation (RGB -> YUV) Timing and Sync Information

25. WWW9 Amsterdam MPEG 2 Meta Data Program Specific Information (PSI) (ISO 13818-1/ITU-T Rec. H222.0, ETS 468) Program Association Table (PAT) Program Map Table (PMT) Conditional Access Table (CAT) Network Information Table (NIT) Service Information (SI by DVB) (ETS 468, TM 1324) Bouquet Assotiation Table (BAT) Service Description Table (SDT) Event Information Table (EIT) Time and Date Table (TDT) Running Status Table (RST) Video Text (ETS 472) Closed Caption (TM 1398)

26. WWW9 Amsterdam MPEG 4 Media Object Description Audio TextToSpeech Score Driven Synthesis: SAOL (Structured Audio Orchestra Language) Video Parametric descriptions of a) a synthetic description of human face and body b) animation streams of the face and body Static and Dynamic Mesh Coding with texture mapping Texture Coding for View Dependent applications Shape and alpha coding

27. WWW9 Amsterdam MPEG 4 Object Content Information Data stream to send textual information along with MPEG-4 content Possibilities: classify content according to pre-defined tables (defined outside of MPEG) giving unique labels to content storing camera parameters

28. WWW9 Amsterdam MPEG 4 Scene Description BIFS: Binary Format for Scenes to compose a set of objects into a scene coded and transmitted together with the media objects follows a hierarchical structure Each node of the graph is a media object audiovisual objects have both a spatial and a temporal extent parameters to the composition layer (e.g. pitch of a sound, the color for a synthetic object)

29. WWW9 Amsterdam MPEG 4 Scene Description II BIFS: Scene Graph

30. WWW9 Amsterdam MPEG 4 Stream Concept

31. WWW9 Amsterdam MPEG 4 Scene Composition

32. WWW9 Amsterdam WAP/WML Wireless Application Protocol (WAP) - specifn. that enables mobile users with wireless devices to easily access and interact with information and services Wireless Markup Language (WML) - designed to send XML over low bandwidth with limited memory and CPU lightweight, subset of HDML (Hand-held Device Markup) specifies presentation/layout of text and images specifies user interaction linear, non-nested flat context Binary XML Content Format Specification (WBML) - compact binary representation of XML documents

33. WWW9 Amsterdam References MPEG-7 http://www.cselt.it/mpeg/ Document N3292 Systems Working Draft 0.3 MPEG-1, MPEG-2, MPEG-4 http://www.cselt.it/mpeg/ WAP/WML http://www.oasis-open.org/cover/wap-wml.html http://www.wapforum.org/ http://www.colorline.no/wap-faq/