The MPEG-7 Standard - A Brief Tutorial - Ali Tabatabai Sony US Research Laboratories February 27, 2001

2 Outline  Objectives of the MPEG-7 Standard  Main Elements of MPEG-7  Scope of MPEG-7  MPEG-7 Application Areas  MPEG 7’s relation with other standards

3 Why do we need MPEG-7 ? N e e d Content Management Fast & Accurate Access Personalized Content Production and Consumption Automation + Support for Advanced Query Visual Audio Sketch

4 MPEG: A Brief History (1)  MPEG: Moving Picture Experts Group  ISO / IEC/JTC1/SC29/WG11 A Working Group of ISO/IEC in charge of the Development of Standards for Coded Representation of Digital Audio and Video Established in 1988

5 MPEG: A Brief History (2)  MPEG-1: Interactive CD and MP311 / 1992  MPEG-2: DTV, STB, DVD11 / 1994  MPEG-4: Web and Mobilityver1: 09 /1998 ver2: 11 /1999  MPEG-7: ??? 08 / 2001  MPEG-21: Multimedia Framework11 / 2001

6 MPEG-7: What Is It ? THE MPEG 7 STANDARD Content Description of Various Audio Visual Information IS NOT a COMPRESSION Standard similar to MPEG-1/2/4 or their Extension IS NOT a STANDARD for FEATURE EXTRACTION/MATCHING Types of Audio Visual Information Audio, Speech Moving video, still pictures, graphics Information on how objects are combined in scenes Types of Audio Visual Information Audio, Speech Moving video, still pictures, graphics Information on how objects are combined in scenes

MPEG-7: Application Areas  Storage and retrieval of audiovisual databases (image, film, radio archives)  Broadcast media selection (radio, TV programs)  Surveillance (traffic control, surface transportation, production chains)  E-commerce and Tele-shopping (searching for clothes / patterns)  Remote sensing (cartography, ecology, natural resources management)  Entertainment (searching for a game, for a karaoke)  Cultural services (museums, art galleries)  Journalism (searching for events, persons)  Personalized news service on Internet (push media filtering)  Intelligent multimedia presentations  Educational applications  Bio-medical applications

8 MPEG-7 Description Scope for AV Content  Description Granularity  Low-level  High-level  Form  Access  Classification  Link  Context

9 MPEG-7: Main Elements  Descriptors (D)  syntax and semantics of each feature representation  Description Schemes (DS)  structure and semantics of the relationships between components  Description Definition Language (DDL)  creation of new DS’s  modification/extension of existing DS’s

10 MPEG-7: Major Functionalities  Systems (ISO / IEC )  Description Definition Language (ISO / IEC )  Visual (ISO / IEC )  Audio (ISO / IEC )  Multimedia Description Schemes (ISO / IEC )  Reference Software (ISO / IEC )

11 DS1 DS2 DS3 D1 D3 D2 Systems DDL Instantiation MPEG-7: Main Elements (2)

12 MPEG-7: Systems It defines tools to:  manage and protect intellectual property  synchronize between content and description  provide for efficient storage and transport

13 MPEG-7: DDL and its Components  Description Definition Language:  Creation of the Ds and DS’s: XML Schema & MPEG-7 Extensions  Instantiation of XML  XML Schema:  Data types  Simple and Complex types  Elements, attributes  Inheritance, Abstract types  MPEG-7 extensions:  Array and Matrix data type

14 MPEG-7: Audio  Sound Effects  Music Instrument Timbre  Spoken Content  Melody Contour

15 MPEG-7: Visual (1)  Color  quantization, dominant, scalable, color-structure, layout, GoF/GoP  Texture  Shape  region-based, contour-based, 3D  Motion  camera motion, motion trajectory, parametric motion, motion activity

16 MPEG-7: Visual (2)  Localization  spatio temporal  Others  face recognition

17 MPEG-7: Basic Visual Structures  Grid Layout  2D-3D Multiple View  Time Series  Spatial 2D Coordinates  Temporal Interpolation

18 Low level Audio Visual descriptors Color Camera motion Motion activity Mosaic Color Motion trajectory Parametric motion Spatio-temporal shape Color Shape Position Texture Video segments Still regions Moving regions Audio segments Spoken content Spectral characterization Music: timbre, melody

19 MPEG-7: MMDS Basic Elements Datatype & structures Link & media localization Basic DSs Basic elements Schema tools Time, Duration, Medialocators Language Annotation,Person, Place Root, Top-level elements, Packages

20 MPEG 7: Content Management and Description Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Title, Creator, Creation location & date, Purpose, Classification, Genre, Review, Parental guidance, etc. (Author generated) Format, Coding, Instances, Identification, Transcoding Hint, etc. (Several instances) Rights holder, Access rights, Usage Record, Financial aspects, etc. (Evolution) Datatype & structures Link & media localization Basic DSs Basic elements Schema tools Viewpoint of the structure: Segments Spatial / temporal structure Audio, video low-level Ds Elementary semantic information. Viewpoint of conceptual notions Events, objects, abstract concepts, and their relation

21 Foreground Example of Segment trees SR1:  Creation,Usage meta information  Mediadescription  Textualannotation  Colorhistogram,Texture SR2:  Shape  ColorHistogram  Textualannotation SR6:  ColorHistogram  Textualannotation SR5:  Shape  Textualannotation SR4:  Shape  ColorHistogram  Textualannotation SR3:  Shape  ColorHistogram  Textualannotation Background

22 Segment Tree Shot1 Shot2 Shot3 Segment 1 Sub-segment 1 Sub-segment 2 Sub-segment 3 Sub-segment 4 segment 2 Segment 3 Segment 4 Segment 5 Segment 6 Segment 7 Semantic DS (Events) Introduction Summary Program logo Studio Overview News Presenter News Items International Clinton Case Pope in Cuba National Twins Sports Closing Time Axis

23 MPEG 7: Navigation and Access Navigation & Access Summary Variation Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Datatype & structures Link & media localization Basic DSs Basic elements Schema tools Efficient support of: discovery, browsing, navigation, visualization Substitution of the original content Adaptation to terminal, network, or user preferences

24 MPEG 7: Hierarchical summary A-V Data Hierarchical Summary HighlightLevel Highlight Segment Highlight Segment Highlight Segment Highlight Segment Highlight Segment Highlight Segment Highlight Segment

25 MPEG 7: Sequential summary Sequential Summary A-V Data Text Property Text Property Frame Property Frame Property Frame Property Sound Property Sound Property Sound Property

26 MPEG 7: Variation Universal Multimedia Access Adapt delivery to network and terminal characteristics (QoS)

27 MPEG-7: Content Organization Navigation & Access Summary Variation Analytic Model Content organization Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Description and organization of collection of documents Collection & Classification Datatype & structures Link & media localization Basic DSs Basic elements Schema tools

28 MPEG-7: Collection

29 MPEG 7: User Interaction Navigation & Access Summary Variation Analytic Model Content organization Content description Content management Creation & production MediaContent Usage Conceptual aspects Structural aspects Collection & Classification UserInteraction User preferences Datatype & structures Link & media localization Basic DSs Basic elements Schema tools User identification and preferences: Filtering, search and browsing

30 MPEG-7 Its Relation with other standards  AHG on “Metadata harmonization”:  SMPTE: Metadata dictionary, KLV encoding  Dublin Core Metadata Initiative  European Broadcast Union  AHG on TV AnyTime Application  Large number of Liaisons:  SMPTE  Dublin Core  W3C (XML Schema)  etc.

Competition: Individual work Definition scope and r MPEG-7: TimeLine - The Work Plan Call for proposals Working draft Committee draft Draft international standard Draft international standard Final committee draft Final committee draft International standard International standard Divergence Convergence

32 Conclusions on AV Content Description and MPEG-7  MPEG -7:  AV content description for interoperable application  Description Definition Language:  XML Schema (flexibility) + Binary version (efficiency)  Description Schemes:  Library of description tools  Covers a wide range of generic needs