1. Concepts of Multimedia Processing and Transmission IT 481, Lecture #9 Hung Nguyen, Ph.D. 11 April, 2005 IT 481, Lecture #10 Dennis McCaughey, Ph.D. 13 November, 2006

2. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 2Outline MPEG-4 –Overview –System Description –Coding of Audio Objects –Coding of Visual Objects –Scene Description –IPR and MPEG-4 –DMIF (Delivery Multimedia Integration Framework) –MPEG-4 Applications

3. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 3 MPEG-4: Overview Flexible Multimedia Communications 5kbps - 50Mbps Audio object compression Video object compression Synthetic Audio/Speech and Video Systems: multiplexing and flexible composition

4. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 4 MPEG-4 Overview (cont’d) Functionalities beyond MPEG-1/2 –Interaction with individual objects: The displayed scene can be composed by the receiver from coded objects –Scalability of contents –Error resilience –Coding of both natural and synthetic audio and video

5. System Description

6. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 6 MPEG-4 : System No need to know –What the terminal device –How to receive the stream Sync Layer –Object Descriptors: Relate to elementary streams to a media object (MO) –Synchronization Layer has two types of timing information Convey the speed of the encoder to the decoder The time stamp to portions of the encoded Audio Visual data

7. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 7 Overview of MPEG-4 System Scene segmentation and depth layering O2O2 O3O3 O1O1 Layered encoding contour motion texture contour motion texture contour motion texture bitstream layer 1 bitstream layer 2 bitstream layer 3 multiplexer demultiplexer Separate decoding compositor AV-objects

8. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 8 MPEG-4: System Delivery of streaming data

9. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 9 The Parts of MPEG-4 Standard Part 1: Systems Part 2: Audio Part 3: Visual Part 4: Conformance Part 5: Reference Software Part 6: DMIF: Delivery Multimedia Integration Framework. A session protocol. –set-up of connection channels (broadcast & interactive) –network becomes transparent to application

10. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 10 Object-Based Coding Entire scene is decomposed into multiple objects –Object segmentation is the most difficult task! –But this does not need to be standardized ☺ Each object is specified by its shape, motion, and texture (color) –Shape and texture both changes in time (specified by motion) MPEG-4 assumes the encoder has a segmentation map available, specifies how to decode shape, motion and texture

11. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 11 The displayed scene is composed by the receiver based on desired view angle and objects of interests

12. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 12 Example of Scene Composition The decoder can compose a scene by including different VOPs (Video Object Planes) in a VOL (Video Object Layer)

13. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 13 Example of Editing, Reusability and Compositing of Video Objects

14. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 14 Example of Editing, Reusability and Compositing of Video Objects

15. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 15 Example of Scalability of Video Objects and Optimizing Regions of Interest

16. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 16 Example of Creativity and Interactivity with Video Objects

17. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 17 Example of 2-D Interactivity

18. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 18 Example of 3-D Multimedia Interactivity

19. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 19 Example of Creating a 3-D illusion from a 2-D Presentation

20. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 20 Example of Interactive Virtual Reality

21. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 21 Example of Multicast and Multi-channel Interoperability

22. Coding of Audio Objects

23. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 23 Progression of MPEG-Audio MPEG-1 Two-Channel coding standard (Nov. 1992) MPEG-2 Extension towards Lower- Sampling-Frequency (LSF) (1994) MPEG-2 Backwards compatible multi- channel coding (1994) MPEG-2 Higher Quality multi-channel standard (MPEG-2 AAC) (1997) MPEG-4 Audio Coding and Added Functionalities (1999, 2000)

24. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 24 MPEG-4 Audio: The Audio Toolbox MPEG-4 Audio integrates the different audio worlds High Quality Audio Coding Speech Coding Representation of Natural Audio MPEG-4 Audio - AAC - AAC Scal - AAC LC - Twin VQ - HILN - HVXC - CELP Sound Synthesis - SAOL (extension toMidi) - Text To Speech (TTS) - Effects Processing - 3-D Localization - HVXC - CELP with different Modes

25. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 25 MPEG-4 Audio Objects

26. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 26 MPEG-4 Audio: Natural and Unnatural (Structured) Audio High Quality Audio Tools: –AAC (Advanced Audio Coding) –Twin VQ (Transform domain weighted interleaved Vector Quantisation) Low Bit-rate Audio Tools: –CELP (Code Excited Linear Predictive) coding –Parametric, i.e. coding based on parametric representation of audio signal HVXC (Harmonic Vector eXcitation Coding) HILN (Harmonic and Individual Line and Noise) SNHC (Synthethic/Natural Hybrid Coding) Audio Tools –SAOL (Structured Audio Orchestra Language) –TTS (Text-To-Speech)

27. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 27 MPEG-4: Audio Objects Coding of Audio Objects (Natural Sound) –HVXC (Harmonic Vector eXcitation Coding) : 2 - 4 kbps –CELP (Code Excitation Linear Predictive) Sampling rates 8kHz : 6 - 12 kbps Sampling rates 16kHz : 18 kbps –MPEG-2 AAC –Examples A base layer : CELP An enhancement layer : AAC

28. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 28 MPEG-4: Audio Objects (cont’d) Synthesized Sound –Text To Speech –Score Driven Synthesis Not a standardized method of synthesis “Scores” - described in Structured Audio Orchestra Language (SOAL), an MPEG-4 audio tool

29. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 29 MPEG-4 Audio Version 2: New Tools (Part 2) Environmental Spatialization –Physical approach, based on description of the acoustical properties of the environment –Perceptual approach, based on high level perceptual description of audio scenes –Version 2 Advanced Audio BIFS (Binary Format for Scene) description CELP Silence Compression MPEG-4 File Format: MP4 –Independent of any particular delivery mechanism –Streamable format, rather than a streaming format –Based on the QuickTime format from Apple Computer Inc. Backchannel Specification –Allows for user-controlled streaming

30. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 30 MPEG-4 Audio Version 2: Profiles and Levels Low Delay Audio Profile –For speech and generic audio coding Scalable Internet Audio Profile –Extends scalable profile of version 1 by small step scalability and Parametric Audio Coding MainPlus Audio Profile –Provides superset of all audio profiles

31. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 31 MPEG-4 Audio: D ifferent Codecs and their typical bit-rate Class A: based on Time/Frequency mapping (T/F), i.e. transform coding Class B: Linear Predicting Coding (LPC) based analysis/synthesis codec Class C: Codecs, based on parametric description of audio signal 2468101214162432 bit-rate (kbit/s)64 48 1624Audio bandwidth (kHz)20 Parametric Core Enhance Modules LPC Core Enhance Modules T/F-based Core + Large Step Enhancement Modules + + Satellite Secure Com Cellular Phone InternetISDN ITU-T G.729

32. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 32 Basics of Speech Coding: Principles of Linear Prediction Coding (LPC) Source: www.cselt.it/leonardo/icjfiles/Mpeg-4_si

33. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 33 Conceptual Diagram LPC-based Audio Coding Preprocessing Excitation Generator LPC Synthesis Filter Spectral Weighting Filter Multiplex Error Minimization Audio Signal Coded Bit-Stream LPC Analysis Filter - Based on synthesis by analysis Performs best for speech between 6 kbit/s and 16 kbit/s per channel Multiple bit-rates Bit-rate and bandwidth scaleability

34. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 34 Conceptual Diagram of MPEG-4 CELP Decoder Source: B. Edler, Speech Coding in MPEG-4 LSP VQ Decoder LSP Interpolation Conversion LSP ---> LPC Adaptive Codebook RPE/MPE Codebooks Amplitude Decoder LP Synthesis Filter Filter x x + Formants Fund. Pitch Excitation Amplitude LSP: Line Spectral Pair RPE: Regular Puls Excitation MPE: Multi-Pulse Excitation

35. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 35 Conceptual Diagram of MPEG-4 HVXC Decoder LSP VQ Decoder LSP Interpolation Conversion LSP ---> LPC Formants LP Synthesis Filter Filter Harmonic Synthesis Stochastic Synthesis Excitation Parameter Decoder v/uv Excitation + Source: B. Edler, Speech Coding in MPEG-4

36. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 36 MPEG-4 Natural Speech Coding Tools used in CELP and HVXC CELP MPE Harmonic VQ LSP VQ HVXC RPE WB 3.85 - 23.8 kbit/s NB/WB 2 - 4 kbit/s HVXC: „ H armonic V ector e X citation C oding“ Source: www.cselt.it/leonardo/icjfiles/Mpeg-4_si

37. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 37 MPEG-4 Natural Speech Coding: Specification of Tools HVXC CELP

38. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 38 MPEG-4 Natural Speech Coding Tools Scalable Bit-rate Coding Input Speech 6 kbit/s 2 kbit/s 10 kbit/s CELP Encode r Decoder A 6 kbit/s Decoder B 8 kbit/s Decoder C 10 kbit/s Decoder D 12 kbit/s Decoder E 22 kbit/s 10 kbit/s high frequency component Core Bit Stream 8 kHz 16 kHz 8 kHz Base Quality High Quality

39. Coding of Visual Objects

40. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 40 MPEG-4: Visual Objects High bitrate tools (interlace) VLBV core Content-based functionalities (shape, scalability) functionality Bit rate 5-64kbit/s 1. Low spatial resolution 2. Low frame rate 3. Conventional rectangular size 4. High coding efficiency 1. High spatial resolution 2. High frame rate 3. Broadcast TV 1. Separate coding of content 2. Content-based interactive 3. Object manipulation 4. Hybrid coding of natural as well as synthetic objects …………………. Classification of Video Coding Tools functionality

41. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 41 VLBC Core and the Generic MPEG-4 Coder Motion (MV) Texture (DCT) MPEG-4 VLBV Core Coder (similar to H.263/MPEG-1) Motion (MV) Texture (DCT) Generic MPEG-4 Coder Shape MPEG-4: Visual Objects

42. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 42 MPEG-4: Visual Objects Natural Video –Conventional and Content-Based Functionality

43. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 43 MPEG-4: Synthetic Visual Objects Parametric descriptions of –A synthetic description of human face and body (body in Version 2) –Animation streams of the face and body (body in Version 2) Static and Dynamic Mesh Coding with texture mapping Texture Coding for View Dependent application

44. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 44 MPEG-4: Synthetic Visual Objects – Facial Animation Facial Definition Parameter Facial Animation Parameter Facial animation needs –The Facial Description Parameter (FDP) in Binary Format Scene –The Face Animation Table within FDPs Example : FAP could say ‘open_jaw’ –The Face Interpolation Technique in BIFS Example : Missing FAP

45. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 45 MPEG-4: Synthetic Visual Objects – Animated Meshes 2D animated meshes –Only triangular meshes View dependent scalability –Taking into account the viewing position in the 3D virtual world –This object is sent and computed both at the encoder and the decoder side

46. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 46 Example of MPEG-4 Visual Objects +

47. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 47 MPEG-4: Scene Description How objects are grouped together and positioned in space & time Attribute Value Selection Other transforms on media objects scene person 2D background furniture audiovisual presentation spritevoiceglobedesk Sprite: region of a VO present in scene throughout a video segment

48. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 48 MPEG-4: Intellectual Property Rights (IPR) IPR generally bestows on its owners the right to exclude others (with certain limited exceptions) from the use or re-use of their intellectual property without a license from the IPR owner. Any intellectual property which is delivered, either freely or for commercial gain, through an MPEG-4 application is afforded such protection. More information on MPEG-4 and IPR on http://www.chiariglione.org/mpeg/faq/mp4- sys/sys-faq-ipmp.htm#IPMP-What-is-IP

49. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 49 MPEG-4: Intellectual Property Rights (IPR) IPMP: Intellectual Property Management and Protection

50. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 50 MPEG-4: Delivery Multimedia Integration Framework (DMIF)

51. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 51 MPEG-4: Delivery Multimedia Integration Framework (DMIF) – cont’d Original DMIFTarget DMIF App App 2 App 1 DMIF

52. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 52 MPEG-4: Applications Scalable & Interactive Application on the WWW “Configurable TV”: choose the elements –subtitles ( +language) –stock values –inset of soccer match Animated talking head with speech synthesis Interactive DVD applications

53. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 53 Interactive content can include anything that enhances the viewer's appreciation of TV— for example, local news headlines, sports scores, program lineups, links to Web sites, advertiser logos, actor/actress profiles, and plot summaries. Interactive TV content is limited only by the imagination. An interactive "i" icon tells viewers that interactive content is available © Microsoft Cooperation MPEG-4 Applications: Configurable TV

54. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 54 MPEG-4 Applications: Internet Viewcam Developed by SHARP (1999.3) Record moving picture (160×120) during 1hour with 32MB Memory Recording Format : ASF of Microsoft

55. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 55 MPEG-4: Other Applications Virtual Work Space –Collaborative applications by 2D or 3D worlds (Server - Clients) Internet e-commerce –Low bit-rate connection –Reduce the original data from several Mbytes to a few Kbytes

56. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 56 MPEG-4: Other Applications (cont’d) MPEG-4 Audio encoding and decoding –Real time : 6~96 kbps, 44.1kHz –PC & DSP card –Digital audio broadcasting via AM Bandwidth Mobile phone High speed computer links Interactive Multimedia Authoring Tool –Compose, modified, combined of images and video clips

57. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 57 MPEG-4: Summary MPEG-4 will be a standard that defines tools for the representation of Multimedia Objects. Both these tools and the AV Objects can be used and re-used, in different services and applications, and on many media

58. 04/10/2006 Hung Nguyen, IT 481, Spring 2006 58References [1] Mohammad Khansari,Overview of MPEG-4, Sharif University of Technology, Tehran, Iran. [2] Gerhard Stoll, MPEG-4 Audio Coding, Institut für Rundfunktechnik, München, Germany [1] Multimedia Communications System II, Video Coding Standards, Yao Wang, Polytechnic University, Brooklyn, NY11201