1. MPEG for the past, present and future of television Leonardo Chiariglione Keynote Speech at ATSC Annual Meeting Washington, DC – 2012/05/08

2. MPEG for the past of television System Transport Stream DSM-CC Timed metadata Video MPEG-2 High Profile Audio MPEG-1 Audio Layer II MPEG-2 AAC 2012/05/02 2 MPEG for the past, present and future of TV

3. An assessment MPEG has provided the means for the television business on air, cable and satellite to migrate from the analogue to the digital age It has been a collective efforts Representatives from the entire spectrum of the broadcasting world provided their requirements Packages of patent licences were created (outside of MPEG) Today there is virtually no broadcasting system that is not based, at least partially, on MPEG standards MPEG for the past, present and future of TV 3 2012/05/08

4. MPEG for the present of television Systems Carriage of MPEG-4 on MPEG-2 TS Dynamic Adaptive Streaming over HTTP (DASH) Video MPEG-4 Advanced Video Coding (AVC) 3D extensions Audio Various forms of Advanced Audio Coding (AAC) Universal Speech and Audio (USAC) 2012/05/08 4 MPEG for the past, present and future of TV

5. An assessment The deployment of 2 nd generation digital broadcasting systems continues successfully Use of the AVC on MPEG-2 TS allows packing twice as many programs as MPEG-2 Video in an analogue channel With IPTV the notion of broadcasting is expanding MPEG is helping the broadcasting industry To enrich the broadcasted user experience Through standard technologies that serve multiple fields MPEG for the past, present and future of TV 5 2012/05/08

6. MPEG for the future of television Well, that’s what I am supposed to talk about… 2012/05/08 6 MPEG for the past, present and future of TV

7. Broadcasting is a great business to be in… Broadcasting is a proven business with a constant revenue flow Many technologies appear that Can help new competing businesses but Can also be used to extend the broadcasting business MPEG is a neutral body devoted to the development of standard technologies for user communities to exploit MPEG for the past, present and future of TV 7 2012/05/08

8. MPEG-H: a video standard for future broadcasting ISO/IEC 23008 High Efficiency Coding and Media Delivery in Heterogeneous Environments A suite of standards for media coding and delivery Currently scheduled to appear in 3 parts MPEG Media Transport High Efficiency Video Coding 3D Audio Timeline MPEG for the past, present and future of TV 8 2012/05/08 PartCDDISFDIS MMT12/0713/0113/07 HEVC12/0212/0713/01 3D Audio14/0414/1015/01

9. Service composition in MMT 3D Earth UD contents Bird is flying… composition information friendly with html5 legacy broadcastingfuture broadcasting MMT Package 9 MPEG for the past, present and future of TV2012/05/02

10. Clouds MMT Encapsulator - Service Component List - Component location - Composition relationship - QoE requirement - How to deliver - How to consume Web video file - Configuration - Adaptive delivery - Sync, Delay, Loss control MMT Frame pull Smart delivery in Clouds 10 2012/05/08

11. Mash-up service Service 1 (MMT Package1)Service 2 (MMT Package2) MMT assets AudioVideo1widgetWeb pageVideo2 11 2012/05/08MPEG for the past, present and future of TV

12. Second screen management Additional Information with Live Content (HTML) Video content on 2 nd device composition information … … MMT Package 12 2012/05/08MPEG for the past, present and future of TV

13. MPEG-H part 2 High Efficiency Video Coding Primary goal: ≥ 50% better compression than AVC For large displays (e.g. 4kx2k) For mobile (e.g. WVGA) Developed jointly with ITU-T by Joint Collaborative Team (JCT) on Video Coding Technically in the same stream of past MPEG Video Coding standards (AVC and prior standards) Currently at Committee Draft level MPEG for the past, present and future of TV 13 2012/05/08

14. HEVC vs AVC Subjective Performance Measurements Preliminary subjective tests carried out for HEVC Committee Draft Reasonably small and non-overlapping confidence intervals Average rate savings at same (interpolated) MOS points HEVC with 67% lower rate in class B (Full HD) HEVC with 49% lower rate in class C (WVGA) HEVC with 58% lower rate overall 2012/05/08MPEG for the past, present and future of TV 14

15. A performance sample 2012/05/08MPEG for the past, present and future of TV 15

16. For those technology aware… Block-based/Variable block sizes Block motion compensation Fractional-pel motion vectors Spatial intra prediction Spatial transform of residual difference Integer-based transform designs Arithmetic or VLC-based entropy coding In-loop filtering to form final decoded picture 2012/05/08MPEG for the past, present and future of TV 16

17. Where we are (May 2012) Activity: level of 1,000 documents per meeting crossed Very diverse participation from industry/academia Significant technical advance over prior standards Challenge: computational/implementation complexity Deliverables: Spec Reference software Conformance Currently only one profile 2012/05/08MPEG for the past, present and future of TV 17

18. HEVC Scalability Extensions/1 Serve the needs of heterogeneous environment of future digital TV distribution Multiple devices: SHDTVs, set-top boxes, tablets, smart phones, PCs etc. Wide range of processing powers, display sizes and power consumption needs Multiple networks: Cable, Satellite, Terrestrial, CDN, Cable Modem, xDSL, WiFi, 4G Wide range of channel bandwidths and QoS Multiple protocols: MPEG-2 TS, DASH, HLS, RTP, UDP Multiple services: Broadcasting, On demand, Streaming, Over The Top 2012/05/08MPEG for the past, present and future of TV 18

19. HEVC Scalability Extensions/2 Options for backward compatible migration paths From 720p60 to 1080p60 From1080p60 to 4kx2kx60 From AVC to HEVC Time Line May 2012 Preliminary Call for Proposals (public) July 2012 Final Call for Proposals (public) October 2012 Evaluation of proposals & start of collaborative design phase Development of the time line associated with standardisation phases (WD, CD/PDAM, FDIS/FDAM, DIS/AMD) 2012/05/08MPEG for the past, present and future of TV 19

20. Service Transitions 1950s: Color TV Analog, backward compatible 1990s: Digital TV New infrastructure required Transitions from SD to HD 2010s: 3D Mixed results from services introduction is Not a single format across all services 2012/05/08MPEG for the past, present and future of TV 20

21. Current Status of 3D Video Increasing investments in 3D production and services Increasingly capable 3D displays in the market (many competing and emerging technologies) Market is primarily stereo (mix of different formats being deployed right now) Focus of current MPEG work: development of a new 3D format and associated compression techniques that could facilitate generation of multiview output 2012/05/08MPEG for the past, present and future of TV 21

22. Existing 3DV Coding Formats Pack pixels from left and right views into a single frame (loss of resolution) Leverage existing infrastructure and equipment, only require additional signaling to de-interleave frame Embraced by broadcasters for first phase of 3D broadcast services Full-resolution coding of stereo and MV video as extension of AVC High coding efficiency achieved via inter-view prediction techniques 2D compatibility supported Adopted as format for 3D Blu-ray Disc Being considered for second phase of broadcast standards 2012/05/08MPEG for the past, present and future of TV 22 LeftRight

23. Targets of Future 3DV Format 2012/05/08MPEG for the past, present and future of TV 23

24. 3DV Reference Framework Limited # Video Inputs (e.g., 2 or 3 views) Depth Estimation Video/Depth Encoder Decoder and View Synthesis Larger # Output Views Binary Representation & Reconstruction Process + 1010001010001 24 2012/05/08MPEG for the past, present and future of TV

25. Overview of Call For Proposals Test Material & Conditions 8 test sequences (1920x1080 & 1024x768) 4 target bit rates per sequence 2-view and 3-view test scenarios AVC and HEVC compatible test categories 23 responses (12 AVC + 11 HEVC) Evaluation in Nov 2011 Objective quality (PSNR of synthesized views) Subjective assessment (stereo and auto-stereo) 2012/05/08MPEG for the past, present and future of TV 25

26. 3DV Testing Scenarios 2012/05/08MPEG for the past, present and future of TV 26 2-view Configuration 3-view Configuration

27. Tool Categories Texture coding Independent of depth E.g., inter-view prediction of color view, inter-view prediction of motion parameters and residual data Using depth data E.g., view synthesis prediction, motion prediction Depth coding Independent of texture E.g., depth modeling modes, weighted prediction, reduced resolution coding Using texture data E.g., motion parameter inheritance, intra prediction 2012/05/08MPEG for the past, present and future of TV 27

28. 3DV Current Status Extensive subjective evaluation of 20+ proposals to the 3DV CfP Considered both AVC and HEVC compatible Many new coding tools proposed for MV texture and depth coding Substantial rate savings compared to capabilities of existing standards AVC and HEVC extensions standardisation under wayd Add support for depth New AVC-based coding tools MV extensions of HEVC Hybrid architectures 2012/05/08MPEG for the past, present and future of TV 28

29. Hybrid Architectures From a pure compression efficiency point of view, it is always best to use the most advanced codec However, when introducing new services, providers must also consider capabilities of existing receivers and transition plan Use Case: Many terrestrial broadcast systems based on MPEG-2 US cable systems based on mix of MPEG-2 and AVC May not be easy to simply switch codecs in near term 2012/05/08MPEG for the past, present and future of TV 29

30. Multiview Extensions of HEVC HEVC Stereo/MV extensions being considered Extensions expected to be completed approximately 12 months after the base specification, i.e., Jan 2014 Reportedly simple extensions of HEVC can achieve 30- 40% bit rate reduction compared to HEVC simulcast Further gains expected if block-level changes to codec are considered 2012/05/08MPEG for the past, present and future of TV 30

31. MPEG-H part 3: 3D Audio Well, there is no Systems and Video without Audio… 2012/05/08MPEG for the past, present and future of TV 31

32. 3D Audio requirements/1 High quality: quality of decoded sound perceptually transparent Localization and Envelopment: Accurate sound localization and very high sense of sound envelopment within a targeted listening area (“sweet spot”) Flexible Loudspeaker Placement: the transmitted audio program to be mapped to a rendering setup in which loudspeakers are in alternate locations Interactivity: Interactive modification of the sound scene rendered from the coded representation MPEG for the past, present and future of TV 32 2012/05/08

33. 3D Audio requirements/2 Rendering on setups with fewer loudspeakers (including headphones): ability to derive a signal from the transmitted program material for reproduction with fewer loudspeakers Audio/visual alignment & consistency: ability to adapt the rendered acoustic scene to be consistent with the visual Efficiency for decoding on different setups: ability to be rendered on all reproduction loudspeaker setups or headphone MPEG for the past, present and future of TV 33 2012/05/08

34. 3D Audio requirements/3 Appropriate computational complexity and system latency for the target application (Broadcasting, Spatial two-(n-)way communication / telepresence Transcoding for low bandwidth devices: A lower bandwidth signal can be extracted from the original program material Backward compatibility: (e.g. to 5.1 channel programs and decoders or transmission of HRTF encoded signals) MPEG for the past, present and future of TV 34 2012/05/08

35. Envisioned Architecture Inputs Audio Channels “normal” content Audio Objects “helicopter overflight” Audio Scene Ambisonics 35 Outputs –Loudspeakers –Headphones Spatialized sound –In-frame loudspeakers “Sound Bar” Channel –500 kb/s primary channel –64 kb/s cellular channel MPEG for the past, present and future of TV2012/05/08

36. Audio, part of an Audio-visual Experience What is new in video? Larger, high-resolution displays Greater sense of envelopment Possibly closer viewing distance More efficient compression Wireless transmission to portable devices Immersive and enveloping audio experience 36 2012/05/08MPEG for the past, present and future of TV

37. Use Case 1 Home Theatre Many loudspeakers 10.1 22.2 “3D” Experience Height loudspeakers for greater envelopment Sense that audio objects are near or distant When near, they are consistent with video image 37 2012/05/08MPEG for the past, present and future of TV

38. 22.2 multichannel sound system (NHK) 2012/05/08MPEG for the past, present and future of TV 38

39. Home Theater – Issues Maintain compatibility with legacy systems: 5.1, stereo Will content providers really step up to 22.2? Will consumers adopt 22.2?: 2 front high How to compress 22.2 for transmission? MPEG AAC 64 kb/s/channel: -> 1.5 mb/s How to render for legacy setups? Relatively slow-growth market 2012/05/08MPEG for the past, present and future of TV 39

40. Use Case 2 Personal TV Small but high-resolution display “super-tablet” Local wireless communications To cable or fiber home hub Possible audio presentation Headphones Loudspeakers around perimeter of display Issue: To what extent can the user get an enveloping experience from only “front” loudspeakers? 40 MPEG for the past, present and future of TV2012/05/08

41. Use Case 3 Mobile TV Hand-held display: Smartphone Headphone listening: Stereo, perhaps with binauralisation Fastest growing market: Quick rollout and adoption of standard What is new? USAC for stereo; MPEG Surround for binauralisation What is the “Wow” factor? To spur adoption of new technology 41 MPEG for the past, present and future of TV2012/05/08

42. User Experience 42 Home Theatre Tablet TV Spatialized on headphones 2012/05/08MPEG for the past, present and future of TV

43. Flexible Rendering Content providers create 22.2 program Consumed on stereo, 5.1, 10.1, 22.2 layouts Consumed on “wrong” layouts Mis-positioned surrounds Missing surrounds Non-standard layouts 2 front high, 5.1 mid 43 MPEG for the past, present and future of TV2012/05/08

44. 3D Audio Test Platform NHK “Loudspeaker Array Frame” (LAF) Suggestions for Alternative platform for assessing “Home Theatre” use case Model for assessing “Personal TV” use case 44 MPEG for the past, present and future of TV2012/05/02

45. Summary Provide compelling immersive audio experience for audio/visual programs Create content once, present on many different loudspeaker layouts or on headphones Provide high compression and high-quality audio presentation 45 MPEG for the past, present and future of TV2012/05/08

46. MPEG-V: Humans have more than two senses 2012/05/08MPEG for the past, present and future of TV 46

47. MARBle Media Servers Service Servers User Local Sensors & Actuators Remote Sensors & Actuators ARAF file or stream Local Real World Environment Local Real World Environment Remote Real World Environment Remote Real World Environment Augmented Reality Application Format (ARAF) MPEG for the past, present and future of TV2012/05/08 47

48. Augmentation Region in ARAF Augmentation Region Broadcaster AR service provider A AR service provider B User A User B MPEG for the past, present and future of TV2012/05/02 48

49. The past, the present and the future, again 25 years ago MPEG selected the digital media way bringing innovation and interoperability …and thrived This year the broadcasting industry has started a move in future broadcast services with interoperability at its core …and it is bound to thrive MPEG looks forward to continuing a cooperation with the broadcasting industry providing the necessary standard technologies 2012/05/08MPEG for the past, present and future of TV 49

50. 50 2012/05/08MPEG for the past, present and future of TV http://mpeg.chiariglione.org/