MPEG-21 Scalable Video Coding and the Stool (UniBS) system Riccardo Leonardi, Alberto Signoroni, Michele Brescianini, Nicola Adami Signals and Communications Lab. – DEA University of Brescia, Italy riccardo.leonardi@ing.unibs.it alberto.signoroni@ing.unibs.it Thessaloniki – October 7-8, 2004

R. Leonardi - A. Signoroni University of Brescia, Italy Outline Scenarios (why SVC is desirable) MPEG Requirements (the “shall” and “should” of SVC) Technologies which enable SVC SVC architectures (t+2D, 2D+t, UniBS) Experimental results R. Leonardi - A. Signoroni University of Brescia, Italy

Video coding paradigms Classical video coding Code-decode at a single operating point (target frame resolution, spatial resolution and quality) Code-transcode at operating points different from the target (complexity, performance loss, cost) Scalability in current standards (few spatio-temporal operating points MPEG-2, coarse or fine grain quality scalability MPEG-4, performance loss) Scalable video coding Code once decode at S x T x Q operating points (full scalability in space, frame-rate and quality) R. Leonardi - A. Signoroni University of Brescia, Italy

SVC scenarios and applications Multi-channel content production and delivery: the same stream will be viewed on a variety of devices with different spatial and/or temporal resolution will be viewed and charged on the basis of delivered quality level devices can decide to trade-off received quality for a longer battery life (complexity scalability) Video streaming over heterogeneous IP networks Advanced surveillance systems Video conferencing Mobile streaming video Others... R. Leonardi - A. Signoroni University of Brescia, Italy

MPEG-21 Ad Hoc Group on SVC at Pattaya Meeting (Dec.2001) MPEG started an AHG to explore Interframe Wavelet Coding – activity successfully completed in Oct.2002 RPI software (prof. J.W.Woods) released based on MCTF+EZBC MPEG started an AHG on Scalable Video Coding (MPEG-21) Call for Evidence on SVC: 9 responses received by July 2003 Call for Proposals on SVC: 14 full proposals (Wavelet based, AVC based) Wavelet based: best selected Microsoft Research Asia (software released) AVC based: best selected HHI-Fraunhofer Core Experiments started at the Monaco (Mar.2004) and Redmond (Jul.2004) meetings. No reference architecture yet specified (MRSA, HHI, Samsung, UNIBS, DANAE-TOMPHSON, UNSW proposals at the next meeting Palma de Mallorca Oct.04) R. Leonardi - A. Signoroni University of Brescia, Italy

Requirements shall should may Last revision: MPEG output doc N6505, Redmond, July 2004 shall should may Spatial scalability 2 levels (e.g. CIF+QCIF) 4 levels From QQCIF to 3610x1536 Temporal scalability (up to 60 Hz) Non-dyadic Quality (SNR) scalability Coarse grain (25% bit-rate) Medium grain (10% bit-rate) FGS or progressive R. Leonardi - A. Signoroni University of Brescia, Italy

R. Leonardi - A. Signoroni University of Brescia, Italy Requirements contd. Combined Scalability (at least for a finite number of points in the spatial-temporal-quality space) Challenging Reqs Coding Efficiency performance (superior to H.264/AVC) Base-layer compatibility (to H.264/AVC) Complexity scalability (for power limited devices) Others (less peculiar to SVC) ROI and object scalability, robustness to transmission errors and on “best-efforts” networks, end-to-end delay, random access capability) Figura a toni di grigio R. Leonardi - A. Signoroni University of Brescia, Italy

Req. + Core Experiment (oct’04) Multiple Adaptation Scalability: “decoding paths” Figura a toni di grigio R. Leonardi - A. Signoroni University of Brescia, Italy

A generic SVC encoder Esempio con immagini e ricordo di contour preservation con nota critica sulla dinamica Sergio p.55 In order to have an efficient scalable codec, it is necessary that each one of the building blocks is operating in a scalable mode. MPEG-21 doc N6520 Scalable Video Model 2.0 July 2004, Redmond, WA, USA R. Leonardi - A. Signoroni University of Brescia, Italy

SVC Enabling Technologies In the MPEG21-SVC framework a series of SVC enabling technologies have been identified Spatial transform: DWT (Discrete Wavelet Transform) 9x7; IntegerDWT 5x3; Decimation/Interpolation; Block based transforms Temporal transform: Haar MCTF (Motion Compensated Temporal Filtering); 5x3 MCTF; UnconstrainedMCTF; … Motion coding: non-scalable; multires. quality scalable explicit (inside the method) / implicit (on the data structure); hierarchical B-frames; … Texture coding: EBCOT, 3D-EBCOT; EZBC; layered H.264/AVC; SNR scalable CABAC based entropic coding; …others (e.g. embedded wavelet coders). Figura p.58 sergio R. Leonardi - A. Signoroni University of Brescia, Italy

R. Leonardi - A. Signoroni University of Brescia, Italy SVC implementations Following the Call for Proposal for SVC (MPEG doc.N5958 Oct.2003) a certain number of implementations have been presented (MPEG doc.M10569 Mar.2004). On the base of the CfP inputs a SVC classification has been made which allows to distinguish some families of SVC implementations: “t+2D” schemes “2D+t” schemes “2D+t+2D” schemes or “Multi-scale Pyramid t+2D” Our STool scheme could be classified in the third category Figura p.58 sergio R. Leonardi - A. Signoroni University of Brescia, Italy

R. Leonardi - A. Signoroni University of Brescia, Italy t+2D schemes Estensione 3d e proprietà Enables temporal scalability Enables spatial scalability Enables quality scalability R. Leonardi - A. Signoroni University of Brescia, Italy

R. Leonardi - A. Signoroni University of Brescia, Italy 2D+t schemes Estensione 3d e proprietà R. Leonardi - A. Signoroni University of Brescia, Italy

UniBS SVC scheme: “STool” Estensione 3d e proprietà 2 SYSTEMS which differs in the EC part 3D-ESCOT (MSRA software) GOF-EMDC (UNI-BS) MEC (Motion Estimation and Coding) + MCTF (MSRA software) R. Leonardi - A. Signoroni University of Brescia, Italy

Results: CITY QCIF 15 Hz 64 kbps Original AVC STool zoom zoom Criteri di filtraggio e esempio slices R. Leonardi - A. Signoroni University of Brescia, Italy

R. Leonardi - A. Signoroni University of Brescia, Italy Results: CITY CIF 30 Hz 256 kbps Original AVC Stool Criteri di filtraggio e esempio slices R. Leonardi - A. Signoroni University of Brescia, Italy

Results: AVC vs MSRA vs Stool Play QCIF 15Hz 96kbps CIF 30Hz 384kbps HARBOUR Criteri di filtraggio e esempio slices QCIF 15Hz 64kbps CIF 30Hz 256kbps CITY R. Leonardi - A. Signoroni University of Brescia, Italy

R. Leonardi - A. Signoroni University of Brescia, Italy Conclusions SVC systems Multipoint spatial-temporal-SNR scalability Comparable or superior in coding performance with respect to AVC A lot of work is currently made on SVC (AVC is mature, SVC is young… :-) STool STool: a new 2D+t+2D system with good performance (UNIBS submission to the next MPEG meeting) Rivedere i prodotti di e la rappresentazione ad albero, commentare i vari utilizzi, inserire immagini (e.g. sclerosi multipla nell’object analysis, reni nella semplificazione…) R. Leonardi - A. Signoroni University of Brescia, Italy