1 Hybrid Bit-Stream Models (January 24-26, 2009 San Jose)

2 Issues to be determined Are we going to use numerical values attached to each category? Decision: TBD

3 Issues to be determined Are we going to mix 10s and 16-24s SRC/PVS? PVS length: QCIF/QVGA –Variable for rebuffering: 16(SRC)/upto24(PVS) –Fixed: 10 sec –Rebuffering is allowed Decision: Don ’ t mix 10s and 16-24s. TBD: Further study on 16-24s SRC/PVS (e.g., single evaluation values for 24 sec, user response to various length of PVSs). May propose a special test for rebuffering, including coding and transmission error impairments.

4 Issues to be determined Evaluators should be seated facing the center of the video display at the specified viewing distance. That means that subject's eyes should be positioned opposite to the video display's center (i.e. centered both vertically and horizontally) Decision: Will be copied from the HD test plan. One viewer for QCIF/QVGS. 24 viewers for each test after screening (similar to MM). Ref: Hoffman’s papers (IEEE)

5 Issues to be determined Are we going to include audio? Decision: No audio.

6 Issues to be determined Display Monitors –QVGA/QCIF: LCD –SD: professional CRT or LCD –HD: LCD or professional CRT Decision: –QVGA/QCIF: LCD –SD: professional CRT –HD: LCD or professional CRT –PDP was voted not to be used.

7 Issues to be determined Are we going to use the same LCD panel if LCD display is used? List of admissible LCD monitors? Decision: Using the same LCD panel is not required. –QCIF/QVGA: TCO 06 or later version –CRT: professional quality –LCD for HD: same as in the HDTV project

8 Issues to be determined For the HD/SD testing, the video will use the full screen dimensions and no background panel or black border will be present. Decision: For the HD/SD testing, the video will use the full screen dimensions and no background panel or black border will be present.

9 Issues to be determined How long will be the session length? How many PVSs are there per session? Decision: For 10s PVS, the number of the total PVSs is about 168. For 15s PVS (SD/HD), the number of the total PVSs is about 130. New vote: 10s (5), 15s (5) => We will use 15s SRC/PVS for SD/HD. For 16-24s PVS, TBD

10 Issues to be determined Randomization: maximum number of viewers per order? Decision: The max number of views for each order is 6 assuming 24 viewers for SD/HD. (full randomization is recommended if possible). For QCIF/QVGA, different ordering for each viewer as in the MM project.

11 Issues to be determined Test design? Full matrix? Decision: TBD

12 Issues to be determined SRC admissibility? (e.g., at least 4 in ACR scale or ILGs will decide?) Decision: “ Good ” or better. Final decision will be made by ILGs.

13 Issues to be determined SRC length –Option #1: extra 2 seconds or 20 frames –Option #2 (see test plan) Decision: extra 2 seconds at the beginning and end for 10s and 15s SRC/PVS.

14 Issues to be determined Cropping & Rescaling (help from Margaret?) Decision: Margaret will revise the table 2 (page 29).

15 Issues to be determined Test materials –MM sources –RRNR sources –NTIA HD –NTIA SD –ITU SD (some of them) –SVT HD –Other materials (Elephant ’ s Dream, Big Buck Bunny) FYI

16 Issues to be determined Codec –H.264 –MPEG2 (SD & HD only) –MPEG4 (QCIF & QVGA only) HRC (bitrate, frame rate) –QVGA/VGA: same as MM –SD: same as RRNR –HD: same as HD Decision: –Codec: H.264 (HD, SD, QVGA), MPEG2 (SD) HRC (bitrate, frame rate) => agreed the following conditions. –QVGA/VGA: same as MM –SD: same as RRNR –HD: same as HD

17 Issues to be determined Max. length of freezing or skipping (10s PVS)? –40% of the total length? Decision: maximum of freezing within 10s: 3s (mandatory 5s) maximum of skipping within 10s: 3s (mandatory 5s) max total loss (including beginning and end): 1s (mandatory 2s) (max total loss=max loss at begin + max loss at end) max total extra (including beginning and end): 1s (mandatory : the entire PVS must be contained in the SRC). Anything can happen in-between (freezing with/without skipping, skipping as long they meet the aforementioned conditions).

18 Issues to be determined Max. length of freezing or skipping (15s PVS)? –40% of the total length? Decision: maximum of freezing within 15s: 3s (mandatory 5s) maximum of skipping within 15s: 3s (mandatory 5s) max total loss (including beginning and end): 1s (mandatory 2s) (max total loss=max loss at begin + max loss at end) max total extra (including beginning and end): 1s (mandatory : the entire PVS must be contained in the SRC). Anything can happen in-between (freezing with/without skipping, skipping as long they meet the aforementioned conditions

19 Issues to be determined Are we going to use the 11 scale ACR (the same decision as the HD project)? Decision: TBD

20 Issues to be determined Are we going to allow proponents to conduct subjective tests? Decision: Yes

21 Issues to be determined How about model entry (e.g., 1. per codec, per resolution)? SD/HD (H.264, MPEG2): QCIF/QVGA (H.264, MPEG4): –=> Different model entry for each codec –=> Four models (each model also include FR, RR, NR). Decision: Three codec-resolution combinations. H.264 (HD/SD): FR, RR, NR H.264 (QVGA): FR, RR, NR MPEG2 (SD): FR, RR, NR A different model is allowed for each codec-resolution. In other words, a proponent may submit model(s) for any one or all of the three types.

22 Number of subjective tests (Proponents) BT 1 (QVGA, SD625, HD50i, HD25p) Ericsson1 (QVGA) DT 1 (HD25p) Ghent Univ. 1 (QVGA, HD30p, HD25p) KDDI 2 (QVGA, SD525, HD60i, HD30p) Lancaster Univ 1 (QVGA) LINK 0 NTT 2 (SD525, HD60i) Opticom 1 (QVGA) Psytechnics ? Symmetricom 1 (SD525, HD60i, HD30p) SQ 1 ? Tektronix Yonsei 3 (QVGA, SD525, HD60i, HD30p) Total: 15 (QVGA, SD, HD50, HD60) HD 60i HD 30p HD 50i HD 25p SD 525 SG 625 QVGA

23 Number of subjective tests (ILG, Jan 2009) CRC 0 INTEL 1 (QVGA, HD60i, HD30p) Acreo 0 or possibly 1 (QVGA, SD625) IRCCyN 1 ? Nortel 0 FUB 1 (QVGA, SD625, HD50i, HD25p) NTIA 0 Verizon 1 (QVGA, SD525, HD60i, HD30p) NOTE: ATIS option may be considered. 4 or 5

24 Issues to be determined Model typeInput Hybrid FRbit stream data, PVS, SRC Hybrid RRbit stream data, PVS, features extracted from SRC Hybrid NRbit stream data, PVS NRPVS Parametric (assumed to be equivalent to P.NAMS (Q12.14) with video only) TS, RTP, UDP, IP headers & (optional) longer freezing Decision: Send a liaison to SG12 about the preferred input requirements and accept their official position. Bitstream 1 (assumed to be equivalent to P.NBAMS with video only, no decoding) TS, RTP, UDP, IP headers, ES information & (optional) longer freezing. Decision : Send a liaison to SG12 about the preferred input requirements and accept their official position. Bitstream 2 (assumed to be equivalent to P.NBAMS with video only, decoding allowed) TS, RTP, UDP, IP headers, ES information & (optional) longer freezing. Decision : Send a liaison to SG12 about the preferred input requirements and accept their official position. Model types and input requirements

25 Issues to be determined Decision –Number of reference decoders (for compatibility check): 1 reference decoder per codec. –Number of encoders: any encoders compatible with the reference decoder. It is preferred that more than one encoder is used. –Number of decoders (for subjective tests and inputs to hybrid models): any decoders compatible with the reference decoder. It is preferred that more than one decoder is used. Encoders & Decoders

26 Issues to be determined Issues with P.NAMS & P.NBAMS Decision: VQEG Parametric (video only) will be validated together with Hybrid if possible depending on the answer from SG12. VQEG Bitstream 1 & 2 (video only) will be validated together with Hybrid if possible depending on the answer from SG12.

27 Issues to be determined Currently, there codecs are considered (H.264, MPEG2, MPEG4). –SD/HD: MPEG2, H.264 –QCIF/QVGA: MPEG4, H.264 Side Channels for RR –HD (56 kbps, 128 kbps, 256 kbps) <= HDTV project –SD (15 kbps, 80 kbps, 256 kbps) <= RRNR-TV –QCIF (1 kbps, 10 kbps) <= MM –CIF (10 kbps, 64 kbps) <= MM Side Channel [TBD] –SD/HD: 15 kbps, 56 kbps, 128 kbps, 256 kbps –QVGA/QCIF: 1kbps, 10 kbps, 64 kbps Took voting and decided option #1 –Option #1 H.264 (SD/HD), MPEG2 (SD), H.264 (QVGA) => Ten votes –Option #2 H.264 (SD/HD), MPEG2 (SD), H.264 (QVGA), MPEG4 (QVGA) => Thee votes Decision: Three codec-resolutions will be tested –H.264 (SD/HD) –MPEG2 (SD) –H.264 (QVGA)

28 Issues to be determined  Are we going to use numerical values attached to each category?  Continuous vs. discrete?  Further study on 16-24s SRC/PVS (e.g., single evaluation values for 24 sec, user response to various length of PVSs). May propose a special test for rebuffering, including coding and transmission error impairments. Decision:

29 Issues to be determined  LCD monitors for SD Decision:

30 Decision from the last meeting (Sep. 2008, Ghent)

31 Test conditions PVS length: SD/HD => 16 –Rebuffering is NOT allowed PVS length: QCIF/QVGA –Variable for rebuffering: 16(SRC)/upto24(PVS) –Fixed: 10 sec –Rebuffering is allowed Definition: Rebuffering is freezing longer than 0.5s without skipping Maximum delay/advance in PVS? (+/ sec): other conditions will be identical as in the multimedia project [TBD] Decision: the maximum time limit for freezing or rebuffering is 8 seconds forQCIF/QVGA.

32 Subjective Test ACR HR (tentatively) for both QVGA/VGA and SD/HD Graphical, continuous ACR version may be also used. Numbers will be used along with semantic terms unless P.910rev is approved. Proposal. (1) Vittorio: Some concerns about ACR-HR, new subjective testing method. => prefer DSCQS, DSIS, at least 10 steps (2) Silvio: Propose to use a scale from 1 to 5 with steps of 1/10, Propose to use MOS (or MOS_adj) as a target value Decision: 11 scale ACR-HR(P.910)=10, DSCQS=1

33 Model Inputs  Model A (P.NAMS): TS, RTP, UDP, IP headers & (optional) longer freezing Model B (P.NBAMS): TS, RTP, UDP, IP headers, ES information & (optional) longer freezing – Decoding is not allowed?  Hybrid models (FR/RR/NR) : bit stream data (TS, RTP, UDP, IP headers, ES information) & PVS (SRC if required)

34 Proponents & Preferred input types TextBit-stream (trace dump with arrival time) Don’t know OPTICOM Yonsei NEC Tektronix Ghent Univ. Symmetricom NTT Lancaster Univ BT KDDI Psytechnics SQ (1) Hybrid models

35 Proponents & Preferred input types Bit-stream (trace dump with arrival time) Don’t know NTT Telchemy Tektronix Ericsson DT Ghent Univ Symmetricom BT KDDI Psytechnics NEC (2) P.NAMS: Use only IP, UDP, RTP and TS headers Note: Potential proponents should declare their intensions to SG12 by Sep. 30.

36 Proponents & Preferred input types TextBitstreamDon’t know Ericsson Ghent Univ DT Symmetricom NTT BT KDDI Psytechnics (3) P.NBAMS

37 ILGs CRC INTEL Acreo IRCCyN Nortel FUB NTIA

38 Reference Decoder [TBD] Reference decoder to ensure error-free bit stream compliance to the standard (open source) –Open-source H.264 codec (JM) Alternative: x264 –Open-source MPEG2 (FFMPEG) –Open-source MPEG4 - momusys, Xvid?? NOTE: Proponents should provide a working system of entire chains (encoder, container, server, capture program, decoder, etc)

39 Capturing bit stream data [TBD] File format: PCAP, RTP dump, MS Network Analyzer format (.cap). Capturing programs: Wireshark, Tcpdump, MS Network Analyzer format Protocol: TCP/IP, UDP??

40 Streaming Server [TBD] Representative streaming server –Live555: working for MPEG2 –VLC: working for H.264 and MPEG4 –Dvbstream: working for ??? –Helix: working for ??? –Ffserver: MPEG2 and MPEG4, H.264(?)

41 Inputs to the model Text file (Hybrid) Bit stream data (Bitstream/Hybrid)

42 Inputs to the model

43 Issues with Inputs to the model [TBD] If bit-stream data which contains transmission errors can ’ t be interpreted by some models, what will we do about the models?

44 IP analyzer: converter & decoder We need to convert the captured bit stream data so that the reference decoder can decode it. programs??? (TCP replay, off-line simulator) Free or open-source SW?? Hire professional developer??

45 IP analyzer Any bit stream data which can be understood by the reference IP analyzer will be admissible. Models are required to handle the bit stream data. Bottom Line: We need working programs for reference encoder, container, server, packet capturer, extractor and reference decoder so that the reference decoder can decode the captured bit stream data. Also the transmission error information should be outputted.

46 Data Analysis Same as the multimedia project – tentatively agreed.

47 Target usage (PLR): TBD SD/HD: (information from Broadband forum may be useful) QCIF/VGA - internet: - mobile: 0-15%

48 Drafting test plan A draft of the hybrid test plan will be done by the co-chairs (JB, CL) and editors (David Hands, Nicolas Staelens, Vittorio Baroncini, Yves Dhondt) by borrowing heavily from the multimedia and HD test plans. The draft will be presented to VQEG for approval.

49 Schedule Proposal for a working system (encoder, server, capturing program, decoder): next VQEG meeting (Jan, 2009) Finalization of test plan: VQEG meeting after the reference IP analyzer is available: VQEG meeting (July, 2009) Model submission: Six months after the test plan is finalized.

50 Remaining issues [TBD] Subjective test –Multimedia project approach: proven, but delays may happen (about 6 month delay after model submission). Easy to correct errors –HDTV project approach: Fast, but there can be impasse if serious PVS problems occur. Difficult to correct errors.

51 Number of subjective tests (Proponents) BT 1 Ericson1 DT 1 Ghent Univ. 1 KDDI 2 Lancaster Univ 1 NEC 1 NTT 1 Opticom 1 Psytechnics 1 Symmetricom 1 SQ 1 Tektronix Yonsei 3 Total: 16

52 Number of subjective tests (ILG) CRC 0 INTEL 0 Acreo 0 or possibly 1 IRCCyN 1 Nortel FUB 1 NTIA 1? NOTE: ATIS option may be considered.

53 Remaining issues [TBD] Calibration limit: currently same as HDTV & Multimedia (will be decided over the reflector)

54 Remaining issues [TBD] Bit-stream models –P.NAMS (audio/video, video only) –P.NBAMS (audio/video, video only) –P.NBAMS with some additional information from decoder such as freeze frames (audio/video, video only) audio/video(P.NAMS) vs. Hybrid (video only) Decision: Initially video only. If enough audio(with video) subjective data is available, models for audio and audio/video will be also validated.