1. 1. hybrid - bitstream models VQEG Berlin Meeting June 2009 for Discussion Jens Berger, Silvio Borer

2. 2. hybrid - bitstream models We should design the test-plan according the following three main steps: 1.Define Scope of the competition → see introduction of testplan (compare with introduction of HDTV testplan). 2.Specify in more detail a) model types and b) model input → see chapter 7 3.Decide about sequence processing and subjective tests General proposition: put decisions directly into the test plan (and not on slides).

3. 3. scope 1.What are the real-world applications hybrid/bitstream models are targeted at? - broadcasting, end-user or studio, lab applications - live monitoring in network (bitstream) / at end-user (hybrid no reference) - lab application (full-reference hybrid) 2.How do the models fit into real-world applications - e.g. if bitstream data contains exactly pvs → how to apply such a hybrid model in real world - is live monitoring a subsequent evaluation of the 8s video sequences analysis?

4. 4. model input encode decodetransmit SRC PVS pcap (bitstream data) 10 s 16 s 18 s 11 s What is used as model input ? - is pvs for model input equal to pvs for subjective test? - bitstream data for bitstream/hybrid model input: contains exactly pvs data? - in subj. test: src and all pvs have exactly the same length?... see next slide... 10 s

5. 5. model input and seq. processing 1.... with reference to previous slide Option1: SRC = 16s src (green box left) PVS = complete transmitted SRC (green box right) incl. codec convergence PCAP = complete bitstream data But: how to apply such a model in real world applications, e.g. with a 5min long stream? Option2: SRC = 10s (blue box) PVS = 10s (red mark) PCAP = ? [ for bitstream model, bitstream data should coincide with the PVS shown in subjective tests] 2. How to check compliance of bitstream data? If 1 reference decoder is used, check this: make database of sample PVS, PCap files and check if reference decoder can handle it (including transmission errors!). 3. Should we leave the evaluation of 10s chunks and go for continous voting of 120s?

6. 6. testplan: open points 1.Introduction: define scope → application areas for models, publication of subjective -, objective data, video databases? restrictions on use of encoders/decoders. 2.chapter 5: test schedule might be tight (e.g. step 12: SRC selection 2 weeks, then 14: PVS creation in 1 month, or 23.-25: 6 weeks from statistical analysis to approval of final report) 3.chapter 6: - 6.1.3: software tools for sequence processing: tools to use for bitstream recording, specify a reference decoder, capturing of PVS - 6.1.5 de-interlacing: like in MM or HD project - test setup should follow scope of the model (i.e. broadcasting, end user experience) - test setup should follow widely the existing MM and HD setups 4.chapter 7: - Specify in detail model input - model input format, section 7.2: naming conventions for bitstream file, format,... - move 7.4 to 6.3 5.chapter 8: Evaluation: - use evaluation like in HD testplan, i.e. use RMSE as main error measure - make subset analysis for samples with/without packet loss, with/without rebuffering - use standard full-reference methods as comparison? - 8.3.3 averaging process implies use of full-matrix design