1 JEG hybrid model Iñigo Sedano June, 2010

2 Three years working at Tecnalia Technology Corporation, Telecom Unit, Broadband networks group, Spain ( Since November 2009 staying at Acreo, Sweden with a specialization grant from Tecnalia and Fundación Centros Tecnológicos Iñaki Goenaga (Basque Government). Pursuing PhD at University of the Basque Country, Spain.

3 JEG wiki and forum JEG wiki, hybrid section jeg.org/index.php5?title=WP3:_Hybrid_Modelhttp://wiki.vqeg- jeg.org/index.php5?title=WP3:_Hybrid_Model JEG forum, WP3:Hybrid model section

4 My personal interest: effect of transmission distortions in the video quality A initial state of the art review led to the identification of important parameters. Hybrid model – Transmission distortions

5 Amount of information lost Packet losses: percentage of packets and bytes lost, or NAL (Network Adaptation Layer) units. An error in an I frame causes more distortion than in a P or B frame. An IDR frame stops the error propagation. I-frame does not reference past frames. I / IDR detector. GOP (Group of Pictures) size. Temporal position of the loss in the GOP. Hybrid model – Transmission distortions

6 If there is a scene change usually many macroblocks are intra coded (no references to previous frames). The effect of a loss is different if it occurs just before, or just after the scene change. Scene change detector Percentage of frame area lost: slice structure and packetization: NAL units per packet or packets per NAL (if fragmented). Packets per frame. Number of macroblocks per slice. Hybrid model – Transmission distortions

7 The spatio-temporal complexity can have error masking effects, and reduce the error concealment effectiveness. Texture can mask errors and motion makes more difficult to conceal them. Measure the spatio-temporal complexity. Precise distortion tracking could be done at macroblock level. Position and percentage of affected macroblock, type of macroblocks and motion vectors. Hybrid model – Transmission distortions

8 Visual attention has a strong influence in the perception of the errors. For example if a distortion affects a human in the scene the perception will be higher. Visual attention algorithms. Variety of error concealment techniques. Hybrid model – Transmission distortions

9 The transmission distortions have different perceptual impact depending on the quality of the video. A transmission distortion can “mask” a compression distortion. Both types must be correctly combined. The temporal distribution of the distortions has a direct effect on the perceived quality: Frequency, clustering degree (how close they are) and forgiveness effect (distance in seconds to the end of the sequence). Hybrid model – Transmission distortions

10 Video content dependence (visual attention, scene changes, spatio-temporal complexity). Decoder error concealment technique, including decoder frame skipping. Temporal alignment between bitstream parameters and PVS. Computational complexity. Limited availability of video databases with subjective tests and different test conditions. Hybrid model main issues

11 The state of the art review led to the following diagram that shows the identified parameters classified by complexity and distortion type (compression or transmission). Hybrid model parameters

Packet Loss: - Percentage of packets and bytes lost Picture, slice & MB QP I / IDR frame detector: - GOP size - Position of loss in GOP Temporal distribution of compression and transmission distortions Computational complexity Video quality Frame type Transmission distortionsCompression distortions … Parameters classified by complexity

Slice structure and packetization: - Percentage of frame area lost Spatial + temporal complexity Visual attention Distortion tracking: - Position and percentage of affected macroblocks (pixels) -Type of macroblock - Motion vectors Scene change Error concealment analysis: -Type of error concealment - Frame freezing - Frame skipping Computational complexity Transmission distortions

14 Modelling A possible model: Estimate the quality of the video (compression distortions) with the QP. Correlate the bitstream parameters with the decoded video. Track the transmission distortions at the macroblock level using motion vectors. Locate the affected pixels. Compute the effect of the affected pixels in the quality with the decoded video (frame freezing, frame skipping, texture, motion, visual attention, scene change). Estimate a MOS from the temporal distribution of compression and transmission distortions.

15 Hybrid model implementation Parse huge XML HMIX1 and HMIX2 files, 500 MB or more… cElementTree Python parser is quick enough. At the moment the model is based on QP (if stable), as a starting point. Publication from Marcus Barkowsky: Analysis of Freely Available Subjective Dataset for HDTV including Coding and Transmission Distortions, Fifth International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM–10), Scottsdale, Arizona, January 13–15, 2010.

16 Database for papers? A database containing the papers sorted by some parameters could be created, but would need to be filled and updated. Author Name Reference type Video bitrate Video size Video type Model parameters Correlation achieved Subjective test conditions

17 How to collaborate: Identify important parameters Recommend or implement papers Ideas for model Information can be found in the JEG wiki: jeg.org/index.php5?title=WP3:_Hybrid_Model jeg.org/index.php5?title=WP3:_Hybrid_Model Any questions? My Thank you!