1. Comparison of subjective test methodologies VQEG Berlin meeting June 2009 P. Le Callet, R. Pépion

2. Context, methodologies and issues Context: HRCs (coder, processing, transmission …) Resolutions Applications and services ACR (5, 11 categories …) Pair Comparison SAMVIQ DSCQS The value (e.g. accuracy, stability) of protocols might depend on the context … and the targeted goals

3. Outline Study 1: ACR 5 vs SAMVIQ HD H264 Study 2: Preference Tests vs SAMVIQ «processing» Study 3: ACR5 vs ACR 11 vs SAMVIQ encoded + processing Study 4: ACR5 for encoded + transmission error

4. Study 1: ACR 5 vs SAMVIQ HD Motivations: HDTV high quality in a short range => quality measure should be precise and discriminative Absolute Category Rating (ACR) - random order - only one viewing - category scale - no explicit reference... Good Subjective Assessment Methodology for Video Quality (SAMVIQ) - user-driven order - multiple viewing (natural?) - continuous scale - explicit reference

5. 5 Previous and new studies [Brotherton, 2006] correlation on CIF (352x288): CC(MOS ACR, MOS SAMVIQ ) = 0.94 New studies: - Resolutions: QVGA, VGA and HD 1080i50 (viewing distance according to the resolution) - HRC: coding artefacts only (H264 AVC and SVC) CC(MOS ACR, MOS SAMVIQ ) = HDTV VGA QVGA 13° 19° 33° 0.969 0.942 0.899 6.73 9.31 14.06 visual field RMSDiff= ACR and seems to provide “equivalent” results up to a certain resolution

6. 6 Accuracy vs Number of observers confidence interval number of observers 24 « Suitable methodology in subjective video quality assessment: a resolution dependent paradigm » Stéphane Péchard, Romuald Pépion and Patrick Le Callet Proceedings of the Third International Workshop on Image Media Quality and its Applications, IMQA2008, Chiba, Japan, September 2008

7. Outline Study 1: ACR 5 vs SAMVIQ HD H264 Study 2: Preference Tests vs SAMVIQ «processing» Study 3: ACR5 vs ACR 11 vs SAMVIQ encoded + processing Study 4: ACR5 for encoded + transmission error

8. Study 2: Preference Test vs SAMVIQ « processing » Motivations: HDTV pre post processing, comparison between format on a 1080p display = > No other impairments 1080p SRC Pre Processing (interleaced and down Scaling) 1080i, 720p Pre Processing (deinterleaced + down Scaling) 720p Post Processing deinterleaced + up Scaling) Post Processing (up Scaling) 1080p PVS

9. Study 2: some results SAMVIQ Preference Test 1080p SRC compared to other PVS 7 categories preference test Generally good agreement but …further analysis is required (Thurstone Mosteller, CI …)

10. Outline Study 1: ACR 5 vs SAMVIQ HD H264 Study 2: Preference Tests vs SAMVIQ «processing» Study 3: ACR5 vs ACR 11 vs SAMVIQ encoded + processing Study 4: ACR5 for encoded + transmission error

11. Study 3: ACR5 vs ACR 11vs SAMVIQ encoded + processing Motivations: Comparison of 1080p50 with other HD and SD formats on a 1080p display => compression + processing Compression: H264 coder All formats (e.g. 1080p or i, 720p …) are coded at 3,6 and 9Mb/s and decoded before post processing. Processing: All formats are displayed in 1080p50 after decoding 1 deinterlacer : Smooth (VirtualDub/Avisynth), 2 Upscalers : Bilinear and Lanczos (VirtualDub/Avisynth).

12. Study 3: PVS generation 1080i50 720p50 1280x 1080i50 1280x 1080p50 SD Deint Upscale 1 Upscale 2 Deint Upscale 1 Upscale 2 Deint Upscale 2 Upscale 1 Upscale 2 29 HRC (8x3 HD +2x2 SD +1Ref) x 3 SRC = 87 PVS 3Mb/s 6Mb/s 9Mb/s Not for SD Upscale 1

13. ACR5 vs ACR11: correlation correlation between ACR 5 and 11: 0.98

14. Study 3: SAMVIQ vs ACR11, PVS generation X 1080i50 720p50 1280x 1080i50 1280x 1080p50 SD Deint Upscale 1 Upscale 2 Deint Upscale 1 Upscale 2 Deint Upscale 1 Upscale 2 10 HRC (8HD +1SD +1Ref) x 2 SRC = 20 PVS

15. Study 3: ACR11 vs SAMVIQ (on 20 PVS) Good correlation between ACR and SAMVIQ (0.97) => may be questionnable for high quality score

16. Study 3: score distribution ACR5 ACR11 SAMVIQ

17. Study 3: CI distribution

18. Outline Study 1: ACR 5 vs SAMVIQ HD H264 Study 2: Preference Tests vs SAMVIQ «processing» Study 3: ACR5 vs ACR 11 vs SAMVIQ encoded + processing Study 4: ACR5 for encoded + transmission error

19. Study 4: ACR5 encoded + transmission error The goal : analyse the relation between the position of the transmission error and the MOS on SD sequences. Each content is coded at 4 or 6Mb/s and some simulation of transmission errors are tested. Advanced FEC and Error concealment technique (ROI based)

20. Study 4: ACR5 encoded + transmission error X 14 HRC (Trans- Mission Errors) = 84 PVS

21. Study 4: ACR5 encoded + transmission error Reminder: coding only (study 3)