1. Towards a unique subjective experiment dataset for 3DTV – « 3DTV phase 1 » Marcus Barkowsky

2. Overview Purpose and scope Presentation of (additional) SRCs Presentation of (some) HRCs Timeline Examples of subjective test methodologies and conditions Discussion 13/12/2011 Marcus Barkowsky: Unique dataset 3DTV phase 1 - 2 -

3. Purpose and scope The scope of this database is to  Provide a unique set of degraded (+reference) sequences Free for distribution Free for publication Well documented Allowing comparisons!  Idea: Collecting a large number of “indices” for the same set of sequences in order to correlate the results and to discover the “ground truth” 13/12/2011 Marcus Barkowsky: Unique dataset 3DTV phase 1 - 3 -

4. Purpose and scope This database allows for  Cross - lab analysis: Run the same subjective experiment in several labs  Cross - methodology: Use different subjective test methodologies  Cross - environment: Use different environments for the subjective experiment, e.g. displays, backgrounds  Cross – scales: Asking observers to rate on various scales such as discomfort, depth quality, preference to 2D, etc. in order to find the relation to “Overall Quality of Experience”  Comparing the effect of training of observers  Evaluating additional inter-observer differences using for example questionnaires, observer characterization tests (prism facility, 3D acuity, sensitivity to asymmetric conditions) 13/12/2011 Marcus Barkowsky: Unique dataset 3DTV phase 1 - 4 -

5. Out of this scope The following points are out of the scope of this evaluation:  Length of sequences 13/12/2011 Marcus Barkowsky: Unique dataset 3DTV phase 1 - 5 -

6. Presentation of additional SRC material database NAMA3DS1: Nantes-Madrid 3D Stereoscopic Video Database 1  Content captured with Panasonic 3DA1  Most of the content captured uncompressed, otherwise ~20Mbit/s per view  Camera parameters mostly available  Depth map estimation available but not satisfying  Involved in the shooting: From UPM: Jesús Gutiérrez, Narciso García From IRCCyN/IVC: Matthieu Urvoy, Marcus Barkowsky, Romain Cousseau, Yohann Pitrey, Yao Koudota, Junle Wang, Patrick Le Callet 13/12/2011 Marcus Barkowsky: Unique dataset 3DTV phase 1 - 6 -

7. Presentation of additional SRC material 13/12/2011 Marcus Barkowsky: Unique dataset 3DTV phase 1 Depth map Filtered depth map - 7 -

8. Presentation of (some) HRCs HRCs currently planned for the database:  Multiview Video Coding (MVC)  MPEG-4 part 10 / H.264 (AVC)  Transmission distortions  Resolution reduction  Framerate reduction  Asymmetric conditions Other ideas:  Conversion S3D  2D+Z  S3D  Geometric distortions  13/12/2011 Marcus Barkowsky: Unique dataset 3DTV phase 1 - 8 -

9. Selection of HRCs How to select the PVSs for a unique subjective experiment ( ~ 160 sequences) from the large set of PVS?  Spread the range of Different content Different visual degradations (coding, blocking, blurring, color distortions, geometric distortions, …) Different scale dimensions (video quality, discomfort, depth, …)  Suggestion: Perform a voting for PVSs in different 3D labs 13/12/2011 Marcus Barkowsky: Unique dataset 3DTV phase 1 - 9 -

10. Timeline Preparation of SRC sequences  2 sets are already available Identification of conditions in large HRC set  End of January? Distribution of PVSs  Finish: End of February Selection of PVSs  Conference Call mid-March Unique dataset  Available end of March Subjective experiments start Analyze the results, prepare a report Determine impact on Recommendations 13/12/2011 Marcus Barkowsky: Unique dataset 3DTV phase 1 - 10 -

11. Dimensions of subjective experiment Ideas of different test methodologies and test setups  Methodologies ACR-HR DSCQS Paired Comparison (subset methods) SAMVIQ  Scales Quality of Experience Video Quality Naturalness Visual Discomfort  Environment Different screens Standardized / Home environment 13/12/2011 Marcus Barkowsky: Unique dataset 3DTV phase 1 - 11 -

12. Questions Adding other SRC sequences  NICT license open enough for free distribution? Who would like to contribute more/different HRCs in that timeframe? Interest in participation in the PVS selection? Who would be willing to run subjective experiments?  With well-prepared setup, e.g. 3D Blu-Ray?  Controlling their own lab setup and experiment method? 13/12/2011 Marcus Barkowsky: Unique dataset 3DTV phase 1 - 12 -