1. Results of the ATIS/T1A1.1 Ad Hoc Group on Full-Reference Video Quality Metrics (FR-VQM) VSF Meeting October 3, 2001 John Pearson Sarnoff Corporation jpearson@sarnoff.com

2. Take Home Messages Tariff’s can now include Visual Quality Metrics (Full Reference) The basis for this is a family of 4 Technical Reports by ATIS/T1A1 The T1A1 approach is extensible to additional Visual Quality Metrics, and does NOT establish a Standard

3. Outline Why is measuring Visual Quality important? Why is measuring Visual Quality hard? International Standards for VQM’s T1A1 Technical Reports

4. FR-VQM Needs of US Telecom Digital video processing can create objectionable noise End-to-End QoS across the networks of multiple companies requires agreement on Quality at Transfer Points (Tariffs) Tariff’s require ANSI sanctioned technical documentation Site of Video Origination (e.g., Denver) Transfer Between Network A & B Site of Video Consumption (e.g., Mexico City) Company A uses VQM-A Company B uses VQM-B Q-A Q-?? Q-B

5. Blocky “Digital” NoiseRandom “Analog” Noise Digital Video Creates “Patterned” Noise... Human visual response to patterned noise highly non-linear... MSE = 27.10 MSE = 21.26 Measures like MSE suitable for Analog noise no longer work for Digital noise

6. Codec Frame Source Frame Difference Map Patterned noise in the sky much more perceptible even though much smaller in terms of pixel differences

7. Visual Quality Metrics... correlate well across scene types, unlike MSE... Visual Discrimination Model Mean-Squared Error Bars show 5% confidence intervals Mean of 80 trials for 20 subjects

8. Vital Role of Subjective Database Goal of VQMs is to approximate subjective quality assessments (SQA) The relevance of the SQA depends on: –Test sequences (SRC’s) –Distortion generators (HRC’s) –Viewing conditions and testing protocols Producing a relevant SQA is hard

9. Three Kinds of VQM’s Full Reference (FR) –a double-ended method and is the subject of this Technical Report. Reduced Reference (RR) –only reduced video reference information is available. This is also a double-ended method. No Reference (NR) –no reference video signal or information is available. This is a single-ended method. It is generally believed that the FR method will provide the most accurate measurement results while the RR and NR methods will be more convenient for QoS monitoring. The T1A1 Technical Reports concern FR methods

10. Full-Reference VQM’s with Normalization

11. International Standards Progress VQEG may be several years from recommending a FR-VQM standard to ITU Its possible that no single FR-VQM will be a clear “winner” The FR-VQM field is young, and significant, steady improvements are expected over the next decade It’s possible that several different FR-VQMs may gain industry acceptance

12. T1A1.1 FR-VQM Strategy … an extensible family of TR’s for FR-VQ, enabling Industry to move ahead without Standards... Provide guidelines for how Industry can –specify its specific FR-VQM needs –assess the suitability of existing documented FR-VQMs –drive the development by FR-VQM proponents of new/improved FR-VQM algorithms and products –inter-operate with different FR-VQMs Provide guidelines for how FR-VQMs can be –documented in algorithms, accuracy and limitations –quantitatively cross-calibrated to each another Extensible framework enabling addition of FR-VQMs –Start by specifying two already disclosed FR-VQMs –Stimulates continued FR-VQM innovation

13. Primary Contributors

14. Family of Technical Reports TR A1: Accuracy and Cross-Calibration (Mike Brill, Sarnoff) –defines accuracy (statistical analysis), limitations of a FR-VQM –defines transformation to common scale, for cross-calibration with other applicable FR-VQMs TR A2: Normalization Methods (David Fibush, Tektronix) –applied to source and processed video before VQM calculation –e.g., spatial/temporal registration, gain/level offset calibration,... –may utilize special test signals TR A3: Peak Signal to Noise Ratio (Steve Wolf, NTIA) –Specify PSNR VQM, following TR A1 and TR A2 guidelines TR A4: Objective Perceptual FR-VQM Using a JND-Based Full Reference Technique (David Fibush, Mike Brill) –Specify JND-based FR-VQM, following TR A1 and TR A2 guidelines

15. “How to” specify VQM accuracy –with respect to subjective assessments –based on defined statistical analysis “How to” specify VQM scope/limitations –type of scene content (“signal”) high/low motion, color/b&w, interlaced/progressive –type/severity of artifacts (“noise”) e.g., encoding techniques, bit-rates, blurring, blockiness –subjective testing characteristics behavior with viewing distance, resolution, gamma, … expert vs non-expert viewers “How to” cross-calibrate VQMs –determination of mathematical transformation relating one VQM’s outputs to another’s TRA1 Defines Basic Methods: SCOPE LIMITATIONS Works well, & has been well tested here

16. VQEG Database: “SRC’s”

17. VQEG Database: “HRC’s”

18. JND/PQR & PSNR Limitations: no H.263

19. Algorithm Documentation: JND/PQR

20. Stripping for JND/PQR Registration

21. Algorithm Documentation: PSNR

22. Normalization Requirements JND/PQR PSNR

23. VQEG data & Logistic-mapped PQR

24. Logistic-mapped PQR for Common Scale … provides approach for cross-calibration...

25. Accuracy -- 3 Methods RMSE Resolving Power Classification of Errors

26. Confidence vs.D-VQM: JND/PQR

27. Confidence vs.D-VQM: PSNR

28. RMSE RMSE: root mean square error between subjective and objective normalized scores

29. Classification of Errors

31. Progress T1A1.1 Ad Hoc Group created Feb. 2001, co-chairs John Grigg, John Pearson Mail Ballot Approval August 2001 Approved by T1A1.1 25 September 2001 Approved at Plenary meeting of T1A1, 28 September 2001