Maria Grazia Albanesi, Riccardo Amadeo University of Pavia, Faculty of Engineering, Computer Department Impact of Fixation Time on Subjective Video Quality Metric: a New Proposal for Lossy Compression Impairment Assessment ICMVIPPA 2011 : International Conference on Machine Vision, Image Processing, and Pattern Analysis Venezia (Mestre), November 28, 2011

The addressed problem: –subjective video quality assessment for lossy compression impairment The tools and the experiments –eye tracking and subjective experiments The goals –Comparison to literature The results and their interpretation A possible application: a new protocol for no- reference video quality assessment Future developments Outline 2 ICMVIPPA Venezia (Mestre) - November 28, 2011

How I can measure the loss of quality due to compression? Field of applications: TV, video services on Internet, video for mobile applications, test of emerging compression algorithms….. Evaluation of multimedia quality user experience Two approaches: objective and subjective metrics Our goal: find objective parameters coming form subjective experiments which reflect the subjective video quality, as perceived by a human observer. The problem 3 ICMVIPPA Venezia (Mestre) - November 28, 2011

Eye tracker: it records the point and the duration of fixation of the eye, when the observer looks at a monitor. Data are subsequently analyzed from a statistical point of view (mean, std. dev….) The tools: eye tracker and subjective QA experiments 4 ICMVIPPA Venezia (Mestre) - November 28, 2011

The set of videos 5 A set of 19 videos downloaded from available online public libraries – (Video trace library of Arizona State University) –ftp://ftp.tnt.uni-hannover.de/pub/svc/testsequences/ (Hannover Liebnitz University video library) – The original files: YUV sequences, 4:2:0, in CIF resolution (352x288) at 30 fps are converted in avi sequences and compressed by a H.264 at 2 bitrates: 450 bps and 150 bps ICMVIPPA Venezia (Mestre) - November 28, 2011

Examples: 6 ICMVIPPA Venezia (Mestre) - November 28, 2011

Visual behavior and impairment 7 Visual path for original «best» video Visual path for compressed (br150 bps) video

Protocol ACR5-HR (absolutely category ranking – hidden reference –MOS scale with five levels: –Only one observation for each video –The observer has no information about the unimpaired version of the video. The subjects: 8 females and 10 males, of age varying from 22 to 27 years old. – Their vision was normal or corrected-to-normal –They had no experience in subjective video quality assessment. –They had normal or good experience in using IT interfaces to watch videos both online and offline. Methodology 8 ICMVIPPA Venezia (Mestre) - November 28, 2011

Our parameter are not related to fixation points, but to the duration of the fixation. Videos are classified according to color content relevance and and movement relevance to create semantic filters Parameters: –Duration of fixation time –MOS, five point scale –Subjective Color Score, three point scale –Subjective Movement Score (SCS e SMS), three point scale. –Removal of «Memory effect» due to the conditioning of ocular motion activity by the visual attention of preceding scenes. Novelties and comparison to literature 9 Starting point: O. Le Meur, A. Ninassi, P. Le Callet, D. Barba, Overt visual attention for free-viewing and quality assessment tasks: Impact of the regions of interest on a video quality metric, Signal Processing Image Communication, 2010, vo. 25, pp

18 tester, 6 in each playlist Each video has three version: reference br450 br150 (57 videos) Each observer looks at only one version of each video. No repetitions are allowed in each playlists. Numero video utilizzati 19 N.IDPlaylist APlaylist BPlaylist C 1ForemanrefBr150Br450 2SilentBr150refBr450 3FlowerBr450Br150ref 4BusBr450refBr150 5TempeterefBr450Br150 6Bridge_closerefBr150Br450 7IceBr150refBr450 8CoastguardBr450refBr150 9Mother_daugBr150Br450ref 10FootballrefBr150Br450 11CrewBr450refBr150 12ParisBr450Br150ref 13ContainerrefBr450Br150 14HighwayBr150Br450ref 15WaterfallBr150refBr450 16HallBr450Br150ref 17StefanrefBr450Br150 18NewsrefBr150Br450 19MobileBr150Br450ref tot: ref766 Br Br Playlists to remove memory effect 10 ICMVIPPA Venezia (Mestre) - November 28, 2011

The MOS really reflect the progressive loss of quality due to compression. Mean Opinion Score 11

Color and movement are considered relevant if the score is > 2 «Highly animated video»: 2, 4, 7, 8, 10, 12, 14, 17, 19 «Highly coloured video» »: 3, 5, 7, 10, 15, 19 N.PlaylistSCSSMS 1Foreman1,671,50 2Silent1,282,28 3Flower2,331,61 4Bus1,672,33 5Tempete2,501,89 6Bridge_close1,441,50 7Ice2,112,33 8Coastguard1,722,33 9Mother_daughter1,781,28 10Football2,392,78 11Crew1,831,94 12Paris1,942,11 13Container1,831,67 14Highway1,392,56 15Waterfall2,721,72 16Hall1,72 17Stefan1,672,50 18News1,781,83 19Mobile2,562,22 SCS e SMS 12

The mean fixation time does not seem to be related to the video quality! Analysis of Mean fixation time (MFT) 13

MFT, semantic filtering Even by filtering by movement or colour, there is not a clear relation between MFT and MOS 14

Even the standard deviation of fixation time does not seem to be related to the video quality! Analysis of Standard deviation of FT 15

RefBr 450Br 150 Average SDoFT, SMS>2155, , ,5709 Standard deviation of SDoFT, SMS>269,736185, ,3062 RefBr 450Br 150 Average SDoFT149, , ,8378 Standard deviation of SDoFT60,671988,298790,7023 The solution: third order statistics! RefBr 450Br 150 Average SDoFT, SCS>2145, , ,9579 Standard deviation of SDoFT, SCS>269, ,821387,1634 Standard deviation of FT is on the average less for videos of high quality The semantic filtering shows that this behaviour is stressened for highly animated videos. 16

The duration of fixation time seems to have a more predictable behavior when the observer watches to a high quality video. If we compute the third order statistics on the fixation time, we can guess a rank of a collection of video which reflects the perceptive visual quality The experiments confirm this behavior for degradation due to lossy compression. The rank according third order statistic reflect the loss of quality and subjective MOS especially for highly animated videos. Future researches: Test on a greater level of quality impairments Test on other kinds of quality impairments Finding a more efficient semantic filtering about color or other criteria. Conclusions and future researches 17 ICMVIPPA Venezia (Mestre) - November 28, 2011