ITU Workshop on “Quality of Service and Quality of Experience of Multimedia Services in Emerging Networks” (Istanbul, Turkey, 9-11 February 2015) QoE evaluation methodologies and results for web browsing Sebastian Egger Scientist, AIT

Goals: Understanding, measuring and managing quality in communication networks ACE 2.0: Focus on mobile broadband QoE, device impact and convergence scenarios ACE 3: Focus on high speed / LTE scenarios, smart devices and web and cloud services Project Partners: FTW A1 Telekom Austria AG Telekom Austria Group AG Vodafone Group Services Limited Vodafone Germany Funding: FFG competence center program COMET Projects: ACE 2.0 and ACE 3

RQ1: What are requirements for subjective testing methodologies that produce reliable and consistent QoE scores for interactive browser based applications? RQ2: Can fundamental relationships of human time perception be utilised to model the relationship between waiting time and QoE in browser based applications? Research questions for browser based applications © FTW - 3 -

Close to real web browsing and people in browsing mode Certain QoS level over time Defined tasks which prevent boredom but still create comparable interactivity levels across subjects Realistic content the subjects can interact with naturally ITU-T P.1501 Subjective testing methodology for web browsing Enables use of real web pages Set of tasks take from everyday web browsing scenarios Condition length of 120s – 180s Creates browsing mode within subjects Requirements for a subjective test methodology for web browsing © FTW - 4 -

Testbed Layout

From Web Pages … Web Page = a HTML (Hyper Text Markup Language) text document with references to other objects embedded (images, scripts, etc.) Typically accessed by user entering URLs/clicking links  triggers several HTTP request/response cycles  page view © FTW

… to Page Views … © FTW

© FTW HTML Page DOM loaded Page loaded User issues request Status bar changes Window blanks Page rendering starts Page fully loaded and displayed Page Load Time (PLT)

… to Web Sessions Web Session = series of page view events and the related timings of the stream of interactions (we’ll come back to that later …) © FTW

Verification of Subjective Test Methodology

© FTW 2012 General Problem for web browsing: waiting for information This is the way the consumer sees the company...it should look good, it should be fast. As long as you see things coming up it’s not nearly as bad as just sitting there waiting and again you don’t know whether you’re stuck. You get a bit spoiled. I guess once you’re used to the quickness, then you want it all the time.  TCP translates every impairment into waiting time BUT: How are waiting times related to QoE?

© FTW 2012 Human time perception Psychology utilises psychophysical laws for: Time estimation error Relationship between waiting time and satisfaction The law of Weber-Fechner Models logarithmic relationship between changes of stimulus S and perception P Mathematical expressions:

© FTW 2012 Psychophysics: The Law of Weber-Fechner Published in 1834 Models logarithmic relationship between changes of stimulus S and perception P Mathematical expressions: Well-known principle e.g. for human vision, hearing, smelling, touching, numerical cognition… Research challenges: Applicability of WFL to QoE VoIP: log. impact of bitrate Web: log. impact of bandwidth Further laws explaining QoE? VoIP: exp. impact of loss  IQX (Cf. Hossfeld et al., 2008) Magnitude of Stimulus

The WQL hypothesis The relationship between W aiting time and its Q oE evaluation on a linear ACR scale is L ogarithmic.

Browser based studies overview © FTW 2012

Verification of the WQL for file downloads

Verification of the WQL for simple web browsing

Rejection of the WQL for complex web browsing?

Potential problem: What is the waiting time for the subject? subjectively perceived page-load time (pPLT) differs from technical PLT (tPLT, 100% loaded) at varying proportions  pPLT (the metric closer to QoE) differs considerably from tPLT (which is typically measured)!

G.1031 Classification

User influence factors Context influence factors – Location: cafeteria, office, on-the-way – Interactivity: high / low level – Task type: business, entertainment etc. – Task urgency: casual vs. urgent System Influence Factors Influence factors

Server-related influence factors – Response time (determined by CPU, OS, memory, software, etc.) – Capacity of the link(s) connecting the server(s) and the Internet. Content-related influence factors – Number of objects – Type of objects – Size of objects – Order of objects – Number of elements – Type of elements – Size of elements – Element appearance on the screen Delivery network influence factors – Network contribution to transaction time (see [b-ITU-T G.1040]) – Available capacity (see [b-ITU-T Y.1540]) – Caching along the delivery network Client influence factors – Resource (webpage) loading procedure – Processing power and other processes demanding processing power – Browser implementation – TCP/IP stack and configuration – Operating system. System Influence factors

RQ1: An appropriate subjective testing methodology for browser based applications has been derived and standardized. RQ2: The WQL hypothesis has been verified for pure waiting time tasks and challenges for the complex web browsing case have been identified A Number of influence factors encountered Conclusions © FTW

Thank you very much! Any questions? Freely available web testing content: