1. Empirically Validated Web Page Design Metrics Melody Y. Ivory, Rashmi R. Sinha, Marti A. Hearst UC Berkeley CHI 2001

2. Empirically Validated Web Page Design Metrics 2 The Usability Gap 196M new Web sites in the next 5 years [Nielsen99] ~20,000 user interface professionals [Nielson99]

3. Empirically Validated Web Page Design Metrics 3 The Usability Gap Most sites have inadequate usability [Forrester, Spool, Hurst] (users can’t find what they want 39- 66% of the time) 196M new Web sites in the next 5 years [Nielsen99] A shortage of user interface professionals [Nielson99]

4. Empirically Validated Web Page Design Metrics 4 The Usability Problem NON-professionals need to create websites Guidelines are helpful, but – There are MANY usability guidelines Survey of 21 web guidelines found little overlap [Ratner et al. 96] Why? – One idea: because they are not empirically validated – Sometimes imprecise – Sometimes conflict

5. Empirically Validated Web Page Design Metrics 5 Possible Solutions: Tools to Help Non-Professional Designers Examples: – A “grammar checker” to assess guideline conformance Imperfect Only suggestions – not dogma – Automatic comparison to highly usable pages/sites – Automatic template suggestions How to create these?

6. Empirically Validated Web Page Design Metrics 6 Current Design Analysis Tools Some tools report on easy-to- measure attributes – Compare measures to thresholds Stein (Rating Game), Theng & Marsden, Thimbley (Gentler) Not empirically validated – Guideline conformance CAST (Bobby), Scholtz & Laskowski (WebSAT), Lift Online – Perceptually based heuristics Faraday (Design Advisor) Small subset of features [Brajnik00] – Simplistic – Not empirically validated

7. Empirically Validated Web Page Design Metrics 7 The WebTANGO Approach Models of good design by looking at existing designs – Empirical foundation for easy-to-measure attributes – Focus on information-centric sites First work to take a large set of sites and analyze them

8. Empirically Validated Web Page Design Metrics 8 The Investigation Using quantitative measures to predict Web site ratings – Followup investigation [HFW00] Given – 1898 pages from 400+ sites – 11 quantitative measures to assess various Web page aspects Questions – Which features distinguish well-designed web pages? – Can metrics predict ratings? – Are there differences for categories of pages?

9. Empirically Validated Web Page Design Metrics 9 Webby Awards 2000 2000 sites initially – 27 topical categories We studied sites from information-centric categories – Finance, education, community, living, health, services 100 judges – International Academy of Digital Arts & Sciences – 3 rounds of judging

10. Empirically Validated Web Page Design Metrics 10 Webby Awards 2000 6 criteria – Content – Structure & navigation – Visual design – Functionality – Interactivity – Overall experience Scale: 1-10 (highest) Nearly normally distributed What are judgments about?

11. Empirically Validated Web Page Design Metrics 11 Webby Awards 2000 Content criterion is best predictor Visual design criterion is worst predictor User study of 57 sites – Ratings reflect usability

12. Empirically Validated Web Page Design Metrics 12 Quantitative Measures: Aspects Impacting Usability Identified 42 attributes from the literature Roughly characterized: – Page Composition words, links, images, … – Page Formatting fonts, lists, colors, … – Overall Page Characteristics information & layout quality, download speed, …

13. Empirically Validated Web Page Design Metrics 13 Quantitative Measures: Word Count

14. Empirically Validated Web Page Design Metrics 14 Quantitative Measures: Body Text %

15. Empirically Validated Web Page Design Metrics 15 Quantitative Measures: Emphasized Body Text %

16. Empirically Validated Web Page Design Metrics 16 Quantitative Measures: Text Positioning Count

17. Empirically Validated Web Page Design Metrics 17 Quantitative Measures: Text Cluster Count

18. Empirically Validated Web Page Design Metrics 18 Quantitative Measures: Link Count

19. Empirically Validated Web Page Design Metrics 19 Quantitative Measures: Page Size (Bytes)

20. Empirically Validated Web Page Design Metrics 20 Quantitative Measures: Graphic %

21. Empirically Validated Web Page Design Metrics 21 Quantitative Measures: Graphic Count

22. Empirically Validated Web Page Design Metrics 22 Quantitative Measures: Color Count

23. Empirically Validated Web Page Design Metrics 23 Quantitative Measures: Font Count

24. Empirically Validated Web Page Design Metrics 24 Characterizing Measures: A View of Web Site Structure [Newman et al. DIS00] Information design – structure, categories of information Navigation design – interaction with information structure Graphic design – visual presentation Courtesy of Mark Newman

25. Characterizing Measures: Web Site Structure Assessed

26. Empirically Validated Web Page Design Metrics 26 Study Method The Webby factor – Principle components analysis of the 6 criteria Accounted for 91% of the variance Two comparisons – Model 1: Highly rated sites (top 33%) vs. the rest Using the overall Webby score – Model 2: Highly rated sites vs. bottom 33% Using the Webby factor

27. Empirically Validated Web Page Design Metrics 27 Findings We can accurately classify web pages – Linear discriminant analysis – Model 1: For highly-rated sites vs. rest 67% correct when not considering content categories 73% correct when taking content categories into account – Model 2: For highly-rated sites vs. bottom 65% correct when not considering content categories 80% correct when taking content categories into account

28. Empirically Validated Web Page Design Metrics 28 Findings Top vs. bottom – Webby factor Linear discriminant analysis Better for categories

29. Empirically Validated Web Page Design Metrics 29 Deeper Analysis Which metrics matter? – Linear regression analysis Backward elimination until adjusted R² reduced – All metrics played a role Compared small, medium, and large pages – Across the board (preliminary profiles) Good pages had significantly smaller graphics percentage Good pages had less emphasized body text Good pages had more colors (on text)

30. Empirically Validated Web Page Design Metrics 30 Small pages (66 words on average) Good small pages have (according to beta coefficients) – Slightly more content – Smaller page sizes – Fewer graphics – More font variations Suggests that these pages – Have faster download times corroborated by a download time metric – Use different fonts for headings vs. the rest of the text Examples – Services (Home Pages) Top Bottom

31. Empirically Validated Web Page Design Metrics 31 Medium pages (230 words on average) Good medium pages – Emphasize less of the body text – Appear to organize text into clusters (e.g., lists and shaded table areas) – Use colors to distinguish headings from body text Suggests that these pages – Are easier to scan

32. Empirically Validated Web Page Design Metrics 32 Large pages (827 words on average) Good large pages have – More headings – More links – Are larger but have fewer graphics Probably attributable to style sheets Suggests that good large pages – Are easier to scan and facilitate information seeking

33. Empirically Validated Web Page Design Metrics 33 Why does this work? Content is most important predictor BUT there’s predictive power in other aspects – Visual and navigation design – Verifies preliminary investigation [HFW00] Possibly: Good design is good design all over Note: we are NOT suggesting we can characterize: – Aesthetics or subjective preferences

34. Empirically Validated Web Page Design Metrics 34 How might we use this? Web Site Design Prediction Similarities Differences Suggestions Analysis Tool Profiles Comparable Designs Favorite Designs

35. Empirically Validated Web Page Design Metrics 35 Future work Distinguish according to page role – Home page vs. content vs. index … Better metrics – More aspects of info, navigation, and graphic design Site level as well as page level Category-based profiles – Use clustering to create profiles of good and poor sites – These can be used to suggest alternative designs

36. Empirically Validated Web Page Design Metrics 36 In Summary Automated tools should help close the Web Usability Gap We have a foundation for a new methodology – Empirical, bottom up We can empirically distinguish good pages – Empirical validation of design guidelines – Can build profiles of good vs. poor sites Eventually build tools to help users assess designs

37. Empirically Validated Web Page Design Metrics 37 More Information http://webtango.berkeley.edu