SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Data Driven Design Using Web Analytics to Improve Information Architectures Andrea Wiggins IA Summit 2007

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Motivation: What Information Architects Want to Know Interviewees said: –Context for making design decisions –Validation of heuristic assumptions –Understand why visitors come to the site & what they seek

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Agenda Overview for Context Insert show of hands here! (topic, tools, data) What is web analytics (WA)? How is it done? –major WA concepts –what the data look like IA questions to answer Rubinoffs user experience audit Some WA measures for heuristic validation

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN What is web analytics? Data mining from web traffic logs –Web server log files –Page tag logs from client-side data collection (end up in server logs) –Cookies to identify unique visitors What for? –Proving web site value (ROI) –Marketing campaign evaluation –Executive decision making - markets & products –Web site design parameters –More…

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN How do you do it? Vendor analysis solutions Hosted ASP –Currently most popular model –Provides traffic stats on-demand Software –Runs on dedicated servers –Scalability: requires significant data storage space and data maintenance Costs –Starts at FREE for Google Analytics and goes way, way up –Large organizations spend $50K/yr and up Open source: not a robust option

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Very Quick Major Concepts Sessionizing (cookie > IP & UA) Hits: all server requests Pageviews: all server requests for page filetypes, variously defined Visits & Visitors: stronger measures from sessionizing, sensitive to time periods

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Sample Logs #Software: Microsoft Internet Information Services 6.0 #Version: 1.0 #Date: :00:35 #Fields: date time cs-method cs-uri-stem cs-username c-ip cs-version cs(User-Agent) cs(Referer) sc- status sc-bytes :10:05 GET /index.htm xx.76.7 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+98) t-280&toggle=1&cop=&ei=UTF :10:29 GET /current.html xx.76.7 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+98) :11:24 GET /tickets.html xx.76.7 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+98) :18:06 GET /index.htm xxx HTTP/1.0 Mozilla/4.0+(compatible;+MSIE+6.0;+AOL+9.0;+Windows+NT+5.1;+SV1;+.NET+CLR ) :20:18 GET /index.htm - 68.xx HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR ) :20:21 GET /classes.html - 68.xx HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR )

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Spiders :49:32 GET /robots.txt - 68.xxx HTTP/1.0 Mozilla/5.0+(compatible;+Yahoo!+Slurp;+ ysearch/slurp) :49:32 GET /plays/completing_dahlia.html - 68.xxx HTTP/1.0 Mozilla/5.0+(compatible;+Yahoo!+Slurp;+ ysearch/slurp)

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN A Few Good Metrics Information Architects want to know: –Confirmation of heuristics Do users leave at first glance of this awful page? Where do they click? What position on the screen or layout produces the most clicks for the same content? Do the users pogo-stick back and forth between pages? What are they comparing? –Ambient findability measures At what hierarchy depth do visitors enter the site? How do they get in on deep pages? Do they ever see the home page? Can they find their way to where we want them to go?

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Searching for IA Answers On-site search behaviors –How many searches do users make? –Do users refine their search results? –What type of queries do users make? –How often are search results the last page? –From what pages are searches initiated? –Do the search terms have context in the page from which the search is initiated? –Why are users querying about chimpanzees?!?

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN What IAs Want Good navigation and content make the online world go round –Where in a process do users leave? Where do they go? Do they re-enter the process? –How do users move through the site? Is there a better route? –What pages dont get visited? What pages get unexpectedly high visits? –What prompts conversion? –Where do search engine spiders go in the site? Is the best content being indexed?

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Everybody Loves Rubinoff UX audit quantifies subjective measures –Offers structure for comparing properties of the site –Completely customizable, use strategically In a perfect world: –Analyst & IA work together to set key performance indicators (KPI) and measurable heuristics –Each independently evaluates the site on the same points and compare the IAs heuristics to user data for validation –They set before-and-after measures to prove value for the entire project

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Rubinoffs Four Categories Using a sample of statements from Rubinoffs model: 1.Branding a)Engaging, memorable brand experience b)Value of multimedia & graphics 2.Functionality a)Server response time & technical errors b)Security & privacy practices 3.Usability a)Error prevention & recovery b)Supporting user goals & tasks 4.Content a)Navigation & site structure b)Search & referrals

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN 1a: Branding Memorable & Engaging Experiences Ratio of new to returning visitors is key; set target KPI specific to site business goals Track trends over time and in relation to cross-channel marketing Median visit length in minutes Average visit length in pages viewed Depth, breadth of visits Segment new and returning visitors to examine visit trends for different audiences

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN 1b: Branding Value of Multimedia & Graphics Flash & AJAX require deciding upon what to measure, programming appropriate data collection, and configuring analysis tools Plan to include measures when designing multimedia applications to prove value Compare clickthrough rates for clickable graphics to rates for standard navigation links Great tools like Crazy Eggs heatmap - easy! (also relevant to navigation, of course)

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Crazy Egg Heatmap Example

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Crazy Egg Overlay Example

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Crazy Egg List Example

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN 2a: Functionality Response Time & Technical Errors Response time is a default log field, easy to measure Check at peak load time to make sure site is responding quickly enough Monitor the rate of 500 (server) errors: this should be an extremely low number

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN 2b: Functionality Security & Privacy Practices A matter of design for measurement, not measurement of design: considerations for designing a site that will be measured –Privacy best practices: Give a short, accurate, easy to understand privacy statement and stand by your word True first-party cookie –Security best practices: (from an IA/analytic POV) SSL encryption on any transactional forms: lead generation, ecommerce, surveys Secure file transfer for & restricted access to raw web analytic data; password restrictions at minimum

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN 3a: Usability Error Prevention & Recovery Percentage of visits experiencing 404 and 500 errors: errors should be < 0.5% of all hits Percentage of visits including an error, that end with an error - frustrated into leaving Where do 404 errors occur? –Use to build a redirect page list to ensure (temporary) continuity of service to bookmarked URLs –Path/navigation analysis: how did users arrive at 404? What did they do after? User errors: identify problems & re-enact or test

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN 3b: Usability Supporting User Goals & Tasks Scenario/conversion analysis –Define tasks and procedures supporting user goals –Examine completion rates, step by step, intervals & overall A to B, B to C, C to D; A to C, B to D; A to D –Look at leakage points Where did they go when they left the process? Did they come back later? –Shopping cart analysis Keep in mind that users shop online for offline purchases Do behaviors suggest a need for a tool like a shipping calculator or product comparison? –Online form completion

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN 4a: Content Navigation & Site Structure Pogo-sticking: jumping back & forth between content or hierarchy levels (what about tabs?) –Need a comparison tool, cant identify product: not enough detail at the right level of site hierarchy or step of the purchase decision process Compare page-level traffic statistics for larger trends, broad navigation analysis: the usual #s Path analysis on navigation tools (by type) to pinpoint navigation and labeling problems –Extensive use of supplemental navigation may indicate need for updates to global navigation

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN 4b: Content Mining Search & Referrals Popularity = value? What about findability? If its not findable, it probably wont be popular. –Compare the contents value (against similar content) with proportions of returning visitors, average page viewing length, external referrals - especially search referrals Search log analysis: what do your users value? –Does user query language match site contents? Are users searching for panties when youre selling pants?

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Validate the Match Between the Site & the Real World More ways to use search log analysis: –Does user vocabulary match site vocabulary? –Do different audiences have different vocabularies, and does the site support them equally? –Brand measurement returns product and industry terminology usage accuracy of brand queries: spelling, inclusion of competitors brands, advertising slogans –Did users find what they expected? How many visits end on search results? Null results are revealing.

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Language Validation

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Conclusions Not much out there in the academic literature on using web analytics (hopefully to change!) WA data is flawed and tough to handle, but ultimately pays off in developing holistic understanding of user behavior Best-suited to case studies WA is ripe for adoption into formal usability frameworks, particularly for persona design and determining design parameters Best used iteratively: beginning, middle, end, annual follow-up…

SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN Thanks! Questions?