Presentation is loading. Please wait.

Presentation is loading. Please wait.

Alternative Approaches for Gathering End User Data Anne de Ridder Sr. User Experience Architect White Horse, Portland, OR UW HCDE Presentation February.

Similar presentations

Presentation on theme: "Alternative Approaches for Gathering End User Data Anne de Ridder Sr. User Experience Architect White Horse, Portland, OR UW HCDE Presentation February."— Presentation transcript:

1 Alternative Approaches for Gathering End User Data Anne de Ridder Sr. User Experience Architect White Horse, Portland, OR UW HCDE Presentation February 26, 2010

2 Overview Introduction The Why and When of using alternate approaches So what are these approaches anyway? –Case Study 1: Remote Unmoderated User Testing –Case Study 2: Low/High-Tech User Testing –Case Study 3a: Online Focus Groups –Case Study 3b: Survey + Moderated Interviews –Case Study 4: Online Card Sort Questions/Discussion

3 Introduction MS in Technical Communication from UW HCDE in 1999 Professional experience in areas of user experience design, usability, technical writing, and marketing communications Worked with big & small companies as both employee and consultant –IBM, Sharp Microelectronics, JPL, Tektronix, Mountain Hardwear, Trane Commercial Systems, several municipalities, environmental consulting firms, and many more! Currently at White Horse –Digital marketing agency specializing in the convergence of emerging and traditional media to create immersive Web experiences. Web development, digital marketing, technical engineering, emerging media, audio/video production –Persona-led Design –Dedicated User Experience team If it is UX-related, we can do it!

4 The Why and When of using alternate approaches Why the need for alternate approaches? –Additional tools in the HCD toolbox When to use alternate approaches? –Defined by the project conditions: Project phase –What is the question you are trying to answer? And why? Project sponsor –Who are your “customers”? –What is their objective? End user data requirements –Type of data; volume of data; quality of data –Geographic distribution and language requirements; interface being evaluated; result objectives Timeframe Cost

5 So what are these approaches anyway? Focus of this presentation is on Online tools A lot of options out there that address many needs –Card sorts, navigation evaluation, surveys, user testing, and more. Variation among options within sub-categories –Considerable differences in quality and depth of data collected The option you select is based on what you’re looking for –Result objectives –End user data requirements

6 Case Study 1: Remote Unmoderated User Testing Testing objective: –Validate that a redesigned corporate Web site addresses the task-based and informational needs of the business’s 7 distinct user segments –Collect data from a large number of geographically dispersed (US) group of users from each segment in a short turn around time with relatively “low” cost Minimum 75 completed tests (5 to 15 tests per segment); maximum 400 tests total Recommendation: Remote Unmoderated User Testing –WebEffective Users invited via email invite and intercept on the Web site. Task-based test using the client’s Web site –Collection of both qualitative and quantitative data, though focus was on the qualitative. –Collection of user behavioral data including: »Clicks, hovers, scrolling, text entries, form field selections

7 Case Study 1: Results 14 days of testing Tests Begun: 1420; Reportable: 224 –Disqualified: 585 (screener; filter); Dropped-Off: 611 (pre-background questions; 331 after background survey) Biggest differences from traditional testing: –It’s a lot of data! 40 hours of data analysis; 70 slides of data summaries—and that was just capturing the high level picture –Need to “recreate” the user experience –Need to recruit/include greater number of participants to off-set drop-offs, disqualifications, outliers, “no-navs”, “give-ups” –(Not so) accurate measurement of confidence ratings

8 Case Study 1: Community Perspective What does the overall research community say? –Independent, informal study shows statistically comparable results from lab-based tests and remote unmoderated tests Sauro, Tullis, Molich, Kirakowski, UPA 2009 Study also highlights the need to collect larger numbers of completed tests in order to have substantial sub-set of “qualifying” data –“Beyond the Usability Lab: Conducting Large-scale Online User Experience Studies” by W. Albert, T. Tullis, and D. Tedesco, 2010

9 Case Study 2: “Low/High-Tech” Testing User acceptance prototype testing “Live” tests via online interface –Project A: Low participant participation on day 1 of listening labs Double-booked day 2 to supplement live sessions Used GotoMeeting to allow users to interact with prototype plus have ability to record sessions (voice; mouse movements) –Project B: 5 hour budget for user testing Three distinct user segments Used GotoMeeting to allow users to interact with prototype plus have ability to record sessions (voice; mouse movements) Problems with firewalls; did not allow users to fully engage with prototype

10 Case Study 3: Online Focus Groups Web site redesign to incorporate more customer-centric approach for finding financial products and services Outcome of Task Analysis and Modeling Phase –Validate that interpretation and synthesis of past research matches actual user behavior Collect feedback on where experience and task flow diagrams match/miss current user behavior and needs 1,200 participants total (for statistical significance) including both demographic and geographic segmentations –Large-scale synchronous online focus groups Allows quicker and cost-effective collection of significant data across personas and geographic locations FacilitatePro ( Online Focus Group

11 Case Study 3b: Survey + Moderated Interviews Design Concept Testing –Goal: Gain insight into user expectations about design elements, interaction flows, and outside influences Gain clear picture of the motivators behind target user’s assessments of design concepts Testing Outcome: Final concept direction selected based on user data; inform content strategy 600 users –Method: iModerate iMpact Collection of both qualitative and quantitative data for each target user group Survey to larger group (via any survey tool of choice) Percentage of users intercepted and “interviewed” by professional moderators

12 Case Study 4: Remote Card Sort Web site redesign for a domain service provider Recommended a complete shift in information architecture and re- categorization of offerings to better meet the needs of site users –Based on competitive benchmarking, buy flow analysis, customer survey Segmentation: business/personal use; web site maintenance experience Remote open card sort ( ) recommended to determine: –Market segment variances in user groupings of particular items and topic category naming –Nomenclature participants used to describe the topic categories Initial scope: 30 “recruited” participants –Limited budget for user research  “How can we get user data in one day?” –Additional segmentation: 4 geographic markets Outcome: –Customer felt confident in recommendations based on competitive benchmarking report and internal stakeholder review –Used card sort as an internal IA tool

13 Other Available Options

14 Remote User Testing Options WebEffective –Including mobile –Data reported: Tasks; participant videos, summaries, ability to “watch” keystrokes/mouse movements/clicks –Use their panel or your own customers; $39 per user Loop11 –Data reported: Task completion rate; Time per task; Most common success page; Most common fail page; Most common first click; Most common navigation path; Detailed participant path analysis; Number of page views to complete tasks –$350 up to 1,000 participants, unlimited tasks/questions – UserZoom –Including mobile –Data reported: Effectiveness ratios; Efficiency ratios; Click-stream paths; Click-mapping; Users' suggestions & feedback; Satisfaction & perception indicators; Cluster analysis & dendograms –

15 Online Focus Group Options FacilitatePro – Artafact – Invoke – Itracks – Qual-vu –Online focus groups incorporating Video Diaries –

16 Survey Tools iModerate – ForeSee –Ability to tie satisfaction ratings to ROI based on proven American Customer Satisfaction index –Initial survey during unique user session –Ability to track impact of satisfaction on future behaviors (true conversion numbers based on cookie tracking) –

17 IA validation tools –Online Card Sort UserZoom –Online Card Sort – Optimal Workshop –Online Card Sort –Site Map testing –Task-based “click test” of mockup or screenshot –


19 Case Study 1: Test development and implementation Test Design: –Test Kit development identical to traditional usability testing More consideration of user fatigue in completing “repetitive” follow-up questions. –Additional UX time to: Input tasks and questions into test system and define test logic, including “minimum” participation requirements Coordinate implementation of technical code on client site QA and optimize test for online experience 33 additional hours Live Testing Period: –Monitor participation on a daily basis at a minimum Quotas; “auto-disqualified” participants; drop-off

20 Case Study 1: Participant Data Number of participants: –Some navigations not captured –Some unofficial task “give-ups” due to lack of interest/time/other? –Inconsistent levels of participation (need to scrub the reportable data even further) –Segment 1: 0 –Segment 2: 2 –Segment 3: 25Better reflection of trends –Segment 4: 173Easy to identify patterns/trends –Segment 5: 11Starting to see trends/patterns –Segment 6: 10 –Segment 7: 3 –Jakob Nielson knows how many users are enough Qualitative Studies Quantitative Studies

Download ppt "Alternative Approaches for Gathering End User Data Anne de Ridder Sr. User Experience Architect White Horse, Portland, OR UW HCDE Presentation February."

Similar presentations

Ads by Google