1. Information Architecture Summit 2004 Uzanto Consulting 1 Rapid User Mental Modeling (RUMM) at eBay.com: A Case Study Jonathan Boutelle Rashmi Sinha & Kirsten Swearingen Uzanto Consulting Larry Cornett & Ido Dan eBay, Inc.

2. Information Architecture Summit 2004 Uzanto Consulting 2 Background to Information Architecture Redesign of eBay.com eBay website had grown in organic fashion, leading to a somewhat inconsistent user experience Although much design and usability work is done on site itself, this project was first effort to delineate user mental models for the entire website. Basic eBay facts Worlds largest online marketplace 16 million listings on any given day (as of Q2, 2003) Almost 34 million active users worldwide Buyers and sellers New and experienced Occasional and full-time Individuals and corporations Different countries and cultures

3. Information Architecture Summit 2004 Uzanto Consulting 3 Challenges New IA would need to: Support sites vision of global economic democracy Serve a large and diverse user population with varying goals and mental models Be future-oriented, allowing for planned change and growth Occur through evolution not revolution -- must accommodate current users who are very used to the site as it exists today Senior management at eBay needed to see proof of the need for a drastic restructuring. First phase of project (background research) to be completed in only 6 weeks

4. Information Architecture Summit 2004 Uzanto Consulting 4 Overview of Design Research Methodology Stage 1: Explore Interviews and free-listing with users and stakeholders Stage 3: Verify / Refine Large-sample closed card-sorting study Rapid User Mental Modeling Stakeholders User mental models Preliminary IA Stage 2: Understand Open card-sorting study Verified categorization Top-level categories Scope of content & functionality

5. Information Architecture Summit 2004 Uzanto Consulting 5 Our Approach to Design Research: Rapid User Mental Modeling (RUMM) Rigorous, three-stage design research methodology to lay foundation for IA redesign Bring together established techniques in a novel way Three stages: Exploration Understanding Verification Individual components complement and build upon each other Process moves from open-ended exploration to focused information-gathering and hypothesis testing Addresses individual and group differences in a systematic way Produces reliable results (can be reproduced) Uses mix of qualitative and quantitative methods Qualitative: rich, open-ended understanding of users and domain Quantitative: generates recommendations firmly grounded in data (can help make case to management)

6. Information Architecture Summit 2004 Uzanto Consulting 6 Towards Persuasive, Defensible Design Research Important to create not just good design, but successful design: Serves user Satisfies the business goals Is accepted by management Results of design research phase must provide rich picture of user, but also be persuasive and defensible. Defensible: firmly grounded in data, can stand up to scrutiny Persuasive: persuades stakeholders that your solution makes sense and will provide benefit to the business

7. Information Architecture Summit 2004 Uzanto Consulting 7 Stage 1: Explore Domain Through Stakeholder Interviews Goals: Explore domain in open-ended fashion Identify both current and planned content and functionality Method: Interviews with eBay users about specific incidents of site usage Interviews with stakeholders: top management, members of design team etc. Identify problems, constraints, and specific plans for future Generate list of current tasks and Horizon Tasks (tasks planned for future) Stakeholder analysis Task list development

8. Information Architecture Summit 2004 Uzanto Consulting 8 Stakeholder Interviews Spoke to 16 eBay managers and designers Asked 3 questions: What do you imagine a user doing on the eBay site two to five years from now that they cant do today? What one problem, if it were solved today, would cause a large increase in the number of transactions executed on the eBay platform? What user population is least well served by the current site?

9. Information Architecture Summit 2004 Uzanto Consulting 9 Purpose of Stakeholder Analysis Gain valuable perspectives for design research: Insure that design aligns with business strategy. Tap into a companys existing knowledge in a systematic fashion. Ensure success of project by getting input at the research phase from those who have ability to say yes or no. Helps stakeholders feel involved in and understand the design research and process. Generate task list: helps map the scope and boundaries of the information domain

10. Information Architecture Summit 2004 Uzanto Consulting 10 Generating the Task List Using Free-Listing Decided not to conduct inventory of site Incredibly time-consuming Might tie new IA too closely to current sites design Focused on tasks, since all eBay content is in context of completing tasks: Pay for an item List an item Generated initial list of 300 tasks by using free-listing exercises with: Users Designers Stakeholders (especially important for identifying future tasks)

11. Information Architecture Summit 2004 Uzanto Consulting 11 The Free-Listing Exercise This simple, semi-structured method helps: Determine the scope of a domain Gain some insight into how the domain is structured Example: List all the animals you can think of Results: Indices of salience: Frequency Rank Indirect indices of similarity: Co-occurrence. How often are cat and dog both mentioned? Average rank distance. Are cat and dog mentioned together more closely than cat and horse? In this case, we asked List all the tasks at eBay.com a person can do / might be able to do in the future.

12. Information Architecture Summit 2004 Uzanto Consulting 12 Refining the Task List Consolidated task list, removing redundant items Divided tasks into groups: Current Tasks (can be done on existing eBay website): Core Tasks form core functionality of website Boundary Tasks mark boundary of functionality offered Horizon Tasks Might be done on future versions of website Important for creating a future-oriented IA Selected representative items from each group Result: comprehensive list of 100 tasks marking scope and boundary of what users can accomplish on the eBay site. We were ready to begin studying user mental models!

13. Information Architecture Summit 2004 Uzanto Consulting 13 Stage 2: Understand User Mental Models Goals: Gain further insight into user mental models and identify top-level categories for site content and functionality Method: open card sorting Used list of 100 tasks (including 20 Horizon Tasks) generated in previous stage. 28 participants, representing the main types of eBay users (sellers and buyers; new and advanced users). 22 completed card-sorts individually 6 worked as a group to create a structure (mix of buyers, sellers, new, and advanced)

14. Information Architecture Summit 2004 Uzanto Consulting 14 Identifying the Top Levels of the Hierarchy Cluster analysis used to generate aggregate mental maps. These were directly used to create the hierarchical structure. Looked at mental maps for four major segments. Identified the inconsistencies between the mental maps of different groups. Understood how to reconcile those to create a new map. Result: preliminary IA, including 5 top-level categories that accounted for all site content and functionality. Aggregate maps from cluster analysis

15. Information Architecture Summit 2004 Uzanto Consulting 15 Stage 3: Verify and Refine 5-Category Scheme Goals: Verify that the five top-level groupings cover all 100 tasks/concepts. Identify tasks/concepts that do not fit in, as well as where users do expect to find such information. Examine differences in groupings among various user groups. Further develop IA for eBay site. Establish method for future verification and refinement of eBay IA.

16. Information Architecture Summit 2004 Uzanto Consulting 16 Stage 3: Verify and Refine 5-Category Scheme Method: closed card sorting Large sample study with more than 1,000 participants representing all segments of eBay users Study conducted online. Participants asked to categorize tasks to one of the five top-level categories (or to select Other if it didnt belong) Result: site-map showing the structure for the main parts of the eBay site.

17. Information Architecture Summit 2004 Uzanto Consulting 17 Analysis of Closed Card Sort Results Category-belonging scores: Percent of users who think a task belongs to a particular category Lack of consensus or high scores in Other category may signal a design issue A color-coded visualization of closed card sorting data

18. Information Architecture Summit 2004 Uzanto Consulting 18 Reconciling Different Mental Models Results suggest structural possibilities--they do not prescribe one solution In both open and closed card sorting, found differences in mental models Reconcile through design, if possible (revisit dendograms) The most important, contentious issues may ultimately require a decision driven by company brand and business goals (revisit stakeholder analysis)

19. Information Architecture Summit 2004 Uzanto Consulting 19 Project Summary Three-stage RUMM methodology used to: Understand business context Delineate hierarchical structure of user mental models Final deliverable: site-map showing structure for main parts of entire eBay website. This map is currently being used to guide the stages of the IA redesign. site.com Top Category 3 Top Category1 Top Category 2 Top Category 4 SubCat1 Sub cat1 SubCat2 xx SubCat3 xx SubCat2 SubCat3 SubCat5 xx SubCat1 Subcat4 Subcat3 Subcat2 xx xxx SubCat3 SubCat5 SubCat4 SubCat6 SubCat4 xxx xx xxx xx xxx SubCat7 SubCat4 SubCat5 SubCat6 SubCat1 SubCat2 xx xxx xx xxx xx SubCat4 xx Preliminary IA

20. Information Architecture Summit 2004 Uzanto Consulting 20 Impact of Project Helped convince management that current site did not match user mental models and that IA redesign was needed. Provided basis for a long-term redesign plan to implement a structured site-wide model. Provided immediate guidance to near-term projects in order to begin moving the site in the long-term direction. Broadened eBay designers perspective on overall design direction and goals by providing: Access to user mental models Site-map for all content and functionality of eBay.com Cleared the road towards a more consistent and unified user experience.