1. Requirements Gathering & Task Analysis – Part 2 of 5 Why, What and How – Methods This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory Abowd, Jim Foley, Diane Gromala, Elizabeth Mynatt, Jeff Pierce, Colin Potts, Chris Shaw, John Stasko, and Bruce Walker. Comments directed to foley@cc.gatech.edu are encouraged. Permission is granted to use with acknowledgement for non-profit purposes. Last revision: Jan 2014. foley@cc.gatech.edu

2. UI Design - Georgia Tech How – Some (Not All) User & Task Analysis Methods Observation Interviews Ethnography Surveys/Questionnaires Focus groups Study documentation Study competitive products Data recording - automated Data recording – diary (log) study

3. UI Design - Georgia Tech Formative & Summative Evaluation Formative evaluation  Conducting this process to help guide the formation (ie, design) of a UI Summative Evaluation  Conducting this process to help summarize (sum up) the effectiveness of an existing or developmental UI Many of the user & task analysis techniques can be used for both formative and summative evaluation  Our focus right now is on formative evaluation  Will revisit some of the methods again later

4. UI Design - Georgia Tech Observation - From a Distance Watch users do what they do  Step-by-step procedures  Pauses to figure out what to do  Mistakes/errors Do this today or tomorrow  Automatic Checkout, ATM, parking meter  Don’t scare off the observees!! Record (with permission)

5. UI Design - Georgia Tech Observation – up Close Often combined with “Thinking Out Loud” Encourage user to verbalize what they are thinking  Put them at ease first!  Not every user is good at this  Hard for user to do for long time; need breaks Record (with permission) Up close and at a distance not mutually exclusive

6. UI Design - Georgia Tech Pros and Cons? Pros Cons

7. UI Design - Georgia Tech Interviews Structured – “Just the facts”  Fixed set of questions –How do you know what questions to ask?  May miss subtleties and big picture Unstructured – A conversation  Go with the flow –May learn what questions to ask Semi-structured – start with focused questions, move to open- ended discussion  Good balance, often appropriate  Training – process + domain knowledge Do some homework first

8. UI Design - Georgia Tech Interviews Know why you are asking questions - don’t waste time Plan for effective question types –How do you perform task x? –Why do you perform task x? –Under what conditions do you perform task x? –What do you do before you perform…? –What information do you need to…? –Whom do you need to communicate with …? –What do you use to…? –What happens after you…? –What is the result or consequence of…? –What is the result or consequence of NOT…?

9. UI Design - Georgia Tech Interviews: Typical Open- Ended Questions Why do you do this (whatever the task is you are studying) How do you do this?  Gets at task-subtask structure  Then ask about each subtask Why do it this way rather than some other way?  Attempts to get user to explain method so you can assess importance of the particular way of doing task What has to be done before you can do this?  To understand sequencing requirements

10. UI Design - Georgia Tech Interviews: More Typical Open- Ended Questions Please show me the results of doing this Do errors ever occur when doing this?  If answer is ‘yes,’ then learn why occur  How do you discover the errors, and how do you correct them? Encourage digressions; ask for elaborations  Pause before asking next question What else should I have asked you?

11. UI Design - Georgia Tech Interviews: Domain Experts Expert may describe how it should be done  Not necessarily how it is done :-)

12. UI Design - Georgia Tech Interviews: Users vs. Management Folks who are not in the trenches doing the work may have a different view than those who are actually doing whatever it is you are studying. In organizations, can be political

13. UI Design - Georgia Tech Ethnography Similar to Observation, but more intense Observe people in their cultural context  Because behavior most meaningful in context Immerse oneself in situation you want to learn about (has anthropological and sociological roots); use  In person observation  Interviews – formal and over coffee  Audio/video recording – after trust develops  Learn to do the work yourself! (biggest difference from observation)

14. UI Design - Georgia Tech Ethnography: Types of Findings Can be both  Qualitative –Observe trends, habits, patterns, …  Quantitative –How often was something done, what per cent of the time did something occur, how many different …

15. UI Design - Georgia Tech Ethnography: Drawbacks Time required  Can take weeks or months for large systems Scale  Most use small numbers of participants just to keep somewhat manageable Type of results  Highly qualitative, may be difficult to present/use Acquired skill – “learn by doing”  Identifying and extracting “interesting” things is challenging

16. UI Design - Georgia Tech Questionnaires Make questions clear and specific Ask some closed questions with range of answers  Sometimes also have a no opinion option, or other answer option Do test run with a few people Use computer-based tools

17. UI Design - Georgia Tech Questionnaires – Likert Scale Seven-point Likert Scale (use odd #) Slightly leading question  Could be seen as favoring one answer –Help was worthwhile –Agent intrusive To avoid issue….

18. UI Design - Georgia Tech Questionnaires – Semantic Differential Annoying Easy to use Attractive Secure Helpful Modern Pleasing Difficult to use Unattractive Not secure Unhelpful Dated X X X X

19. UI Design - Georgia Tech Questionnaires – Typical Questions Rank the importance of each of these tasks (give a list of tasks) List the four most important tasks that you perform (this is an open question) List the pieces of information you need to have before making a decision about X, in order of importance Are there any other points you would like to make? (open-ended opinion question; good way to end) Same questions can be used in interview and in questionnaire; difference is in follow-up opportunity

20. Questionnaire Pros/Cons ProsCons UI Design - Georgia Tech

21. Focus Groups Group of individuals - 3 to 10  Use several different groups with different roles or perspectives  Careful about power relationships – separate groups  Careful about few people dominating discussion Use structured set of questions  More specific at beginning, more open as progresses  Allow digressions before coming back on track Relatively low cost, quick way to learn a lot Audio or video record, with permission Blog – open-ended focus group Pros/cons?

22. UI Design - Georgia Tech Study Documentation Similar in some ways to the expert interview Often describe how things should be done rather than how they are done  Try to understand why not done “by the book”

23. UI Design - Georgia Tech Study Competitive Products Looking for both good and bad ideas  Functionality  UI style Try to understand why are successful or unsuccessful Measure task completion times for new and experienced users  Upper bounds for new product

24. UI Design - Georgia Tech Data Recording – Time-stamped Event Log High-level, such as  Web page visits  Commands used Low-level, such as  Keystroke and mouse events  Device/person location; think Aware Home Lots of data - need analysis tools Pros/Cons?

25. UI Design - Georgia Tech Data Recording - Manual (Log Study) Participants keep a log for days or weeks Asked to record relevant events May do follow-up interview Lot of detailed text to analyze after the fact Pros/cons

26. UI Design - Georgia Tech Methods to Use Depends on… Resources Current knowledge of tasks and users Context  Can’t use talking out loud if tasks involve two people working together Essential to use some methods Not likely you will use all methods Which best for class project? Understand pros/cons of each  On own and relative to other methods

27. End of Part 2 UI Design - Georgia Tech

29. Ethnography: Observations Carefully observe everything about users and their environment  Structure and language used in work  Individual and group actions  Culture affecting work  Explicit and implicit aspects of work; what is said, what is left unsaid Example: Office work environment  Business practices, rooms, artifacts, work standards, relationships between workers, managers, …

30. UI Design - Georgia Tech Why is Ethnography Useful? Helps gain rich assessment of user needs  Help to define requirements Uncovers true nature of user’s job  Discovers things that are outside of job description or documentation Allows you to play role of end-user better  Can sit in when real users not available Open-ended and unbiased nature promotes discovery