Software Usability Course notes for CSI University of Ottawa Section 4: Usability in the Software Engineering Process Timothy C. Lethbridge

Lethbridge - CSI Section 4: Usability and the SE Process2 An Assertion Lack of usability is the most critical problem facing software engineering In other words, the biggest gains in software quality productivity cost-reduction user satisfaction profitability, etc. … would come from focusing on usability and related issues.

Lethbridge - CSI Section 4: Usability and the SE Process3 Evidence for the assertion Failures of projects due in significant part to usability E.g. FAA air traffic control systems Observations of ‘ great ’ software vs. not-so-great with respect to usability Great: —Google (many of its products) —Microsoft Office (some of it) —Mac-OS (much of it) Not-so-great: —Much in-house custom software —Software development environments (many aspects) —Web sites (far too many of them) I have observed many users formally and informally My survey of practitioners (next slide) Analysis of benefits and cost savings

Lethbridge - CSI Section 4: Usability and the SE Process4 Practitioner survey about 75 topics taught to computer science students Lethbridge, T.C. (2000), “ What Knowledge is Important to a Software Engineer? ”, IEEE Computer, May, pp

Lethbridge - CSI Section 4: Usability and the SE Process5 More on the survey HCI was second in terms of “ knowledge gap ” Where importance most exceeds current knowledge The top 5 (out of 75) Negotiation HCI/user interfaces Leadership Real-time system design Management I believe not much has changed since 1998

Lethbridge - CSI Section 4: Usability and the SE Process6 Sources of resistance to adopting usability practices Primarily: Inadequate education of students, practitioners and managers Which leads to: Lack of introspection ability of software developers —Unable to think like users Lack of integration of usability into core development processes Persistent beliefs —Most software is usable enough —Usability can be fixed at the “ end ” —Usability should be left to the HCI experts

Lethbridge - CSI Section 4: Usability and the SE Process7 Causes of the problems in education Faculty don ’ t have the background either The field of HCI is seen as distinct and separate The field is perceived as too soft: There is a tendency to focus teaching and research on hard deterministic areas There is little industry push for research in this area There is little employer pull for greater education

Lethbridge - CSI Section 4: Usability and the SE Process8 Some partial solutions The SE and HCI communities need to work more closely together Core UI development and evaluation topics should be considered jointly part of the two fields Improve education by ensuring HCI permeates curriculum models, certifications, accreditation, etc. Enable corporations to be certified for their usability capabilities Would help “ pull ” education up by creating a need for professionals Clients might come to learn that better software comes from such companies

Lethbridge - CSI Section 4: Usability and the SE Process9 Should aspects of HCI be considered also integral to Software Engineering? I argue yes: Usability should have no special status as compared to reliability, efficiency, maintainability, etc. Design involving users and their needs must drive software development

Lethbridge - CSI Section 4: Usability and the SE Process10 Should aspects of HCI be considered also integral to Software Engineering? Yes, but this does not preclude HCI specialists Analogies: —A software engineer must be capable of designing architectural elements for storing data and information -But there will always need to be database specialists —Similar argument can be made for -Security experts, performance experts, requirements experts, etc. Core UCD techniques (coming slides) can be part of the SE field and part of a broader HCI field

USER-CENTRED DESIGN REVISITED Lethbridge - CSI Section 4: Usability and the SE Process11

Lethbridge - CSI Section 4: Usability and the SE Process12 User centered design (UCD) A term encompassing a range of software engineering approaches that focus on the user We have talked about many of the techniques

Lethbridge - CSI Section 4: Usability and the SE Process13 UCD main stages Pre-design Design stage Post-design

Lethbridge - CSI Section 4: Usability and the SE Process14 UCD Pre-design techniques: Understand classes of user Understand tasks Perform a competitive analysis i.e. usability testing of competing products Set usability goals using the metrics we discussed earlier

Lethbridge - CSI Section 4: Usability and the SE Process15 UCD Design-stage techniques Parallel design Many developers generate UI ideas in parallel on their own They meet together and pick the best ideas Has maximum chance of discovering good ideas, without constraining the possibilities too soon

Lethbridge - CSI Section 4: Usability and the SE Process16 UCD Design-stage techniques - 2 Involving users Critiquing design Discovering problems —with the UI —with task descriptions —through user testing Contributing ideas But users are not designers so don't ask them what do you want?

Lethbridge - CSI Section 4: Usability and the SE Process17 UCD Design-stage techniques - 3 Coordinating the interfaces of all parts of the system Including documentation, training etc. Applying guidelines and heuristic rules Prototyping

Lethbridge - CSI Section 4: Usability and the SE Process18 UCD Design-stage techniques - 4 Performing empirical testing using simulations and prototypes To discover problems Measuring usability of the simulations and prototypes Performing iterative design Design Evaluate Redesign

Lethbridge - CSI Section 4: Usability and the SE Process19 UCD Post-Design stage Gather on-going feedback about potential improvements

Another term related to UCD “Outside-in development” Lethbridge - CSI Section 4: Usability and the SE Process20

ECONOMICS OF USABILITY Lethbridge - CSI Section 4: Usability and the SE Process21

Lethbridge - CSI Section 4: Usability and the SE Process22 An example of economic analysis in UCD: Economics of heuristic evaluation We will look at Number of people needed Cost-benefit analysis

Lethbridge - CSI Section 4: Usability and the SE Process23 How many evaluators should you use in a heuristic evaluation? Given: N = total number of problems to be found —Clearly depends on: -Quality of the original design or work performed so far -Size and complexity of the system  = proportion of all usability problems found by a single evaluator: —Studies have shown this averages 34% (range 19% to 51%) The percent of the total number of different usability problems found by aggregating reports from i evaluators is: PercentProblemsFound(i) = 1-(1-  ) i

Lethbridge - CSI Section 4: Usability and the SE Process24 We can plot the effect of this formula for different 

Lethbridge - CSI Section 4: Usability and the SE Process25 We can also look at specific figures for the average case where  =34% EvaluatorsPercent Problems found Given a desired percentage of problems to be found, we can determine the number of evaluators needed by inverting the formula: i = ln(1-P)/ln(1- ) It is not economical to try to find every problem but where should we stop?

Lethbridge - CSI Section 4: Usability and the SE Process26 How much cost savings can we achieve by finding a given problem? This depends on many factors, but The worst problems are the easiest to find and will be found first As we put more effort into finding problems, we will find problems that are less and less severe We can model the cost savings from fixing a problem as seconds of wasted time per user session Very approximate, but useful —Some problems will only occur once, then never again because the user will learn to avoid them

Lethbridge - CSI Section 4: Usability and the SE Process27 Visualizing the diminishing returns Estimated number of seconds per session saved by fixing the nth worst problem found in a particular project.

Lethbridge - CSI Section 4: Usability and the SE Process28 Deciding what to fix and the resulting savings From the previous chart, find the total savings if we just solve the first n problems We can divide this by the n to find the average savings per problem had we just found the first n A typical answer for this would be 2 seconds

Lethbridge - CSI Section 4: Usability and the SE Process29 Deciding what to fix and the resulting savings Some data from the previous chart, where there were about 200 problems eventually found nth problems/session savedaverage savings had we just foundfound the first n problems 110.6s10.6s 107.4s8.9s 204.9s7.4s 402.2s5.4s 601.0s4.1s s3.2s s2.6s In a large organization, we can gather data to calibrate our own charts and tables like the above.

Lethbridge - CSI Section 4: Usability and the SE Process30 Deciding how many evaluators to include: Costs Fixed costs to organize the evaluation: (no matter how many evaluators): —Planning —Preparing materials —Writing report and communicating results to developers —Typical estimate for a small system -= 75 hours work -= $4500

Lethbridge - CSI Section 4: Usability and the SE Process31 Costs continued Variable costs for each evaluator —Weighted salary for time spent —Cost of analyzing the evaluator's report —Typical estimate for an average system -8 hours work for each evaluator ($480) -2 hours work analyzing the report ($120) -Total = $600 Total costs = $ i —Where i is the number of evaluators

Lethbridge - CSI Section 4: Usability and the SE Process32 Deciding how many evaluators to include: Benefits Total benefits vary substantially depending on nature of system. In some published studies, the benefit averaged $15000 per problem On the following slides are two benefit calculation examples

Lethbridge - CSI Section 4: Usability and the SE Process33 Benefits example 1: small mass-market system Three factors to consider: 1. Time saved per session per problem Will obviously vary from problem to problem and session to session Reasonable average 2s saved per session per problem (similar to data shown earlier) 2. Typical number of sessions of use per system Assume an average of 50 sessions 3. Number of users (licenses) Assume an average of 3000 users

Lethbridge - CSI Section 4: Usability and the SE Process34 Benefits example 1 - continued Therefore: total time = 2s * 50 sessions * 3000 users = seconds per problem! = 83.3 hours of saved time If weighted at $60 per hour: Potential benefit is $5000 per problem Assuming only half the problems will get fixed: Benefit = $2500 per problem found Or added value to a license per problem fixed = 2s * 50 sessions = 100s = $1.66

Lethbridge - CSI Section 4: Usability and the SE Process35 Benefits example 2: small in-house system The same three factors to consider: 1. Same average time savings per session (2s) 2. More average sessions over lifetime (200) 3. Fewer users (40) Therefore: total time = 2s * 200 sessions * 40 users = s / problem = 4.4 hours of saved time Using the same weighted value of $60 per hour: Potential benefit is $266 per problem or $133 per problem actually fixed.

Lethbridge - CSI Section 4: Usability and the SE Process36 Cost benefit analysis So will a usability study be economical? Let ’ s put the costs and benefits together

Lethbridge - CSI Section 4: Usability and the SE Process37 Example 1: Mass-market system InspTotal benefit % probstotalben/cost cost/prob foundbenefitsratio

Lethbridge - CSI Section 4: Usability and the SE Process38 Example 2: In-house system InspTotal benefit % probstotalben/cost cost/prob foundbenefitsratio

ODDS AND ENDS Lethbridge - CSI Section 4: Usability and the SE Process39

Lethbridge - CSI Section 4: Usability and the SE Process40 Prototyping A prototyping tool should: Be easy to learn and use Possess good visual capabilities Allow easy extension Give full support for the types of interface being developed Support modular programming

Lethbridge - CSI Section 4: Usability and the SE Process41 Types of prototyping Horizontal prototyping Complete UI, but no underlying functionality Vertical prototyping In-depth functionality of only certain parts Scenario prototyping Selected paths only

Lethbridge - CSI Section 4: Usability and the SE Process42 The design rationale document Allows you to keep control of the various alternatives considered and decisions made Explain your usability decisions so others know the reasoning you followed

Lethbridge - CSI Section 4: Usability and the SE Process43 Discount usability engineering User and task observation Cheap prototyping and scenarios Simplified thinking aloud Heuristic evaluation Key goal, communicate qualitative information to developers in order to improve interfaces as opposed to qualitative measures A study was performed of iterative design involving discount usability engineering Improvements of 38% after each stage

Lethbridge - CSI Section 4: Usability and the SE Process44 Consumability - Closely allied to usability Ability for customers to get rapid return on investment. Factors include all aspects of usability installability / uninstallability lack of inter-product dependency/interaction low footprint: lack of run-time resource consumption backward-compatibility integratability appropriate prioritization of iterations and features to yield business value low purchase cost

RATING CORPORATE MATURITY WITH RESPECT TO USABILITY Lethbridge - CSI Section 4: Usability and the SE Process45

Lethbridge - CSI Section 4: Usability and the SE Process46 The Model for Usability Maturity (MUM) (proposed by Lethbridge!) A 5-level CMM-like categorization of organizations —Based on their interaction with human end-users in software development —(Each level beyond 1 builds on the previous levels) Level 1: Haphazard Level 2: Defined input from users / usability awareness Level 3: Iterative interaction with users / design for usability Level 4: Controlled and measured involvement of users Level 5: Continually improving usability

Lethbridge - CSI Section 4: Usability and the SE Process47 Level 1: Haphazard If you can get users to use the system, then it is considered good enough!

Lethbridge - CSI Section 4: Usability and the SE Process48 Level 2: Defined input from users / usability awareness Users involved in requirements reviews Feedback from users at reviews is incorporated into the next stage Design team members have basic training in usability Design team adheres to usability standards for look and feel Reuse of well-understood controls, styles etc.

Lethbridge - CSI Section 4: Usability and the SE Process49 Level 3: Iterative interaction with users / design for usability Users actively involved in decision making Use case / task analysis Competitive analysis Design with careful attention to usability guidelines Usability design decisions are carefully analyzed and the decisions are recorded E.g. options analysis, tables of pros and cons

Lethbridge - CSI Section 4: Usability and the SE Process50 Level 3: Continued User input and feedback with repeated prototypes From paper prototypes to functional prototypes Informal qualitative observations of users or heuristic evaluations Discount usability engineering

Lethbridge - CSI Section 4: Usability and the SE Process51 Level 4: Controlled and measured involvement of users User input and feedback at all stages Design team has trained usability experts or cognitive scientists Careful cost-benefit optimization including usability Setting of quantitative usability objectives For all parts of the system that will be subject to regular or critical-situation use For learnability, efficiency of use, etc.

Lethbridge - CSI Section 4: Usability and the SE Process52 Level 4: Continued User-centered design with formal usability studies Measurement of usability so as to determine progress towards goals

Lethbridge - CSI Section 4: Usability and the SE Process53 Level 5: Continually improving usability Active development of new UI understanding and innovations Formal experiments to validate new UI modalities metaphors, controls, widgets, styles etc. Anthropological studies of human tasks to enable optimized UI design Scientific study of Users and their work practices Usability and the software engineering process

Lethbridge - CSI Section 4: Usability and the SE Process54 Potential benefits of MUM It provides a visible framework that companies can use to incrementally improve their practices If even a few companies adopted it and were certified, they might have a competitive advantage in some situations