Presentation is loading. Please wait.

Presentation is loading. Please wait.

The art and science of measuring people l Reliability l Validity l Operationalizing.

Similar presentations


Presentation on theme: "The art and science of measuring people l Reliability l Validity l Operationalizing."— Presentation transcript:

1 The art and science of measuring people l Reliability l Validity l Operationalizing

2 Overview of design and analysis Posing a usability question l Conceptualizing the question l Operationalizing the related concepts l Identifying Independent, Dependent, & Controlled Variables l Developing the Hypothesis

3 Choosing the testing method What method is appropriate for the current situation? (experiment, empirical user study, observation, surveys etc.) >> choice of method as a trade off between control realism l Experimental, Quasi-Experimental and Non- Experimental Methods.

4 Collecting data The art of finding and recruiting participants l A practical view of randomization: Randomization and Pseudo Randomization l Random Selection and Random Assignment. l Practical issues about sample size and statistical power.

5 Experimental designs: Single Factor Experiments Statistical Hypothesis Testing l Estimates of Experimental Error l Estimates of Treatment Effects l Evaluation of the Null Hypothesis l Various ANOVA models

6 Multi Factor Experiments Advantages of the factorial design l Interaction Effects l The power of within-subjects designs (reduction of variance) l The two factorial experiment l Higher Order Factorial Designs

7 Statistics for Experimental Designs Means, Medians, Standard Deviations, Standard Errors l CHI Square l T-tests: Within Subjects and Independent Samples l ANOVA: l Univariate and Multivariate l Independent Samples and Repeated Measures l Single Factor and Multiple Factors l Estimates of Effect Size

8 Statistics for Non-Experimental Designs Correlations, Scatter Plots, Partial Corrs l Multiple Regression l Factor Analysis, Cluster Analysis and Multidimensional Scaling

9 Surveys and Questionnaires The design of surveys and questionnaires l How to frame questions l Kinds of scales: Likert, Semantic Differential etc. l Analyzing survey data: which items are useful, Item Response Theory Forming a scale to measure an attribute, e.g., satisfaction. Reliability, validity of scale

10 Measuring Individual Differences How to test for individual differences within users l Kinds of individual differences variables: -demographic: such a age, gender etc. -situational: motivation, interest, fatigue -cognitive: memory, cognitive style etc., -personality: internal/external locus of control l How to analyze existing data to identify individual differences, and how to design studies to test for individual differences?

11 Plethora of Methods: Perception of download speed and all the ways to study it...

12 Survey: Are people bothered by long download times? Sample Question: – How often do you leave a site without waiting for the first page to download? »0-5% of times »5-10% of times »10 % and higher –In your opinion, how important are the below web site characteristics. Rate their relative importance. »Download speed »site content »site interactivity Possibilities: Task based surveys

13 Observation: Do people seem to like fast (without graphics) sites as compared to slow (with graphics) sites? Method: Make two versions of a site, one with sophisticated graphics (slower site) and the other mostly text (faster site). Ask subjects to browse / complete a task on both sites. Measurement: Watch participants for signs of frustration or satisfaction with speed of site

14 Experiment: Relationship of perceived download times to actual download times? Method: Find similar sites with differential speeds. Ask people to complete the same tasks on both sites. Give them some interesting and some boring tasks, and less than enough time to complete the task. Measurement: Log the clicks of the users as they traverse the sites. How many of the interesting and how many boring tasks did they complete. Relate that to download speed of site. Do some users tend to be more frustrated with slower sites.

15 User Logs: Do people leave sites while waiting for slow pages to download? Method: Find similar sites with differential speeds. Analyze the server logs for the sites. Measurement: Log the clicks of the users as they traverse the sites. How many of the interesting and how many boring tasks did they complete. Relate that to download speed of site. Do some users tend to be more frustrated with slower sites.


Download ppt "The art and science of measuring people l Reliability l Validity l Operationalizing."

Similar presentations


Ads by Google