1. Gary Suh Vesna Memisevic
Methodology Matters: Doing Research in the Social and Behavioral Sciences Joseph E. McGrath
Gary Suh
Vesna Memisevic
Fall 2007
Informatics 231 Human-Computer Interaction

2. Informatics 231 Human-Computer Interaction
Outline
Introduction
Some Basic Features of the Research Process
Substantive Domain
Conceptual Domain
Methodological Domain
Research Methods as Opportunities and Limitations
Research Strategies: Choosing a Setting For a Study
Quadrant I: The Field Strategies
Quadrant II: The Experimental Strategies
Quadrant III: The Respondent Strategies
Quadrant IV: The Theoretical Strategies
Some Strategic Issues
Fall 2007
Informatics 231 Human-Computer Interaction

3. Informatics 231 Human-Computer Interaction
Outline
Study design
Comparison techniques
Baserates
The Correlation Question
The Difference Question
Randomization and “True experiments”
Sampling, allocation and statistical inference
Validity of findings
Potential classes of measures in social psychology
Strengths and Weaknesses of Types of Measures
Techniques for Manipulating Variables
Selection, direct intervention, induction
Conclusion
Fall 2007
Informatics 231 Human-Computer Interaction

4. Informatics 231 Human-Computer Interaction
Introduction
Doing research – the systematic use of some set of theoretical and empirical tools to try to increase our understanding of some set of phenomena or events
Some of the tools with which
researchers in the social and
behavioral sciences go about
doing research (strategy,
tactics, and operations)
Fall 2007
Informatics 231 Human-Computer Interaction

5. Some Basic Features of the Research Process
Always involves bringing together three sets of things:
some content that is of interest,
some ideas that give meaning to that content, and
some techniques or procedures by means of which those ideas and content can be studied.
Fall 2007
Informatics 231 Human-Computer Interaction

6. Some Basic Features of the Research Process
These three sets of things more formally, as three distinct, though interrelated domains:
The Substantive domain, from which we draw contents that seem worthy of our study and attention;
The Conceptual domain, from which we draw ideas that seem likely to give meaning to our results; and
The Methodological domain, from which we draw techniques that seem useful in conducting that research.
Fall 2007
Informatics 231 Human-Computer Interaction

7. Some Basic Features of the Research Process
Fall 2007
Informatics 231 Human-Computer Interaction

8. Informatics 231 Human-Computer Interaction
Substantive Domain
Phenomena and Patterns of phenomena are the object of our study
The phenomena of interest involve the states and actions of some human systems and the conditions and processes that give rise to and follow from those states and actions.
Fall 2007
Informatics 231 Human-Computer Interaction

9. Informatics 231 Human-Computer Interaction
Substantive Domain
Example – an individual casting a vote in a county election
Fall 2007
Informatics 231 Human-Computer Interaction

10. Informatics 231 Human-Computer Interaction
Conceptual Domain
Properties of the states and actions of those human systems that are the focus of study
Relations refer to any of a variety of possible ways in which two or more elements can be connected
Examples – causal connections, logical relations, chronological relations
Fall 2007
Informatics 231 Human-Computer Interaction

11. Informatics 231 Human-Computer Interaction
Conceptual Domain
Examples – attitude, cohesiveness, power, social pressure, status
Fall 2007
Informatics 231 Human-Computer Interaction

12. Methodological Domain
Basic sets of elements or “tools” by which social and behavioral scientists systematically gather empirical information:
Measuring
Manipulating
Controlling
Distributing Impact
Fall 2007
Informatics 231 Human-Computer Interaction

13. Methodological Domain
Measuring
For assessing the state or magnitude of some property of some actors-behaving-in-context so that the researcher can determine what value or level that feature has for each “case” to be studied
Examples – questionnaire, rating scale, personality test, instruments for observing and recording communications, techniques for assessing the quality of some products resulting from individual or group task performance
Fall 2007
Informatics 231 Human-Computer Interaction

14. Methodological Domain
Manipulating
Techniques for manipulating some property of an actor-behavior-context
Experimental manipulation – making a feature have one particular predetermined value or level for certain “cases” to be studied and another specific preordained value or level for certain other “cases,” so that the effect of differences in that property can be assessed by comparing those two sets of “cases”
Fall 2007
Informatics 231 Human-Computer Interaction

15. Methodological Domain
Manipulating
Social psychologists have tried to manipulate features of the systems they study by a number of techniques, such as:
(a) giving instruction to participants
(b) imposing constraints on features of the environment
(c) selecting materials for use
(d) giving feedback about prior performances
(e) using experimental confederates
(More is said about techniques for manipulating variables near the end of this chapter)
Fall 2007
Informatics 231 Human-Computer Interaction

16. Methodological Domain
Controlling
A set of techniques for controlling the impact of features that are important but that you are not going to measure or manipulate in a particular study
These include:
Techniques for experimental control – you make certain features take the same predetermined value for all cases in the study
Techniques for statistical control – you try to nullify the effects of variations in a given property within a study by “removing” those variations by statistical means
Techniques for distributing the impact – so that such impact can be taken into account in interpretation of results
Fall 2007
Informatics 231 Human-Computer Interaction

17. Methodological Domain
Distributing Impact
Techniques for distributing the impact of a number of features of the system and its context-without directly manipulating or controlling any one of them-so that such impact can be taken into account in interpretation of results
Randomization – the most prominent means; refers to procedures for the allocation of “cases” among various conditions within the study
Fall 2007
Informatics 231 Human-Computer Interaction

18. Research Methods as Opportunities and Limitations
Methods enable but also limit evidence.
All methods are valuable, but all have weaknesses or limitations.
You can offset the different weaknesses of various methods by using multiple methods.
You can choose such multiple methods so that they have patterned diversity; that is so that strengths of some methods offset weaknesses of others.
Fall 2007
Informatics 231 Human-Computer Interaction

19. Research Methods as Opportunities and Limitations
The fundamental principle, in behavioral and social science is that credible empirical knowledge requires consistency or convergence of evidence across studies based on different methods.
Fall 2007
Informatics 231 Human-Computer Interaction

20. Research Strategies: Choosing a Setting For a Study
When you gather a batch of research evidence, you are always trying to maximize three desireable features or criteria:
A. Generalizability of the evidence over the populations of Actors.
B. Precision of the measurement of the behaviors that are being studied (and precision of control over extraneous factors that are not being studied).
C. Realism of the situation or Context within which the evidence is gathered, in relation to the contexts to which you want your evidence to apply.
Fall 2007
Informatics 231 Human-Computer Interaction

21. Research Strategies: Choosing a Setting For a Study
Although you always want to maximize all three of these criteria, A, B and C simultaneously, you cannot do so.
This is the fundamental dilemma of the research process.
Fall 2007
Informatics 231 Human-Computer Interaction

22. Research Strategies: Choosing a Setting For a Study
Fall 2007
Informatics 231 Human-Computer Interaction

23. Quadrant I: The Field Strategies
Fall 2007
Informatics 231 Human-Computer Interaction

24. Quadrant I: The Field Strategies
The two research strategies in quadrant I are the Field Study and the Field Experiment.
Field study – the researcher sets out to make direct observations of “natural”, ongoing systems, while intruding on and disturbing those systems as little as possible.
Field experiment – also works within an ongoing natural system as unobtrusively as possible, except for intruding on that system by manipulating one major feature of that system.
Fall 2007
Informatics 231 Human-Computer Interaction

25. Quadrant I: The Field Strategies
The essence of both of the strategies in quadrant I, the field study and the field experiment, is that the behavior system under study is “natural”, in the sense that it would occur whether or not the researcher were there and whether or not it were being observed as part of a study.
Fall 2007
Informatics 231 Human-Computer Interaction

26. Quadrant II: The Experimental Strategies
Fall 2007
Informatics 231 Human-Computer Interaction

27. Quadrant II: The Experimental Strategies
Laboratory experiment – the investigator deliberately concocts a situation or behavior setting or context, defines the rules for its operation, and then induces some individuals or groups to enter the concocted system and engage in the behaviors called for by its rules and circumstances.
Experimental simulation – the researcher attempts to achieve much of the precision and control of the laboratory experiment but to gain some of the realism (or apparent realism) of field studies.
Fall 2007
Informatics 231 Human-Computer Interaction

28. Quadrant II: The Experimental Strategies
The two strategies in Quadrant II, in contrast to those of Quadrant I, involve concocted rather than natural settings.
The laboratory experiment and the experimental simulation are strategies that involve “actor-behavior-context” systems that would not exist at all were it not for the researcher’s interest in doing the study.
Fall 2007
Informatics 231 Human-Computer Interaction

29. Quadrant III: The Respondent Strategies
Fall 2007
Informatics 231 Human-Computer Interaction

30. Quadrant III: The Respondent Strategies
Sample survey – the investigator tries to obtain evidence that will permit him or her to estimate the distribution of some variables, and/or some relationships among them, within a specified population
Examples – public opinion surveys on voting intentions, political preferences, buying intentions
Fall 2007
Informatics 231 Human-Computer Interaction

31. Quadrant III: The Respondent Strategies
The strategies of Quadrant III concentrate on the systematic gathering of responses of the participants to questions or stimuli formulated by the experimenter, in contrast to the observation of behaviors of the participants within an ongoing behavior system
Studies are usually done under “neutral” conditions of room temperature, lighting, chair comfort to nullify any effects of the behavior setting or context on the judgments that are the topic of study.
Fall 2007
Informatics 231 Human-Computer Interaction

32. Quadrant IV: The Theoretical Strategies
Fall 2007
Informatics 231 Human-Computer Interaction

33. Quadrant IV: The Theoretical Strategies
Formal theory – the researcher focuses on formulating general relations among a number of variables of interest
Computer simulation – a complete and closed system that models the operation of the concrete system without any behavior by any system participants
Fall 2007
Informatics 231 Human-Computer Interaction

34. Quadrant IV: The Theoretical Strategies
The inclusion of these two strategies reminds us of the importance of the theoretical side of the research process.
One of the more powerful general strategies for research is the simultaneous use of one of the theoretical strategies (say, the formulation of a general theory) and one of the empirical strategies (for example, a laboratory experiment).
Fall 2007
Informatics 231 Human-Computer Interaction

35. Informatics 231 Human-Computer Interaction
Some Strategic Issues
Does the material, as presented, properly reckon with the strengths and weaknesses of the research strategies it encompasses?
To what extent is the research evidence on each problem or topic based on use of only a single research strategy, and therefore limited by the weaknesses of that strategy; and to what extent is that body of evidence based on use of multiple, complementary strategies, with agreement or convergence among the findings attained via the different strategies? 35
Fall 2007
Informatics 231 Human-Computer Interaction

36. Informatics 231 Human-Computer Interaction
Study design
Gathering observations
Aggregation and partitioning
Comparison on the data set
Fall 2007
Informatics 231 Human-Computer Interaction

37. Informatics 231 Human-Computer Interaction
Study design
Comparison depends on:
What’s included in the study (what phenomena, what properties, what model of treatment for variables)
What system works (which substantive system we study, paradigms, strategies)
What conceptual relations have been posted (which properties are linked)
What comparison techniques are available
Fall 2007
Informatics 231 Human-Computer Interaction

38. Comparison Techniques
Three base forms:
Baserates (how often?)
Correlations (are properties related; do they occur together?)
Differences (comparison or difference)
Fall 2007
Informatics 231 Human-Computer Interaction

39. Informatics 231 Human-Computer Interaction
Base rates
How often Y occurs in the general case, as a basis for deciding whether the rate of Y in some particular case is “notably” high or low
Problems - difference in interpretation - various political, economic and social issues
Fall 2007
Informatics 231 Human-Computer Interaction

40. The Correlation Question
Is there covariation in the values of two properties or features of system?
Correlations: High – Low; Positive – Negative; Zero
Linear or nonlinear relation between two or more variables
Can asses conceptual relations that imply covariation between two or more variables, but cannot asses any conceptual relations that are causal in their implications
Fall 2007
Informatics 231 Human-Computer Interaction

41. The Difference Question
Whether Y is present under condition where X is present and whether Y is absent when X is absent?
Samples are separated in two groups, one with members which have performance X, and another with members who don’t have performance X.
Some set of tasks is given to both of groups, and average task performance is compared for both groups.
Groups must be comparable on factors that might affect task performances!
Informatics 231 Human-Computer Interaction

42. Randomization and True Experiments
Randomization = using random assignment procedure to allocate cases to conditions (in order to remove artifacts and observations that happen by chance)
In order for the study to be called a “true experiment”, study must include some randomization
Random allocation procedure doesn’t guarantee an equal distribution of any of the potential factors among the conditions being compared
Fall 2007
Informatics 231 Human-Computer Interaction

43. Randomization and True Experiments
Possible problems:
Reduce the scope of study, as some variables are hold constant, and therefore experimental variables will occur only at a few levels
Reduce the realism of context of your study, designed the tasks serves us - not the participants' - purposes.
Fall 2007
Informatics 231 Human-Computer Interaction

44. Sampling, Allocation and Statistical Inference
The way we choose cases which will be included in our study (from larger population of potential cases) effect credibility of the evidence resulting from the study
How to choose right nature of the “random sample” population?
Fall 2007
Informatics 231 Human-Computer Interaction

45. Sampling, Allocation and Statistical Inference
We need to use random sample in that way that we can apply results of the study to the population of which cases constitute a random sample.
Fall 2007
Informatics 231 Human-Computer Interaction

46. Sampling, Allocation and Statistical Inference
Sampling case procedure: which cases from larger population will be included in study.
Allocation case procedure: which condition every given cases (already selected as a part of the study) will be assigned to
Random refers to procedure, not outcome!
Fall 2007
Informatics 231 Human-Computer Interaction

47. Sampling, Allocation and Statistical Inference
Size of the samples?
The larger the number the more distributed those cases will approach the idealized random distribution
Uneven distribution doesn’t occur often if only chance is operating.
Fall 2007
Informatics 231 Human-Computer Interaction

48. Informatics 231 Human-Computer Interaction
Validity of Findings
Four different types of validity (Cook & Campbell):
Statistical conclusion validity
Internal validity
Construct validity
External validity
Fall 2007
Informatics 231 Human-Computer Interaction

49. Informatics 231 Human-Computer Interaction
Validity of Findings
Statistical conclusion validity:
Difference arisen just by chance?
Relationship between cause and effect
Internal validity:
How close can you come to
asserting that the present of X
caused the altered level of Y values?
Difference in Y associated with a
difference in X does not necessarily
imply a causal role for X
Are there other factors which may
have been covary with X and they, rather
than X, might have produced the change in Y
Fall 2007
Informatics 231 Human-Computer Interaction

50. Informatics 231 Human-Computer Interaction
Validity of Findings
Construct validity:
How well defined are the theoretical ideas in our study?
How clearly understood are the conceptual relations
being explored?
How well are mapping of concepts and relations
External validity:
How confident you can be that your
findings will hold upon replication or
how general are your our findings.
How confident you can be that
you can make predictions ?
Fall 2007
Informatics 231 Human-Computer Interaction

51. Potential Classes of Measures In Social Psychology
Different techniques to measure presence or values of the specific features of the human system
Participants awareness:
Natural or unnatural behavior
Reactivity of measures
Who makes which record of behavior:
Participants
Knowingly make (self reports)
Unwittingly make (trace measures)
Investigators
Observations (visible or hidden observer)
Third party (for non research purposes)
Archival records (private or public behavior)
Fall 2007
Informatics 231 Human-Computer Interaction

52. Strengths and Weaknesses of Types of Measures
Self reports
Observations by visible observer
Observations by hidden observer
Archival records of public behavior
Archival records of private behavior
Trace measures
Reactivity
High potential
(Some participants are aware that their response will be recorded, that may influence how they response – they might try to make good impression, to give socially desirable answers or to try to help investigation)
High
Low
None
Versatility
Versatile to potential contents and to the population to which they would apply to
Can only be used on overt behavior (not feelings, expectations, etc)
Med-Low
Low (many concepts not available)
Costs
Medium
High (sometimes)
Dross Rates (discarded info.)
High (per observer hour bases)
High(per observer hour bases)
Very high (links to concept sometimes very week)
Time
Errors
Vulnerable to observer error
Ethical concerns, Vulnerable to observer error
No possibility to cross validate
Fall 2007
Informatics 231 Human-Computer Interaction

53. Techniques for Manipulating Variables
Techniques for manipulating variables are not very well specified
Three classes of techniques:
Selection
Direct intervention
Inductions
Manipulating variables are use to create an experimental manipulation of features of a situation
Fall 2007
Informatics 231 Human-Computer Interaction

54. Informatics 231 Human-Computer Interaction
Selection
Advantage: Selection is the most convenient means to make sure that all cases of a given conditions are alike on a certain variable and that all cases of another comparison condition differ o that variable
Disadvantage: conclusion drawn is more or less unreliable unless all other potential factors are removed (uncertainty in nature of variable manipulated (additional meaning))
Fall 2007
Informatics 231 Human-Computer Interaction

55. Informatics 231 Human-Computer Interaction
Direct intervention
Advantages:
Surest way of achieving a definite and specific variable manipulation
Permits random allocation (distributing impact of other factors and variables, that we are not studying)
Nor likely to be costly or time consuming
Low dross rate
Disadvantages:
Applicable only for relatively overt
and tangible variables
May suffer from reactivity effects
and experimental demands
Fall 2007
Informatics 231 Human-Computer Interaction

56. Informatics 231 Human-Computer Interaction
Induction
Three major forms
Use of misleading instruction to the participants
Use of false feed back
Use of experimental confederates
Advantages:
Fairy versatile in context of variables
Normally low in cost and time
Disadvantages:
Involves deception (raises ethical concerns)
Risk of detection
Risk of experimental demands
Fall 2007
Informatics 231 Human-Computer Interaction

57. Informatics 231 Human-Computer Interaction
Conclusion
Results depend on methods. All methods have limitations. Hence, any set of results is limited.
It is not possible to maximize all desirable features of method in any one study; tradeoffs and dilemmas are involved.
Each study (each set of results) must be interpreted in relation to other evidence bearing on the same questions.
Fall 2007
Informatics 231 Human-Computer Interaction

58. Informatics 231 Human-Computer Interaction
Questions?
Fall 2007
Informatics 231 Human-Computer Interaction