Methodology Matters: Doing Research in the Behavioral and Social Sciences ICS 205 Ha Nguyen Chad Ata

The way we do research matters  Because the meaning of the discovered evidence is based on the way it is obtained  3 domains are always involved when doing research –Substantive  content of interest –Conceptual  ideas that give meaning to the content –Methodological  techniques for conducting research  3 important keywords –Relations which are between elements within a context or embedding system

The Substantive & Conceptual Domains  Substantive Domain –Elements = Phenomena –Relations = Patterns of phenomena –Studying the “actors behaving towards objects in context”  Conceptual Domain –Elements = Properties of the actors’ behaviors... –Relations = Interconnections between elements

The Methodological Domain  Elements = methods = Modes of Treatment –Includes techniques for Measuring Manipulating –Giving instruction to participants –Imposing constraints on the environment –Selecting materials for use –Giving feedback about prior performances –Using experimental confederates Controlling Distributing impact

… Methodological Domain…  Relations = Comparison Techniques –Involve 3 features of the system under study Dependent variables or features –These are measured Independent variables or features –These are manipulated All of the rest of the system’s features –These are relevant but not the focus of the study –Comparison performed to assess covariation or association between dependents and independents

LevelsSubstantiveConceptualMethodological Elements PhenomenaPropertiesModes of Treatment Relations PatternsRelationsComparison Techniques Contexts Ongoing Systems Conceptual Systems Research Strategies The Domains

Research Methods Are Like Tools  Each method offers opportunities not available by other means  Each method also has inherent limitations –Therefore you should use multiple methods in order to counterbalance the flaws in each individual method Make sure to chose methods such that one strength can compensate for another’s weakness

The Research Methods Abstract UnobtrusiveObtrusive Concrete I II III IV Generalizability Maximum Precision Maximum Realism Maximum

Quadrant I: Field Strategies Abstract UnobtrusiveObtrusive Concrete II III IV Generalizability Maximum Precision Maximum Realism Maximum

Quadrant I: Field Strategies  Field Study –Natural & unobtrusive  Field Experiment –Manipulating one variable in an otherwise natural system  decrease in realism –A bit obtrusive

Quadrant II: Experimental Strategies Abstract UnobtrusiveObtrusive Concrete I III IV Generalizability Maximum Precision Maximum Realism Maximum

Quadrant II: Experimental Strategies  Laboratory Experiment –Concocted system Increased precision Decreased realism & generalizability  Experimental Simulation –Similar to lab experiment, but strives to increase realism

Quadrant III: Respondent Strategies Abstract UnobtrusiveObtrusive Concrete I II IV Generalizability Maximum Precision Maximum Realism Maximum

Quadrant III: Respondent Strategies  Sample Survey –Sampling of the population  high generalizability –Low precision  Judgment Study –Here the focus is on the stimulus material, rather than respondents’ attributes  High precision –“actors of convenience”  Low generalizability over the population  Both often done in neutral settings  low realism

Quadrant IV: Theoretical Strategies Abstract UnobtrusiveObtrusive Concrete I II III Generalizability Maximum Precision Maximum Realism Maximum

Quadrant IV: Theoretical Strategies  Formal Theory –Formulating general relations between variables of interest  high generalizability –No concrete system involved  low in realism –No observation  low precision  Computer Simulation –Like experimental simulation, except human participants are not included –Modeling the real world  high realism –Outcomes are predictable  low precision –Modeling a particular system  low generalizability

Questions to keep in mind…  Does the presented material coincide with the strengths & weaknesses of the research strategies that were used?  Is the research evidence well supported by a single or by multiple research strategies? –Are the strategies’ weaknesses and strengths offset such that they counterbalance one another?

Study Design  For every empirical study, observations have to be gathered, combined, and compared –Ex: Given a pair of properties, X and Y, can we tell if X caused Y? X is independent variable, Y is dependent variable –What techniques are available for comparison? Baserates Correlation Difference

Comparison Technique #1  Baserates –How often (at what rate, or what proportion of time) does Y occur? If I do not know how often Y occurs in the general case, then I cannot decide whether the rate of Y in the study is significant (particularly high or low).

Comparison Technique #2  Difference Question –Is Y present under conditions where X is present? Is Y absent when X is absent?

Comparison Technique #3  Correlational Question –Do the values of property X covary with the values of property Y? If X is high, is it likely that Y will also be high? If X is low, is it likely that Y will also be low? –High positive correlation, high negative correlation, little or no correlation  This technique tells you whether X and Y go together but it does not help in determining causality.

Randomization  Randomization – using a random assignment procedure to allocate “cases” to “conditions” –Each case must be equally likely to end up in any given condition –This procedure does not guarantee to rule out all other factors that could affect the results.

Validity of Findings  4 types of validity –Internal validity How close can you come to asserting that the presence of X caused the altered level of Y? –Statistical conclusion validity Is the given result due to chance? –Construct validity How well defined are the theoretical ideas in the study? –External validity How well will the results hold up for replication and how generalizable are the results?

Measure Techniques  Categorized by two questions –Who makes the record of the behavior? –Is the participant aware that his/her behavior is being recorded for research purposes?

6 Major Types of Measures Participant Is AwareParticipant Is Not Aware Self-ReportTrace Measure Observation w/Visible Observer Observation w/Hidden Observer Records of Public Behavior Records of Private Behavior

Strengths and Weaknesses of the Measures  Self-reports –Strength - Low costs, does not require a lot of time to create and use –Weakness - Reactive  Trace measures –Strength – unobtrusive –Weakness – not very versatile, time consuming to gather

Strengths and Weaknesses of the Measures  Observations –Strength – relatively versatile –Weakness – can only be used on overt behavior, vulnerable to observer errors, can be reactive for visible observers, possible ethical issues for hidden observers  Archival records –Strength – good for researching the past, studies for extensive periods of time –Weakness – public records can be reactive, versatility is limited

Techniques for Manipulating Variables  Selection – selecting cases with desired values and assigning them to appropriate conditions  Direct Intervention – manipulating the specific variable directly

Techniques for Manipulating Variables  Inductions –3 types Misleading instructions False feedback Experimental confederates –Ethical issues

Concluding Comments  Results depend on methods. All methods have limitations. Therefore, any set of results is limited.  It is not possible to maximize all desirable features of different methods in any one study.  Each set of results must be interpreted in relation to other evidence relating to the same questions.