1. McGraw-Hill/Irwin Copyright © 2011 by The McGraw-Hill Companies, Inc. All rights reserved. Using Nonexperimental Research

2. 2 Purely observational research has two features Correlational designs that do not involve manipulating independent variables Use trained observers

3. 3 A behavioral category includes the general and specific classes of behavior to be observed Also called coding schemes or ethnograms (in animal research) Categories must be operationally defined Developing behavioral categories may be easy or challenging Behavioral categories must be clearly defined to avoid confusion Begin with clear goals for research Clearly define all hypotheses Keep categories as simple as possible Avoid temptation to accomplish too much in one study

4. 4 Two methods can be used to help develop behavioral categories Make preliminary observations of behavior Become familiar with behavior observed Create a list of behaviors of interest that can be condensed later Conduct a literature search to find research that used behavioral categories You might find “perfect” categories or you can adapt categories for your research Take the time to carefully construct your categories Adjustments are more easily made before rather than during your research

5. 5 Frequency Method Record the frequency with which a behavior occurs within a time period Duration Method Record how long a behavior lasts Intervals Method Divide the observation period into several discrete time intervals (e.g., ten 2-minute intervals), and record whether a behavior occurs within each interval

6. 6 Observational research traditionally involves recording single events in an observation Behavior sequences can also be recorded Recording a number of behaviors occurring sequentially Provides a more complex picture of behavior than recording single events Requires more effort than recording single events, but data yielded is worth the effort

7. 7 Time Sampling Scan subjects for a specific period (e.g., 30 seconds), and then record your observations during the next period Individual Sampling Select a subject and observe behavior for a given period (e.g., 30 seconds), and then shift to another subject and repeat observations

8. 8 Event Sampling Select one behavior for observation and record all instances of that behavior It is best if one behavior can be specified as more important than others Recording Use a recording device to make a record of behavior for later review

9. 9 Multiple observers are used to increase reliability of observations You must establish reliability of observations from multiple observers (interrater reliability) Methods for evaluating interrater reliability Percent agreement Simplest method (total agreements/total observations)/100 Percent agreement should be around 70% Percent agreement may underestimate agreement

10. 10 Cohen’s Kappa Popular method Assesses the amount of agreement actually observed relative to the amount of agreement that would be expected by chance A Cohen’s Kappa of.70 indicates acceptable reliability Pearson Product-Moment Correlation Correlate ratings of multiple observers with Pearson r Simple and easy method to evaluate interrater reliability Two sets of scores may correlate highly, but may still differ markedly Intraclass Correlation (ICC) Extension of Analysis of Variance logic to interrater reliability A powerful and flexible tool for evaluating interrater reliability

11. 11 If multiple observers disagree you must deal with the disagreement A number of methods can be used to handle disagreements Average across observers Have observers meet to resolve disagreements Designate one observer as the “main observer” and the other a “secondary observer” Make selections before you begin your research Data from main observer used as data for analysis and data from secondary observer used to establish reliability

12. 12 Bias may be a problem in observational research A number of potential sources of bias must be considered Observer bias occurs when knowledge of goals of research or hypotheses influence observations Using a blind observer avoids observer bias Observers interpreting behavior rather than recording what they see Make sure observers do not interpret behavior when making observations Interpretations can be done after observations made

13. 13 Recording quantitative data involves quantifying behavior numerically (e.g., rate behavior on a scale) Can apply a wide range of statistics to quantitative data Not all behavior can be recorded quantitatively Recording qualitative data involves making careful notes during observation periods No numbers generated nor are counts of behavior made Qualitative record analyzed for themes Specialized techniques needed to analyze data Either or both types of data can be recorded if necessary

14. 14 Naturalistic Observation Unobtrusive observations of subjects’ naturally occurring behavior are made Habituation can be used if you cannot remain hidden Can use indirect measures of behavior Advantage: Behavior not tainted by artificial setting Disadvantage: Purely descriptive, cannot infer causality Ethnography The researcher becomes immersed in the behavioral or social system being studied May be conducted as a participant or non-participant observation study

15. 15 A number of factors must be considered Gaining entry to field setting Gaining entry into group to be studied Making yourself “invisible” in the group Making observations and recording behavior How to analyze data Sociometry You identify and measure interpersonal relationships within a group Members of a group rate each other on some dimension Can be used as a stand alone technique or one in a wider study Construct a sociogram to graphically display interpersonal relationships

16. 16 Case History You observe and report on a single case Descriptive technique that cannot be used to establish causality Archival Research You use existing records (e.g., police records) as your source of data You should have specific research questions and hypotheses Gaining access to records may be difficult or easy, depending on records needed Incomplete records may be a problem Descriptive technique that cannot be used to establish causality

17. 17 Content Analysis You analyze spoken or written records for the occurrence of specific categories of events (e.g., a word or phrase) Both recording units and context units are recorded

18. 18 Used to analyze a written or spoken record for occurrence of specific behaviors or events Archival sources often used as sources for data Appears simple, but may be complex Should be used within a clearly developed study, including hypotheses to be tested Response categories must be clearly defined A method for quantifying behavior must be defined

19. 19 Clearly defined response categories are essential Two units of analysis Recording unit: Element of the material you are going to record (e.g., instances of a certain word) Context unit: Context within which material analyzed appears Observers doing content analysis must be blind so that bias will not enter the analysis

20. 20 Materials to be analyzed should be chosen carefully to increase generality A content analysis of biased materials produces biased results Cannot be used to establish causal connections among variables

21. 21 A meta-analysis applies statistical techniques to evaluate a body of literature in a research area You can combine or compare studies Differs from a traditional literature review which involves summarizing research literature Three steps involved in doing a meta-analysis Identifying relevant variables Locating research to review Conducting the meta-analysis

22. 22 Full reference citation Names and addresses of authors Sex of experimenter Sex of subjects used in each experiment Characteristics of subject sample (e.g., how obtained, number) Task required of subjects and other details about the dependent variable

23. 23 Design of the study (including any unusual features) Control groups and procedures included to reduce confoundings Results from statistical tests that bear directly on the issue being considered in the meta-analysis (effect sizes, values of inferential statistics, p values)

24. 24 Quality of research varies across different journals Research from different journals may have to be differentially weighted No agreement on how research should be weighted Studies in a meta-analysis may used very different methodologies May not be a serious problem: Different methodologies analogous to different subjects in a study Practical problems Incomplete information Imprecise information

25. 25 Meta-analysis yields different conclusions than traditional review in some aspects More likely to detect differences Larger effects of variables detected No difference between meta-analysis and traditional review in evaluating methodology It may be better to base decisions about effects of variables on statistical analysis than more subjective observations