Presentation is loading. Please wait.

Presentation is loading. Please wait.

Organizational Research Methods Week 2: Causality.

Similar presentations

Presentation on theme: "Organizational Research Methods Week 2: Causality."— Presentation transcript:

1 Organizational Research Methods Week 2: Causality

2 What Do We Mean By Causality? Relationship between two events where one is a consequence of the other Determinism: A (cause) leads to B (effect) “In the strict formulation of the law of causality—if we know the present, we can calculate the future—it is not the conclusion that is wrong but the premise”. On an implication of the uncertainty principle. Werner Heisenberg

3 Heisenberg & Uncertainty Principle Certain properties of subatomic particles are linked so the more accurately you know one, the less accurately you know the other –We can compute probabilities not certainties –Argues against determinism “Physics should only describe the correlation of observations; there is no real world with causality” Heisenberg, 1927, Zeitschrift für Physik –Psychology, like quantum physics, is probabalistic

4 Cause Versus Effect Effect of a Cause (Description) –What follows a cause? Cause of an Effect (Explanation) –Why did the effect happen? Do bacteria “cause” disease? –Actually toxins cause disease –Actually certain chemical reactions are cause Holland, P. W. (1988). Causal inference, path analysis, and recursive structural equations models. Sociological Methodology, 18,

5 Three Elements of Causal Case Cause and effect are related Cause preceded effect No plausible alternative explanations John Stuart Mill

6 Experiment Vary something to discover effects –Shows association –Shows time sequence –Can rule out only some alternatives Confounds Boundary conditions (generalizability) Good for causal description not explanation

7 Encouragement Design Manipulation of instructions/messages Subjects “encouraged” to do certain things Subjects self-select level of condition Holland, P. W. (1988). Causal inference, path analysis, and recursive structural equations models. Sociological Methodology, 18,

8 Studying and Performance Students randomly assigned to study amount Test scores as DV Did studying lead to test results? –Encouragement led to test results –Impact on studying unclear –Effect of studying unclear What was cause of test results? Holland, P. W. (1988). Causal inference, path analysis, and recursive structural equations models. Sociological Methodology, 18,

9 Nonexperimental Research Strategy Determine covariation Test for time sequence –Daily diary –Longitudinal design –Quasi-experiment Rule out plausible alternatives –Based on data –Logical

10 Inus Condition Multiple causes/mechanisms for a phenomenon –The same thing can occur for different reasons –Sufficient but unnecessary conditions –Multiple motives –People do things for different reasons Are social phenomena a hierarchy of inus conditions? Should we expect strong relationships between variables if inus conditions exist?

11 Inus Conditions For Turnover Better job offer Bullying Disability Dissatisfaction Poor skill-job match Pursue other interests (e.g., Olympics) Spouse transfer Strategic reasons: Part of plan

12 Confirmation/Falsification Observation used to –Confirm/support theories –Falsify/disconfirm theories Confirmation: All swans are white –Must observe all swans in existence Falsification: One black swan –Easier to falsify than confirm –Null hypothesis testing disconfirmation –Based on construct validity Poor measure might falsely falsify

13 Scientific Skepticism Science not completely based on objective reality Observations based on theory of construct Construct validity is theoretical interpretation of what numbers represent Theories could be wrong: Biased measurement –NA as bias (Watson, Pennebaker, Folger) –Social constructions (Salancik & Pfeffer) Science based on trust of methods: Faith –Experiments –SEM –Statistical control (Meehl)

14 Shadish et al. Skepticisms (p. 30) “…scientists tend to notice evidence that confirms their preferred hypotheses and to overlook contradictory evidence.” “They make routine cognitive errors of judgment” The react to peer pressures to agree with accepted dogma” They are partly motivated by sociological and economic rewards”

15 Statistical Control Including controls common practice Atinc & Simmering review –Micro-studies 3.7 controls/study –Macro-studies 7.7 controls/study Purification principle –Assumption that controls more accurate –Implicit causal conclusions

16 Bias Versus Spuriousness Third variable that affects observations Bias: Affect on measurement Spuriousness: Affect on construct

17 xy C XY xy C XY BiasSpuriousness

18 No Bias or Spuriousness Remove effects of interest before testing Baby with the bathwater

19 Performance Dimension 1 Performance Dimension 2 Liking Performance Dimension 1 Performance Dimension 2

20 Meehl 1971: High School Yearbooks What is Meehl’s issue with Schwarz and common practice? Schwarz approach –Control SES because it relates to schizophrenia and social participation –Does not consider plausible alternatives Schwarz accepts without skepticism (Shadish) Meehl no Automatic Inference Machine

21 Alternative Mechanisms: NA Negative Affectivity (NA) –Routinely controlled as a bias –Based on observed correlations –Watson, Pennebaker, Folger Ignores feasible alternative mechanisms –Spector, Zapf, Chen, Frese 2000

22 MechanismDescription PerceptionSee world/job in a negative or positive way Hyper-responsivityOver or under-react emotionally SelectionPersonality affects selection into better vs. worse jobs Situation creationPeople create good vs. bad situations for themselves MoodMood affects all measures in a study including job satisfaction Reverse CausalityExperience affects personality

23 Automatic Inference Machine Idea that statistics can provide tests for causation There is no such thing as a “causal test” There is no such thing as a “test” for mediation Statistical controls do not provide the “true” relationship between variables Statistics are only numbers: They don’t know where they came from Inference is in the design Inference is in the mind: Logical reasoning

24 Commonest Methodological Vice Meehl 1971 Assuming certain variables are fixed and therefore must be causal –SES –Demographics –Personality But these variables can’t be effects. Can they?

25 Can Job Satisfaction Cause Gender? Correlation of Gender and satisfaction = group mean differences Satisfaction can’t cause someone’s gender Satisfaction can be the cause of gender distribution of a sample Suppose Females have higher satisfaction than Males Multiple reasons

26 Alternative Gender-Job Satisfaction Model Females more likely to quit dissatisfying jobs Dissatisfaction causes gender distribution Gender moderates relation of satisfaction with quitting

27 Satisfaction Males Females … …

28 More Alternatives 1.Women less likely to take dissatisfying job (better job decisions) 2.Women less likely to be hired into dissatisfying jobs (protected) 3.Women less likely to be bullied/mistreated 4.Women given more realistic previews (lower expectations) 5.Women more socially skilled at getting what they want at work

29 How To Use Controls Controls great devices to test hypotheses/theory Rule in/out plausible alternatives Best based on theory Sequence of tests No automatic/blind use/inference Tests with controls not more conclusive and often less

30 Control Strategy 1.Test that A and B are related Salary relates to job satisfaction 2.Confirm/disconfirm control variable Gender relates to both 3.Generate/test alternative explanations for control variable Differential expectations Differential hiring rate Differential job experience Differential turnover rate

31 Mediation Explanatory variable X  M  Y Routinely tested in cross-sectional designs Mediation is a causal conclusion

32 Stone-Romero & Rosopa Advocates experimental designs –Overstates advantages –Ignores limitations Confounds Construct invalidity Experimenter effects External validity

33 Experimental Approach To Mediation Basic logic –Show X  M –Show M  Y –Therefore X  M  Y Limitations –How do you manipulate M without X? –How can you be sure that X works through M? Inus conditions

34 X ZMY

35 Example p. 340 Study 1: Goals to Norms Study 2: Norms to Performance When goals lead to performance was it through norms?

36 Demonstrate Mediation Show the chain of events –Multiple measurements –Show X leads to M leads to Y What does it take to build a mediation case? –Difficult to demonstrate causality among 3 variables –Show relationship –3-way time sequence –Rule out alternatives

37 Medical Perspective Three types of variables –Correlate: Variable shown to correlate. –Proxy risk factor: Predictor that precedes but has no demonstrable effect on outcome. –Causal risk factor: Precedes and affects outcome when manipulated or changed. Kraemer, H. C., Stice, E., Kazdin, A., Offord, D., & Kupfer, D. (2001). How do risk factors work together? Mediators, moderators, and independent, overlapping, and proxy risk factors. American Journal of Psychiatry, 158,

38 Three-step Strategy Step 1: Show relationship exists Step 2: Show prediction over time Step 3: Manipulate X and show effects on Y

39 Week 3: Validity, Threats And Qualitative Methods Validity –Interpretation of constructs/results –Inference based on purpose Hypothesized causal connections among constructs Nature of constructs Population of interest –People –Settings Not a property of designs or measures themselves

40 Four Types of Design Validity Statistical conclusion validity –Appropriate statistical method to make desired inference Internal validity –Causal conclusions reasonable based on design Construct validity –Interpretation of measures External validity –Generalizeability to population of interest

41 Threats to Validity Statistical Conclusion Validity –Statistics used incorrectly –Low power –Poor measurement Internal Validity

42 Eight Threats From Shaddish 1.Ambiguous Temporal Precedence 2.Selection: Differences in groups 3.History: Events occurring between testings 4.Maturation: Natural change in subjects over time 5.Regression To the Mean 6.Attrition: Especially differential 7.Testing: Affects of repeated testings 8.Instrumention: Change over time or conditions Bolded: Most problematic for organizational research

43 Threats To Validity 2 Construct Validity –Inadequate specification of theoretical construct –Unreliable measurement –Biases –Poor content validity External Validity –Inadequate specification of population –Poor sampling of population Subjects Settings

44 Points From Shadish et. al Evaluation of validity based on subjective judgment Scientists conservative about accepting results/conclusions that run counter to belief Scientists market their ideas; try to convince colleagues Design controls preferable to statistical Statistical controls based on assumptions –Untrue –Untested –Untestable

45 Points From Shadish et al. 2 People in same workplace more similar than people across workplaces –When might this be true? Multiple tests –Probability compounding assumes independent tests Molar vs. molecular –First determine molar effect –Then breakdown to determine molecular elements

46 Qualitative Methods What are qualitative methods –Collection/analysis of written/spoken text –Direct observation of behavior Participant observation Ethnography Case study Interview Written materials –Existing documents –Open-ended questions

47 Purpose – Erickson 2012 Describes social action –What people do in light of their attributions and meanings Interpretive –“describing what people in a local setting do in terms that make contact with the meaning perspectives within their actions make sense, from their points of view”, p. 686 Understand cause in local situation Erickson, F. (2012). Comments on causality in qualitative inquiry. Qualitative Inquiry, 18,

48 Qualitative Research Approach Accept subjectivity of science –Is this an excuse? Less driven by hypothesis Assumption that reality a social construction –If no one knows I’ve been shot, am I really dead? Interested in subject’s viewpoint More open-ended More interested in context Less interested in general principles Focus more on interpretation than quantification

49 Analysis Content Analysis –Interviews –Written materials –Open-ended questions –Audio or video recordings –Quantifying Counts of behaviors/events Categorization of incidents Multiple raters with high agreement Nonquantitative –Analysis of case –Narrative description Ethnography

50 Qualitative Organizational Research: Job Stress Quantitative survey dominates Role ambiguity and conflict dominated in 1980s & 1990s (Katz & Kahn) Dominated by Rizzo et al. weak scales Studies linked RA & RC to potential consequences and moderators Qualitative approach (Keenan & Newton, 1985) Stressful incidents

51 Keenan & Newton’s SIR Stress Incident Record –Describe event in prior 2 weeks –Aroused negative emotion Top stressful events for engineers –Time/effort wasted –Interpersonal conflict –Work underload –Work overload –Conditions of employment RA & RC rare (1.2% and 4.3%)

52 Subsequent SIR Research Comparison of occupations –Clerical: Work overload, lack of control –Faculty: Interpersonal conflict, time wasters –Sales clerks: Interpersonal conflict, time wasters –Narayanan, L., Menon, S., & Spector, P. E. (1999). Stress in the workplace: A comparison of gender and occupations. Journal of Organizational Behavior, 20, Informed subsequent quantitative studies –Focus on more common stressors Interpersonal conflict Organizational constraints –Forget RA & RC

53 Cross-Cultural SIR Research Comparison of university support staff India vs. U.S. StressorIndiaUS Overload0%25.6% Lack of control0%22.6% Lack of structure26.5%0% Constraints (Equipment)15.4%0% Conflict16.5%12.3%

54 Value of Qualitative Approach Richer more complete picture –Doesn’t reduce complex to few variables More information on context Doesn’t constrain subjects –Open ended responses Raw material for hypothesis and theory Can be quantified Can be used to test hypotheses

55 Limitations of Qualitative Approach If not quantified –Subjective: One person’s perspective –Not science without systematic methods that are replicable Quality of material content analyzed Potential biases –Fundamental attribution error: Attributing others’ behavior to stable traits. –Attribute success internally, failure externally

56 Research As Craft Scholarly research as expertise not bag of tricks Logical case Go beyond sheer technique –Research not just formulaic/trends –Not just using right design, measures, stats –Can’t go wrong with Big Five, SEM, Meta- analysis –SEM on meta-analytic correlation matrix of Big Five

57 Meta-Analytic Correlation Matrix Analysis Do meta-analysis compiling correlations Conduct Regression or SEM on mean correlations Problems: –Mean correlations based on different sets of samples –Often (usually) moderators affect correlations across studies –Populations differ between correlations –Results not meaningful

58 Meta-Analytic Correlation Analyses II Regression –Regression of Y on X1 and X2 –Want to conclude that X2 incremental over X1 –Can only conclude that X2 in one population incremental over X1 in another –Example: Job Satisfaction = Conscientiousness and Autonomy Conclusion: Conscientiousness in police & nurses predicts better than Autonomy in military &teachers

59 Meta-Analytic Correlation Analyses Conditions To Use –All sample representative from same population –All samples have all variables

60 Developing the Craft Experience Trying different things –Constructs –Designs/methods –Problems –Statistics Reading Reviewing Teaching Thinking/discussing Courses necessary but not sufficient Lifelong learning—you are never done

61 Developing the Craft 2 Field values novelty and rigor Don’t be afraid of exploratory research –Not much contribution if answer known in advance Look for surprises Don’t be afraid to follow intuition Ask interesting question without a clear answer Focus on interesting variables Good papers tell stories –Variables are characters –Relationships among variables

62 Week 4 Construct & External Validity and Method Variance

63 Constructs Theoretical level –Conceptual definitions of variables –Basic building blocks of theories Measurement level –Operationalizations –Based on theory of construct

64 What We Do With Constructs Define Operationalize/Measure Establish relations with other constructs –Covariation –Causation

65 Construct Validity Case based on weight of evidence Theory of the construct –What is its nature? –What does it relate to? Strength based on –Adequate definition –Adequate operationalization –Control for confounding

66 Steps To Building the Case 1.Define Construct 2.Operationalize construct: Scale development 3.Construct theory: What it relates to 4.Validation evidence 1.Correlation with other variables 1.Cross-sectional 2.Predictive 2.Known groups 3.Convergent validity 4.Discriminant validity 5.Factorial validity

67 Points By Shadish et al. Construct confounding –Assessment of unintended constructs –SD and NA –Race and income –Height and gender O = T + E + Bias –Bias = Extra unintended stuff

68 Points by Shadish 2 Mono-operation bias –Not clear on what it is Advocates converging operations Multiple operationalizations What is a different operationalization? –Different item formats –Different raters –Different experimenters –Different training programs

69 Points by Shadish 3 Compensatory equalization: Extra to control group Compensatory de-equalization: Extra to experimental group FMHI Study –Random assignment to FMHI vs. state hospital –Staff violated

70 Construct Validity: Example of Weak Link Deviance: Violation of norms Theoretical construct weakness –Whose norms? Society, organization, workgroup Operationalization weakness –List of behaviors with no reference to norms –Norms assumed from behavior Retaliation: Response to unfairness –Asks behaviors plus motive –Retaliation mentioned in instructions

71 External Validity: Population Link between sample and theoretical population Define theoretical population Identify critical characteristics Compare sample to population –Employed individuals –Do students qualify?

72 External Validity: Setting Link between current setting and other settings –Organization –Occupation Identify critical characteristics of settings Compare setting to others –Lab to field

73 External Validity: Treatment/IV Is IV a reasonable analog? Encouragement design idea Link between current treatment/IV and others Will treatment in study work in nonresearch setting?

74 External Validity: Outcome/DV Link between current outcome/DV and others Will results in study work similarly in nonresearch condition? Will different operationalizations of outcome have same result? –Supervisor rating of performance vs. objective –Safety behavior versus accidents/injuries

75 Facts Are the Enemy of Truth When Facts Oppose Belief Gender bias in medical studies (Shadish p. 87) Women are neglected in medical research Treatments not tested on women New grant rules require women Study of 724 studies (Meinart et al.) –55.2% both genders –12.2% males only –11.2% females only –21.4% not specified –355,000 males, 550,000 females

76 When Politics Attack Science Evolution IQ and performance Differential validity of IQ tests Others?

77 Method Variance Method Variance: Variance attributable to the method itself rather than trait Campbell & Fiske 1959 Assumed to be ubiquitous V Total = V Trait + V Error + V Method

78 Campbell & Fiske, 1959 “…features characteristic of the method being employed, features which could also be present in efforts to measure other quite different traits.”

79 Common Method Variance CMV –Mono-method bias –Same source bias When method component shared V Total = V Trait + V Error + V Method Assumes same method has same V method Assumed to inflate correlations Only raised with self-reports

80 Evidence Vs. Truth Truth: CMV = Everything with same method correlated Evidence: Boswell et al. JAP 2004, n = 1601 Leverage seeking Separation seeking -.04 Career satisfaction Perceived alternatives Reward importance

81 Potential Universal Biases Truth: Specific biases widespread Evidence –Social Desirability Meta-analysis Moorman & Podsakoff Mean r =.05 Highest -.17 role ambiguity;.17 job satisfaction Lowest.01 Performance Lower with employees (.03) than students (.09) –Social Desirability Control Study Role clarity-job satisfaction –r =.46 to.45 (when SD controlled)

82 Single-Source Vs. Multi-Source Truth: Multisource correlations smaller Evidence: Crampton & Wagner Meta-analysis –Compared single-source vs. multi-source –26.6% single-source higher (job sat and perform) –62.2% no difference (job sat and turnover) –11.2% multi-source higher (job sat and absence)

83 Conclusions About CMV CMV is a myth –Variance does not reside in the method –No constant inflation of correlations Oversimplification of complex situation If it existed, it would be easy to solve Techniques to fix the problem ineffective and counterproductive –Enhance publishability

84 Cynical Use of CMV Tests Lindell & Whitney Test Marker variable expected to be unrelated Partial out effect of marker to yield relationship free of CMV Strawman test of something that isn’t there Great for demonstrating to reviewer you had no CMV problem Lindell, M. K., & Whitney, D. J. (2001). Accounting for common method variance in cross- sectional research designs. Journal of Applied Psychology, 86,

85 If Not CMV CMV myth but bias is not Bias: Unintended influences on measure Bias a function of construct plus method Unshared bias: Correlation attenuation Shared bias: Correlation inflation No automatic inference machine to eliminate all biases Some known, many likely unknown

86 Solutions Eliminate term “method variance” Bias or 3 rd variables Construct validity and potential 3 rd variables Interpret results cautiously Choose methods to control feasible 3 rd variables Alternate sources –Not always accurate Converging operations Use strategy of first establishing correlation –Rule out 3 rd variables in series of steps

87 Potential Sources of Bias Difficult to distinguish bias from substantive effect Personality –Social desirability –NA Culture –Response styles –Asians avoid strong positives Mood

88 Potential Sources of Bias II Cognitive Processes –Schema –Attribution errors Fundamental attribution error –Halo and Leniency Priming –Effects of instructions or prior questions Item overlap

89 Item Overlap Similar content in scales of different constructs Stress –Confound of stressors and strains –Stressor: How often do you feel stressed by … –Strain: How often do you feel anxious

90 CWB-OCB Biases Dalal’s three artifacts Antithetical items (overlap) Agreement vs. frequency –Agreement more subject to halo Source: Self versus supervisor –Supervisor more subject to halo Dalal, R. S. (2005). A Meta-analysis of the relationship between organizational citizenship behavior and counterproductive work behavior. Journal of Applied Psychology, 90,

91 CWB – OCB Overlap OCBCWB Does not take extra breaks Takes undeserved breaks Taken a longer break than you were allowed to take Obeys company rules and regulations Purposely failed to follow instructions Consumes a lot of time complaining about trivial matters Complained about insignificant things at work Conducts personal business on company time I used working time for private affairs

92 Dalal’s Meta-Analysis Mean correlations Antitheticals: yes vs. no: -.54 vs Agreement vs. frequency: -.55 vs Supervisor vs. self: -.60 vs Spector et al. isolated effects: –Order Antithetical, format, source Spector, P. E., Bauer, J. A., & Fox, S. (2010). Measurement artifacts in the assessment of counterproductive work behavior and organizational citizenship behavior: Do we know what we think we know? Journal of Applied Psychology, 95,

93 Conway & Lance 2010 Distinguished method variance from effects p. 325 Note: Different methods can share bias, p. 327 –Survey versus interview and SD Different sources measure different things, p. 328 Did Heisenberg really say we change things by measuring them?, p. 329 Judge et al. were clear that self-report of job characteristics were perceptions and not objective, p. 329

94 Conway & Lance 2010 cont. Construct validity evidence does not rule out possible biases, p. 329 Be concerned about item overlap, p. 330

95 Week 5: Quasi-Experimental Design Design: Structure of an investigation –Number of groups –Assignment of subjects –Sequence of conditions –Sequence of assessments Experimental vs. Nonexperimental

96 Experimental Design What is an experiment? –Random assignment –Creation of Conditions? –Naturally occurring experiment Simple to Complex –Number of independent variables –Number of dependent variables –Sequence of DV assessment

97 Two-Group Experiment Treatment Group: X T O Control Group:X C O Comparison of two conditions Hold everything constant that is possible Many ways to design the control group Report effect size Power

98 Two-Group Design: Analysis Independent group t-test One-way ANOVA (2 groups) t 2 (n 1 + n 2 – 2) = F (1, n 1 + n 2 -2) r (with IV dummy coded) r = f(t)

99 Data Structure 2 group t-test Treatment Control

100 Data Structure IV by DV Correlation IV DV

101 t vs. r Entered data set into SAS Independent group t-test –t = 2.31, p =.0497 Correlation –r = –t-test for significance of r = , p =.0497

102 Variance Partitioning Total: All variance among all subjects Between: Variance between groups –Variation in means Within: Variance within groups –Nested across groups Total = Between + Within Significance: Ratio of B/W –Is there more B variance than expected by chance?

103 One IV: Multiple Groups One IV More than two groups Categorical IV –Different treatments Continuous IV –Different levels of treatment –Sessions of training ANOVA

104 Treatment Group Independence Between Subject Within subject –Repeated measures Nested or hierarchical –Subjects in teams in conditions Affects analyses –ANOVA –Multilevel modeling for nested

105 Between Group Advantages –Independence of conditions –No carry over effects –Conceptually simple Disadvantages –Sample size requirement –Limited to one level, usually people

106 Within Group Advantages –Greater power by controlling error within people –Efficiency: One group of subjects –Direct comparison of conditions on same sample Disadvantages –Contamination/carryover effects of conditions

107 Nested People in higher social units –Departments –Classes –Organizations –Teams

108 Nested Advantages –Allows study of multiple levels –People and teams –Allows study of cross-level effects Disadvantages –Requires larger samples N at both levels: n people per group and n of groups –Analysis/interpretation more complex

109 Factorial Designs Multiple IVs Two-Way design –Two IVs totally crossed –Every combination of conditions –Orthogonal: equal sample size –Nonorthogonal: Unequal sample size Confounding of IVs Level of A predicts likely level of B Interpretation of IV effects difficult ANOVA

110 Main Effects and Interactions Main effect –Average effect of levels of one IV ignoring the other Interaction –Joint effect of both IVs –Shows effect of one IV affected by the other

111 Main Effect A1A2 B1 B2 DV

112 Interaction A1A2 B1 B2 DV

113 Interaction A1A2 B1 B2 DV

114 Interaction A1A2 B1 B2 DV

115 2x2 Design A1A2Mean B1A1B1A2B1(A1B1+A2B1) / 2 B2A1B2A2B2(A1B2+ A2B2) /2 Mean(A1B1 + A1B2) /2(A2B1 + A2B2) /2 A Main effect: (A1B1 + A2B1) vs. (A1B2 + A2B2) B Main effect: (A1B1 + A1B2) vs. (A2B1 + A2B2) Interaction: (A1B1 + A2B2) vs. (A2B1 + A1B2)

116 Larger Designs Any number of IVs Three-way interaction complex to interpret Four or more way difficult to interpret Takes large number of subjects Not always feasible for experiments

117 Moderator Tests Another term for interaction Term used with continuous variables Used with multiple regression Same idea of one variable affecting effects of another. Nonparallel lines Slope = f(moderator)

118 Mixed Design Between, Within, and Nested Test of CWB-OCB Artifacts Response format between-S –Randomly assigned Antithetical yes vs. no: Within subject –Everyone got both Source: Nested –Subject versus supervisor nested within dyad Spector et al. 2010

119 Quasi-experiment Design without random assignment Comparison of conditions –Trained vs. Not trained Researcher created or existing Can characteristics of people be an IV? –Gender –Personality Is a survey a quasi-experiment? –Question about condition

120 Settings Laboratory vs. field Laboratory –Setting in which phenomenon doesn’t naturally occur Field –Setting in which phenomenon naturally occurs Classroom field for educational psychologist Classroom lab for us

121 Lab vs. Field Strengths/Weaknesses Lab –High level of control –Easy to do experiments –Limits to what can be studied –Limited external validity of population/setting Field –Limited control –Difficult to do experiments –Wide range of what can be studied –High reliance on self-report –High external validity

122 Design Versus Setting Design: Structure of investigation Setting: Place in which study occurs Lab ≠ Experiment Field ≠ Nonexperiment

123 Applied Versus Basic Refers to areas of psychology Basic: Experimental Areas –Cognitive, Neuro, Social?, Developmental? Applied: Clinical, I/O Not useful distinction What is applied? “Applied psychologists” use same methods to address the same questions.

124 Lab in I/O Research What’s the role of lab in I/O research? Stone suggests lab is as generalizeable as field. Do you agree? Stone says I/O field biased against lab. Is it? When should we do lab vs. field studies?

125 Quasi-Experimental Compromise Quasi-experiments –Compromise when true experiment isn’t possible –Built in confounds Requires more data than experiment to rule out confounds Inference complex Logic puzzle not cookbook Can’t just assume IV caused DV

126 Quasi-Experiment and Control Use of design AND statistical controls “Statistical adjustment only after the best design controls have been used” Shadish, p. 161 Control through comparison groups Control through retesting –Pretest-posttest –Multiple pretests/posttests –Long-term follow-ups –Trends Statistical control: 3 rd variables & potential confounds

127 Single Group Designs Posttest only X O When (if ever) is this useful? Pretest-posttest O X O When is this useful? What are the limitations? Program evaluation

128 Nonequivalent Groups Design Preexisting groups assigned treatment vs. control X O O Establishes difference between groups Limited inference

129 Nonequivalent Groups Design Limitations What are the main limitations? –Groups could have been different initially –Interaction of group characteristics and treatment –Differential history causing differences

130 Coping With Preexisting Group Differences 1.Assess preexisting differences Pretest 2.Assess trends Multiple pretests and posttests 3.Assign multiple groups Random assignment of groups if possible 4.Replicate 5.Additional control groups 6.Matching 7.Statistical adjustment of potential confounds 8.Switching replications—Give treatment to control

131 Switching Replications Group 1: XO--O Group 2: --OXO Group 2 control for Group 1 Replicate effect with Group 2 Can have more groups Power and replicability

132 Matching Selecting similar participants from each group –Choose one or more matching variables –Assess variables –Choose pairs that are close matches Difficult to match on multiple variables –Sample size reduction Might bias samples –High in one sample low in another –Meaning of high/low can vary –LOC: Internal Chinese is external New Zealander

133 Randomly Assign Groups Identify multiple groups Randomly assign to conditions Groups need to be isolated –Contamination of control by treated group –Contamination of control by supervisors who know about study Creates potential levels issue –Subjects nested in groups

134 Case-Control Design Compare sample meeting criterion with sample not meeting Must match to same population –Employees who quit vs. all other employees –Employees who were promoted vs. other employees –CEOs vs. line employees –Employees assaulted/bullied vs. others Assess other variables to compare

135 Case-Control Typically we have the case sample at hand Controls may not be easily accessible Often cases compared to a “normal” population –Cancer patients vs. norms for general public Could compare cases in organization with employees in general –E.g., absence from case group vs. absence rate in company

136 Is Case-Control Useful To Us? What might we use this design to study in Organizations? What is the Case Group? What is the Control Group? What variables do we compare?

137 Limits To Case-Control Design Defining groups from same population Effect size uncertain –All cases have X –Small proportion of people with X are cases –Asymmetrical prediction Groups may differ on more than case variable Retrospective assessment of supposed cause –Quitting caused report of lower satisfaction

138 Week 6: Design Issues Experimental Design –Random assignment –Creation of conditions Randomized experiment –Time sequence built into design –Still must rule out plausible alternatives Construct validity of IV and DV External validity for lab studies Is “real science” so real?

139 Random Assignment Random sample (external validity) Random assignment (internal validity) –Probability of assignment equal –Expected value of characteristics equal –Not all variables equal Type 1 errors Faith in random assignment Differential attrition

140 What Are We Really Assigning To? Encouragement Design –Ask subjects to do certain things What features of complex condition are critical? Confounds in IV

141 Control Groups No treatment Waiting list Placebo treatment Currently accepted treatment Comparisons to isolate variables

142 Bias In Experiments Construct validity of DV and IV Bias in Assessment of DV Bias/confounding in IV Bias affects –Subjects –Experimenters –Samples –Conditions (Contamination and Distortion) –Designs –Instruments

143 Humans Used As Instruments Self vs. other reports Bias in judgments of others –Schema & stereotypes –Implicit theories –Attractiveness Pretty blondes are dumb –Physical ability Athletes are dumb –Height Tall are better leaders

144 Demand Characteristics Implicit meaning of experimental condition IV not accurately perceived Subject motivated to do well Subject tries to figure out experiment –Response not natural for situation

145 Lie Detection Lab Vs. Field Lab Study Two trials of detection Detect Trial 1, Harder to detect Trial 2 Not detect Trial 1, Easier to detect Trial 2 Opposite to field experience Hypothesized that motive important –Want to fool—being detected makes it easier to detect –Get caught—being detected makes it harder to detect

146 Lie Detection Study 2 2 trials x 2 conditions –Told intelligent can fool Anxious when caught –Told sociopath can fool Relax when caught Percent Detected Trial 2 MotiveDetect Trial 1Not Detect Trial 1 Fool (intelligent)94% Anxious 19% Relaxed Catch (sociopath)25% Relaxed 88% Anxious

147 Subject Expectancies Hawthorne Effects –Knowledge of being in an experiment –Does this really happen? Placebo Effects Blind procedures

148 Experimenter Effects Observer Errors –Late 1700s Greenwich Observatory –Maskelyne fires Kinnebrook for errors –Astronomer Bessel: Widespread errors –About 1% of observation have errors –75% direction of hypothesis Experimenter expectancy—self fulfilling prophecy Clever Hans Dull/Bright rat study Double blind procedure

149 Experimenter Behavior Smiling at subjects –12% at males –70% at females Mixed gender S-E longer to complete Videotape of S-E interactions (Female E) AuditoryVisual Male subjectFriendlyNonfriendly Female subjectNonfriendlyFriendly

150 Cross-Sectional Design All data at once Variables assessed once Most common design in I/O & OB/HR Often done with questionnaires Can establish relationships Cannot rule out most threats Cheap and efficient Good first step

151 Data Collection Method Ways of collecting data –Self-report questionnaire Formats –Interview Degree of structure –Observation Behavior checklist vs. rating –Open-ended questionnaire

152 Data Source Incumbent Supervisor Coworker Significant other Observer Existing materials –Job description

153 Single-Source All data from one source Usually also mono-method Usually survey Many areas usually self-report –Well-being Some area other-report –Performance

154 Multi-Source Same variables from different sources –Convergent validity –Confirmation of results Different variables from different sources –Rules out some biases and 3 rd variables –Some biases can be shared Not panacea

155 Multisource Discriminant Validity StudyVariablesSelf- Report Other- Report Dalal 2005 metaCWB-OCB Spector et al CWB-OCB Spector, Fox 2005Autonomy-Job Characteristics Glick et al. 86Job Characteristics-Satisfaction Note: Dalal meta-analysis; Spector, Fox mean correlation across 4 job characteristics; Glick et al. multiple correlation.

156 Bias Can Affect All Raters Self-Efficacy—outward signs of confidence –Gives impression of effortless performance –Coworker perception of employee’s constraints Doesn’t appear to have constraints –Supervisor perceptions of performance Looks like a great performer

157 Self-Efficacy Self-report Constraints Self-report Job Performance Self-report

158 Self-Efficacy Self-report Constraints Coworker Job Performance Self-report

159 Self-Efficacy Self-report Constraints Coworker Job Performance Supervisor

160 Week 7: Longitudinal Designs Design introducing element of time Same variable measured repeatedly Different variables separated in time –Turnover How much time needed to be longitudinal?

161 Advantages of Longitudinal Design Can establish relationships Can sometimes establish time sequence Can rule out some plausible alternatives –Some biases –Occasion factors –Mood

162 Proper Time Sequence Before and after an event –Turnover Precursors assessed prior –Job satisfaction Difficult to know when satisfaction occurred Arbitrary points in time not helpful –Steady state results same as cross-sectional

163 Predicting Change Showing that X predicts change in Y Relation of X & Y controlling for prior levels Weak evidence for causality Regression to mean effects Basement/ceiling effects

164 Attrition Problem Attrition between time periods –From organization –From study Attrition not random Mean change due to attrition Interaction of attrition and variables –Those most/least affected quit

165 Practical Issues Tracking subjects Matching responses –Loss of anonymity –Use of secret codes Subject might not remember –Anonymous identifiers First street lived on Name of first grade teacher Grandmother’s first name Participation incentives Time to complete study

166 Pretest-Posttest Design Single Group O 1 O 2 Two Group O 1 X O 2 O 1 O 2

167 Trends Over Time Single group Time Series O 1 O 2 O 3 X O 4 O 5 O 6 Multigroup Time Series O 1 O 2 O 3 X O 4 O 5 O 6 O 1 O 2 O 3 O 4 O 5 O 6

168 Discontinuity Change in trend around X Single group –Can’t rule out other causes Multigroup –Control group to rule out alternatives

169 Zapf et al. Stress area Relationships small over time Inus conditions –Strains caused by 15 factors –Each accounts for 7% of variance –.26 correlation if measurement perfect Attrition of least healthy Relationships not always linear Choose appropriate time frame

170 Longitudinal Multi-Group Design Identify classification variable –Assaulted, Smoking Assess two times Group employees –Yes/Yes –Yes/No –No/Yes –No/No Compare groups Manning, M. R., Osland, J. S., & Osland, A. (1989). Work-related consequences of smoking cessation. Academy of Management Journal, 32,

171 Manning Design Time 1: SmokeTime 1: No Smoke Time 2: SmokeSmokersStarters Time 2: No SmokeQuittersNonsmokers

172 Yang Design Time 1: AssaultedTime 1: Not Assaulted Time 2: AssaultedConstant strainStrain increased Time 2: Not AssaultedStrain decreased (recovery) Constant strain

173 Experience Sampling Diary Study Multiple measures on same person –Daily –Multiple times per day –1-2 weeks Look at within person variation –Changes in DV as a function of IV

174 Experience Sampling Analysis Hierarchical linear modeling (HLM) –Level 1 within person –Level 2 between person Multiple regression –DV 2 = IV 1 + DV 1 –Time 1 IV on Time 2 DV control for Time 1 DV –To see if change in DV is associated with IV

175 HLM Deals with hierarchical structure of data –Observations nested –Individuals in groups, departments, organizations Experience sampling –Observations nested in people –Separates variance into between versus within –Analysis of within person change Relationship of fluctuations of IV vs. DV

176 Curvilinear Stressor-Strain Two studies –CISMS2: Anglo n = 1470 Spector-Jex, 1991, n =232 Stressors –Conflict, Constraints, Role ambiguity, Workload Strains –anxiety, anger, depression, frustration, intent, job satisfaction, symptoms

177 Analyses Curvilinear regression Strain = Stressor + Stressor 2 Plot by substituting values of Stressor –Similar to plotting moderated regression

178 Example Y = X +.2X 2 X ranges from 0 to 20 Substitute values 5 points apart (0, 5, 10, 15, 20)

179 Computations Xb 1 X (b 1 = -2) X2X2 b 2 X 2 (b 2 =.2) b 1 X+b 2 X b 1 X+b 2 X


181 Results Significance for workload Limited significance for –Conflict –Constraints –Role ambiguity

182 StrainCISMSSpector-JexDirection Anxietyns*U Frustration--*U Intent**U Job Satisfaction **Inverted U Symptoms*nsU

183 Week 8: Field Research and Evaluation Field Research –Done in naturalistic settings Experimental Quasi-experimental Observational Evaluation – Organizational Effectiveness –Figuring out if things work Organizations Programs Interventions

184 Challenges To Field Research Access to organizations/subjects Lack of control –Distal contact with subjects (surveys) –Who participates –Contaminating conditions Participants discussing study Lack of full cooperation Organizational resistance to change

185 Creative and Varied Approaches

186 Accessing Subjects Define population needed for your purpose –People –Jobs –Organizations List likely locations to access populations Consider ways to access locations

187 Defining Populations: People Characteristics of people –Demographics Age, Education, Gender, Race –KSAOs –Occupations Do variables of interest vary across occupations? Single or multiple occupations Single controls variety of factors Multiple –More variance –Tests of occupation differences –Greater generalizeability

188 Defining Populations: Organizations Characteristics of organizations needed –Occupations represented –Characteristics of people represented –Characteristics of practices Single versus multiple organizations –Single adds control –Multiple adds Variance Tests of organization characteristics Generalizeability

189 Accessing Participants: Students Psychology student subject pool Employed students in classes, e.g., night Advantages/Limitations –Easily accessed on many campuses –Cheap –Cooperative –Younger and more educated than average –Heterogeneous jobs/organizations –Often part-time and temporary jobs –Potential work-school conflict

190 Accessing Participants: Nonstudents Organizations: Access can be a problem Association mailing lists: Single occupations Web search: Government employees Clubs, churches, nonwork organizations Unions General surveys –Phone, mail, door-to-door, street corner

191 Approaching Organizations Sell to management Appeal to value of science not ideal What’s in it for them? Partnership –Free service: Employee survey, job analysis –Address their problem –Piggyback your interest

192 Modes of Approach Personal contact: Networking –Give talks to local managers, e.g., SHRM –Students in class –Approach based on known need Hospitals and violence Consider the audience –Psychologist vs. nonpsychologist –HR vs. nonHR –Level of sophistication about problem –Don’t assume you know more than organization about their problem

193 Project Prospectus One page nontechnical prospectus –Purpose: Clear and succinct –What you need from them –What it will cost (e.g., staff time) –What’s in it for them –What products you will provide to them –Timeline

194 Example Determine factors leading to patient assaults on nurses in hospitals Need to survey 200 nurses with questionnaire Questionnaire will take minutes –Can be taken on break or home Will provide report to organization about –How many nurses have been assaulted –The impact of the assaults on them –Factors that might be addressed to reduce the problem Would like to conduct study next month, and provide report within 60 days of completion.

195 Partnerships Academics and nonacademics Projects come from mutual interests Piggyback onto organizational project Internship –Johannes Rank’s training evaluation Issues –Proprietary results –Organizational confidentiality

196 Program Evaluation/Organizational Effectiveness Program Evaluation –Education –Human Services –Determining if program is effective Organizational Effectiveness –More generic –Determining effectiveness of organization –Determining effectiveness of activity/unit

197 Formative Approach Focus on processes Often used in developmental approach Can be qualitative Can be quantitative Action research –Identify problem, try solution, evaluate, revise

198 Summative Approach Assess if things work Often quantitative Experimental or quasi-experimental design Compare to control group/s Utility –Return on investment (ROI) Private sector Profitability –Cost/outcome (bang for buck) Military—literally Nonmilitary—cost/unit of outcome

199 Steps In Determining Effectiveness 1.Define goals/objectives 2.Determine criteria for success 3.Choose design 1.Single group vs. multigroup 4.Pick measures 5.Collect data 6.Analyze/draw conclusion 7.Report/Feedback 8.Program improvement

200 Week 9 Survey Methods & Constructs Survey methods Sampling Cross-cultural challenges –Measurement equivalence/invariance Reflective Vs. Formative scales Artifactual constructs

201 Survey Settings Within employer organization Within other organization –University –Professional association –Community group –Club General population –Phone book –Door-to-door

202 Methods Questionnaire –Paper-and-pencil – –Web Interview –Face-to-face –Phone –Video-phone – ? –Instant Message ?

203 Population Single organization Multiple organizations –Within industry/sector Single occupation Multiple occupations General population –Employed students

204 Sample Versus Population Survey everyone in population vs. sample –Single organization or unit of organization Often survey goes to everyone –Multiple organizations Kessler: All psychology faculty –Other organization Professional association Often survey everyone –General population

205 Sampling Definitions Population – Aggregate of cases meeting specification –All humans –All working people –All accountants –Not always directly measurable Sampling frame – List of all members of a population to be sampled –List of all USF support personnel

206 Sampling Definitions cont. Stratum – Segment of a population Divided by a characteristic –Demographics Male vs. female –Job level Manager vs. nonmanager –Job title –Occupation –Department/division of organization

207 Representativeness of Samples Representative –Sample characteristics match population Non representative –Sample characteristics do not match population Some procedures more likely to yield representative samples

208 Nonprobability Sampling Nonprobability sample – Every member of population doesn’t have equal chance Representativeness not assured Types –Accidental or convenience –Snowball –Quota – Accidental but stratified Choose half male/female –Purposive – Handpick cases that meet criteria Pick full-time employees in a class

209 Probability Sampling Random sample from defined population Stratified random sample –More efficient than random Cluster or multistage –Random selection of aggregates Select organizations stratified by industry

210 International Research Methods Cross-cultural vs. cross-national (CC/CN) Purposes –Research within a country/culture (emic) –Generalize finding/theory –Compare countries/cultures (etic) –Test culture hypotheses across groups defined by culture CC/CN differences Within country Across countries Across regions –North America vs. Latin America

211 Challenges of CC/CN Research Equivalence of samples Measurement Equivalence/Invariance (MEI)

212 Sample Equivalence What is it about samples that causes differences? Confounding of country with sample characteristics –Occupations Can vary across countries –Industry sectors Private sector doesn’t exist universally –Organization characteristics –People characteristics (e.g., demographics) Gender breakdown differs across countries

213 Instrument Issues Linguistic meaning –Translation – Back-translation Calibration –Numerical equivalence –Cultural response tendencies Asian modesty Latin expansiveness Measurement equivalence/invariance –Construct validity –Factor Structure

214 Measurement Equivalence/Invariance MEI Construct Validity –Same interpretation across groups SEM and IRT approaches –Based on item inter-relationship similarity –Factor structure –Item characteristic curves

215 SEM Approach Equality of item variance/covariance Equal corresponding loadings Form invariance –Equal number of factors –Same variables load per factor

216 IRT Approach Equivalent item behavior For unidimensional scales Better developed for ability tests Often conclusion similar to SEM

217 Eastern versus Western Control Beliefs at Work Paul E. Spector, USF Juan I. Sanchez, Florida International University Oi Ling Siu, Lingnan University, Hong Kong Jesus Salgado, University of Santiago, Spain Jianhong Ma, Zhejiang University, PRC Applied Psychology: An International Review, 2004

218 Background Cross-cultural study of control beliefs Americans Vs. Chinese Locus of control beliefs vary –Chinese very external vs. Americans –Suggests Chinese passive view of world –Look to others for direction

219 Primary Vs. Secondary Control Primary: Direct control of environment Secondary: Adapt self to environment –Predictive: Enhance ability to predict events –Illusory: Focusing on chance, i.e., gambling –Vicarious: Associate with powerful others –Interpretive: Looking for meaning Asians more secondary Rothbaum, Weisz, & Snyder

220 Socioinstrumental Control Control through social networks Build social networks Cultivate relationships Juan Sanchez

221 Purpose Develop new control scales –Secondary control –Socioinstrumental control Avoid ethnocentricism by using international item writers

222 Pilot Study Method Develop definitions of constructs International team wrote 87 items –1 American, 2 Chinese, 2 Spanish Administered 126 Americans Item analysis

223 Sample Items Secondary I take pride in the accomplishments of my superiors at work (Vicarious control) In doing my work, I sometimes consider failure in my work as payment for future success (Interpretative control) Socioinstrumental It is important to cultivate relationships with superiors at work if you want to be effective You can get your own way at work if you learn how to get along with other people

224 Pilot Study Results Secondary control scale –11 items –Alpha =.75 –r = -.44 Work LOC Socioinstrumental control 24 items –Alpha =.87 –r =.26 Work LOC Two scales r =.12 (nonsignificant)

225 Main Study Method Subjects from HK, PRC, US –Ns = 130, 146, 254 –Employed students & university support –US from FIU and USF Work LOC & New Scales Stressors –Autonomy, conflict, role ambiguity & conflict Strains –Job satisfaction, work anxiety, life satisfaction

226 Coefficient Alphas ScaleHKPRCUS Secondary Socio- instrumental

227 Mean Differences VariableHKPRCUSR2R2 Second43.8 A 46.0 B 45.6 B.02 Socio93.4 A 97.1 B 91.9 A.01 Work LOC 51.0 B 57.0 C 40.2 A.38

228 Correlations With Work LOC VariableHKPRCUS Secondary Socio- instrumental

229 Significant Correlations VariableHKPRCUS SecondaryJob sat, Autonomy Job satAll Socio- instrumental Role conflictJob satAutonomy Work LOCJob sat, Autonomy, Conflict NoneAll

230 Conclusions Procedure created internally consistent scale Little mean difference China vs. US Work LOC huge mean difference Relationships different across samples

231 Nature of Indicators Reflective Vs. Formative Determines meaningful statistics Affects conclusions

232 Reflective or Effect Indicator Indicator caused by or reflects underlying construct Change in construct  Change in indicators Classical test theory Measures of attitudes and personality Needs internal consistency Factor analysis meaningful…..sometimes

233 Formative or Causal Indicator Indicator defines underlying construct Items don’t reflect single construct Items not interchangeable Change in indicator  Change in construct Examples –Socio-economic status Education, Income, Occupation Status –Behavior checklist (CWB or OCB) –Symptom checklist Internal consistency not always high Factor analysis might not be meaningful

234 Formative Indicator Example: Personality and CWB Trait anger and trait anxiety: Spielberger STPI CWB: Counterproductive Work Behavior Checklist N = 78 miscellaneous employees, community Trait anger & CWB r =.37 Trait anxiety & CWB r =.30 Can we assume anger & anxiety relate to all behaviors? Fox, Spector, Miles, 2001, Journal of Vocational Behavior

235 Item Trait AngerTrait Anxiety Purposely wasted your employer’s materials/supplies Daydreamed rather than did your work.45 *.42 * Complained about insignificant things at work.52 *.34 * Purposely did your work incorrectly Came to work late without permission Stayed home from work and said you were sick when you weren’t Purposely damaged a piece of equipment or property Purposely dirtied or littered your place of work Stolen something belonging to your employer Took supplies or tools home without permission Tried to look busy while doing nothing.34 *.36 * Took money from your employer without permission

236 How Do You Know? Theoretical interpretation Are items equivalent forms of construct? Do items correlate? Time sequencing—which changed first? –Does increase in SES affect education and income equally? No statistical test exists No automatic inference machine

237 Artifactual Constructs: Overinterpretation of Factor Analysis Tendency to assume factors = constructs If items load on different factors they reflect different constructs Sometimes item characteristics are confounded with factors –Wording direction

238 General Assumptions About Item Relationships Related items reflect same construct Unrelated items reflect different constructs Clusters of items reflect the same construct Factor analysis is magic

239 Dominance Model Assumptions About Measurement People agree with items in direction of position –If I have a favorable attitude, I will agree with all favorable items People disagree with items opposite to direction of position –If I have a favorable attitude, I will disagree with all unfavorable items Responses to oppositely worded items are a mirror image of one another –If I moderately agree with positive items, I will moderately disagree with negative items

240 Ideal Point Principle: Thurstone Items vary along a continuum. People’s positions vary along a continuum People agree only with items near their position Oppositely worded items not always mirror image Items of same value relate strongly Items of different value relate weakly

241 Agree Disagree Item value on construct continuum Person

242 Ideal Point Principle PessimismOptimism ABCDE

243 Difficulty Factors Ability tests Items vary in difficulty Items of same difficulty relate well –Those who get 1 easy will tend to get all easy Items of varying difficulty relate less well –Those who get hard tend to get easy –Those who get easy don’t all get hard

244 Example PeopleEasy ItemsHard Items Low Ability80% Correct0% Correct High Ability100% Correct80% Correct

245 Effects On Statistics Easy items strongly correlated Hard items strongly correlated Easy items relate modestly to Hard Factor analysis produces factors based on difficulty Difficulty factors reflect item characteristics not people characteristics

246 Summated Ratings Items of same scale value relate strongly Items of different value relate modestly Oppositely worded items have different scale values Scatterplots triangular not elliptical –High-Low, Low-High, and Low-Low common –Few High-High Often distributions are skewed Mixed value items produce factors according to scale value of item Might not reflect underlying constructs

247 Plot of Moderate Positively Vs. Negatively Worded Job Satisfaction Items


249 Plot of Extreme Positively Vs. Negatively Worded Job Satisfaction Items


251 Conclusions Be wary of factors where content is confounded with item direction Be wary when assumption of homoscedasticity is violated Be wary when items are extremely worded More evidence than factor analysis

252 Week 10 Theory

253 What Is A Theory? Bernstein –Set of propositions that account for, predict and control phenomena Muchinsky –Statement that explains relationships among phenomena Webster –General or abstract principles of science –Explanation of phenomenon

254 Types of Theories Deductive –Starts with theory –Data used to support/refute theory Inductive –Starts with data –Theory explains observations

255 Deduction Reasoning from premise to case Example –All doctoral students own laptops –Chris is a doctoral student –Therefore: Chris owns a laptop 

256 Induction Reasoning from observed cases to all cases Example –All doctoral students in my class have laptops –Therefore: All doctoral students have laptops

257 Inference To Best Explanation Conclusion based on likelihood Example –Apartment lock broken and laptop missing –Most feasible Must have been a thief –Alternative Roommate stole and then faked break-in

258 What Does Probability Mean? Frequency –.25 chance of coal miners get lung disease –25% of coal miners will get lung disease Subjective Interpretation –.01 chances of finding life on Mars –Does not mean that 1% of Mars-like planets have life –Our confidence in finding life

259 Induction and Probability Inductive approach based on chance our inference is correct Inferential statistics –Conclusion based on sample data –Generalizing to broader population

260 Covering Law Model of Science Hempel Covering Law –General laws –Particular facts –Phenomenon Begin with research question –Why did my plant die? –Plants need water. –I didn’t water my plant. –Therefore: My plant died.

261 Advantages Integrates and summarizes large amounts of data Can help predict Guides research Helps frame good research questions

262 Disadvantages Biases researchers “Theory, like mist of eyeglasses, obscures facts” (Charlie Chan in Muchinsky) “Facts are the enemy of truth” (Levine’s boss) A distraction as research does not require theory (Skinner)

263 Hypothesis Statement of expected relationships among variables Tentative More limited than a theory Doesn’t deal with process or explanation

264 Model Representation of a phenomenon Description of a complex entity or process –Webster Boxes and arrows showing causal flow

265 Observable Vs. Unobservable Some phenomena directly observable –Easily detectable with our senses –E.g., some behaviors Some phenomena unobservable –Inferred indirectly –E.g., internal cognitive states Theories about unobservable are underdetermined –Competing theories can explain observations

266 Theoretical Construct Abstract representation of a characteristic of people, situation, or thing Building blocks of theories

267 Paradigm Accepted scientific practice Rules and standards for scientific practice Law, theory, application and instrumentation that provide models for research. –Thomas Kuhn

268 What Are Our Paradigms? Behaviorism? Environment-perception-outcome approach Surveys

269 Structure of Scientific Revolutions Thomas Kuhn “An apparently arbitrary element, compounded of personal and historical accident, is always a formative ingredient of the beliefs espoused by a given scientific community at a given time.”, p. 4 “research as a strenuous and devoted attempt to force nature into the conceptual boxes supplied by professional education.”, p. 5

270 History of Theory in Psychology Behaviorism: Rejection of theory –More consistent with natural science –Avoid the unobservable –Dustbowl empiricism criticism “Cognitive revolution”: Embracing models and theory –Unobservables commonly studied Organizational research –Theory as paramount The empiricists strike back? –Hambrick and Locke

271 Current State of Theory Almost required in introductions –Marginalize importance of data –Ideas more important than facts Scholarship vs. Science –Scholarly writing—making good arguments –Scientific writing—describing/explaining phenomena based on data

272 Misuse of Theory Posthoc: Pretending theory drove research Citing theories as evidence Claiming hypothesis is based on a theory it is not based on Sprinkling cites to irrelevant theories –(Sutton & Staw)

273 Example from Stress Research Hobfol’s Conservation of Resources Theory –People are motivated to acquire and conserve resources –Demands on resources and threats to resources are stressful People routinely cite COR theory in support of stressor-strain hypotheses –No measure of resources or threat –Using a theory to support a hypothesis that does not derive from the theory

274 Why Do People Do This? Pressure for theory Everyone else is doing it –Descriptive norms Think this is real science Playing the game

275 Backlash Increasing criticism of the obsession with theory –Hambrick & Locke –Harry Barrick: Half-life of models in cognitive –AOM sessions –One unnamed reviewer –Informal interactions

276 Proper Role of Theory in Science Goal of science is to understand the world Science is evidence-based not intuition-based –Data is the heart of science –Theory is current state of understanding how/why things work Theory is the tail not the dog There is a place for both empiricism and theory

277 Natural Science More focused on data Longer timeframe –Decades and centuries of data before theory “Social science theory a smokescreen to hide weak data” USF chemist

278 Levels of Explanation Atomic or chemical Neural Individual cognitive Social Difficult to generalize across levels –Deduce properties from one to another Lower level is not more “scientific” or valuable


280 Social CognitiveCog/NeuroNeuroscienceBehavioral GeneticsI/O I/O is just applied Social. Social is just applied Cognitive. Cognitive is just applied Cog/Neuro. Cog/Neuro is Just applied Neuroscience. It’s nice to be on top.

281 Use Theory Properly Hypotheses: Explicitly derive from a theory Don’t claim support from a theory Often better to mention theories in the discussion Multiple theories useful in comparative test

282 Additional Reading Accessible overview of philosophy of science Okasha, S. (2002). Philosophy of Science. Oxford, UK: Oxford University Press.

283 Week 11 Levels of Analysis

284 Level Nature of the sampling unit –Person –Couple –Family –Group/Team –Department –Organization –Industry sector –Country

285 Individual Vs. Higher Level Most psychological constructs person level –Attitude –Performance Some constructs higher (aggregate) level –Organizational climate –Team performance

286 Types of Aggregates Sum of individuals –Sales team performance Consensus of individuals –Mean of individuals –Majority votes Aggregate level data –Job analysis observer ratings for job title –Organization profitability –Team characteristic (size, gender breakdown) –Turnover rates

287 Aggregate As Sum of Individuals Sum individual characteristics –Ask individuals about own values –Sum values Direct assessment of aggregate –Ask individuals about people in their unit –“How do your team members feel about…?” –Sum values

288 Aggregate As Consensus Shared perceptions –Climate People within unit should agree Assess extent of agreement –Intraclass correlation, ICC(1) –Rwg

289 Partition Variance Total variation among subjects Between variation among group means Within variation among subjects within groups Total = Between + Within

290 Intraclass Correlation ICC(1) Multiple raters of multiple targets Extent of within rater agreement Var (Total) = Var (Between target) + Var (Within target) ICC = Var (Between) / Var (Total) Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86,

291 Rwg Compares observed to expected (by chance) variance of ratings Rwg = (Var (expected) – Var (Obs) ) / Var (Expected) Var (Expected) = (A 2 – 1)/12 –A = Number of rating categories –Assumes uniform distribution of responses James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within- group interrater reliability with and without response bias. Journal of Applied Psychology, 69,

292 Example 10 Raters: 5, 2, 3, 5, 2, 3, 1,4, 3, 4 A = 5 (5 rating choices) Var (Expected) = (5 2 – 1)/12 = 2 Var (Observed) = 1.73 Rwg = (2 – 1.73) / 2 =.135

293 Independence of Observations Independence a statistical assumption Subjects nested in groups –Subjects influence one another –Observations non-independent Example –Effects of supervisory style on OCB –Subjects nested in workgroups –Style ratings within supervisor nonindependent –Subjects influence one another –Shared biases

294 Confounding of Levels Individual case per unit –One person per organization Nonindependence issue resolved Confounding of individual vs. organization –Is relation due to individual or organization? Potential problem for inference –Self report of satisfaction and org performance –Could be shared bias—happy employee reports greater performance

295 Ecological Fallacy Drawing inferences from one level to another When measurement and question don’t match –Job satisfaction vs. group morale –Individual behavior vs. group behavior Improper inference Can’t draw conclusions across levels Empirically data only reflect own level

296 If it works for individuals, why won’t it work for groups? Correlation and Subgroups

297 Individual: No correlation

298 Individual: No correlation, Group positive correlation

299 Individual within group no correlation; Group positive

300 Individual within group no correlation; Group negative

301 Individual within group positive correlation: Group none

302 Individual within group positive correlation; Group negative

303 Pay and Job Satisfaction Question 1: Job level –Do better paying jobs have more satisfied people? Question 2: Individual level –Are better paid within jobs more satisfied?

304 Job Satisfaction Salary Nurses Physicians Pay-Job Satisfaction correlation Mixed jobs r =.17 (Spector, 1985) Single job r =.50 (Rice et al. 1990)

305 Pooled Within-Group Correlation Remove Effects of Group Differences in Means

306 Variance/Covariance SS = Σ(X-GM X ) 2 = Σ(X-GM X )(X-GM X ) SP = Σ(X-GM X )(Y-GM Y )

307 Group Versus Grand Means Group 1 Grand MeanGroup 2 Grand Mean Group 2...

308 Pooled Within-Group Correlation Remove Effects of Group Differences in Means Correlation Pooled WG where 1 and 2 refer to groups 1 and 2

309 Multi-level Studies Lower to higher levels numbered Multiple teams across multiple organizations in multiple countries –Level 1 = person –Level 2 = team –Level 3 = organization –Level 4 = country

310 Two-Level Most studies two-level Need multiple units at each level for adequate power Can analyze within level or across –Main effects for people and for teams –Interaction of level 1 with level 2

311 Experience Sampling or Daily Diary Multiple observations on same person –Daily –Multiple times/day Two-level –Level 1: Within person –Level 2: Between person Can test if IV fluctuation leads to DV fluctuation within people Can look at DV before and after event

312 Hierarchical Linear Modeling Statistical technique Deals with nonindependence Analyze data at two or more levels –Individuals in teams –Teams in organizations Cross levels –Does team-level diversity affect relation between individual satisfaction and OCB? –Does individual level satisfaction affect team level performance?

313 Levels of Behavior Aggregation How should behaviors be combined?

314 Overall Index Sum of multiple behaviors Skarlicki-Folger Retaliation = 17 items Bennett-Robinson Deviance = 24 items Spector-Fox Counterproductive Work Behavior Checklist (CWB-C) = 45

315 Distinguish Target Robinson-Bennett 1995 –Organization versus person target –Bennett-Robinson deviance scale –CWB-C Bennett, R. J., & Robinson, S. L. (2000). Development of a measure of workplace deviance. Journal of Applied Psychology, 85, Robinson, S. L., & Bennett, R. J. (1995). A typology of deviant workplace behaviors: A multidimensional scaling study. Academy of Management Journal, 38,

316 Five Dimensions of CWB-C Abuse Production Deviance Sabotage Theft Withdrawal Spector, P. E., Fox, S., Penney, L. M., Bruursema, K., Goh, A., & Kessler, S. (2006). The dimensionality of counterproductivity: Are all counterproductive behaviors created equal? Journal of Vocational Behavior, 68,

317 More Dimensions Abuse – Physical – Verbal Production deviance – Loafing – Damaging Sabotage Theft – From coworker – From Organization Withdrawal

318 Even Finer Grained Abuse –Physical Weapon No Weapon –Slap –Punch Injure vs. not Injury needs medical treatment vs. not Reactive vs. proactive

319 Does It Matter? CWB-C –45 items Job Satisfaction N = 312

320 Correlation With Job Satisfaction Total -.32 * –CWB-organization -.35 * –CWB-person -.19 * Abuse -.31 * Withdrawal -.22 * Production deviance -.19 * Sabotage -.14 * Theft -.05 –Individual Items40% Significant

321 Itemr Ran down employer to others -.60 Called in sick when not -.24 Worked slowly on purpose -.22 Insulted someone about their work -.20 Stole from employer -.08 Purposely dirtied or littered workplace -.03 Purposely damaged equipment -.02 Spread rumor -.02

322 Conclusion Behavior checklist is formative not reflective –Items are not interchangeable –High levels of composite do not mean high levels of all components –Same score from different sets of behaviors Do not cross levels with inference –Assume components have same relation as composite

323 Week 12 Statistical Inference

324 Null Hypothesis Signficance Testing Purposes –Reduce subjectivity –Agreed upon rules for claims –Based on consistent metric –Communicates in common language Based on probability

325 What Does Significance Mean? Results unlikely due to chance If the null were true, observed results would be expected less that 5% of the time Null hypothesis is a comparison standard –There is a zero relationship between variables –Alternative: There is a nonzero relationship

326 Limitations Sensitive to sample size –Small samples low power –Large samples small effects can be significant Problem is interpretation not the method itself Does not indicate probability that point estimates are correct. –Point estimates have confidence intervals Failure to find significance does not mean null is true

327 Critics’ Claims Can’t be certain true effect is zero if test is nonsignificant. –So what? Promotes dichotomous thinking –This is limitation of researcher not statistics Significance tests are misused.

328 Suggested Alternative Report confidence intervals and effect sizes –Can do whether or not you test significance Report CI and show that it does or does not include zero –How is this different from reporting significance?

329 Benefits of the Controversy Emphasis on appropriate use of significance testing More complete reporting –Effect sizes –Confidence intervals

330 Rule of Thumb Cutoffs Situations where significance tests are unavailable –Cannot reduce situations to probabilities –Based on opinions of authorities Jum Nunnally Peter Bentler Larry James –Past experience –Simulations

331 Become Widely Accepted Internal consistency (Nunnally) –Coefficient alpha (α ≥.70) –Basement for new research –Modified from 1 st edition (α ≥.60) SEM fits statistics (Bentler & Bonett) –Criterion ≥.90 –<.90: model can be substantially improved –Many fit statistics with different properties

332 Lance, Butts, Michels 2006 Close look at basis for cutoffs Attributed sources said something more complex –Nunnally –Bentler & Bonett Attributed source didn’t really say –James No specific source –Eigenvalue > 1 criterion

333 Four Criteria Well established –Coefficient alpha ≥.70 –SEM fit stats ≥.90 Other criteria sometimes used (e.g.,.95) See occasionally –R WG ≥.70 No longer very common –Eigenvalue > 1 –Sometimes used as a starting point –Scree more common

334 Statistical and Methodological Myths and Urban Legends SMA 2003 –“Three Authors Speak Out On the Peer Review Process” Art Bedeian, Mark Martinko, Paul Spector, Bob Vandenberg Discussion afterwards with Larry Williams AOM 2004 First SMMUL Standing room only SIOP (5 times)

335 SMMUL Publication Organizational Research Methods –Special feature Lance & Vandenberg 2009 edited book SMMULs, NY: Routledge

336 Decimal Dust Bedeian et al. Misplace precision in reporting statistics Point estimates are not precise, given typical sample sizes Don’t over-report significant digits Correlation: 2 digits Means and SDs: 1 digit right of decimal

337 Statistical Power Probability of finding a true effect Type II error –Failure to detect a true effect. Four factors –One or two tailed –Significance level –Sample size –Effect size


339 Increase power 1.One-tailed 2.Less stringent p level 3.Greater n 4.Bigger effect size 1, 2 Move rejection line to right 3 Distributions get tighter 4 Move distribution to left

340 Standard Errors Factors that increase standard errors –Small sample size –Multicolinearity Larger correlations among predictors

341 Regression Coefficients Y = X 1 + X 2

342 Regression Coefficients Standard Errors

343 Imprecision of prediction Sum of Squares of predictor Multicolinearity

344 Regression Coefficients Standard Errors Imprecision of prediction Sum of squares of predictor Multicolinearity

345 Appropriate Comparisons Conclusions require direct tests Cannot conclude differences based on different p levels Example: Do groups differ on correlations? –Group 1: r =.20 significant –Group 2: r =.15 nonsignificant –N = 100 –Z-test =.3593: Nonsignificant –Differences of this size likely by chance

346 Comparing Correlation Magnitude Draw conclusions about size of relationship based on observed correlations Are correlations of that magnitude significant? Caution in multiple regression Will results replicate?

347 Do Data Tell Consistent Story? An Example Cech, E., Rubineau, B., Silbey, S., & Seron, C. (2011). Professional role confidence and gendered persistence in engineering. American Sociological Review, 76, Does role confidence lead to women’s failure to complete an engineering degree?

348 Prospective Survey Study Freshman year –Major –Confidence Expertise: developing skills in college Professional: satisfaction with career Senior year –Completed major –Transferred to STEM vs. nonSTEM major

349 Author’s Conclusion “Instead, we find that professional role confidence is significantly associated with engineering persistence, and that its differential distribution between men and women contributes to gender segregation in engineering.”, p. 656

350 Their Basis for Conclusion Gender differences in confidence –Expertise: Female = 3.09, Male = 3.44 –Career-fit: Female = 2.67, Male = 3.01 –Both significant (t-test), –N: Female = 125, Male = 163 Gender differences in completion –Female = 77.2%, Male = 82.5% (significant t-test) Expertise (not professional) confidence significant in regression –11 control variables (including dummy coded college) –6 predictors

351 Closer Look Correlations in appendix Correlations with engineering completion –Expertise confidence r =.006 (ns) –Career-fit confidence r = (ns) Confidence does not predict completion Why significant in regression?

352 No Direct Gender Comparison Authors logic –Gender difference in confidence –Confidence relates to completion –Therefore confidence causes gender difference What do we need to draw this conclusion?

353 Where Do Students Go? Females more likely to switch within STEM –Female = 16.7% vs Male = 6.3% (Significant) Females more likely to graduate in STEM –Female = 93.9% vs. Male = 88.8%

354 Reasonable Conclusion Males more likely to stick with engineering Women more likely to switch to other STEM disciplines Confidence unrelated to engineering persistence Not clear why men more likely to stick with engineering

355 Approach To Data Analysis Clean dataset –Check inputting for accuracy –Check out of range observations Frequencies Ranges –Item analysis Item-remainders Coefficient alpha –Means and correlations make sense Compare to other studies

356 Analysis From Simple To Complex Descriptive statistics –Means, SDs, ranges, frequency distributions –Look for restriction of range –Does everything look reasonable? Simple relationships –Correlations –T-tests –One-way ANOVAs Study results at length

357 Complex Statistics Complex only after you have a feel for simple results Run complex sequentially with increasing complexity Compare complex with simple –Consistency in conclusions –Inconsistencies need to be explained Avoid assuming complex correct Complex and simple test different things Goal is a coherent explanation

358 Cautions Be aware of assumptions Be aware of relative power of analyses being compared Don’t overlook methodological artifacts Go beyond cookbook interpretations Don’t let hypotheses/theories blind you Avoid premature conclusions Avoid pure data mining –Type 1 error hunt Don’t be afraid to test alternatives

359 Week 13 Literature Reviews

360 Narrative Review Summary of research findings Qualitative analysis “Expert” analysis Based on evidence Room for subjectivity Classical approach

361 Meta-analysis Quantitative cumulation of findings Based on common metric Many approaches Many decision rules Room for subjectivity in decision rules

362 Meta-Analysis  Hunter- Schmidt Approach There are MANY ways to conduct meta-analysis

363 Use of Narrative Review Used almost exclusively before 1990s Psychological Bulletin In depth literature summary Brief overview vs. comprehensive –Brief overview part of empirical articles Can contrast very different studies –Constructs –Designs –Measures Small number of studies

364 Limitations to Narratives One person’s subjective impression Different reviews – different conclusions Lacks decision rules for drawing conclusions –What if half studies are significant? Difficulty with conflicting results Narratives often hard to read Narratives difficult to write

365 Narrative Review Procedure Define domain Decide scope (how comprehensive) Inclusion rules Identify/obtain studies Read studies/take notes Organize review –Outline of topics –Assign studies to topic Write sections Draw conclusions

366 Meta-Analysis NO AUTOMATIC INFERENCE MACHINE Does NOT provide absolute truth Does NOT provide population parameters –Provides parameter estimates, i.e. statistics –Samples not always random or representative Has not revolutionized research Is just another tool that you need

367 Just Another Tool

368 Use of Meta-Analysis Dominant procedure today for reviews Published in most journals Can help settle inconsistencies in conclusions Often descriptive and superficial Allows for hypothesis tests –Moderators Requires highly similar studies –Constructs –Designs –Measures

369 Settling an Inconsistency Does HONS moderate job characteristics  Outcomes Six reviews—inconsistent conclusions –2 no effect –1 weak effect –2 positive effect –1 no conclusion Settle with meta-analysis Spector, P. E. (1985). Higher-order need strength as a moderator of the job scope-employee outcome relationship: A meta-analysis. Journal of Occupational Psychology, 58,

370 The Meta-Analysis Literature search –20 studies –Correlations of core characteristics with outcomes –Compared high versus low HONS groups Results –8 of 11 cases moderator effect found HONS moderator supported Low power likely contributed to inconsistency

371 Limitations To Meta-Analysis Small number of studies meeting criteria Convenience sample of convenience samples Subjectivity of decision rules –Inclusion/exclusion rules –Statistics used –Procedures to gather studies Journals Dissertations Unpublished Different reviewers, different conclusions Sometimes data are made up Need lots of studies

372 Meta-Analysis Procedure Define domain and scope Inclusion rules Decide on M-A method –Artifact adjustments? Identify/obtain studies Code data from studies Conduct analyses Prepare tables Write paper/interpret results

373 Define Domain Choose topic Specify domain –Personality: Big Five vs. Individual traits Define populations –Employees vs. Students Define settings –Workplace vs. Home Types of studies –Group comparisons vs. correlations Define variable operationalizations –Self-reports vs. other reports

374 Apples Vs. Oranges Quantitative estimate of population parameter –What is the population? Mean effect size across samples Assumes sample statistics assess same thing Cumulating results across different constructs not meaningful

375 Inclusion Rules Operationalizations parallel forms –Measures of NA, neuroticism, emotional stability, trait anxiety –All trait measures Samples from same population –All full-time working adults –Full-time = > 30 hours/week –All American samples Designs equivalent –All cross-sectional self-report Journal published studies vs. others

376 Meta-Analysis Method Many to choose from Nature of studies –Group comparisons –Correlations Rosenthal –Describe distribution of rs –Moderators as specific variables to test Hunter-Schmidt –Adjust for artifacts –Moderators as more variance than expected

377 Effect Size Estimates Combine effect sizes Correlation as amount of shared variance Magnitude of mean differences Where d is difference in means in SD units

378 Identify/Obtain Studies Electronic databases (PsycINFO) Other reviews Reference lists of papers Conference programs/proceedings Listservs Write authors in area

379 Coding Choose variables to code Judgments about inclusion rules How to handle multiple statistics –Independent samples –Dependent samples: Average Sometimes ratings made, e.g., quality –Interrater agreement

380 Variables To Code Effect sizes N Reliability of measures Name of measures Sample description –Demographics –Job types –Organization types –Country Design

381 Analysis Meta-analysis software Statistical package –Excel, SAS, SPSS Organize results –Tables by IV or DV Analysis of moderators

382 Interpret Often descriptive –Little insight other than mean correlations –Nothing new if results have been consistent Often superficial Can test hypotheses Effects of moderators Can inconsistencies be resolved? Suggest new directions or research gaps?

383 Rosenthal Approach Convert statistics to r –Chi square from 2x2 table –Independent group t-test –Two-level between group ANOVA Convert r to z Compute descriptive statistics Describe results in tables Meta-analysis as summary of studies

384 Rosenthal Descriptives Mean effect size Weighted mean Median Mode Standard deviation Confidence interval

385 Rosenthal Moderators Identify moderator and relate to effect sizes Correlate characteristic of study with r Shows if r is a function of moderator

386 Moderator Example Satisfaction – turnover Unemployment as moderator Found studies Contacted authors where/when conducted Database of unemployment rates Correlated unemployment to r of study Unemployment –r with satisfaction- turnover Carsten-Spector 1987 Journal of Applied Psychology

387 Schmidt-Hunter Convert effect sizes to r Compute descriptive statistics on r Collect artifact data –Theoretical variability –Unreliability –Restriction of range –Quality of study Artifact distributions to estimate missing data Adjust observed mean r to estimate rho Compare observed SD to theoretical after adjustments Residual variance = moderators

388 Estimating Missing Artifacts Estimate = Make up data “The magic of statistics cannot create information where none exists” Wainer Existing data to guess what missing might have been Hall-Brannick JAP 2000 it is inaccurate Science of what might be rather than what is

389 Value of Artifact Adjustments Variability in r is what is/isn’t expected Show that variance due to differential reliability, restriction of range, etc. Requires you have artifact data

390 Rosenthal Vs. H-S Both identify/code studies Both compute descriptive statistics r to z transformation –Rosenthal yes, H-S no H-S artifact adjustments H-S rho vs. Rosenthal mean r H-S advocate estimating unobservables Rosenthal deals only with observables Begin the same, H-S goes farther Rosenthal similar to H-S bare bones

391 Why I Prefer Rosenthal Rho is parameter for undefined population –Convenience sample of convenience samples –Population = studies that were done/found Unavailable artifact data –Uncomfortable in estimating missing data Prefer to deal with observables Don’t believe in automatic inference Lot’s of competing methods

392 Week 14 Ethics and Research Integrity

393 Appropriate Research Practices Conducting Research –Treatment of human subjects –Treatment of organizational subjects Data Analysis/Interpretation Disseminating Results –Publication Peer reviewing

394 Ethical Codes Appropriate moral behavior/practice Accepted practices Basic Principle: Do no harm Protect dignity, health, rights, well-being Codes –APA –AOM

395 American Psychological Association Code Largely Practice oriented Five principles –Beneficence and Nonmaleficence [Do no harm] –Fidelity and Responsibility –Integrity –Justice –Respect for People’s Rights and Dignity Standards and practices Applies to APA members

396 Preamble Psychologists are committed to increasing scientific and professional knowledge of behavior and people's understanding of themselves and others and to the use of such knowledge to improve the condition of individuals, organizations, and society. Psychologists respect and protect civil and human rights and the central importance of freedom of inquiry and expression in research, teaching, and publication. They strive to help the public in developing informed judgments and choices concerning human behavior. In doing so, they perform many roles, such as researcher, educator, diagnostician, therapist, supervisor, consultant, administrator, social interventionist, and expert witness.

397 APA Conflict Between Profession and Ethical Principles Restriction of Advertising –Violation of the law Maximization of income for members Tolerance of torture –Convoluted statements Other associations manage to avoid such conflicts

398 Academy of Management Code Largely academically oriented Three Principles –Responsibility –Integrity –Respect for people’s rights and dignity Responsibility to –Students –Advancement of managerial knowledge –AOM and larger profession –Managers and practice of management –All people in the world

399 Professional Principles Our professional goals are to enhance the learning of students and colleagues and the effectiveness of organizations through our teaching, research, and practice of management.

400 AOM Vs. APA AOM –Consistent principles –Simpler –More directly relevant to organizational practice and research –No attempt to compromise ethics for profit

401 Principles Vs. Practice Principles clear in theory Ethical line not always clear Ethical dilemmas –Harm can be done no matter what is done –Conflicting interests between parties Employee versus organization Whose rights take priority?

402 Example: Exploitive Relationships Principle –Psychologists do not exploit persons over whom they have supervisory, evaluative, or other authority What does it mean to exploit? Professor A hires Student B to be an RA –How much pay/compensation is exploitive? –How many hours/week demanded? What if student gets publication?

403 Example: Assessing Performance In academic and supervisory relationships, psychologists establish a timely and specific process for providing feedback to students and supervisees. Not giving an evaluation is unethical? How often? How detailed? What if honest feedback harms the person’s job situation?

404 Conducting Research Privacy Informed consent Safety Debriefing Inducements

405 Privacy Anonymity: Best protection –Procedures to match data without identities Confidentiality –Security of identified data Locked computer/cabinet/lab Encoding data Code numbers cross-referenced to names –Removing names and identifying information

406 Informed Consent Subject must know what is involved –Purpose –Disclosure of risk –Benefits of research Researcher/society Subject –Privacy/confidentiality Who has access to data Who has access to identity –Right to withdraw –Consequences of withdrawal

407 Safety Minimize exposure to risk –Workplace safety study: Control group Physical and psychological risk

408 Debriefing Subject right to know Educational experience for students Written document Presentation Surveys: Provide contact for follow-up Provide results in future upon request

409 Inducements Pure Volunteer – no inducement Course requirement –Is this coercion? Extra credit Financial payment –Is payment coercion?

410 Institutional Review Board: IRB Original Purpose: Protection of human subjects Current Purpose: Protection of institution –Federal government requirement We pay for government atrocities of the past –Government sanctions –Bureaucratic –Often absurd –Designed for invasive medical research

411 IRB Jurisdiction Institutions receiving federal research funds All funded research under jurisdiction Cross-country differences Canada like US China doesn’t exist

412 Types of Review Full –One year Expedited: Research with limited risk –Data from audio/video recordings –Research on individual or group characteristics or behavior (including, but not limited to, research on perception, cognition, motivation, identity, language, communication, cultural beliefs or practices, and social behavior) or research employing survey, interview, oral history, focus group, program evaluation, human factors evaluation, or quality assurance methodologies. –One year Exempt –Five years

413 Exempt Project doesn’t get board review Determined by staff member You can’t determine own exemption Five year Surveys, interviews tests, observations Unless Subjects identified AND potential for harm –Legal liability –Financial standing –Employability –Reputation

414 IRB Impact Best: Minor bureaucratic inconvenience –Protects institution –Protects investigator Worst: Chilling effect on research –Prevents certain projects –Ties up investigators for months

415 IRB: What Goes Wrong? Inadequate expertise –Lack of understanding of research –Apply medical model to social science Going beyond authority –Copyright issues Abuse of power Refuge of the petty and small minded tyrant

416 Research Vs. Practice Research = Purpose not activity Dissemination intent = research –Presentation –Publication Class demo not research Management project not research –Consulting projects as research projects Don’t ethics apply to class demos? –Not IRB purview

417 Dealing with Organizations Who needs protection –Employee –Organization Who owns and can see the data? –Researcher –Organization What if organization won’t play by IRB rules? –IRB has no jurisdiction off campus

418 Anticipate Ethical Conflicts Avoid issues –Don’t know can’t tell Negotiate issues –Confidentiality –Nature of report –Ownership of data –Procedures

419 Integrity Issues: Analysis Honesty in research Report what was done –Why Hunter-Schmidt procedures aren’t unethical Estimation procedures transparent Bad data practice –Fabrication –Deleting disconfirming cases: Trimming –Data mining: Type 1 Error hunt –Selective reporting: Only the significant findings

420 Dissemination Authorship credit Referencing Sharing Data Editorial issues –Editor –Reviewers

421 Author Credit Authors: Substantive contributions –What is substantive? –People vary in generosity Order of authorship –Order of contribution –Not by academic rank –Dissertation/thesis special case –Last for senior person Authorship agreed to up front Potential for student/junior colleague exploitation

422 Slacker Coauthors When do you drop from coauthorship Late Not at all Poor quality Less than you expected

423 Submission One journal at a time One conference at a time Can submit to conference and journal –Prior to paper being in press Almost all are electronic submission –Can be difficult and tedious Break paper into multiple documents Enter each coauthor Most reviewing is blind –Only editor knows authors/reviewers

424 Journal Review All 1 st submissions are rejects –Don’t want to see again –Revise and resubmit (R&R) Will consider revision if you insist (high risk) Encourages resubmission Desk rejections: No review Feedback from 1 to 4 reviewers (mode 2) Feedback from editor Multiple cycles of R&R can be required –Can be rejected at any step Tentative accept: Needs minor tweaks Full accept: Congratulations!

425 Steps To Publication Submit R&R Revise –Include response to feedback Provisional acceptance –Minor revision Acceptance –Copyright release –Proofs In print –Entire process 1 year or more

426 R&R More likely accepted than rejected –Depends on editor –Good editor has few R&R rejects Work hard to incorporate feedback Argue points of disagreement –Additional analyses –Prior literature –Logical argument Don’t be argumentative –Choose your battles Give high priority

427 Author Role Make good faith effort to revise Incorporate feedback Be honest in what was done –Don’t claim you tried things you didn’t Treat editor/reviewers with respect

428 Editor Role Be an impartial judge Weight input from authors and reviewers Be decisive Keep commitments –R&R is promise to publish if things fixed Treat everyone with respect

429 Reviewer Role Objective review No room for politics Reveal biases to editor Disclose ghost-reviewer to editor –E.g., doctoral student –Pre-approval Private recommendation to editor Feedback to author/s Keep commitments Treat author with respect

430 Reviewer As Ghostwriter Art Bedeian Notes reviewers go too far –Dictating question asked, hypotheses, analyses, interpretation Review inflation over the years –Sometimes feedback longer than papers Reviewers subjective Poor inter-rater agreement Abuse of power?

431 Reviewer Problems Reviewers late Reviewers nasty Overly picky Factually inaccurate Overly dogmatic –Favorite stats (CFA/SEM) –Edit out ideas they disagree with –Insist on own theoretical position –Assume there’s only one right way Not knowledgeable Miss obvious Careless

432 Scientific Progress Through Dispute Work is based on prior work –Testing theories –Integrating findings/theories Build a case for an argument or conclusion Disseminate Colleagues build case for alternative –Scientific dispute Two camps battle producing progress –Dispute motivates work –Literature enriched

433 Crediting Sources Must reference anything borrowed –Cite findings/ideas –Quote direct passages –Little quoting done in psychology Stealing work –Plagiarism: not quoting quotes –Borrowing ideas Papers People Reviewed papers –Easy to forget you didn’t have idea

434 Strategy For Successful Publication Choose topic field likes –Existing hot topic –Tomorrow’s hot topic (hard to predict) Conduct high quality study Craft good story –Make a strong case for conclusions –Theoretical arguments in introduction –Strong data to test Write clearly and concisely Pay attention to current practice Lead don’t follow

435 Dealing With Journals Be patient and persistent Match paper to journal –Journal interests –Quality of paper Count on extensive revision Learn from rejection –Consider feedback –Only fix things you agree with –Look for trends over reviewers

436 Fragmented Publication Multiple submission from same project Discouraged in theory Required in practice –Single purpose –Tight focus Different purposes –Minimize overlap –Cross-cite –Disclose to editor

437 Example: CISMS Four major papers Unreliability of Hofstede measure –Applied Psychology: An International Review Universality of Work LOC and well-being –Academy of Management Journal Country level values and well-being –Journal of Organizational Behavior Work-family pressure and well-being –Personnel Psychology

438 How Much Overlap Is Too Much? A: Aquino, K., Grover, S. L., Bradfield, M., & Allen, D. G. (1999). The effects of negative affectivity, hierarchical status, and self-determination on workplace victimization, Academy of Management Journal, 42, B: Aquino, K. (2000). Structural and individual determinants of workplace victimization: The effects of hierarchical status and conflict management style. Journal of Management, 26,

439 Purpose from Abstract A: “Conditions under which employees are likely to become targets of coworkers’ aggressive actions” B: “…when employees are more likely to perceive themselves as targets of co- workers’ aggressive actions”

440 Procedure A:“Two surveys were administered to employees of a public utility as part of an organizational assessment. Although the surveys differed in content, both versions contained an identical set of items measuring workplace victimization” B: “Two different surveys were administered to employees of a public utility as part of an organizational assessment. Although the surveys differed in content, both versions contained an identical set of items measuring workplace victimization.”

441 Sample/Measures A: n = 371, 76% response, mean age 40.7, tenure 11.5, 65% male, 72% African American B: n = 369, 76% response, mean age 40.7, tenure 11.5, 65% male, 72% African American A: PANAS, Hierarchical status (Haleblian), self- determination, Victimization (14 items) B: Rahim Organizational Conflict Inventory-II, Hierarchical status (Haleblian), Victimization (14 items)

442 Factor Analysis/Table 1 A: Exploratory FA of Victimization, CFA of 8 items on holdout sample B: Exploratory FA of Victimization, CFA of 8 items on holdout sample A: “Factor loadings and lamdas for victimization items a ” [Note misspelling of lambda] B: “Factor loadings and lamdas for victimization items 1 ” [Note misspelling of lambda]

443 Table 3/Hypothesis Tests A: “Results of hierarchical regression analysis” B: “Results of hierarchical regression analysis” A: “Two regression equations were fitted: one predicting direct victimization and the other predicting indirect victimization.” B: “Two regression equations were fitted: one predicting direct victimization and the other predicting indirect victimization.”

444 Limitations A: “This study has several limitations that deserve comment. Perhaps the most serious is its cross- sectional research design. The victim precipitation model is based on the assumption that victims either intentionally or unintentionally instigate some negative acts.” B: “This study has several limitations that deserve comment. Perhaps the most serious is its cross- sectional research design. The victim precipitation model is based on the assumption that victims either intentionally or unintentionally instigate some negative acts…”

445 Research Support: Grants Funding needed for many studies –Expands what can be done Some research very cheap –Shoestring because lack of funding? –Lack of funding because research is cheap? Universities encourage grants –Diminishing state support

446 Grant Pros Tangible –Covers direct costs of research Equipment/supplies Subject fees/inducements Human resources (research assistants) –Rewards to investigator Summer salary Course buyout Conference travel –Support students Intangible –Forces you to plan study in detail –Prestige –Administrative admiration (rewards)

447 Grant Cons Tough to get: Competitive –Time consuming Requires resubmission with long cycle time Administrative burden Takes time from teaching/research Can redirect research focus –Not always a bad thing Confuse path with goal –Grant is not a research contribution

448 Sources Federal (Highest status—indirects to university) Foundations Internal University Grants –Small –Not as competitive New faculty –New investigator and small grants Doctoral students –ERC pilot grants –SIOP grants –Dissertation grants

449 Federal Grants Challenging High rejection rate Takes multiple submissions (R&R) Must link to priorities—not everything fundable

450 Grant Strategy Develop grant writing skill Tie to fundable –Workplace health and safety Musculoskeletal Disorders (MSD) and …. Workplace violence Intervention research in demand Interdisciplinary Use of consultants Pilot studies Programmatic and strategic

451 Week 15 Wrap-Up

452 Successful Research Career Conducting good research –Lead don’t follow Visibility –Good journals –Conferences –Other outlets –Quantity First authored publications –Important more early in career Impact Grants

453 Programmatic Program of research –More conclusive –Multiple tests –Boundary conditions –More impact through visibility –Helps getting jobs –Helps with tenure/promotion –Can have more than one focus

454 Conducting Successful Research Develop an interesting question –Based on theory –Based on literature –Based on observation –Based on organization need Link question to literature –Theoretical perspective –Place in context of what’s been done –Multiple types of evidence –Consider other disciplines

455 Conducting Successful Research 2 Design one or more research strategies –Lab vs. field –Data collection technique Survey, interview, observation, etc. –Design Experimental, quasi-experimental or observational Cross-sectional or longitudinal Single-source or multisource –Instrumentation Existing or ad hoc

456 Conducting Successful Research 3 Analysis Hierarchy of methods simple to complex –Descriptives –Bi-variable relationships –Test for controls –Complex relationships Multiple regression Factor analysis HLM SEM

457 Conducting Successful Research 4 Conclusions –What’s reasonable based on data –Alternative explanations –Speculation –Theoretical development –Suggestions for future

458 KSAOs Needed Content knowledge Methods expertise Writing skill Presentation skill Creativity Thick skin

459 Pipeline Body of work at various stages –In press –Under review –Writing/revising –In progress –Planning Set priorities –Don’t let revisions sit –Get work under review –Always work on next project Collaboration to multiply productivity Time management

460 Authorship Order First takes the lead on paper –Most of writing –Most input in project Important early in career to be first Balance quantity with order Sometimes most senior person is last –PI on project –Senior member of lab

461 Impact Effect of work on field/world Citations –Sources ISI Thomson Harzing’s Publish or Perish Others –Self-citation –Citation studies Individuals (e.g., Podsakoff et al. Journal of Management 2008) Programs (e.g., Oliver et al. TIP, 2005) Being attacked

462 Partnering State/local government Corporations Tying research to consulting Partnership with practitioner –In kind

463 Grants Expands what you can do Good for career –Current employer –Potential future employers

464 Grantsmanship Develop grant writing skill –Start as a student Small grant at first Proposal somewhat different from article –Background that establishes need for study –Demonstrates ability to conduct Expertise of team Letters of agreement/support –High likelihood of success Pilot data important Low risk Address funding agency priority

465 Final Advice Be a leader not a follower –Address problem that is not being addressed –Find creative ways of doing things Be evolutionary not revolutionary –Too different unlikely to be accepted –Most creative often in lesser journals Follow-up studies in better journals Critical mass –Need multiple publications on topic to be noticed –Programmatic Build on the past, don’t tear it down –Positive rather than negative citation

466 Final Advice cont. Be flexible in thinking –Don’t get prematurely locked into Conclusion, Idea, Method, Theory Use theory inductively –A good theory explains findings Don’t take yourself too seriously Have a thick skin Enjoy your work

Download ppt "Organizational Research Methods Week 2: Causality."

Similar presentations

Ads by Google