Organizational Research Methods Week 2: Causality

Organizational Research Methods Week 2: Causality

What Do We Mean By Causality?
Relationship between two events where one is a consequence of the other Determinism: A (cause) leads to B (effect) “In the strict formulation of the law of causality—if we know the present, we can calculate the future—it is not the conclusion that is wrong but the premise”. On an implication of the uncertainty principle. Werner Heisenberg

Heisenberg & Uncertainty Principle
Certain properties of subatomic particles are linked so the more accurately you know one, the less accurately you know the other We can compute probabilities not certainties Argues against determinism “Physics should only describe the correlation of observations; there is no real world with causality” Heisenberg, 1927, Zeitschrift für Physik Psychology, like quantum physics, is probabalistic

Cause Versus Effect Effect of a Cause (Description)
What follows a cause? Cause of an Effect (Explanation) Why did the effect happen? Do bacteria “cause” disease? Actually toxins cause disease Actually certain chemical reactions are cause Holland, P. W. (1988). Causal inference, path analysis, and recursive structural equations models. Sociological Methodology, 18,

Three Elements of Causal Case
Cause and effect are related Cause preceded effect No plausible alternative explanations John Stuart Mill

Experiment Vary something to discover effects
Shows association Shows time sequence Can rule out only some alternatives Confounds Boundary conditions (generalizability) Good for causal description not explanation

Encouragement Design Manipulation of instructions/messages
Subjects “encouraged” to do certain things Subjects self-select level of condition Holland, P. W. (1988). Causal inference, path analysis, and recursive structural equations models. Sociological Methodology, 18,

Studying and Performance
Students randomly assigned to study amount Test scores as DV Did studying lead to test results? Encouragement led to test results Impact on studying unclear Effect of studying unclear What was cause of test results? Holland, P. W. (1988). Causal inference, path analysis, and recursive structural equations models. Sociological Methodology, 18,

Nonexperimental Research Strategy
Determine covariation Test for time sequence Daily diary Longitudinal design Quasi-experiment Rule out plausible alternatives Based on data Logical

Inus Condition Multiple causes/mechanisms for a phenomenon
The same thing can occur for different reasons Sufficient but unnecessary conditions Multiple motives People do things for different reasons Are social phenomena a hierarchy of inus conditions? Should we expect strong relationships between variables if inus conditions exist?

Inus Conditions For Turnover
Better job offer Bullying Disability Dissatisfaction Poor skill-job match Pursue other interests (e.g., Olympics) Spouse transfer Strategic reasons: Part of plan

Confirmation/Falsification
Observation used to Confirm/support theories Falsify/disconfirm theories Confirmation: All swans are white Must observe all swans in existence Falsification: One black swan Easier to falsify than confirm Null hypothesis testing disconfirmation Based on construct validity Poor measure might falsely falsify

Scientific Skepticism
Science not completely based on objective reality Observations based on theory of construct Construct validity is theoretical interpretation of what numbers represent Theories could be wrong: Biased measurement NA as bias (Watson, Pennebaker, Folger) Social constructions (Salancik & Pfeffer) Science based on trust of methods: Faith Experiments SEM Statistical control (Meehl)

Shadish et al. Skepticisms (p. 30)
“…scientists tend to notice evidence that confirms their preferred hypotheses and to overlook contradictory evidence.” “They make routine cognitive errors of judgment” The react to peer pressures to agree with accepted dogma” They are partly motivated by sociological and economic rewards”

Statistical Control Including controls common practice
Atinc & Simmering review Micro-studies 3.7 controls/study Macro-studies 7.7 controls/study Purification principle Assumption that controls more accurate Implicit causal conclusions

Bias Versus Spuriousness
Third variable that affects observations Bias: Affect on measurement Spuriousness: Affect on construct

C x y x y X Y X Y C Bias Spuriousness

No Bias or Spuriousness
Remove effects of interest before testing Baby with the bathwater

Performance Dimension 1 Performance Dimension 2
Liking

Meehl 1971: High School Yearbooks
What is Meehl’s issue with Schwarz and common practice? Schwarz approach Control SES because it relates to schizophrenia and social participation Does not consider plausible alternatives Schwarz accepts without skepticism (Shadish) Meehl no Automatic Inference Machine

Alternative Mechanisms: NA
Negative Affectivity (NA) Routinely controlled as a bias Based on observed correlations Watson, Pennebaker, Folger Ignores feasible alternative mechanisms Spector, Zapf, Chen, Frese 2000

Mechanism Description Perception See world/job in a negative or positive way Hyper-responsivity Over or under-react emotionally Selection Personality affects selection into better vs. worse jobs Situation creation People create good vs. bad situations for themselves Mood Mood affects all measures in a study including job satisfaction Reverse Causality Experience affects personality

Automatic Inference Machine
Idea that statistics can provide tests for causation There is no such thing as a “causal test” There is no such thing as a “test” for mediation Statistical controls do not provide the “true” relationship between variables Statistics are only numbers: They don’t know where they came from Inference is in the design Inference is in the mind: Logical reasoning

Commonest Methodological Vice
Meehl 1971 Assuming certain variables are fixed and therefore must be causal SES Demographics Personality But these variables can’t be effects. Can they?

Can Job Satisfaction Cause Gender?
Correlation of Gender and satisfaction = group mean differences Satisfaction can’t cause someone’s gender Satisfaction can be the cause of gender distribution of a sample Suppose Females have higher satisfaction than Males Multiple reasons

Alternative Gender-Job Satisfaction Model
Females more likely to quit dissatisfying jobs Dissatisfaction causes gender distribution Gender moderates relation of satisfaction with quitting

... .. . ... .. …. . Satisfaction Males Females

More Alternatives Women less likely to take dissatisfying job (better job decisions) Women less likely to be hired into dissatisfying jobs (protected) Women less likely to be bullied/mistreated Women given more realistic previews (lower expectations) Women more socially skilled at getting what they want at work

How To Use Controls Controls great devices to test hypotheses/theory
Rule in/out plausible alternatives Best based on theory Sequence of tests No automatic/blind use/inference Tests with controls not more conclusive and often less

Control Strategy Test that A and B are related
Salary relates to job satisfaction Confirm/disconfirm control variable Gender relates to both Generate/test alternative explanations for control variable Differential expectations Differential hiring rate Differential job experience Differential turnover rate

Mediation Explanatory variable X  M  Y
Routinely tested in cross-sectional designs Mediation is a causal conclusion

Stone-Romero & Rosopa Advocates experimental designs
Overstates advantages Ignores limitations Confounds Construct invalidity Experimenter effects External validity

Experimental Approach To Mediation
Basic logic Show X  M Show M  Y Therefore X  M  Y Limitations How do you manipulate M without X? How can you be sure that X works through M? Inus conditions

X Z M Y

Example p. 340 Study 1: Goals to Norms Study 2: Norms to Performance
When goals lead to performance was it through norms?

Demonstrate Mediation
Show the chain of events Multiple measurements Show X leads to M leads to Y What does it take to build a mediation case? Difficult to demonstrate causality among 3 variables Show relationship 3-way time sequence Rule out alternatives

Medical Perspective Three types of variables
Correlate: Variable shown to correlate. Proxy risk factor: Predictor that precedes but has no demonstrable effect on outcome. Causal risk factor: Precedes and affects outcome when manipulated or changed. Kraemer, H. C., Stice, E., Kazdin, A., Offord, D., & Kupfer, D. (2001). How do risk factors work together? Mediators, moderators, and independent, overlapping, and proxy risk factors. American Journal of Psychiatry, 158,

Three-step Strategy Step 1: Show relationship exists
Step 2: Show prediction over time Step 3: Manipulate X and show effects on Y

Week 3: Validity, Threats And Qualitative Methods
Interpretation of constructs/results Inference based on purpose Hypothesized causal connections among constructs Nature of constructs Population of interest People Settings Not a property of designs or measures themselves

Four Types of Design Validity
Statistical conclusion validity Appropriate statistical method to make desired inference Internal validity Causal conclusions reasonable based on design Construct validity Interpretation of measures External validity Generalizeability to population of interest

Threats to Validity Statistical Conclusion Validity Internal Validity
Statistics used incorrectly Low power Poor measurement Internal Validity

Eight Threats From Shaddish
Ambiguous Temporal Precedence Selection: Differences in groups History: Events occurring between testings Maturation: Natural change in subjects over time Regression To the Mean Attrition: Especially differential Testing: Affects of repeated testings Instrumention: Change over time or conditions Bolded: Most problematic for organizational research

Threats To Validity 2 Construct Validity External Validity
Inadequate specification of theoretical construct Unreliable measurement Biases Poor content validity External Validity Inadequate specification of population Poor sampling of population Subjects Settings

Points From Shadish et. al
Evaluation of validity based on subjective judgment Scientists conservative about accepting results/conclusions that run counter to belief Scientists market their ideas; try to convince colleagues Design controls preferable to statistical Statistical controls based on assumptions Untrue Untested Untestable

Points From Shadish et al. 2
People in same workplace more similar than people across workplaces When might this be true? Multiple tests Probability compounding assumes independent tests Molar vs. molecular First determine molar effect Then breakdown to determine molecular elements

Qualitative Methods What are qualitative methods
Collection/analysis of written/spoken text Direct observation of behavior Participant observation Ethnography Case study Interview Written materials Existing documents Open-ended questions

Purpose – Erickson 2012 Describes social action Interpretive
What people do in light of their attributions and meanings Interpretive “describing what people in a local setting do in terms that make contact with the meaning perspectives within their actions make sense, from their points of view”, p. 686 Understand cause in local situation Erickson, F. (2012). Comments on causality in qualitative inquiry. Qualitative Inquiry, 18,

Qualitative Research Approach
Accept subjectivity of science Is this an excuse? Less driven by hypothesis Assumption that reality a social construction If no one knows I’ve been shot, am I really dead? Interested in subject’s viewpoint More open-ended More interested in context Less interested in general principles Focus more on interpretation than quantification

Analysis Content Analysis Nonquantitative Interviews Written materials
Open-ended questions Audio or video recordings Quantifying Counts of behaviors/events Categorization of incidents Multiple raters with high agreement Nonquantitative Analysis of case Narrative description Ethnography

Qualitative Organizational Research: Job Stress
Quantitative survey dominates Role ambiguity and conflict dominated in 1980s & 1990s (Katz & Kahn) Dominated by Rizzo et al. weak scales Studies linked RA & RC to potential consequences and moderators Qualitative approach (Keenan & Newton, 1985) Stressful incidents

Keenan & Newton’s SIR Stress Incident Record
Describe event in prior 2 weeks Aroused negative emotion Top stressful events for engineers Time/effort wasted Interpersonal conflict Work underload Work overload Conditions of employment RA & RC rare (1.2% and 4.3%)

Subsequent SIR Research
Comparison of occupations Clerical: Work overload, lack of control Faculty: Interpersonal conflict, time wasters Sales clerks: Interpersonal conflict, time wasters Narayanan, L., Menon, S., & Spector, P. E. (1999). Stress in the workplace: A comparison of gender and occupations. Journal of Organizational Behavior, 20, Informed subsequent quantitative studies Focus on more common stressors Interpersonal conflict Organizational constraints Forget RA & RC

Cross-Cultural SIR Research
Comparison of university support staff India vs. U.S. Stressor India US Overload 0% 25.6% Lack of control 22.6% Lack of structure 26.5% Constraints (Equipment) 15.4% Conflict 16.5% 12.3%

Value of Qualitative Approach
Richer more complete picture Doesn’t reduce complex to few variables More information on context Doesn’t constrain subjects Open ended responses Raw material for hypothesis and theory Can be quantified Can be used to test hypotheses

Limitations of Qualitative Approach
If not quantified Subjective: One person’s perspective Not science without systematic methods that are replicable Quality of material content analyzed Potential biases Fundamental attribution error: Attributing others’ behavior to stable traits. Attribute success internally, failure externally

Research As Craft Scholarly research as expertise not bag of tricks
Logical case Go beyond sheer technique Research not just formulaic/trends Not just using right design, measures, stats Can’t go wrong with Big Five, SEM, Meta-analysis SEM on meta-analytic correlation matrix of Big Five

Meta-Analytic Correlation Matrix Analysis
Do meta-analysis compiling correlations Conduct Regression or SEM on mean correlations Problems: Mean correlations based on different sets of samples Often (usually) moderators affect correlations across studies Populations differ between correlations Results not meaningful

Meta-Analytic Correlation Analyses II
Regression Regression of Y on X1 and X2 Want to conclude that X2 incremental over X1 Can only conclude that X2 in one population incremental over X1 in another Example: Job Satisfaction = Conscientiousness and Autonomy Conclusion: Conscientiousness in police & nurses predicts better than Autonomy in military &teachers

Meta-Analytic Correlation Analyses
Conditions To Use All sample representative from same population All samples have all variables

Developing the Craft Experience Trying different things Reading
Constructs Designs/methods Problems Statistics Reading Reviewing Teaching Thinking/discussing Courses necessary but not sufficient Lifelong learning—you are never done

Developing the Craft 2 Field values novelty and rigor
Don’t be afraid of exploratory research Not much contribution if answer known in advance Look for surprises Don’t be afraid to follow intuition Ask interesting question without a clear answer Focus on interesting variables Good papers tell stories Variables are characters Relationships among variables

Week 4 Construct & External Validity and Method Variance

Constructs Theoretical level Measurement level
Conceptual definitions of variables Basic building blocks of theories Measurement level Operationalizations Based on theory of construct

What We Do With Constructs
Define Operationalize/Measure Establish relations with other constructs Covariation Causation

Construct Validity Case based on weight of evidence
Theory of the construct What is its nature? What does it relate to? Strength based on Adequate definition Adequate operationalization Control for confounding

Steps To Building the Case
Define Construct Operationalize construct: Scale development Construct theory: What it relates to Validation evidence Correlation with other variables Cross-sectional Predictive Known groups Convergent validity Discriminant validity Factorial validity

Points By Shadish et al. Construct confounding O = T + E + Bias
Assessment of unintended constructs SD and NA Race and income Height and gender O = T + E + Bias Bias = Extra unintended stuff

Points by Shadish 2 Mono-operation bias
Not clear on what it is Advocates converging operations Multiple operationalizations What is a different operationalization? Different item formats Different raters Different experimenters Different training programs

Points by Shadish 3 Compensatory equalization: Extra to control group
Compensatory de-equalization: Extra to experimental group FMHI Study Random assignment to FMHI vs. state hospital Staff violated

Construct Validity: Example of Weak Link
Deviance: Violation of norms Theoretical construct weakness Whose norms? Society, organization, workgroup Operationalization weakness List of behaviors with no reference to norms Norms assumed from behavior Retaliation: Response to unfairness Asks behaviors plus motive Retaliation mentioned in instructions

External Validity: Population
Link between sample and theoretical population Define theoretical population Identify critical characteristics Compare sample to population Employed individuals Do students qualify?

External Validity: Setting
Link between current setting and other settings Organization Occupation Identify critical characteristics of settings Compare setting to others Lab to field

External Validity: Treatment/IV
Is IV a reasonable analog? Encouragement design idea Link between current treatment/IV and others Will treatment in study work in nonresearch setting?

External Validity: Outcome/DV
Link between current outcome/DV and others Will results in study work similarly in nonresearch condition? Will different operationalizations of outcome have same result? Supervisor rating of performance vs. objective Safety behavior versus accidents/injuries

Facts Are the Enemy of Truth When Facts Oppose Belief
Gender bias in medical studies (Shadish p. 87) Women are neglected in medical research Treatments not tested on women New grant rules require women Study of 724 studies (Meinart et al.) 55.2% both genders 12.2% males only 11.2% females only 21.4% not specified 355,000 males, 550,000 females

When Politics Attack Science
Evolution IQ and performance Differential validity of IQ tests Others?

Method Variance Method Variance: Variance attributable to the method itself rather than trait Campbell & Fiske 1959 Assumed to be ubiquitous VTotal = VTrait + VError + VMethod

Campbell & Fiske, 1959 “…features characteristic of the method being employed, features which could also be present in efforts to measure other quite different traits.”

Common Method Variance
CMV Mono-method bias Same source bias When method component shared VTotal = VTrait + VError + VMethod Assumes same method has same Vmethod Assumed to inflate correlations Only raised with self-reports

Evidence Vs. Truth -.04 -.10 -.14 .07 -.09 .20 .16 .05 -.03 .02
Truth: CMV = Everything with same method correlated Evidence: Boswell et al. JAP 2004, n = 1601 Leverage seeking Separation seeking -.04 Career satisfaction -.10 -.14 Perceived alternatives .07 -.09 .20 Reward importance .16 .05 -.03 .02

Potential Universal Biases
Truth: Specific biases widespread Evidence Social Desirability Meta-analysis Moorman & Podsakoff Mean r = .05 Highest -.17 role ambiguity; .17 job satisfaction Lowest .01 Performance Lower with employees (.03) than students (.09) Social Desirability Control Study Role clarity-job satisfaction r = .46 to .45 (when SD controlled)

Single-Source Vs. Multi-Source
Truth: Multisource correlations smaller Evidence: Crampton & Wagner Meta-analysis Compared single-source vs. multi-source 26.6% single-source higher (job sat and perform) 62.2% no difference (job sat and turnover) 11.2% multi-source higher (job sat and absence)

Conclusions About CMV CMV is a myth
Variance does not reside in the method No constant inflation of correlations Oversimplification of complex situation If it existed, it would be easy to solve Techniques to fix the problem ineffective and counterproductive Enhance publishability

Cynical Use of CMV Tests
Lindell & Whitney Test Marker variable expected to be unrelated Partial out effect of marker to yield relationship free of CMV Strawman test of something that isn’t there Great for demonstrating to reviewer you had no CMV problem Lindell, M. K., & Whitney, D. J. (2001). Accounting for common method variance in cross-sectional research designs. Journal of Applied Psychology, 86,

If Not CMV CMV myth but bias is not
Bias: Unintended influences on measure Bias a function of construct plus method Unshared bias: Correlation attenuation Shared bias: Correlation inflation No automatic inference machine to eliminate all biases Some known, many likely unknown

Solutions Eliminate term “method variance” Bias or 3rd variables
Construct validity and potential 3rd variables Interpret results cautiously Choose methods to control feasible 3rd variables Alternate sources Not always accurate Converging operations Use strategy of first establishing correlation Rule out 3rd variables in series of steps

Potential Sources of Bias
Difficult to distinguish bias from substantive effect Personality Social desirability NA Culture Response styles Asians avoid strong positives Mood

Potential Sources of Bias II
Cognitive Processes Schema Attribution errors Fundamental attribution error Halo and Leniency Priming Effects of instructions or prior questions Item overlap

Item Overlap Similar content in scales of different constructs Stress
Confound of stressors and strains Stressor: How often do you feel stressed by … Strain: How often do you feel anxious

CWB-OCB Biases Dalal’s three artifacts Antithetical items (overlap)
Agreement vs. frequency Agreement more subject to halo Source: Self versus supervisor Supervisor more subject to halo Dalal, R. S. (2005). A Meta-analysis of the relationship between organizational citizenship behavior and counterproductive work behavior. Journal of Applied Psychology, 90,

CWB – OCB Overlap OCB CWB Does not take extra breaks
Takes undeserved breaks Taken a longer break than you were allowed to take Obeys company rules and regulations Purposely failed to follow instructions Consumes a lot of time complaining about trivial matters Complained about insignificant things at work Conducts personal business on company time I used working time for private affairs

Dalal’s Meta-Analysis
Mean correlations Antitheticals: yes vs. no: vs. -.16 Agreement vs. frequency: vs. -.23 Supervisor vs. self: vs. -.12 Spector et al. isolated effects: Order Antithetical, format, source Spector, P. E., Bauer, J. A., & Fox, S. (2010). Measurement artifacts in the assessment of counterproductive work behavior and organizational citizenship behavior: Do we know what we think we know? Journal of Applied Psychology, 95,

Conway & Lance 2010 Distinguished method variance from effects p. 325
Note: Different methods can share bias, p. 327 Survey versus interview and SD Different sources measure different things, p. 328 Did Heisenberg really say we change things by measuring them?, p. 329 Judge et al. were clear that self-report of job characteristics were perceptions and not objective, p. 329

Conway & Lance 2010 cont. Construct validity evidence does not rule out possible biases, p. 329 Be concerned about item overlap, p. 330

Week 5: Quasi-Experimental Design
Design: Structure of an investigation Number of groups Assignment of subjects Sequence of conditions Sequence of assessments Experimental vs. Nonexperimental

Experimental Design What is an experiment? Simple to Complex
Random assignment Creation of Conditions? Naturally occurring experiment Simple to Complex Number of independent variables Number of dependent variables Sequence of DV assessment

Two-Group Experiment Treatment Group: XT O Control Group: XC O
Comparison of two conditions Hold everything constant that is possible Many ways to design the control group Report effect size Power

Two-Group Design: Analysis
Independent group t-test One-way ANOVA (2 groups) t2 (n1 + n2 – 2) = F (1, n1 + n2 -2) r (with IV dummy coded) r = f(t)

Data Structure 2 group t-test
Treatment Control

Data Structure IV by DV Correlation
IV DV

t vs. r Entered data set into SAS Independent group t-test Correlation
t = 2.31, p = .0497 Correlation r = t-test for significance of r = , p = .0497

Variance Partitioning
Total: All variance among all subjects Between: Variance between groups Variation in means Within: Variance within groups Nested across groups Total = Between + Within Significance: Ratio of B/W Is there more B variance than expected by chance?

One IV: Multiple Groups
More than two groups Categorical IV Different treatments Continuous IV Different levels of treatment Sessions of training ANOVA

Treatment Group Independence
Between Subject Within subject Repeated measures Nested or hierarchical Subjects in teams in conditions Affects analyses ANOVA Multilevel modeling for nested

Between Group Advantages Disadvantages Independence of conditions
No carry over effects Conceptually simple Disadvantages Sample size requirement Limited to one level, usually people

Within Group Advantages Disadvantages
Greater power by controlling error within people Efficiency: One group of subjects Direct comparison of conditions on same sample Disadvantages Contamination/carryover effects of conditions

Nested People in higher social units Departments Classes Organizations
Teams

Nested Advantages Disadvantages Allows study of multiple levels
People and teams Allows study of cross-level effects Disadvantages Requires larger samples N at both levels: n people per group and n of groups Analysis/interpretation more complex

Factorial Designs Multiple IVs Two-Way design ANOVA
Two IVs totally crossed Every combination of conditions Orthogonal: equal sample size Nonorthogonal: Unequal sample size Confounding of IVs Level of A predicts likely level of B Interpretation of IV effects difficult ANOVA

Main Effects and Interactions
Average effect of levels of one IV ignoring the other Interaction Joint effect of both IVs Shows effect of one IV affected by the other

Main Effect B2 DV B1 A1 A2

Interaction B2 DV B1 A1 A2

2x2 Design A Main effect: (A1B1 + A2B1) vs. (A1B2 + A2B2)
Mean B1 A1B1 A2B1 (A1B1+A2B1) / 2 B2 A1B2 A2B2 (A1B2+ A2B2) /2 (A1B1 + A1B2) /2 (A2B1 + A2B2) /2 A Main effect: (A1B1 + A2B1) vs. (A1B2 + A2B2) B Main effect: (A1B1 + A1B2) vs. (A2B1 + A2B2) Interaction: (A1B1 + A2B2) vs. (A2B1 + A1B2)

Larger Designs Any number of IVs
Three-way interaction complex to interpret Four or more way difficult to interpret Takes large number of subjects Not always feasible for experiments

Moderator Tests Another term for interaction
Term used with continuous variables Used with multiple regression Same idea of one variable affecting effects of another. Nonparallel lines Slope = f(moderator)

Mixed Design Between, Within, and Nested Test of CWB-OCB Artifacts
Response format between-S Randomly assigned Antithetical yes vs. no: Within subject Everyone got both Source: Nested Subject versus supervisor nested within dyad Spector et al. 2010

Quasi-experiment Design without random assignment
Comparison of conditions Trained vs. Not trained Researcher created or existing Can characteristics of people be an IV? Gender Personality Is a survey a quasi-experiment? Question about condition

Settings Laboratory vs. field Laboratory Field
Setting in which phenomenon doesn’t naturally occur Field Setting in which phenomenon naturally occurs Classroom field for educational psychologist Classroom lab for us

Lab vs. Field Strengths/Weaknesses
High level of control Easy to do experiments Limits to what can be studied Limited external validity of population/setting Field Limited control Difficult to do experiments Wide range of what can be studied High reliance on self-report High external validity

Design Versus Setting Design: Structure of investigation
Setting: Place in which study occurs Lab ≠ Experiment Field ≠ Nonexperiment

Applied Versus Basic Refers to areas of psychology
Basic: Experimental Areas Cognitive, Neuro, Social?, Developmental? Applied: Clinical, I/O Not useful distinction What is applied? “Applied psychologists” use same methods to address the same questions.

Lab in I/O Research What’s the role of lab in I/O research?
Stone suggests lab is as generalizeable as field. Do you agree? Stone says I/O field biased against lab. Is it? When should we do lab vs. field studies?

Quasi-Experimental Compromise
Quasi-experiments Compromise when true experiment isn’t possible Built in confounds Requires more data than experiment to rule out confounds Inference complex Logic puzzle not cookbook Can’t just assume IV caused DV

Quasi-Experiment and Control
Use of design AND statistical controls “Statistical adjustment only after the best design controls have been used” Shadish, p. 161 Control through comparison groups Control through retesting Pretest-posttest Multiple pretests/posttests Long-term follow-ups Trends Statistical control: 3rd variables & potential confounds

Single Group Designs Posttest only X O When (if ever) is this useful?
Pretest-posttest O X O When is this useful? What are the limitations? Program evaluation

Nonequivalent Groups Design
Preexisting groups assigned treatment vs. control X O O Establishes difference between groups Limited inference

Nonequivalent Groups Design Limitations
What are the main limitations? Groups could have been different initially Interaction of group characteristics and treatment Differential history causing differences

Coping With Preexisting Group Differences
Assess preexisting differences Pretest Assess trends Multiple pretests and posttests Assign multiple groups Random assignment of groups if possible Replicate Additional control groups Matching Statistical adjustment of potential confounds Switching replications—Give treatment to control

Switching Replications
Group 1: X O -- O Group 2: -- O X O Group 2 control for Group 1 Replicate effect with Group 2 Can have more groups Power and replicability

Matching Selecting similar participants from each group
Choose one or more matching variables Assess variables Choose pairs that are close matches Difficult to match on multiple variables Sample size reduction Might bias samples High in one sample low in another Meaning of high/low can vary LOC: Internal Chinese is external New Zealander

Randomly Assign Groups
Identify multiple groups Randomly assign to conditions Groups need to be isolated Contamination of control by treated group Contamination of control by supervisors who know about study Creates potential levels issue Subjects nested in groups

Case-Control Design Compare sample meeting criterion with sample not meeting Must match to same population Employees who quit vs. all other employees Employees who were promoted vs. other employees CEOs vs. line employees Employees assaulted/bullied vs. others Assess other variables to compare

Case-Control Typically we have the case sample at hand
Controls may not be easily accessible Often cases compared to a “normal” population Cancer patients vs. norms for general public Could compare cases in organization with employees in general E.g., absence from case group vs. absence rate in company

Is Case-Control Useful To Us?
What might we use this design to study in Organizations? What is the Case Group? What is the Control Group? What variables do we compare?

Limits To Case-Control Design
Defining groups from same population Effect size uncertain All cases have X Small proportion of people with X are cases Asymmetrical prediction Groups may differ on more than case variable Retrospective assessment of supposed cause Quitting caused report of lower satisfaction

Week 6: Design Issues Experimental Design Randomized experiment
Random assignment Creation of conditions Randomized experiment Time sequence built into design Still must rule out plausible alternatives Construct validity of IV and DV External validity for lab studies Is “real science” so real?

Random Assignment Random sample (external validity)
Random assignment (internal validity) Probability of assignment equal Expected value of characteristics equal Not all variables equal Type 1 errors Faith in random assignment Differential attrition

What Are We Really Assigning To?
Encouragement Design Ask subjects to do certain things What features of complex condition are critical? Confounds in IV

Control Groups No treatment Waiting list Placebo treatment
Currently accepted treatment Comparisons to isolate variables

Bias In Experiments Construct validity of DV and IV
Bias in Assessment of DV Bias/confounding in IV Bias affects Subjects Experimenters Samples Conditions (Contamination and Distortion) Designs Instruments

Humans Used As Instruments
Self vs. other reports Bias in judgments of others Schema & stereotypes Implicit theories Attractiveness Pretty blondes are dumb Physical ability Athletes are dumb Height Tall are better leaders

Demand Characteristics
Implicit meaning of experimental condition IV not accurately perceived Subject motivated to do well Subject tries to figure out experiment Response not natural for situation

Lie Detection Lab Vs. Field
Lab Study Two trials of detection Detect Trial 1, Harder to detect Trial 2 Not detect Trial 1, Easier to detect Trial 2 Opposite to field experience Hypothesized that motive important Want to fool—being detected makes it easier to detect Get caught—being detected makes it harder to detect

Lie Detection Study 2 Percent Detected Trial 2 Motive Detect Trial 1
2 trials x 2 conditions Told intelligent can fool Anxious when caught Told sociopath can fool Relax when caught Percent Detected Trial 2 Motive Detect Trial 1 Not Detect Trial 1 Fool (intelligent) 94% Anxious 19% Relaxed Catch (sociopath) 25% 88%

Subject Expectancies Hawthorne Effects Placebo Effects
Knowledge of being in an experiment Does this really happen? Placebo Effects Blind procedures

Experimenter Effects Observer Errors
Late 1700s Greenwich Observatory Maskelyne fires Kinnebrook for errors Astronomer Bessel: Widespread errors About 1% of observation have errors 75% direction of hypothesis Experimenter expectancy—self fulfilling prophecy Clever Hans Dull/Bright rat study Double blind procedure

Experimenter Behavior
Smiling at subjects 12% at males 70% at females Mixed gender S-E longer to complete Videotape of S-E interactions (Female E) Auditory Visual Male subject Friendly Nonfriendly Female subject

Cross-Sectional Design
All data at once Variables assessed once Most common design in I/O & OB/HR Often done with questionnaires Can establish relationships Cannot rule out most threats Cheap and efficient Good first step

Data Collection Method
Ways of collecting data Self-report questionnaire Formats Interview Degree of structure Observation Behavior checklist vs. rating Open-ended questionnaire

Data Source Incumbent Supervisor Coworker Significant other Observer
Existing materials Job description

Single-Source All data from one source Usually also mono-method
Usually survey Many areas usually self-report Well-being Some area other-report Performance

Multi-Source Same variables from different sources
Convergent validity Confirmation of results Different variables from different sources Rules out some biases and 3rd variables Some biases can be shared Not panacea

Multisource Discriminant Validity
Study Variables Self-Report Other-Report Dalal 2005 meta CWB-OCB -.12 -.60 Spector et al. 2010 -.00 -.42 Spector, Fox 2005 Autonomy-Job Characteristics .54 .67 Glick et al. 86 Job Characteristics-Satisfaction .024 .598 Note: Dalal meta-analysis; Spector, Fox mean correlation across 4 job characteristics; Glick et al. multiple correlation.

Bias Can Affect All Raters
Self-Efficacy—outward signs of confidence Gives impression of effortless performance Coworker perception of employee’s constraints Doesn’t appear to have constraints Supervisor perceptions of performance Looks like a great performer

Constraints Self-report Job Performance Self-report Self-Efficacy Self-report

Constraints Coworker Job Performance Self-report Self-Efficacy Self-report

Constraints Coworker Job Performance Supervisor Self-Efficacy Self-report

Week 7: Longitudinal Designs
Design introducing element of time Same variable measured repeatedly Different variables separated in time Turnover How much time needed to be longitudinal?

Advantages of Longitudinal Design
Can establish relationships Can sometimes establish time sequence Can rule out some plausible alternatives Some biases Occasion factors Mood

Proper Time Sequence Before and after an event
Turnover Precursors assessed prior Job satisfaction Difficult to know when satisfaction occurred Arbitrary points in time not helpful Steady state results same as cross-sectional

Predicting Change Showing that X predicts change in Y
Relation of X & Y controlling for prior levels Weak evidence for causality Regression to mean effects Basement/ceiling effects

Attrition Problem Attrition between time periods Attrition not random
From organization From study Attrition not random Mean change due to attrition Interaction of attrition and variables Those most/least affected quit

Practical Issues Tracking subjects Matching responses
Loss of anonymity Use of secret codes Subject might not remember Anonymous identifiers First street lived on Name of first grade teacher Grandmother’s first name Participation incentives Time to complete study

Pretest-Posttest Design
Single Group O O2 Two Group O1 X O2 O O2

Trends Over Time Single group Time Series O1 O2 O3 X O4 O5 O6
Multigroup Time Series O1 O2 O O4 O5 O6

Discontinuity Change in trend around X Single group Multigroup
Can’t rule out other causes Multigroup Control group to rule out alternatives

Zapf et al. Stress area Relationships small over time Inus conditions
Strains caused by 15 factors Each accounts for 7% of variance .26 correlation if measurement perfect Attrition of least healthy Relationships not always linear Choose appropriate time frame

Longitudinal Multi-Group Design
Identify classification variable Assaulted, Smoking Assess two times Group employees Yes/Yes Yes/No No/Yes No/No Compare groups Manning, M. R., Osland, J. S., & Osland, A. (1989). Work-related consequences of smoking cessation. Academy of Management Journal, 32,

Manning Design Time 1: Smoke Time 1: No Smoke Time 2: Smoke Smokers
Starters Time 2: No Smoke Quitters Nonsmokers

Strain decreased (recovery)
Yang Design Time 1: Assaulted Time 1: Not Assaulted Time 2: Assaulted Constant strain Strain increased Time 2: Not Assaulted Strain decreased (recovery)

Experience Sampling Diary Study Multiple measures on same person
Daily Multiple times per day 1-2 weeks Look at within person variation Changes in DV as a function of IV

Experience Sampling Analysis
Hierarchical linear modeling (HLM) Level 1 within person Level 2 between person Multiple regression DV2 = IV1 + DV1 Time 1 IV on Time 2 DV control for Time 1 DV To see if change in DV is associated with IV

HLM Deals with hierarchical structure of data Experience sampling
Observations nested Individuals in groups, departments, organizations Experience sampling Observations nested in people Separates variance into between versus within Analysis of within person change Relationship of fluctuations of IV vs. DV

Curvilinear Stressor-Strain
Two studies CISMS2: Anglo n = 1470 Spector-Jex, 1991, n =232 Stressors Conflict, Constraints, Role ambiguity, Workload Strains anxiety, anger, depression, frustration, intent, job satisfaction, symptoms

Analyses Curvilinear regression Strain = Stressor + Stressor2
Plot by substituting values of Stressor Similar to plotting moderated regression

Example Y = 10 - 2X + .2X2 X ranges from 0 to 20
Substitute values 5 points apart (0, 5, 10, 15, 20)

Computations X b1X (b1 = -2) X2 b2X2 (b2 = .2) b1X+b2X2 10 +b1X+b2X2 5 -10 25 -5 -20 100 20 15 -30 225 45 -40 400 80 40 50

Results Significance for workload Limited significance for Conflict
Constraints Role ambiguity

Strain CISMS Spector-Jex Direction Anxiety ns * U Frustration -- Intent Job Satisfaction Inverted U Symptoms

Week 8: Field Research and Evaluation
Done in naturalistic settings Experimental Quasi-experimental Observational Evaluation – Organizational Effectiveness Figuring out if things work Organizations Programs Interventions

Challenges To Field Research
Access to organizations/subjects Lack of control Distal contact with subjects (surveys) Who participates Contaminating conditions Participants discussing study Lack of full cooperation Organizational resistance to change

Creative and Varied Approaches

Accessing Subjects Define population needed for your purpose
People Jobs Organizations List likely locations to access populations Consider ways to access locations

Defining Populations: People
Characteristics of people Demographics Age, Education, Gender, Race KSAOs Occupations Do variables of interest vary across occupations? Single or multiple occupations Single controls variety of factors Multiple More variance Tests of occupation differences Greater generalizeability

Defining Populations: Organizations
Characteristics of organizations needed Occupations represented Characteristics of people represented Characteristics of practices Single versus multiple organizations Single adds control Multiple adds Variance Tests of organization characteristics Generalizeability

Accessing Participants: Students
Psychology student subject pool Employed students in classes, e.g., night Advantages/Limitations Easily accessed on many campuses Cheap Cooperative Younger and more educated than average Heterogeneous jobs/organizations Often part-time and temporary jobs Potential work-school conflict

Accessing Participants: Nonstudents
Organizations: Access can be a problem Association mailing lists: Single occupations Web search: Government employees Clubs, churches, nonwork organizations Unions General surveys Phone, mail, door-to-door, street corner

Approaching Organizations
Sell to management Appeal to value of science not ideal What’s in it for them? Partnership Free service: Employee survey, job analysis Address their problem Piggyback your interest

Modes of Approach Personal contact: Networking Consider the audience
Give talks to local managers, e.g., SHRM Students in class Approach based on known need Hospitals and violence Consider the audience Psychologist vs. nonpsychologist HR vs. nonHR Level of sophistication about problem Don’t assume you know more than organization about their problem

Project Prospectus One page nontechnical prospectus
Purpose: Clear and succinct What you need from them What it will cost (e.g., staff time) What’s in it for them What products you will provide to them Timeline

Example Determine factors leading to patient assaults on nurses in hospitals Need to survey 200 nurses with questionnaire Questionnaire will take minutes Can be taken on break or home Will provide report to organization about How many nurses have been assaulted The impact of the assaults on them Factors that might be addressed to reduce the problem Would like to conduct study next month, and provide report within 60 days of completion.

Partnerships Academics and nonacademics
Projects come from mutual interests Piggyback onto organizational project Internship Johannes Rank’s training evaluation Issues Proprietary results Organizational confidentiality

Program Evaluation/Organizational Effectiveness
Education Human Services Determining if program is effective Organizational Effectiveness More generic Determining effectiveness of organization Determining effectiveness of activity/unit

Formative Approach Focus on processes
Often used in developmental approach Can be qualitative Can be quantitative Action research Identify problem, try solution, evaluate, revise

Summative Approach Assess if things work Often quantitative
Experimental or quasi-experimental design Compare to control group/s Utility Return on investment (ROI) Private sector Profitability Cost/outcome (bang for buck) Military—literally Nonmilitary—cost/unit of outcome

Steps In Determining Effectiveness
Define goals/objectives Determine criteria for success Choose design Single group vs. multigroup Pick measures Collect data Analyze/draw conclusion Report/Feedback Program improvement

Week 9 Survey Methods & Constructs
Sampling Cross-cultural challenges Measurement equivalence/invariance Reflective Vs. Formative scales Artifactual constructs

Survey Settings Within employer organization Within other organization
University Professional association Community group Club General population Phone book Door-to-door

Methods Questionnaire Interview Paper-and-pencil E-mail Web
Face-to-face Phone Video-phone ? Instant Message ?

Population Single organization Multiple organizations
Within industry/sector Single occupation Multiple occupations General population Employed students

Sample Versus Population
Survey everyone in population vs. sample Single organization or unit of organization Often survey goes to everyone Multiple organizations Kessler: All psychology faculty Other organization Professional association Often survey everyone General population

Sampling Definitions Population – Aggregate of cases meeting specification All humans All working people All accountants Not always directly measurable Sampling frame – List of all members of a population to be sampled List of all USF support personnel

Sampling Definitions cont.
Stratum – Segment of a population Divided by a characteristic Demographics Male vs. female Job level Manager vs. nonmanager Job title Occupation Department/division of organization

Representativeness of Samples
Sample characteristics match population Non representative Sample characteristics do not match population Some procedures more likely to yield representative samples

Nonprobability Sampling
Nonprobability sample – Every member of population doesn’t have equal chance Representativeness not assured Types Accidental or convenience Snowball Quota – Accidental but stratified Choose half male/female Purposive – Handpick cases that meet criteria Pick full-time employees in a class

Probability Sampling Random sample from defined population
Stratified random sample More efficient than random Cluster or multistage Random selection of aggregates Select organizations stratified by industry

International Research Methods
Cross-cultural vs. cross-national (CC/CN) Purposes Research within a country/culture (emic) Generalize finding/theory Compare countries/cultures (etic) Test culture hypotheses across groups defined by culture CC/CN differences Within country Across countries Across regions North America vs. Latin America

Challenges of CC/CN Research
Equivalence of samples Measurement Equivalence/Invariance (MEI)

Sample Equivalence What is it about samples that causes differences?
Confounding of country with sample characteristics Occupations Can vary across countries Industry sectors Private sector doesn’t exist universally Organization characteristics People characteristics (e.g., demographics) Gender breakdown differs across countries

Instrument Issues Linguistic meaning Calibration
Translation – Back-translation Calibration Numerical equivalence Cultural response tendencies Asian modesty Latin expansiveness Measurement equivalence/invariance Construct validity Factor Structure

Measurement Equivalence/Invariance MEI
Construct Validity Same interpretation across groups SEM and IRT approaches Based on item inter-relationship similarity Factor structure Item characteristic curves

SEM Approach Equality of item variance/covariance
Equal corresponding loadings Form invariance Equal number of factors Same variables load per factor

IRT Approach Equivalent item behavior For unidimensional scales
Better developed for ability tests Often conclusion similar to SEM

Eastern versus Western Control Beliefs at Work Paul E
Eastern versus Western Control Beliefs at Work Paul E. Spector, USF Juan I. Sanchez, Florida International University Oi Ling Siu, Lingnan University, Hong Kong Jesus Salgado, University of Santiago, Spain Jianhong Ma, Zhejiang University, PRC Applied Psychology: An International Review, 2004

Background Cross-cultural study of control beliefs
Americans Vs. Chinese Locus of control beliefs vary Chinese very external vs. Americans Suggests Chinese passive view of world Look to others for direction

Primary Vs. Secondary Control
Primary: Direct control of environment Secondary: Adapt self to environment Predictive: Enhance ability to predict events Illusory: Focusing on chance, i.e., gambling Vicarious: Associate with powerful others Interpretive: Looking for meaning Asians more secondary Rothbaum, Weisz, & Snyder

Socioinstrumental Control
Control through social networks Build social networks Cultivate relationships Juan Sanchez

Purpose Develop new control scales
Secondary control Socioinstrumental control Avoid ethnocentricism by using international item writers

Pilot Study Method Develop definitions of constructs
International team wrote 87 items 1 American, 2 Chinese, 2 Spanish Administered 126 Americans Item analysis

Sample Items Secondary
I take pride in the accomplishments of my superiors at work (Vicarious control) In doing my work, I sometimes consider failure in my work as payment for future success (Interpretative control) Socioinstrumental It is important to cultivate relationships with superiors at work if you want to be effective You can get your own way at work if you learn how to get along with other people

Pilot Study Results Secondary control scale
11 items Alpha = .75 r = -.44 Work LOC Socioinstrumental control 24 items Alpha = .87 r = .26 Work LOC Two scales r = .12 (nonsignificant)

Main Study Method Subjects from HK, PRC, US Work LOC & New Scales
Employed students & university support US from FIU and USF Work LOC & New Scales Stressors Autonomy, conflict, role ambiguity & conflict Strains Job satisfaction, work anxiety, life satisfaction

Coefficient Alphas Scale HK PRC US Secondary .87 .70 .76
Socio-instrumental .91 .88

Mean Differences Variable HK PRC US R2 Second 43.8A 46.0B 45.6B .02
Socio 93.4A 97.1B 91.9A .01 Work LOC 51.0B 57.0C 40.2A .38

Correlations With Work LOC
Variable HK PRC US Secondary .33 -.55 -.21 Socio-instrumental .51 -.59 .23

Significant Correlations
Variable HK PRC US Secondary Job sat, Autonomy Job sat All Socio-instrumental Role conflict Autonomy Work LOC Job sat, Autonomy, Conflict None

Conclusions Procedure created internally consistent scale
Little mean difference China vs. US Work LOC huge mean difference Relationships different across samples

Nature of Indicators Reflective Vs. Formative
Determines meaningful statistics Affects conclusions

Reflective or Effect Indicator
Indicator caused by or reflects underlying construct Change in construct  Change in indicators Classical test theory Measures of attitudes and personality Needs internal consistency Factor analysis meaningful…..sometimes

Formative or Causal Indicator
Indicator defines underlying construct Items don’t reflect single construct Items not interchangeable Change in indicator  Change in construct Examples Socio-economic status Education, Income, Occupation Status Behavior checklist (CWB or OCB) Symptom checklist Internal consistency not always high Factor analysis might not be meaningful

Formative Indicator Example: Personality and CWB
Trait anger and trait anxiety: Spielberger STPI CWB: Counterproductive Work Behavior Checklist N = 78 miscellaneous employees, community Trait anger & CWB r = .37 Trait anxiety & CWB r = .30 Can we assume anger & anxiety relate to all behaviors? Fox, Spector, Miles, 2001, Journal of Vocational Behavior

Item Trait Anger Trait Anxiety Purposely wasted your employer’s materials/supplies .09 .08 Daydreamed rather than did your work .45* .42* Complained about insignificant things at work .52* .34* Purposely did your work incorrectly .07 .11 Came to work late without permission .10 Stayed home from work and said you were sick when you weren’t .20 .13 Purposely damaged a piece of equipment or property .14 .04 Purposely dirtied or littered your place of work -.02 .19 Stolen something belonging to your employer .16 -.08 Took supplies or tools home without permission .05 -.03 Tried to look busy while doing nothing .36* Took money from your employer without permission -.05

How Do You Know? Theoretical interpretation
Are items equivalent forms of construct? Do items correlate? Time sequencing—which changed first? Does increase in SES affect education and income equally? No statistical test exists No automatic inference machine

Artifactual Constructs: Overinterpretation of Factor Analysis
Tendency to assume factors = constructs If items load on different factors they reflect different constructs Sometimes item characteristics are confounded with factors Wording direction

General Assumptions About Item Relationships
Related items reflect same construct Unrelated items reflect different constructs Clusters of items reflect the same construct Factor analysis is magic

Dominance Model Assumptions About Measurement
People agree with items in direction of position If I have a favorable attitude, I will agree with all favorable items People disagree with items opposite to direction of position If I have a favorable attitude, I will disagree with all unfavorable items Responses to oppositely worded items are a mirror image of one another If I moderately agree with positive items, I will moderately disagree with negative items

Ideal Point Principle: Thurstone
Items vary along a continuum. People’s positions vary along a continuum People agree only with items near their position Oppositely worded items not always mirror image Items of same value relate strongly Items of different value relate weakly

Agree Disagree Person Item value on construct continuum

Ideal Point Principle A B C D E Pessimism Optimism

Difficulty Factors Ability tests Items vary in difficulty
Items of same difficulty relate well Those who get 1 easy will tend to get all easy Items of varying difficulty relate less well Those who get hard tend to get easy Those who get easy don’t all get hard

Example People Easy Items Hard Items Low Ability 80% Correct
High Ability 100% Correct

Effects On Statistics Easy items strongly correlated
Hard items strongly correlated Easy items relate modestly to Hard Factor analysis produces factors based on difficulty Difficulty factors reflect item characteristics not people characteristics

Summated Ratings Items of same scale value relate strongly
Items of different value relate modestly Oppositely worded items have different scale values Scatterplots triangular not elliptical High-Low, Low-High, and Low-Low common Few High-High Often distributions are skewed Mixed value items produce factors according to scale value of item Might not reflect underlying constructs

Plot of Moderate Positively Vs
Plot of Moderate Positively Vs. Negatively Worded Job Satisfaction Items

Plot of Extreme Positively Vs. Negatively Worded Job Satisfaction Items

Conclusions Be wary of factors where content is confounded with item direction Be wary when assumption of homoscedasticity is violated Be wary when items are extremely worded More evidence than factor analysis

Week 10 Theory

What Is A Theory? Bernstein Muchinsky Webster
Set of propositions that account for, predict and control phenomena Muchinsky Statement that explains relationships among phenomena Webster General or abstract principles of science Explanation of phenomenon

Types of Theories Deductive Inductive Starts with theory
Data used to support/refute theory Inductive Starts with data Theory explains observations

Deduction Reasoning from premise to case Example
All doctoral students own laptops Chris is a doctoral student Therefore: Chris owns a laptop

Induction Reasoning from observed cases to all cases Example
All doctoral students in my class have laptops Therefore: All doctoral students have laptops

Inference To Best Explanation
Conclusion based on likelihood Example Apartment lock broken and laptop missing Most feasible Must have been a thief Alternative Roommate stole and then faked break-in

What Does Probability Mean?
Frequency .25 chance of coal miners get lung disease 25% of coal miners will get lung disease Subjective Interpretation .01 chances of finding life on Mars Does not mean that 1% of Mars-like planets have life Our confidence in finding life

Induction and Probability
Inductive approach based on chance our inference is correct Inferential statistics Conclusion based on sample data Generalizing to broader population

Covering Law Model of Science
Hempel Covering Law General laws Particular facts Phenomenon Begin with research question Why did my plant die? Plants need water. I didn’t water my plant. Therefore: My plant died.

Advantages Integrates and summarizes large amounts of data
Can help predict Guides research Helps frame good research questions

Disadvantages Biases researchers
“Theory, like mist of eyeglasses, obscures facts” (Charlie Chan in Muchinsky) “Facts are the enemy of truth” (Levine’s boss) A distraction as research does not require theory (Skinner)

Hypothesis Statement of expected relationships among variables
Tentative More limited than a theory Doesn’t deal with process or explanation

Model Representation of a phenomenon
Description of a complex entity or process Webster Boxes and arrows showing causal flow

Observable Vs. Unobservable
Some phenomena directly observable Easily detectable with our senses E.g., some behaviors Some phenomena unobservable Inferred indirectly E.g., internal cognitive states Theories about unobservable are underdetermined Competing theories can explain observations

Theoretical Construct
Abstract representation of a characteristic of people, situation, or thing Building blocks of theories

Paradigm Accepted scientific practice
Rules and standards for scientific practice Law, theory, application and instrumentation that provide models for research. Thomas Kuhn

What Are Our Paradigms? Behaviorism?
Environment-perception-outcome approach Surveys

Structure of Scientific Revolutions Thomas Kuhn
“An apparently arbitrary element, compounded of personal and historical accident, is always a formative ingredient of the beliefs espoused by a given scientific community at a given time.”, p. 4 “research as a strenuous and devoted attempt to force nature into the conceptual boxes supplied by professional education.”, p. 5

History of Theory in Psychology
Behaviorism: Rejection of theory More consistent with natural science Avoid the unobservable Dustbowl empiricism criticism “Cognitive revolution”: Embracing models and theory Unobservables commonly studied Organizational research Theory as paramount The empiricists strike back? Hambrick and Locke

Current State of Theory
Almost required in introductions Marginalize importance of data Ideas more important than facts Scholarship vs. Science Scholarly writing—making good arguments Scientific writing—describing/explaining phenomena based on data

Misuse of Theory Posthoc: Pretending theory drove research
Citing theories as evidence Claiming hypothesis is based on a theory it is not based on Sprinkling cites to irrelevant theories (Sutton & Staw)

Example from Stress Research
Hobfol’s Conservation of Resources Theory People are motivated to acquire and conserve resources Demands on resources and threats to resources are stressful People routinely cite COR theory in support of stressor-strain hypotheses No measure of resources or threat Using a theory to support a hypothesis that does not derive from the theory

Why Do People Do This? Pressure for theory Everyone else is doing it
Descriptive norms Think this is real science Playing the game

Backlash Increasing criticism of the obsession with theory
Hambrick & Locke Harry Barrick: Half-life of models in cognitive AOM sessions One unnamed reviewer Informal interactions

Proper Role of Theory in Science
Goal of science is to understand the world Science is evidence-based not intuition-based Data is the heart of science Theory is current state of understanding how/why things work Theory is the tail not the dog There is a place for both empiricism and theory

Natural Science More focused on data Longer timeframe
Decades and centuries of data before theory “Social science theory a smokescreen to hide weak data” USF chemist

Levels of Explanation Atomic or chemical Neural Individual cognitive
Social Difficult to generalize across levels Deduce properties from one to another Lower level is not more “scientific” or valuable

Social is just applied Cognitive. Cog/Neuro is Just applied Neuroscience. It’s nice to be on top. I/O is just applied Social. Cognitive is just applied Cog/Neuro. I/O Social Cognitive Cog/Neuro Neuroscience Behavioral Genetics

Use Theory Properly Hypotheses: Explicitly derive from a theory
Don’t claim support from a theory Often better to mention theories in the discussion Multiple theories useful in comparative test

Additional Reading Accessible overview of philosophy of science
Okasha, S. (2002). Philosophy of Science. Oxford, UK: Oxford University Press.

Week 11 Levels of Analysis

Level Nature of the sampling unit Person Couple Family Group/Team
Department Organization Industry sector Country

Individual Vs. Higher Level
Most psychological constructs person level Attitude Performance Some constructs higher (aggregate) level Organizational climate Team performance

Types of Aggregates Sum of individuals Consensus of individuals
Sales team performance Consensus of individuals Mean of individuals Majority votes Aggregate level data Job analysis observer ratings for job title Organization profitability Team characteristic (size, gender breakdown) Turnover rates

Aggregate As Sum of Individuals
Sum individual characteristics Ask individuals about own values Sum values Direct assessment of aggregate Ask individuals about people in their unit “How do your team members feel about…?”

Aggregate As Consensus
Shared perceptions Climate People within unit should agree Assess extent of agreement Intraclass correlation, ICC(1) Rwg

Partition Variance Total variation among subjects
Between variation among group means Within variation among subjects within groups Total = Between + Within

Intraclass Correlation ICC(1)
Multiple raters of multiple targets Extent of within rater agreement Var(Total) = Var(Between target) + Var(Within target) ICC = Var(Between) / Var(Total) Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86,

Rwg Compares observed to expected (by chance) variance of ratings
Rwg = (Var(expected) – Var(Obs)) / Var(Expected) Var(Expected) = (A2 – 1)/12 A = Number of rating categories Assumes uniform distribution of responses James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69,

Example 10 Raters: 5, 2, 3, 5, 2, 3, 1 ,4, 3, 4 A = 5 (5 rating choices) Var(Expected) = (52 – 1)/12 = 2 Var(Observed) = 1.73 Rwg = (2 – 1.73) / 2 = .135

Independence of Observations
Independence a statistical assumption Subjects nested in groups Subjects influence one another Observations non-independent Example Effects of supervisory style on OCB Subjects nested in workgroups Style ratings within supervisor nonindependent Shared biases

Confounding of Levels Individual case per unit
One person per organization Nonindependence issue resolved Confounding of individual vs. organization Is relation due to individual or organization? Potential problem for inference Self report of satisfaction and org performance Could be shared bias—happy employee reports greater performance

Ecological Fallacy Drawing inferences from one level to another
When measurement and question don’t match Job satisfaction vs. group morale Individual behavior vs. group behavior Improper inference Can’t draw conclusions across levels Empirically data only reflect own level

If it works for individuals, why won’t it work for groups?
Correlation and Subgroups

Individual: No correlation

Individual: No correlation, Group positive correlation

Individual within group no correlation; Group positive

Individual within group no correlation; Group negative

Individual within group positive correlation: Group none

Individual within group positive correlation; Group negative

Pay and Job Satisfaction
Question 1: Job level Do better paying jobs have more satisfied people? Question 2: Individual level Are better paid within jobs more satisfied?

Pay-Job Satisfaction correlation Mixed jobs r = .17 (Spector, 1985)
Nurses Physicians Salary Pay-Job Satisfaction correlation Mixed jobs r = .17 (Spector, 1985) Single job r = .50 (Rice et al. 1990)

Pooled Within-Group Correlation Remove Effects of Group Differences in Means

Variance/Covariance SS = Σ(X-GMX)2 = Σ(X-GMX)(X-GMX)
SP = Σ(X-GMX)(Y-GMY)

Group Versus Grand Means
. . Group 2 . Grand Mean Group 1 Group 1 Grand Mean Group 2

Pooled Within-Group Correlation Remove Effects of Group Differences in Means
Pooled WG where 1 and 2 refer to groups 1 and 2

Multi-level Studies Lower to higher levels numbered
Multiple teams across multiple organizations in multiple countries Level 1 = person Level 2 = team Level 3 = organization Level 4 = country

Two-Level Most studies two-level
Need multiple units at each level for adequate power Can analyze within level or across Main effects for people and for teams Interaction of level 1 with level 2

Experience Sampling or Daily Diary
Multiple observations on same person Daily Multiple times/day Two-level Level 1: Within person Level 2: Between person Can test if IV fluctuation leads to DV fluctuation within people Can look at DV before and after event

Hierarchical Linear Modeling
Statistical technique Deals with nonindependence Analyze data at two or more levels Individuals in teams Teams in organizations Cross levels Does team-level diversity affect relation between individual satisfaction and OCB? Does individual level satisfaction affect team level performance?

Levels of Behavior Aggregation
How should behaviors be combined?

Overall Index Sum of multiple behaviors
Skarlicki-Folger Retaliation = 17 items Bennett-Robinson Deviance = 24 items Spector-Fox Counterproductive Work Behavior Checklist (CWB-C) = 45 Judge 2006 used Bennett deviance as single score

Distinguish Target Robinson-Bennett 1995
Organization versus person target Bennett-Robinson deviance scale CWB-C Bennett, R. J., & Robinson, S. L. (2000). Development of a measure of workplace deviance. Journal of Applied Psychology, 85, Robinson, S. L., & Bennett, R. J. (1995). A typology of deviant workplace behaviors: A multidimensional scaling study. Academy of Management Journal, 38,

Five Dimensions of CWB-C
Abuse Production Deviance Sabotage Theft Withdrawal Spector, P. E., Fox, S., Penney, L. M., Bruursema, K., Goh, A., & Kessler, S. (2006). The dimensionality of counterproductivity: Are all counterproductive behaviors created equal? Journal of Vocational Behavior, 68,

More Dimensions Abuse Production deviance Sabotage Theft Withdrawal
Physical Verbal Production deviance Loafing Damaging Sabotage Theft From coworker From Organization Withdrawal

Even Finer Grained Abuse Physical Weapon No Weapon Injure vs. not
Slap Punch Injure vs. not Injury needs medical treatment vs. not Reactive vs. proactive

Does It Matter? CWB-C 45 items Job Satisfaction N = 312

Correlation With Job Satisfaction
Total * CWB-organization * CWB-person * Abuse * Withdrawal * Production deviance * Sabotage * Theft Individual Items 40% Significant

-.60 -.24 -.22 -.20 Item r Ran down employer to others
Called in sick when not -.24 Worked slowly on purpose -.22 Insulted someone about their work -.20 Stole from employer -.08 Purposely dirtied or littered workplace -.03 Purposely damaged equipment -.02 Spread rumor

Conclusion Behavior checklist is formative not reflective
Items are not interchangeable High levels of composite do not mean high levels of all components Same score from different sets of behaviors Do not cross levels with inference Assume components have same relation as composite

Week 12 Statistical Inference

Null Hypothesis Signficance Testing
Purposes Reduce subjectivity Agreed upon rules for claims Based on consistent metric Communicates in common language Based on probability

What Does Significance Mean?
Results unlikely due to chance If the null were true, observed results would be expected less that 5% of the time Null hypothesis is a comparison standard There is a zero relationship between variables Alternative: There is a nonzero relationship

Limitations Sensitive to sample size
Small samples low power Large samples small effects can be significant Problem is interpretation not the method itself Does not indicate probability that point estimates are correct. Point estimates have confidence intervals Failure to find significance does not mean null is true

Critics’ Claims Can’t be certain true effect is zero if test is nonsignificant. So what? Promotes dichotomous thinking This is limitation of researcher not statistics Significance tests are misused.

Suggested Alternative
Report confidence intervals and effect sizes Can do whether or not you test significance Report CI and show that it does or does not include zero How is this different from reporting significance?

Benefits of the Controversy
Emphasis on appropriate use of significance testing More complete reporting Effect sizes Confidence intervals

Rule of Thumb Cutoffs Situations where significance tests are unavailable Cannot reduce situations to probabilities Based on opinions of authorities Jum Nunnally Peter Bentler Larry James Past experience Simulations

Become Widely Accepted
Internal consistency (Nunnally) Coefficient alpha (α ≥ .70) Basement for new research Modified from 1st edition (α ≥ .60) SEM fits statistics (Bentler & Bonett) Criterion ≥ .90 < .90: model can be substantially improved Many fit statistics with different properties

Lance, Butts, Michels 2006 Close look at basis for cutoffs
Attributed sources said something more complex Nunnally Bentler & Bonett Attributed source didn’t really say James No specific source Eigenvalue > 1 criterion

Four Criteria Well established See occasionally No longer very common
Coefficient alpha ≥ .70 SEM fit stats ≥ .90 Other criteria sometimes used (e.g., .95) See occasionally RWG ≥ .70 No longer very common Eigenvalue > 1 Sometimes used as a starting point Scree more common

Statistical and Methodological Myths and Urban Legends
SMA 2003 “Three Authors Speak Out On the Peer Review Process” Art Bedeian, Mark Martinko, Paul Spector, Bob Vandenberg Discussion afterwards with Larry Williams AOM 2004 First SMMUL Standing room only SIOP (5 times)

SMMUL Publication Organizational Research Methods
Special feature 2006 2011 Lance & Vandenberg 2009 edited book SMMULs, NY: Routledge

Decimal Dust Bedeian et al. Misplace precision in reporting statistics
Point estimates are not precise, given typical sample sizes Don’t over-report significant digits Correlation: 2 digits Means and SDs: 1 digit right of decimal

Statistical Power Probability of finding a true effect Type II error
Failure to detect a true effect. Four factors One or two tailed Significance level Sample size Effect size

Increase power One-tailed Less stringent p level Greater n Bigger effect size 3 Distributions get tighter 4 Move distribution to left 1, 2 Move rejection line to right

Standard Errors Factors that increase standard errors
Small sample size Multicolinearity Larger correlations among predictors

Regression Coefficients Y = X1 + X2

Regression Coefficients Standard Errors

Sum of Squares of predictor Imprecision of prediction Multicolinearity

Sum of squares of predictor Imprecision of prediction Multicolinearity

Appropriate Comparisons
Conclusions require direct tests Cannot conclude differences based on different p levels Example: Do groups differ on correlations? Group 1: r = .20 significant Group 2: r = .15 nonsignificant N = 100 Z-test = .3593: Nonsignificant Differences of this size likely by chance

Comparing Correlation Magnitude
Draw conclusions about size of relationship based on observed correlations Are correlations of that magnitude significant? Caution in multiple regression Will results replicate?

Do Data Tell Consistent Story? An Example
Cech, E., Rubineau, B., Silbey, S., & Seron, C. (2011). Professional role confidence and gendered persistence in engineering. American Sociological Review, 76, Does role confidence lead to women’s failure to complete an engineering degree?

Prospective Survey Study
Freshman year Major Confidence Expertise: developing skills in college Professional: satisfaction with career Senior year Completed major Transferred to STEM vs. nonSTEM major

Author’s Conclusion “Instead, we find that professional role confidence is significantly associated with engineering persistence, and that its differential distribution between men and women contributes to gender segregation in engineering.”, p. 656

Their Basis for Conclusion
Gender differences in confidence Expertise: Female = 3.09, Male = 3.44 Career-fit: Female = 2.67, Male = 3.01 Both significant (t-test), N: Female = 125, Male = 163 Gender differences in completion Female = 77.2%, Male = 82.5% (significant t-test) Expertise (not professional) confidence significant in regression 11 control variables (including dummy coded college) 6 predictors

Closer Look Correlations in appendix
Correlations with engineering completion Expertise confidence r = .006 (ns) Career-fit confidence r = (ns) Confidence does not predict completion Why significant in regression?

No Direct Gender Comparison
Authors logic Gender difference in confidence Confidence relates to completion Therefore confidence causes gender difference What do we need to draw this conclusion?

Where Do Students Go? Females more likely to switch within STEM
Female = 16.7% vs Male = 6.3% (Significant) Females more likely to graduate in STEM Female = 93.9% vs. Male = 88.8%

Reasonable Conclusion
Males more likely to stick with engineering Women more likely to switch to other STEM disciplines Confidence unrelated to engineering persistence Not clear why men more likely to stick with engineering

Approach To Data Analysis
Clean dataset Check inputting for accuracy Check out of range observations Frequencies Ranges Item analysis Item-remainders Coefficient alpha Means and correlations make sense Compare to other studies

Analysis From Simple To Complex
Descriptive statistics Means, SDs, ranges, frequency distributions Look for restriction of range Does everything look reasonable? Simple relationships Correlations T-tests One-way ANOVAs Study results at length

Complex Statistics Complex only after you have a feel for simple results Run complex sequentially with increasing complexity Compare complex with simple Consistency in conclusions Inconsistencies need to be explained Avoid assuming complex correct Complex and simple test different things Goal is a coherent explanation

Cautions Be aware of assumptions
Be aware of relative power of analyses being compared Don’t overlook methodological artifacts Go beyond cookbook interpretations Don’t let hypotheses/theories blind you Avoid premature conclusions Avoid pure data mining Type 1 error hunt Don’t be afraid to test alternatives

Week 13 Literature Reviews

Narrative Review Summary of research findings Qualitative analysis
“Expert” analysis Based on evidence Room for subjectivity Classical approach

Meta-analysis Quantitative cumulation of findings
Based on common metric Many approaches Many decision rules Room for subjectivity in decision rules

Meta-Analysis  Hunter-Schmidt Approach
There are MANY ways to conduct meta-analysis

Use of Narrative Review
Used almost exclusively before 1990s Psychological Bulletin In depth literature summary Brief overview vs. comprehensive Brief overview part of empirical articles Can contrast very different studies Constructs Designs Measures Small number of studies

Limitations to Narratives
One person’s subjective impression Different reviews – different conclusions Lacks decision rules for drawing conclusions What if half studies are significant? Difficulty with conflicting results Narratives often hard to read Narratives difficult to write

Narrative Review Procedure
Define domain Decide scope (how comprehensive) Inclusion rules Identify/obtain studies Read studies/take notes Organize review Outline of topics Assign studies to topic Write sections Draw conclusions

Meta-Analysis NO AUTOMATIC INFERENCE MACHINE
Does NOT provide absolute truth Does NOT provide population parameters Provides parameter estimates, i.e. statistics Samples not always random or representative Has not revolutionized research Is just another tool that you need

Just Another Tool

Use of Meta-Analysis Dominant procedure today for reviews
Published in most journals Can help settle inconsistencies in conclusions Often descriptive and superficial Allows for hypothesis tests Moderators Requires highly similar studies Constructs Designs Measures

Settling an Inconsistency
Does HONS moderate job characteristics  Outcomes Six reviews—inconsistent conclusions 2 no effect 1 weak effect 2 positive effect 1 no conclusion Settle with meta-analysis Spector, P. E. (1985). Higher-order need strength as a moderator of the job scope-employee outcome relationship: A meta-analysis. Journal of Occupational Psychology, 58,

The Meta-Analysis Literature search Results HONS moderator supported
20 studies Correlations of core characteristics with outcomes Compared high versus low HONS groups Results 8 of 11 cases moderator effect found HONS moderator supported Low power likely contributed to inconsistency

Limitations To Meta-Analysis
Small number of studies meeting criteria Convenience sample of convenience samples Subjectivity of decision rules Inclusion/exclusion rules Statistics used Procedures to gather studies Journals Dissertations Unpublished Different reviewers, different conclusions Sometimes data are made up Need lots of studies

Meta-Analysis Procedure
Define domain and scope Inclusion rules Decide on M-A method Artifact adjustments? Identify/obtain studies Code data from studies Conduct analyses Prepare tables Write paper/interpret results

Define Domain Choose topic Specify domain Define populations
Personality: Big Five vs. Individual traits Define populations Employees vs. Students Define settings Workplace vs. Home Types of studies Group comparisons vs. correlations Define variable operationalizations Self-reports vs. other reports

Apples Vs. Oranges Quantitative estimate of population parameter
What is the population? Mean effect size across samples Assumes sample statistics assess same thing Cumulating results across different constructs not meaningful

Inclusion Rules Operationalizations parallel forms
Measures of NA, neuroticism, emotional stability, trait anxiety All trait measures Samples from same population All full-time working adults Full-time = > 30 hours/week All American samples Designs equivalent All cross-sectional self-report Journal published studies vs. others

Meta-Analysis Method Many to choose from Nature of studies Rosenthal
Group comparisons Correlations Rosenthal Describe distribution of rs Moderators as specific variables to test Hunter-Schmidt Adjust for artifacts Moderators as more variance than expected

Effect Size Estimates Combine effect sizes
Correlation as amount of shared variance Magnitude of mean differences Where d is difference in means in SD units

Identify/Obtain Studies
Electronic databases (PsycINFO) Other reviews Reference lists of papers Conference programs/proceedings Listservs Write authors in area

Coding Choose variables to code Judgments about inclusion rules
How to handle multiple statistics Independent samples Dependent samples: Average Sometimes ratings made, e.g., quality Interrater agreement

Variables To Code Effect sizes N Reliability of measures
Name of measures Sample description Demographics Job types Organization types Country Design

Analysis Meta-analysis software Statistical package Organize results
Excel, SAS, SPSS Organize results Tables by IV or DV Analysis of moderators

Interpret Often descriptive Often superficial Can test hypotheses
Little insight other than mean correlations Nothing new if results have been consistent Often superficial Can test hypotheses Effects of moderators Can inconsistencies be resolved? Suggest new directions or research gaps?

Rosenthal Approach Convert statistics to r Convert r to z
Chi square from 2x2 table Independent group t-test Two-level between group ANOVA Convert r to z Compute descriptive statistics Describe results in tables Meta-analysis as summary of studies

Rosenthal Descriptives
Mean effect size Weighted mean Median Mode Standard deviation Confidence interval

Rosenthal Moderators Identify moderator and relate to effect sizes
Correlate characteristic of study with r Shows if r is a function of moderator

Moderator Example Satisfaction – turnover Unemployment as moderator
Found studies Contacted authors where/when conducted Database of unemployment rates Correlated unemployment to r of study Unemployment –r with satisfaction-turnover Carsten-Spector 1987 Journal of Applied Psychology

Schmidt-Hunter Convert effect sizes to r
Compute descriptive statistics on r Collect artifact data Theoretical variability Unreliability Restriction of range Quality of study Artifact distributions to estimate missing data Adjust observed mean r to estimate rho Compare observed SD to theoretical after adjustments Residual variance = moderators

Estimating Missing Artifacts
Estimate = Make up data “The magic of statistics cannot create information where none exists” Wainer Existing data to guess what missing might have been Hall-Brannick JAP 2000 it is inaccurate Science of what might be rather than what is

Value of Artifact Adjustments
Variability in r is what is/isn’t expected Show that variance due to differential reliability, restriction of range, etc. Requires you have artifact data

Rosenthal Vs. H-S Both identify/code studies
Both compute descriptive statistics r to z transformation Rosenthal yes, H-S no H-S artifact adjustments H-S rho vs. Rosenthal mean r H-S advocate estimating unobservables Rosenthal deals only with observables Begin the same, H-S goes farther Rosenthal similar to H-S bare bones

Why I Prefer Rosenthal Rho is parameter for undefined population
Convenience sample of convenience samples Population = studies that were done/found Unavailable artifact data Uncomfortable in estimating missing data Prefer to deal with observables Don’t believe in automatic inference Lot’s of competing methods

Week 14 Ethics and Research Integrity

Appropriate Research Practices
Conducting Research Treatment of human subjects Treatment of organizational subjects Data Analysis/Interpretation Disseminating Results Publication Peer reviewing

Ethical Codes Appropriate moral behavior/practice Accepted practices
Basic Principle: Do no harm Protect dignity, health, rights, well-being Codes APA AOM

American Psychological Association Code
Largely Practice oriented Five principles Beneficence and Nonmaleficence [Do no harm] Fidelity and Responsibility Integrity Justice Respect for People’s Rights and Dignity Standards and practices Applies to APA members

Preamble Psychologists are committed to increasing scientific and professional knowledge of behavior and people's understanding of themselves and others and to the use of such knowledge to improve the condition of individuals, organizations, and society. Psychologists respect and protect civil and human rights and the central importance of freedom of inquiry and expression in research, teaching, and publication. They strive to help the public in developing informed judgments and choices concerning human behavior. In doing so, they perform many roles, such as researcher, educator, diagnostician, therapist, supervisor, consultant, administrator, social interventionist, and expert witness.

APA Conflict Between Profession and Ethical Principles
Restriction of Advertising Violation of the law Maximization of income for members Tolerance of torture Convoluted statements Other associations manage to avoid such conflicts

Academy of Management Code
Largely academically oriented Three Principles Responsibility Integrity Respect for people’s rights and dignity Responsibility to Students Advancement of managerial knowledge AOM and larger profession Managers and practice of management All people in the world

Professional Principles
Our professional goals are to enhance the learning of students and colleagues and the effectiveness of organizations through our teaching, research, and practice of management.

AOM Vs. APA AOM Consistent principles Simpler
More directly relevant to organizational practice and research No attempt to compromise ethics for profit

Principles Vs. Practice
Principles clear in theory Ethical line not always clear Ethical dilemmas Harm can be done no matter what is done Conflicting interests between parties Employee versus organization Whose rights take priority?

Example: Exploitive Relationships
Principle Psychologists do not exploit persons over whom they have supervisory, evaluative, or other authority What does it mean to exploit? Professor A hires Student B to be an RA How much pay/compensation is exploitive? How many hours/week demanded? What if student gets publication?

Example: Assessing Performance
In academic and supervisory relationships, psychologists establish a timely and specific process for providing feedback to students and supervisees. Not giving an evaluation is unethical? How often? How detailed? What if honest feedback harms the person’s job situation?

Conducting Research Privacy Informed consent Safety Debriefing
Inducements

Privacy Anonymity: Best protection Confidentiality
Procedures to match data without identities Confidentiality Security of identified data Locked computer/cabinet/lab Encoding data Code numbers cross-referenced to names Removing names and identifying information

Informed Consent Subject must know what is involved Purpose
Disclosure of risk Benefits of research Researcher/society Subject Privacy/confidentiality Who has access to data Who has access to identity Right to withdraw Consequences of withdrawal

Safety Minimize exposure to risk Physical and psychological risk
Workplace safety study: Control group Physical and psychological risk

Debriefing Subject right to know Educational experience for students
Written document Presentation Surveys: Provide contact for follow-up Provide results in future upon request

Inducements Pure Volunteer – no inducement Course requirement
Is this coercion? Extra credit Financial payment Is payment coercion?

Institutional Review Board: IRB
Original Purpose: Protection of human subjects Current Purpose: Protection of institution Federal government requirement We pay for government atrocities of the past Government sanctions Bureaucratic Often absurd Designed for invasive medical research

IRB Jurisdiction Institutions receiving federal research funds
All funded research under jurisdiction Cross-country differences Canada like US China doesn’t exist

Types of Review Full Expedited: Research with limited risk Exempt
One year Expedited: Research with limited risk Data from audio/video recordings Research on individual or group characteristics or behavior (including, but not limited to, research on perception, cognition, motivation, identity, language, communication, cultural beliefs or practices, and social behavior) or research employing survey, interview, oral history, focus group, program evaluation, human factors evaluation, or quality assurance methodologies. Exempt Five years

Exempt Project doesn’t get board review Determined by staff member
You can’t determine own exemption Five year Surveys, interviews tests, observations Unless Subjects identified AND potential for harm Legal liability Financial standing Employability Reputation

IRB Impact Best: Minor bureaucratic inconvenience
Protects institution Protects investigator Worst: Chilling effect on research Prevents certain projects Ties up investigators for months

IRB: What Goes Wrong? Inadequate expertise Going beyond authority
Lack of understanding of research Apply medical model to social science Going beyond authority Copyright issues Abuse of power Refuge of the petty and small minded tyrant

Research Vs. Practice Research = Purpose not activity
Dissemination intent = research Presentation Publication Class demo not research Management project not research Consulting projects as research projects Don’t ethics apply to class demos? Not IRB purview

Dealing with Organizations
Who needs protection Employee Organization Who owns and can see the data? Researcher What if organization won’t play by IRB rules? IRB has no jurisdiction off campus

Anticipate Ethical Conflicts
Avoid issues Don’t know can’t tell Negotiate issues Confidentiality Nature of report Ownership of data Procedures

Integrity Issues: Analysis
Honesty in research Report what was done Why Hunter-Schmidt procedures aren’t unethical Estimation procedures transparent Bad data practice Fabrication Deleting disconfirming cases: Trimming Data mining: Type 1 Error hunt Selective reporting: Only the significant findings

Dissemination Authorship credit Referencing Sharing Data
Editorial issues Editor Reviewers

Author Credit Authors: Substantive contributions Order of authorship
What is substantive? People vary in generosity Order of authorship Order of contribution Not by academic rank Dissertation/thesis special case Last for senior person Authorship agreed to up front Potential for student/junior colleague exploitation

Slacker Coauthors When do you drop from coauthorship Late Not at all
Poor quality Less than you expected

Submission One journal at a time One conference at a time
Can submit to conference and journal Prior to paper being in press Almost all are electronic submission Can be difficult and tedious Break paper into multiple documents Enter each coauthor Most reviewing is blind Only editor knows authors/reviewers

Journal Review All 1st submissions are rejects
Don’t want to see again Revise and resubmit (R&R) Will consider revision if you insist (high risk) Encourages resubmission Desk rejections: No review Feedback from 1 to 4 reviewers (mode 2) Feedback from editor Multiple cycles of R&R can be required Can be rejected at any step Tentative accept: Needs minor tweaks Full accept: Congratulations!

Steps To Publication Submit R&R Revise Provisional acceptance
Include response to feedback Provisional acceptance Minor revision Acceptance Copyright release Proofs In print Entire process 1 year or more

R&R More likely accepted than rejected
Depends on editor Good editor has few R&R rejects Work hard to incorporate feedback Argue points of disagreement Additional analyses Prior literature Logical argument Don’t be argumentative Choose your battles Give high priority

Author Role Make good faith effort to revise Incorporate feedback
Be honest in what was done Don’t claim you tried things you didn’t Treat editor/reviewers with respect

Editor Role Be an impartial judge
Weight input from authors and reviewers Be decisive Keep commitments R&R is promise to publish if things fixed Treat everyone with respect

Reviewer Role Objective review No room for politics
Reveal biases to editor Disclose ghost-reviewer to editor E.g., doctoral student Pre-approval Private recommendation to editor Feedback to author/s Keep commitments Treat author with respect

Reviewer As Ghostwriter
Art Bedeian Notes reviewers go too far Dictating question asked, hypotheses, analyses, interpretation Review inflation over the years Sometimes feedback longer than papers Reviewers subjective Poor inter-rater agreement Abuse of power?

Reviewer Problems Reviewers late Reviewers nasty Overly picky
Factually inaccurate Overly dogmatic Favorite stats (CFA/SEM) Edit out ideas they disagree with Insist on own theoretical position Assume there’s only one right way Not knowledgeable Miss obvious Careless

Scientific Progress Through Dispute
Work is based on prior work Testing theories Integrating findings/theories Build a case for an argument or conclusion Disseminate Colleagues build case for alternative Scientific dispute Two camps battle producing progress Dispute motivates work Literature enriched

Crediting Sources Must reference anything borrowed Stealing work
Cite findings/ideas Quote direct passages Little quoting done in psychology Stealing work Plagiarism: not quoting quotes Borrowing ideas Papers People Reviewed papers Easy to forget you didn’t have idea

Strategy For Successful Publication
Choose topic field likes Existing hot topic Tomorrow’s hot topic (hard to predict) Conduct high quality study Craft good story Make a strong case for conclusions Theoretical arguments in introduction Strong data to test Write clearly and concisely Pay attention to current practice Lead don’t follow

Dealing With Journals Be patient and persistent Match paper to journal
Journal interests Quality of paper Count on extensive revision Learn from rejection Consider feedback Only fix things you agree with Look for trends over reviewers

Fragmented Publication
Multiple submission from same project Discouraged in theory Required in practice Single purpose Tight focus Different purposes Minimize overlap Cross-cite Disclose to editor

Example: CISMS Four major papers Unreliability of Hofstede measure
Applied Psychology: An International Review Universality of Work LOC and well-being Academy of Management Journal Country level values and well-being Journal of Organizational Behavior Work-family pressure and well-being Personnel Psychology

How Much Overlap Is Too Much?
A: Aquino, K., Grover, S. L., Bradfield, M., & Allen, D. G. (1999). The effects of negative affectivity, hierarchical status, and self-determination on workplace victimization, Academy of Management Journal, 42, B: Aquino, K. (2000). Structural and individual determinants of workplace victimization: The effects of hierarchical status and conflict management style. Journal of Management, 26,

Purpose from Abstract A: “Conditions under which employees are likely to become targets of coworkers’ aggressive actions” B: “…when employees are more likely to perceive themselves as targets of co-workers’ aggressive actions”

Procedure A:“Two surveys were administered to employees of a public utility as part of an organizational assessment. Although the surveys differed in content, both versions contained an identical set of items measuring workplace victimization” B: “Two different surveys were administered to employees of a public utility as part of an organizational assessment. Although the surveys differed in content, both versions contained an identical set of items measuring workplace victimization.”

Sample/Measures A: n = 371, 76% response, mean age 40.7, tenure 11.5, 65% male, 72% African American B: n = 369, 76% response, mean age 40.7, tenure 11.5, 65% male, 72% African American A: PANAS, Hierarchical status (Haleblian), self-determination, Victimization (14 items) B: Rahim Organizational Conflict Inventory-II, Hierarchical status (Haleblian), Victimization (14 items)

Factor Analysis/Table 1
A: Exploratory FA of Victimization, CFA of 8 items on holdout sample B: Exploratory FA of Victimization, CFA of 8 items on holdout sample A: “Factor loadings and lamdas for victimization itemsa” [Note misspelling of lambda] B: “Factor loadings and lamdas for victimization items1” [Note misspelling of lambda]

Table 3/Hypothesis Tests
A: “Results of hierarchical regression analysis” B: “Results of hierarchical regression analysis” A: “Two regression equations were fitted: one predicting direct victimization and the other predicting indirect victimization.” B: “Two regression equations were fitted: one predicting direct victimization and the other predicting indirect victimization.”

Limitations A: “This study has several limitations that deserve comment. Perhaps the most serious is its cross-sectional research design. The victim precipitation model is based on the assumption that victims either intentionally or unintentionally instigate some negative acts.” B: “This study has several limitations that deserve comment. Perhaps the most serious is its cross-sectional research design. The victim precipitation model is based on the assumption that victims either intentionally or unintentionally instigate some negative acts…”

Research Support: Grants
Funding needed for many studies Expands what can be done Some research very cheap Shoestring because lack of funding? Lack of funding because research is cheap? Universities encourage grants Diminishing state support

Grant Pros Tangible Rewards to investigator Intangible
Covers direct costs of research Equipment/supplies Subject fees/inducements Human resources (research assistants) Rewards to investigator Summer salary Course buyout Conference travel Support students Intangible Forces you to plan study in detail Prestige Administrative admiration (rewards)

Grant Cons Tough to get: Competitive
Time consuming Requires resubmission with long cycle time Administrative burden Takes time from teaching/research Can redirect research focus Not always a bad thing Confuse path with goal Grant is not a research contribution

Sources Federal (Highest status—indirects to university) Foundations
Internal University Grants Small Not as competitive New faculty New investigator and small grants Doctoral students ERC pilot grants SIOP grants Dissertation grants

Federal Grants Challenging
High rejection rate Takes multiple submissions (R&R) Must link to priorities—not everything fundable

Grant Strategy Develop grant writing skill Tie to fundable
Workplace health and safety Musculoskeletal Disorders (MSD) and …. Workplace violence Intervention research in demand Interdisciplinary Use of consultants Pilot studies Programmatic and strategic

Week 15 Wrap-Up

Successful Research Career
Conducting good research Lead don’t follow Visibility Good journals Conferences Other outlets Quantity First authored publications Important more early in career Impact Grants

Programmatic Program of research More conclusive Multiple tests
Boundary conditions More impact through visibility Helps getting jobs Helps with tenure/promotion Can have more than one focus

Conducting Successful Research
Develop an interesting question Based on theory Based on literature Based on observation Based on organization need Link question to literature Theoretical perspective Place in context of what’s been done Multiple types of evidence Consider other disciplines

Conducting Successful Research 2
Design one or more research strategies Lab vs. field Data collection technique Survey, interview, observation, etc. Design Experimental, quasi-experimental or observational Cross-sectional or longitudinal Single-source or multisource Instrumentation Existing or ad hoc

Analysis Hierarchy of methods simple to complex Descriptives Bi-variable relationships Test for controls Complex relationships Multiple regression Factor analysis HLM SEM

Conclusions What’s reasonable based on data Alternative explanations Speculation Theoretical development Suggestions for future

KSAOs Needed Content knowledge Methods expertise Writing skill
Presentation skill Creativity Thick skin

Pipeline Body of work at various stages Set priorities
In press Under review Writing/revising In progress Planning Set priorities Don’t let revisions sit Get work under review Always work on next project Collaboration to multiply productivity Time management

Authorship Order First takes the lead on paper
Most of writing Most input in project Important early in career to be first Balance quantity with order Sometimes most senior person is last PI on project Senior member of lab

Impact Effect of work on field/world Citations Being attacked Sources
ISI Thomson Harzing’s Publish or Perish Others Self-citation Citation studies Individuals (e.g., Podsakoff et al. Journal of Management 2008) Programs (e.g., Oliver et al. TIP, 2005) Being attacked

Partnering State/local government Corporations
Tying research to consulting Partnership with practitioner In kind

Grants Expands what you can do Good for career Current employer
Potential future employers

Grantsmanship Develop grant writing skill Small grant at first
Start as a student Small grant at first Proposal somewhat different from article Background that establishes need for study Demonstrates ability to conduct Expertise of team Letters of agreement/support High likelihood of success Pilot data important Low risk Address funding agency priority

Final Advice Be a leader not a follower
Address problem that is not being addressed Find creative ways of doing things Be evolutionary not revolutionary Too different unlikely to be accepted Most creative often in lesser journals Follow-up studies in better journals Critical mass Need multiple publications on topic to be noticed Programmatic Build on the past, don’t tear it down Positive rather than negative citation

Final Advice cont. Be flexible in thinking Use theory inductively
Don’t get prematurely locked into Conclusion, Idea, Method, Theory Use theory inductively A good theory explains findings Don’t take yourself too seriously Have a thick skin Enjoy your work

Organizational Research Methods Week 2: Causality

Similar presentations

Presentation on theme: "Organizational Research Methods Week 2: Causality"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Organizational Research Methods Week 2: Causality

Similar presentations

Presentation on theme: "Organizational Research Methods Week 2: Causality"— Presentation transcript:

Similar presentations

About project

Feedback