1. Rick Hoyle Duke Dept. of Psychology & Neuroscience rhoyle@duke.edu
Is Observation Oriented Modeling a Way Forward for the Behavioral Sciences?
Rick Hoyle
Duke Dept. of Psychology & Neuroscience

2. Crisis of confidence in psychological science
2011 – Stapel suspended for fabricating data used in research publications leading to 58 retractions
2011 – Bem’s paper reported findings from 9 experiments seeming to show pre-cognition
2012 – John et al. publish findings from survey or researchers showing that more than half routinely engage in questionable research practices
2011 – Bakker & Wicherts reported widespread statistical reporting errors, some of which, if corrected, would change the inference

3. Are the prevailing research methods capable of yielding an authentic science of psychology?
[I]s significance testing the appropriate tool for evaluating data?
Is the randomized controlled experiment sufficient for determining causality?
Are parametric statistics appropriate for the attributes studied by psychologists?
Grice (2011, p. 2)

4. Problems to be solved
concerns about pervasive Type 1 errors
uncertainty about reproducibility of effects
concerns about validity of measurement
dearth of translatable research outcomes that could inform policy and practice
It’s time for something new!

5. OOM is an appealing alternative to the status quo
more than just an attempt to fix what is broken
shifts the focus
from population focused group-level patterns and comparisons and variable based data modeling
to observation-focused methods
What might a specific person think or feel or do under certain circumstances?
“. . . observation oriented modeling shifts the focus of analysis away from computed aggregates such as means and variances onto the observations themselves.” (Grice, 2011, p. 45)

6. How do we get there from here? things to stop doing
NHST
treating measures as if they were continuous (or even interval scaled)
testing hypotheses using summary statistics
inferring from sample to population

7. How do we get there from here? things to start doing
using techniques that do not make untenable assumptions about samples and measures
addressing generalizability and reproducibility concerns through replication and theory-development rather than sampling and p-values
aligning hypotheses and methods to focus directly on what individuals (rather than populations) do
designing studies that involve intensive repeated assessments

9. The worst-case scenario is clearly dire: It is plausible that inattention to nonergodicity and a lack of group-to-individual generalizability threaten the veracity of countless studies, conclusions, and best- practice recommendations even in the best-case scenario, we should not think of a correlation in group data as an estimate that generalizes to any given individual in the population. Stated bluntly, this implies that the temptation to use aggregate estimates to draw inferences at the basic unit of social and psychological organization—the person—is far less accurate or valid than it may appear in the literature. Fisher et al. (2018)

11. Practical matters
How receptive are journals likely to be to the OOM approach?
How would findings from observation-oriented research be synthesized?
Is there a place for “big data” in the behavioral sciences given the concerns that OOM addresses?
Is it time to abandon classical statistics in behavioral science research?