1. New Directions Integrated framework
Full Bayesian approach that applies to internal and external validity (Tenglong Li)
Value added: for a teacher or school rated as ineffective, how many students would you have to replace with average students to achieve an effective rating? (Qinyun Lin)
Application to Comparative Interrupted Time Series (CITS) – Spiro Maroulis
Mediation (Qinyun Lin)
Non-linear (e.g., logistic)
What is a null hypothesis case?
P(success) is independent of predictor
multilevel regression
Can be thought of as weighted least squares, assumes weights not related to replacement of cases
Propensity score matching
How many bad matches would you have to have to invalidate inference (Rubin liked this idea)

2. Integrated Framework
In case replacement, πi represents the probability of a case being replaced.
In the omitted variables case πi is constant across all i.
The omitted variable = f(yideal -yobserved ) where f is any linear function
In the weighted case, 0< πi < 1, varying across cases.

3. Presented at AEFP, Portland Oregon, March 2018

4. Interaction and replacement cases predicting test score
DATASET ACTIVATE DataSet2.
RECODE g10ctrl1 (1=0) (2=1) (ELSE=SYSMIS) INTO catholic.
VARIABLE LABELS catholic 'dummy, for catholic school'.
EXECUTE.
GRAPH
/SCATTERPLOT(BIVAR)=f1ses WITH f1txcomp BY catholic
/MISSING=LISTWISE.

5. Difference-in-Difference Counterfactual Assumption
Can swing red dashed line and see who much swing we need to invalidate inference
It can swing because of 1) what happens at Ypre 2 or 2) between Ypost and Y pre 2
Sharpens two different confounding conversations
Can compare % bias to invalidate with the std err of pretreatment slopes
SPIRO: try all of this on the JAMA suicide attempt example
What if we treated the trend covariate as a covariate for which there was “balance” – what does it mean when we do that – it’s a comparision to a theoretical experiments that has many, many “states”
We are in world where rvt is low and rvy is high (v is the pre-trend) .
Outcome . .
Tpre 2 . .
Tpre 1
Cpost
I’m only borrowing the slope from the control group to use in the counterfactual. What could make my estimate of the slope be off?
Cpre 2 .
Cpre 1
Time
Pre-treatment
period
Treatment
occurs, t
Post-treatment
period

6. Diffence-in-Difference Counterfactual Assumption (Spiro Maroulis).
Outcome
Tpost
D-i-D Treatment effect . .
Counterfactual: What would have happened to treatment group if they had not actually received the treatment
Tpre 2 . .
Tpre 1
Cpost
Cpre 2 .
Cpre 1
Time
Pre-treatment
period
Treatment
occurs, t
Post-treatment
period

7. Reflection What part if most confusing to you?
Why?
More than one interpretation?
Talk with one other, share
Find new partner and problems and solutions

8. Robustness and Replicability

9. https://osf.io/ezcuj/wiki/home/
Original study effect size versus replication effect size (correlation coefficients).
data
Original study effect size versus replication effect size (correlation coefficients). Diagonal line represents replication effect size equal to original effect size. Dotted line represents replication effect size of 0. Points below the dotted line were effects in the opposite direction of the original. Density plots are separated by significant (blue) and nonsignificant (red) effects.
Open Science Collaboration Science 2015;349:aac4716
Published by AAAS

10. THE BAYESIAN PARADIGM OF ROBUSTNESS INDICES OF CAUSAL INFERENCES Doctoral dissertation by Tenglong Li, Michigan State February 2018
ABSTRACT
The validity of a causal inference hinges on a research design with both strong internal validity and strong external validity (Shadish et al. 2002). Unfortunately, such research is rare so that causality is typically inferred through a small-scale randomized experiment or a large-scale observational study (Schneider et al. 2007). In light of this gap, the robustness indices of causal inferences have been proposed by Frank et al. (2013) to measure the robustness of causal inference by quantifying the proportion of the observed sample that needs to be replaced with unfavorable unobserved cases.
Drawing on the Bayesian discussion in Frank & Min (2007), this dissertation purposes developing the Bayesian framework of the robustness indices of causal inferences for causal research with either limited internal validity or limited external validity. This dissertation has two chapters: The first chapter lays the foundation of the Bayesian paradigm of robustness indices by formally defining prior as distribution built on an unobserved sample. For a particular family of prior and likelihood distributions, the posterior can be interpreted as distribution built on an ideal sample. The Bayesian paradigm of robustness indices of causal inferences focuses on the relationship between the posterior probability of invalidating an inference and the unobserved sample statistics and the central task is to locate the threshold of an unobserved sample statistics with regard to a given value of the posterior probability of invalidating an inference. Considering the first chapter targets the simple group-mean-difference estimator only, the second chapter extends the Bayesian paradigm of robustness indices to regression models. This dissertation promotes the scientific discourse of causality and critical thinking by linking the probability of invalidating an inference to detailed thought experiments characterized by the thresholds of sufficient statistics pertaining to an unobserved sample.

11. 'Crosnoe, Robert' <crosnoe@austin. utexas. edu>; rwerum2@unl
'Crosnoe, Robert' Muller, Chandra L Spiro Maroulis 'Ben Kelcey' 'Minh Duong' 'James Moody' 'William Carbonaro' 'Mark Berends' Min Sun Kaitlin Torphy I-Chien Chen Sarah Galey Jihyun Kim qinyun Lin Aaron Zimmerman Kim Jansen zixi chen John Lane Guan K. Saw Ran Xu Christopher Lee Rachel White Angelo Joseph Garcia Tingqiao Chen Yincheng Ye 刘雨甲 Ruth N. Lopez Turley David Williamson Shaffer Eric Camburn Geoffrey Borman Cavanagh, Shannon E Dedrick, Robert John Lockwood Lou Mariano Joshua Cowen T Gmail Yu Xie Konstantopoulos, Spyros Kimberly S. Maier Schmidt, William Houang, Richard 'Sarah Galey' 'Jihyun Kim' 'qinyun Lin' 'Aaron Zimmerman' 'Kaitlin Obenauf' 'I-Chien Chen' 'Kim Jansen' 'zixi chen' 'John Lane' 'Guan K. Saw' 'Soobin Jang' 'Ran Xu' 'Christopher Lee' 'Rachel White' Angelo Joseph Garcia 'Tingqiao Chen' Youngs, Peter Alexander (pay2n Susanna Loeb Guanglei Hong Christina Prell McCaffrey, Daniel F William Penuel Allison Atteberry Henry, Adam D - (adhenry Adam Gamoran Eric Grodsky 'Jim Spillane' Rob Greenwald Miller, Jason
'Crosnoe, Robert' Muller, Chandra L Spiro Maroulis 'Ben Kelcey' 'Minh Duong' 'James Moody' 'William Carbonaro' 'Mark Berends' Min Sun Kaitlin Torphy I-Chien Chen Sarah Galey Jihyun Kim qinyun Lin Aaron Zimmerman Kim Jansen zixi chen John Lane Guan K. Saw Ran Xu Christopher Lee Rachel White Angelo Joseph Garcia Tingqiao Chen Yincheng Ye 刘雨甲 Ruth N. Lopez Turley David Williamson Shaffer Eric Camburn Geoffrey Borman Cavanagh, Shannon E Dedrick, Robert John Lockwood Lou Mariano Joshua Cowen T Gmail Yu Xie Konstantopoulos, Spyros Kimberly S. Maier Schmidt, William Houang, Richard 'Sarah Galey' 'Jihyun Kim' 'qinyun Lin' 'Aaron Zimmerman' 'Kaitlin Obenauf' 'I-Chien Chen' 'Kim Jansen' 'zixi chen' 'John Lane' 'Guan K. Saw' 'Soobin Jang' 'Ran Xu' 'Christopher Lee' 'Rachel White' Angelo Joseph Garcia 'Tingqiao Chen' Youngs, Peter Alexander (pay2n Susanna Loeb Guanglei Hong Christina Prell McCaffrey, Daniel F William Penuel Allison Atteberry Henry, Adam D - (adhenry Adam Gamoran Eric Grodsky 'Jim Spillane' Rob Greenwald Miller, Jason
'Crosnoe, Robert' Muller, Chandra L Spiro Maroulis 'Ben Kelcey' 'Minh Duong' 'James Moody' 'William Carbonaro' 'Mark Berends' Min Sun Kaitlin Torphy Chen, I-Chien Sarah Galey Jihyun Kim qinyun Lin Aaron Zimmerman Jansen, Kimberly Ann zixi chen Li, Tenglong John Lane Guan K. Saw Martin, Kacy Lynn Ran Xu Christopher Lee Rachel White Angelo Joseph Garcia chentin4 Yincheng Ye chentin4 Chen, I-Chien 刘雨甲 Ruth N. Lopez Turley David Williamson Shaffer Eric Camburn Geoffrey Borman Cavanagh, Shannon E Dedrick, Robert John Lockwood Lou Mariano Cowen, Joshua T Gmail Yu Xie Konstantopoulos, Spyros Maier, Kimberly Schmidt, William Houang, Richard Reckase, Mark 'Sarah Galey' 'Jihyun Kim' 'qinyun Lin' 'Aaron Zimmerman' 'Kaitlin Obenauf' Chen, I-Chien Jansen, Kimberly Ann 'zixi chen' 'John Lane' 'Guan K. Saw' 'Soobin Jang' 'Ran Xu' 'Christopher Lee' 'Rachel White' Angelo Joseph Garcia chentin4 Youngs, Peter Alexander (pay2n Susanna Loeb Guanglei Hong Christina Prell McCaffrey, Daniel F Schneider, Barbara William Penuel Allison Atteberry Henry, Adam D - (adhenry Adam Gamoran Eric Grodsky 'Jim Spillane' Rob Greenwald Miller, Jason 'Allison Atteberry' John Selby Paul Bruno Edward Cremata Strunk, Katharine Plümper, Thomas Chad Hazlett Carlos L. K. Cinelli Judea Pearl