1. Network A/B Testing: From Sampling to Estimation
Ya Xu‡
Joint work with Huan Gui† Anmol Bhasin‡ Jiawei Han†
† University of Illinois at Urbana-Champaign, Urbana
‡ LinkedIn Corporation

2. introduction

3. A/B Testing Uniformly Random Control Treatment
Average Treatment Effect

4. A/B Testing – Two parallel universes
Assumption
Two parallel universes
Parallel Universe 1
(control, )
Real World
(Observations, )
Parallel Universe 2 (treatment, )

5. Network A/B Testing
Interactions between nodes in networks

6. Examples Experiment on feed ranking algorithms
Treatment feed algorithm ranks more relevant items higher
Adam (treatment) clicks on a feed update(X)
X shows up higher for Adam’s friend Ben (control)
Ben (control) clicks on X
Experiment on People You May Know recommendations …

7. Assumption: SUTVA SUTVA (Stable Unit Treatment Value Assumption)
Treatment Assignment Vector
Response function
Each individual’s response is affected only by their own treatment assignments.

8. Network A/B Testing Framework

9. Framework Experimental Design Experimental Analysis
Randomize assignment to minimize interactions
Experimental Analysis
Adjust for network effect post experiment

10. Experimental Design Partition the network/graph
Randomize at cluster level
Minimize the links between clusters
Minimize the interactions between treatment and control
Minimize information leakage
Smaller bias for ATE

11. Balanced Graph Partition
If the cluster sizes are the same for all clusters
No matter what users’ responses are, the covariance is zero, leading to non-biased estimator.
See Middleton and Aronow 2011 for derivation

12. Clustering Real Network
Heterogeneous & large scale (350MM+)
An employee network from LinkedIn
3-net clustering (Ugander et. al.,KDD’13)

13. Randomized Balanced Graph Partition
Random Shuffling on Label Propagation
Randomly initialize clusters (equal size)
Select two nodes and swap their labels if it results in fewer edges between clusters.
Randomly Shuffle x% of labels
Repeat until convergence.
Break local optimal

14. Clustering Results Network Statistics Edges # within each clusters
Nodes #
Edges #
Max Degree
Avg. Degree
7.26e4
2.88e6
3997
39.67
Method LP
RSLP MM
# of edges(1e6)
2.161
2.355
2.359
RSLP can be easily distributed as Label Propagation Algorithm, while achieves comparable performance as Modularity Maximization.

15. Experimental Analysis
Exposure Models
SUTVA
Neighborhood Exposure (Ugander et. al., KDD’13)
Definition: i is neighborhood exposed to treatment if (1) i is in treatment, and (2) At least θ% of i’s neighbors are in treatment
Assumption: i’s response under neighborhood exposure is the same as if everyone receives treatment.

16. Bias-Variance Tradeoff
θ = 0.9
θ = 0.3
About 80% of data points would be invalid
(high variance)
Stronger assumption
Yi(θ= 0.3) = Yi(θ= 1)
(large bias)

17. Fraction Neighborhood Exposure
Users’ responses are determined by
the treatment assignment
the fraction of neighbors having the same treatment assignment.
E.g., Additive Models
can be arbitrary function

18. Example
Additive Model I
ATE

19. Example
Additive Model II
ATE

20. Simulations & real experiments

21. Simulations Real network graph Generation model (Eckles et al. 2014)
Compare bias & variance of five estimators

22. Increasing treatment% Increasing treatment%
Bias
Variance

23. Increasing Network Effect Increasing Network Effect
Bias
Variance

24. Real Online Experiment
Select a country
Apply randomized balanced graph partitioning to assign treatment/control
Apply two Feed ranking algorithms to treatment/control
Estimate ATE using various approaches

25. Real Online Experiment
Picked Netherlands
600 clusters  300/300 in treatment/control
Conducted A/A test to ensure no bias

26. Real Online Experiments
Results
Method
ATE for social gesture
SUTVA
0.168
Network Exposure θ = 0.75
0.264
Network Exposure θ = 0.9
0.520
Hajek. Network Exposure θ = 0.75
0.625
0.133
Fraction Exposure (Additive I)
0.687
Fraction Exposure (Additive II)
0.714

27. Key Takeaways Network effect in A/B Testing
Experimental Design: Balanced Graph Partition
Experimental Analysis: Fraction Neighborhood Exposure Model
Experiments
Simulation
Real Online Experiments
Lots of future work!

29. Percentage of Units in Treatment
The distribution of changes with percentage of units in treatment.
is not representative.

30. Graph Cluster Randomization (Ugander et. al., KDD’13)
Partition the social network
How to cluster? Any constraints?
Randomization on the cluster level
Users in the same cluster receive the same treatment assignment (treatment/control).
Estimate Average Treatment Effect
Any assumptions?