1. Causality Workbenchclopinet.com/causality Results of the Causality Challenge Isabelle Guyon, Clopinet Constantin Aliferis and Alexander Statnikov, Vanderbilt Univ. André Elisseeff and Jean-Philippe Pellet, IBM Zürich Gregory F. Cooper, Pittsburg University Peter Spirtes, Carnegie Mellon

2. Causality Workbenchclopinet.com/causality Causal discovery Which actions will have beneficial effects? …your health? …climate changes? … the economy? What affects…

3. Causality Workbenchclopinet.com/causality The system Systemic causality External agent

4. Causality Workbenchclopinet.com/causality Feature Selection X Y Predict Y from features X 1, X 2, … Select most predictive features.

5. Causality Workbenchclopinet.com/causality X Y Causation Predict the consequences of actions: Under “manipulations” by an external agent, some features are no longer predictive. Y

6. Causality Workbenchclopinet.com/causality Challenge Design

7. Causality Workbenchclopinet.com/causality Available data A lot of “observational” data. Correlation  Causality! Experiments are often needed, but: –Costly –Unethical –Infeasible This challenge, semi-artificial data: –Re-simulated data –Real data with artificial “probes”

8. Causality Workbenchclopinet.com/causality Four tasks Toy datasets Challenge datasets

9. Causality Workbenchclopinet.com/causality On-line feed-back

10. Causality Workbenchclopinet.com/causality Difficulties Violated assumptions: –Causal sufficiency –Markov equivalence –Faithfulness –Linearity –“Gaussianity” Overfitting (statistical complexity): –Finite sample size Algorithm efficiency (computational complexity): –Thousands of variables –Tens of thousands of examples

11. Causality Workbenchclopinet.com/causality Evaluation Fulfillment of an objective Prediction of a target variable Predictions under manipulations Causal relationships: Existence Strength Degree

12. Causality Workbenchclopinet.com/causality Setting Predict a target variable (on training and test data). Return the set of features used. Flexibility: –Sorted or unsorted list of features –Single prediction or table of results Complete entry = xxx0, xxx1, xxx2 results (for at least one dataset).

13. Causality Workbenchclopinet.com/causality Metrics Results ranked according to the test set target prediction performance “Tscore”: We also assess directly the feature set with a “Fscore”, not used for ranking.

14. Causality Workbenchclopinet.com/causality Toy Examples

15. Causality Workbenchclopinet.com/causality Lung Cancer SmokingGenetics Coughing Attention Disorder Allergy AnxietyPeer Pressure Yellow Fingers Car Accident Born an Even Day Fatigue LUCAS 0 : natural Causality assessment with manipulations

16. Causality Workbenchclopinet.com/causality LUCAS 1 : manipulated Lung Cancer Smoking Genetics Coughing Attention Disorder Allergy AnxietyPeer Pressure Yellow Fingers Car Accident Born an Even Day Fatigue Causality assessment with manipulations

17. Causality Workbenchclopinet.com/causality Lung Cancer SmokingGenetics Coughing Attention Disorder Allergy AnxietyPeer Pressure Yellow Fingers Car Accident Born an Even Day Fatigue LUCAS 2 : manipulated Causality assessment with manipulations

18. Causality Workbenchclopinet.com/causality Goal driven causality 0 9 4 11 6 1 102 3 7 5 8 We define: V=variables of interest (e.g. MB, direct causes,...) We assess causal relevance: Fscore=f(V,S). 4 11 2 3 1 Participants return: S=selected subset (ordered or not).

19. Causality Workbenchclopinet.com/causality Causality assessment without manipulation?

20. Causality Workbenchclopinet.com/causality Using artificial “probes” Lung Cancer SmokingGenetics Coughing Attention Disorder Allergy AnxietyPeer Pressure Yellow Fingers Car Accident Born an Even Day Fatigue LUCAP 0 : natural Probes P1P1 P2P2 P3P3 PTPT

21. Causality Workbenchclopinet.com/causality Probes Lung Cancer SmokingGenetics Coughing Attention Disorder Allergy AnxietyPeer Pressure Yellow Fingers Car Accident Born an Even Day Fatigue P1P1 P2P2 P3P3 PTPT LUCAP 1&2 : manipulated Using artificial “probes”

22. Causality Workbenchclopinet.com/causality Scoring using “probes” What we can compute (Fscore): –Negative class = probes (here, all “non-causes”, all manipulated). –Positive class = other variables (may include causes and non causes). What we want (Rscore): –Positive class = causes. –Negative class = non-causes. What we get (asymptotically): Fscore = (N TruePos /N Real ) Rscore + 0.5 (N TrueNeg /N Real )

23. Causality Workbenchclopinet.com/causality Results

24. Causality Workbenchclopinet.com/causality Challenge statistics Start: December 15, 2007. End: April 30, 2000 Total duration: 20 weeks. Last (complete) entry ranked: Number of ranked submissions Number of ranked entrants

25. Causality Workbenchclopinet.com/causality Learning curves

26. Causality Workbenchclopinet.com/causality AUC distribution

27. Causality Workbenchclopinet.com/causality REGED

28. Causality Workbenchclopinet.com/causality SIDO

29. Causality Workbenchclopinet.com/causality CINA

30. Causality Workbenchclopinet.com/causality MARTI

31. Causality Workbenchclopinet.com/causality Pairwise comparisons

32. Causality Workbenchclopinet.com/causality Top ranking methods According to the rules of the challenge: –Yin Wen Chang: SVM => best prediction accuracy on REGED and CINA. Prize: $400 donated by Microsoft. –Gavin Cawley: Causal explorer + linear ridge regression ensembles => best prediction accuracy on SIDO and MARTI. Prize: $400 donated by Microsoft. According to pairwise comparisons: –Jianxin Yin and Prof. Zhi Geng’s group: Partial Orientation and Local Structural Learning => best on Pareto front, new original causal discovery algorithm. Prize: free WCCI 2008 registration.

33. Causality Workbenchclopinet.com/causality Pairwise comparisons REGEDSIDO CINA MARTI

34. Causality Workbenchclopinet.com/causality Conclusion We have found good correlation between causation and prediction under manipulations. Several algorithms have demonstrated effectiveness of discovering causal relationships. We still need to investigate what makes then fail in some cases. We need to capitalize on the power of classical feature selection methods.