1. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Graphical Models of Probability for Causal Reasoning Thursday 07 November 2002 William H. Hsu Laboratory for Knowledge Discovery in Databases Department of Computing and Information Sciences Kansas State University http://www.kddresearch.org This presentation is: http://www.kddresearch.org/KSU/CIS/Math-20021107.ppt KSU Math Department Colloquium

2. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Overview Graphical Models of Probability –Markov graphs –Bayesian (belief) networks –Causal semantics –Direction-dependent separation (d-separation) property Learning and Reasoning: Problems, Algorithms –Inference: exact and approximate Junction tree – Lauritzen and Spiegelhalter (1988) (Bounded) loop cutset conditioning – Horvitz and Cooper (1989) Variable elimination – Dechter (1996) –Structure learning K2 algorithm – Cooper and Herskovits (1992) Variable ordering problem – Larannaga (1996), Hsu et al. (2002) Probabilistic Reasoning in Machine Learning, Data Mining Current Research and Open Problems

3. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Stages of Data Mining and Knowledge Discovery in Databases Adapted from Fayyad, Piatetsky-Shapiro, and Smyth (1996)

4. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Graphical Models Overview [1]: Bayesian Networks P(20s, Female, Low, Non-Smoker, No-Cancer, Negative, Negative) = P(T) · P(F) · P(L | T) · P(N | T, F) · P(N | L, N) · P(N | N) · P(N | N) Conditional Independence –X is conditionally independent (CI) from Y given Z (sometimes written X  Y | Z) iff P(X | Y, Z) = P(X | Z) for all values of X, Y, and Z –Example: P(Thunder | Rain, Lightning) = P(Thunder | Lightning)  T  R | L Bayesian (Belief) Network –Acyclic directed graph model B = (V, E,  ) representing CI assertions over  –Vertices (nodes) V: denote events (each a random variable) –Edges (arcs, links) E: denote conditional dependencies Markov Condition for BBNs (Chain Rule): Example BBN X1X1 X3X3 X4X4 X5X5 Age Exposure-To-Toxins Smoking Cancer X6X6 Serum Calcium X2X2 Gender X7X7 Lung Tumor

5. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Graphical Models Overview [2]: Markov Blankets and d-Separation Property Z XEY (1) (2) (3) Z Z From S. Russell & P. Norvig (1995) Adapted from J. Schlabach (1996) Motivation: The conditional independence status of nodes within a BBN might change as the availability of evidence E changes. Direction-dependent separation (d- separation) is a technique used to determine conditional independence of nodes as evidence changes. Definition: A set of evidence nodes E d-separates two sets of nodes X and Y if every undirected path from a node in X to a node in Y is blocked given E. A path is blocked if one of three conditions holds:

6. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Graphical Models Overview [3]: Inference Problem Adapted from slides by S. Russell, UC Berkeley http://aima.cs.berkeley.edu/http://aima.cs.berkeley.edu/ Multiply-connected case: exact, approximate inference are #P-complete

7. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Other Topics in Graphical Models [1]: Temporal Probabilistic Reasoning Goal: Estimate Filtering: r = t –Intuition: infer current state from observations –Applications: signal identification –Variation: Viterbi algorithm Prediction: r < t –Intuition: infer future state –Applications: prognostics Smoothing: r > t –Intuition: infer past hidden state –Applications: signal enhancement CF Tasks –Plan recognition by smoothing –Prediction cf. WebCANVAS – Cadez et al. (2000) Adapted from Murphy (2001), Guo (2002)

8. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org General-Case BBN Structure Learning: Use Inference to Compute Scores Optimal Strategy: Bayesian Model Averaging –Assumption: models h  H are mutually exclusive and exhaustive –Combine predictions of models in proportion to marginal likelihood Compute conditional probability of hypothesis h given observed data D i.e., compute expectation over unknown h for unseen cases Let h  structure, parameters   CPTs Posterior ScoreMarginal Likelihood Prior over StructuresLikelihood Prior over Parameters Other Topics in Graphical Models [2]: Learning Structure from Data

9. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Propagation Algorithm in Singly-Connected Bayesian Networks – Pearl (1983) C1C1 C2C2 C3C3 C4C4 C5C5 C6C6 Upward (child-to- parent) messages  ’ (C i ’ ) modified during message-passing phase Downward  messages P ’ (C i ’ ) is computed during  message-passing phase Adapted from Neapolitan (1990), Guo (2000) Multiply-connected case: exact, approximate inference are #P-complete (counting problem is #P-complete iff decision problem is NP-complete)

10. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Inference by Clustering [1]: Graph Operations (Moralization, Triangulation, Maximal Cliques) Adapted from Neapolitan (1990), Guo (2000) A D BE G C H F Bayesian Network (Acyclic Digraph) A D BE G C H F Moralize A1A1 D8D8 B2B2 E3E3 G5G5 C4C4 H7H7 F6F6 Triangulate Clq6 D8D8 C4C4 G5G5 H7H7 C4C4 Clq5 G5G5 F6F6 E3E3 Clq4 G5G5 E3E3 C4C4 Clq3 A1A1 B2B2 Clq1 E3E3 C4C4 B2B2 Clq2 Find Maximal Cliques

11. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Inference by Clustering [2]: Junction Tree – Lauritzen & Spiegelhalter (1988) Input: list of cliques of triangulated, moralized graph G u Output: Tree of cliques Separators nodes S i, Residual nodes R i and potential probability  (Clq i ) for all cliques Algorithm: 1. S i = Clq i  (Clq 1  Clq 2  …  Clq i-1 ) 2. R i = Clq i - S i 3. If i >1 then identify a j < i such that Clq j is a parent of Clq i 4. Assign each node v to a unique clique Clq i that v  c(v)  Clq i 5. Compute  (Clq i ) =  f(v) Clqi = P(v | c(v)) {1 if no v is assigned to Clq i } 6. Store Clq i, R i, S i, and  (Clq i ) at each vertex in the tree of cliques Adapted from Neapolitan (1990), Guo (2000)

12. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Inference by Clustering [3]: Clique-Tree Operations Clq6 D8D8 C4C4G5G5 H7H7 C4C4 Clq5 G5G5 F6F6 E3E3 Clq4 G5G5 E3E3 C4C4 Clq3 A1A1 B2B2 Clq1 E3E3 C4C4 B2B2 Clq2  (Clq5) = P(H|C,G)  (Clq2) = P(D|C) Clq 1 Clq3 = {E,C,G} R3 = {G} S3 = { E,C } Clq1 = {A, B} R1 = {A, B} S1 = {} Clq2 = {B,E,C} R2 = {C,E} S2 = { B } Clq4 = {E, G, F} R4 = {F} S4 = { E,G } Clq5 = {C, G,H} R5 = {H} S5 = { C,G } Clq6 = {C, D} R5 = {D} S5 = { C}  (Clq 1 ) = P(B|A)P(A)  (Clq2) = P(C|B,E)  (Clq3) = 1  (Clq4) = P(E|F)P(G|F)P(F) AB BEC ECG EGF CGH CD B EC CGEG C R i : residual nodes S i : separator nodes  (Clq i ): potential probability of Clique i Clq 2 Clq 3 Clq 4 Clq 5 Clq 6 Adapted from Neapolitan (1990), Guo (2000)

13. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Inference by Loop Cutset Conditioning Split vertex in undirected cycle; condition upon each of its state values Number of network instantiations: Product of arity of nodes in minimal loop cutset Posterior: marginal conditioned upon cutset variable values X3X3 X4X4 X5X5 Exposure-To- Toxins Smoking Cancer X6X6 Serum Calcium X2X2 Gender X7X7 Lung Tumor X 1,1 Age = [0, 10) X 1,2 Age = [10, 20) X 1,10 Age = [100,  ) Deciding Optimal Cutset: NP-hard Current Open Problems –Bounded cutset conditioning: ordering heuristics –Finding randomized algorithms for loop cutset optimization

14. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Inference by Variable Elimination [1]: Intuition Adapted from slides by S. Russell, UC Berkeley http://aima.cs.berkeley.edu/http://aima.cs.berkeley.edu/

15. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Inference by Variable Elimination [2]: Factoring Operations Adapted from slides by S. Russell, UC Berkeley http://aima.cs.berkeley.edu/http://aima.cs.berkeley.edu/

16. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Inference by Variable Elimination [3]: Example A BC F G Season Sprinkler Rain Wet Slippery D Manual Watering P(A|G=1) = ? d = G D F B C A λ G (f) = Σ G=1 P(G|F) P(A), P(B|A), P(C|A), P(D|B,A), P(F|B,C), P(G|F) P(G|F) P(D|B,A) P(F|B,C) P(B|A) P(C|A) P(A) G=1 Adapted from Dechter (1996), Joehanes (2002)

17. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org [2] Representation Evaluator for Learning Problems Genetic Wrapper for Change of Representation and Inductive Bias Control D: Training Data : Inference Specification D train (Inductive Learning) D val (Inference) [1] Genetic Algorithm α Candidate Representation f(α) Representation Fitness Optimized Representation Genetic Algorithms for Parameter Tuning in Bayesian Network Structure Learning

18. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Supervised and Unsupervised Learning: Decision Support in Insurance Pricing Hsu, Welge, Redman, Clutter (2002) Data Mining and Knowledge Discovery, 6(4):361-391

19. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Treatment 1 (Control) Treatment 2 (Pathogen) Messenger RNA (mRNA) Extract 1 Messenger RNA (mRNA) Extract 2 cDNA DNA Hybridization Microarray (under LASER) Adapted from Friedman et al. (2000) http://www.cs.huji.ac.il/labs/compbio/http://www.cs.huji.ac.il/labs/compbio/ Computational Genomics and Microarray Gene Expression Modeling Learning Environment G = (V, E) Specification Fitness (Inferential Loss) B = (V, E,  ) [B] Parameter Estimation G1G1 G2G2 G3G3 G4G4 G5G5 [A] Structure Learning G1G1 G2G2 G3G3 G4G4 G5G5 D val (Model Validation by Inference) D: Data (User, Microarray)

20. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Domain-Specific Repositories Experimental Data Source Codes and Specifications Data Models Ontologies Models Data Entity and Source Code Repository Index for Bioinformatics Experimental Research Personalized Interface Domain-Specific Collaborative Filtering New Queries Learning and Inference Components Historical Use Case & Query Data Decision Support Models Users of Scientific Document Repository Interface(s) to Distributed Repository Example Queries: What experiments have found cell cycle-regulated metabolic pathways in Saccharomyces? What codes and microarray data were used, and why? DESCRIBER: An Experimental Intelligent Filter

21. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Tools for Building Graphical Models Commercial Tools: Ergo, Netica, TETRAD, Hugin Bayes Net Toolbox (BNT) – Murphy (1997-present) –Distribution page http://http.cs.berkeley.edu/~murphyk/Bayes/bnt.html http://http.cs.berkeley.edu/~murphyk/Bayes/bnt.html –Development group http://groups.yahoo.com/group/BayesNetToolbox http://groups.yahoo.com/group/BayesNetToolbox Bayesian Network tools in Java (BNJ) – Hsu et al. (1999-present) –Distribution page http://bndev.sourceforge.net http://bndev.sourceforge.net –Development group http://groups.yahoo.com/group/bndev http://groups.yahoo.com/group/bndev –Current (re)implementation projects for KSU KDD Lab Continuous state: Minka (2002) – Hsu, Guo, Perry, Boddhireddy Formats: XML BNIF (MSBN), Netica – Guo, Hsu Space-efficient DBN inference – Joehanes Bounded cutset conditioning – Chandak

22. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org References [1]: Graphical Models and Inference Algorithms Graphical Models –Bayesian (Belief) Networks tutorial – Murphy (2001) http://www.cs.berkeley.edu/~murphyk/Bayes/bayes.html http://www.cs.berkeley.edu/~murphyk/Bayes/bayes.html –Learning Bayesian Networks – Heckerman (1996, 1999) http://research.microsoft.com/~heckerman http://research.microsoft.com/~heckerman Inference Algorithms –Junction Tree (Join Tree, L-S, Hugin): Lauritzen & Spiegelhalter (1988) http://citeseer.nj.nec.com/huang94inference.html http://citeseer.nj.nec.com/huang94inference.html –(Bounded) Loop Cutset Conditioning: Horvitz & Cooper (1989) http://citeseer.nj.nec.com/shachter94global.html http://citeseer.nj.nec.com/shachter94global.html –Variable Elimination (Bucket Elimination, ElimBel): Dechter (1986) http://citeseer.nj.nec.com/dechter96bucket.html –Recommended Books Neapolitan (1990) – out of print; see Pearl (1988), Jensen (2001) Castillo, Gutierrez, Hadi (1997) Cowell, Dawid, Lauritzen, Spiegelhalter (1999) –Stochastic Approximation http://citeseer.nj.nec.com/cheng00aisbn.htmlhttp://citeseer.nj.nec.com/cheng00aisbn.html

23. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org References [2]: Machine Learning, KDD, and Bioinformatics Machine Learning, Data Mining, and Knowledge Discovery –K-State KDD Lab: literature survey and resource catalog (2002) http://www.kddresearch.org/Resources http://www.kddresearch.org/Resources –Bayesian Network tools in Java (BNJ): Hsu, Guo, Joehanes, Perry, Thornton (2002) http://bndev.sourceforge.net http://bndev.sourceforge.net –Machine Learning in Java (BNJ): Hsu, Louis, Plummer (2002) http://mldev.sourceforge.net http://mldev.sourceforge.net –NCSA Data to Knowledge (D2K): Welge, Redman, Auvil, Tcheng, Hsu http://www.ncsa.uiuc.edu/STI/ALG Bioinformatics –European Bioinformatics Institute Tutorial: Brazma et al. (2001) http://www.ebi.ac.uk/microarray/biology_intro.htm http://www.ebi.ac.uk/microarray/biology_intro.htm –Hebrew University: Friedman, Pe’er, et al. (1999, 2000, 2002) http://www.cs.huji.ac.il/labs/compbio/ http://www.cs.huji.ac.il/labs/compbio/ –K-State BMI Group: literature survey and resource catalog (2002) http://www.kddresearch.org/Groups/Bioinformatics http://www.kddresearch.org/Groups/Bioinformatics

24. Kansas State University Department of Computing and Information Sciences Kansas State University KDD Lab (www.kddresearch.org)www.kddresearch.org Acknowledgements Kansas State University Lab for Knowledge Discovery in Databases –Graduate research assistants: Haipeng Guo (hpguo@cis.ksu.edu), Roby Joehanes (robbyjo@cis.ksu.edu)hpguo@cis.ksu.edurobbyjo@cis.ksu.edu –Other grad students: Prashanth Boddhireddy, Siddharth Chandak, Ben B. Perry, Rengakrishnan Subramanian –Undergraduate programmers: James W. Plummer, Julie A. Thornton Joint Work with –KSU Bioinformatics and Medical Informatics (BMI) group: Sanjoy Das (EECE), Judith L. Roe (Biology), Stephen M. Welch (Agronomy) –KSU Microarray group: Scot Hulbert (Plant Pathology), J. Clare Nelson (Plant Pathology), Jan Leach (Plant Pathology) –Kansas Geological Survey, Kansas Biological Survey, KU EECS Other Research Partners –NCSA Automated Learning Group (Michael Welge, Tom Redman, David Clutter, Lisa Gatzke) –University of Manchester (Carole Goble, Robert Stevens) –The Institute for Genomic Research (John Quackenbush, Alex Saeed) –International Rice Research Institute (Richard Bruskiewich)