1. Graphical Models - Inference - Wolfram Burgard, Luc De Raedt, Kristian Kersting, Bernhard Nebel Albert-Ludwigs University Freiburg, Germany PCWP CO HRBP HREKG HRSAT ERRCAUTER HR HISTORY CATECHOL SAO2 EXPCO2 ARTCO2 VENTALV VENTLUNG VENITUBE DISCONNECT MINVOLSET VENTMACH KINKEDTUBE INTUBATIONPULMEMBOLUS PAPSHUNT ANAPHYLAXIS MINOVL PVSAT FIO2 PRESS INSUFFANESTHTPR LVFAILURE ERRBLOWOUTPUT STROEVOLUMELVEDVOLUME HYPOVOLEMIA CVP BP Mainly based on F. V. Jensen, „Bayesian Networks and Decision Graphs“, Springer-Verlag New York, 2001. Advanced I WS 06/07 Junction Tree

2. Bayesian Networks Advanced I WS 06/07 Outline Introduction Reminder: Probability theory Basics of Bayesian Networks Modeling Bayesian networks Inference (VE, Junction tree) Excourse: Markov Networks Learning Bayesian networks Relational Models

3. Bayesian Networks Advanced I WS 06/07 Why Junction trees ? Multiple inference, e.g., For each i do variable elimination? No, instead reuse information resp. computations - Inference (Junction Tree)

4. Bayesian Networks Advanced I WS 06/07 A potential f A over a set of variables A is a function that maps each configuration into a non-negative real number. dom f A =A (domain of f A ) Potentials Examples: Conditional probability distribution and joint probability distributions are special cases of potentials - Inference (Junction Tree)

5. Bayesian Networks Advanced I WS 06/07 Ex: A potential  A,B,C over the set of variables {A,B,C}. A has four states, and B and C has three states. domf A,B,C ={A,B,C} Potentials - Inference (Junction Tree)

6. Bayesian Networks Advanced I WS 06/07 Let F be a set of potentials (e.g. CPDs), and let X be a variable. X is eliminated from F: 1.Remove all potentials in F with X in the domain. Let F X that set. 2.Calculate 3.Add to F 4.Iterate VE using Potentials Potential where X is not a member of the domain - Inference (Junction Tree)

7. Bayesian Networks Advanced I WS 06/07 Domain Graph A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A6 Bayesian Network Moralizing Domain Graph (Moral Graph) moral link Let F={f 1,...,f n } be potentials over U ={A 1,...,A m } with dom f i = D i. The domain graph for F is the undirected graph with variables of U as nodes and with a link between pairs of variables being members of the same D i. - Inference (Junction Tree)

8. Bayesian Networks Advanced I WS 06/07 Moralizing A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A6 Bayesian Network Moralizing Domain Graph (Moral Graph) - Inference (Junction Tree)

9. Bayesian Networks Advanced I WS 06/07 Moralizing A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A6 Bayesian Network Moralizing Domain Graph (Moral Graph) - Inference (Junction Tree) CPDs

10. Bayesian Networks Advanced I WS 06/07 Moralizing A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A6 Bayesian Network Moralizing Domain Graph (Moral Graph) - Inference (Junction Tree) CPDs CPD = potential

11. Bayesian Networks Advanced I WS 06/07 Moralizing A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A6 Bayesian Network Moralizing Domain Graph (Moral Graph) - Inference (Junction Tree) CPDs CPD = potential Potentials induce undirected edges

12. Bayesian Networks Advanced I WS 06/07 Moralizing A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A6 Bayesian Network Moralizing Domain Graph (Moral Graph) - Inference (Junction Tree) CPDs CPD = potential moral link Potentials induce undirected edges

13. Bayesian Networks Advanced I WS 06/07 Elimination Sequence A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A6 Bayesian Network Moralizing Moral Graph A1 A4 A2 A5 A6 Eliminating A3 - Inference (Junction Tree)

14. Bayesian Networks Advanced I WS 06/07 Elimination Sequence A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A6 Bayesian Network Moralizing Moral Graph A1 A4 A2 A5 A6 Eliminating A3 - Inference (Junction Tree)

15. Bayesian Networks Advanced I WS 06/07 Reminder: Markov Networks Undirected Graphs Nodes = random variables Cliques = potentials (~ local jpd) VE on potentials works for Markov networks - Inference (Junction Tree)

16. Bayesian Networks Advanced I WS 06/07 Elimination Sequence A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A6 Bayesian Network Moralizing Moral Graph A1 A4 A2 A5 A6 fill-ins: work with a potential over a domain that was not present originally Eliminating A3 GOAL:Elimination sequence that does not introduce fill-ins - Inference (Junction Tree)

17. Bayesian Networks Advanced I WS 06/07 Perfect Elimination Sequence... A1 A3 A4 A2 A5 A6 - Inference (Junction Tree)

18. Bayesian Networks Advanced I WS 06/07 Perfect Elimination Sequence... A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A6A5 - Inference (Junction Tree)

19. Bayesian Networks Advanced I WS 06/07 Perfect Elimination Sequence... A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A6A5 A1 A3 A4 A2 A3 - Inference (Junction Tree)

20. Bayesian Networks Advanced I WS 06/07 Perfect Elimination Sequence... A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A1 A4 A2 A6A5 A1 A3 A4 A2 A3 A1 - Inference (Junction Tree)

21. Bayesian Networks Advanced I WS 06/07 Perfect Elimination Sequence... A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A1 A4 A2 A4 A2 A6A5 A2 A1 A3 A4 A2 A3 A1 - Inference (Junction Tree)

22. Bayesian Networks Advanced I WS 06/07 Perfect Elimination Sequence... A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A1 A4 A2 A4 A2 A4 A6A5 A2 A1 A3 A4 A2 A3 A1 - Inference (Junction Tree)

23. Bayesian Networks Advanced I WS 06/07 Perfect Elimination Sequence... A1 A3 A4 A2 A5 A6... do not introduce fill-ins A1 A3 A4 A2 A5 A1 A4 A2 A4 A2 A4 A6A5 A2 A1 A3 A4 A2 A3 A1 There are several perfect elimination sequences ending in A4 But they have different complexities - Inference (Junction Tree)

24. Bayesian Networks Advanced I WS 06/07 Characterized by the set of domains Set of domains of potentials produced during the elimination (potentials that are subsets of other potentials are removed). A6,A5,A3,A1,A2,A4: {{A6,A3},{A2,A3,A5},{A1,A2,A3},{A1,A2},{A2,A4}} Complexity of Elimination Sequence - Inference (Junction Tree)

25. Bayesian Networks Advanced I WS 06/07 Complexity of Elimination Sequence A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A1 A4 A2 A4 A2 A4 A6A5 A2 A1 A3 A4 A2 A3 A1 - Inference (Junction Tree) {A3,A6} {A2,A3,A5} {A1,A2,A3} {A1,A2} {A2,A4}

26. Bayesian Networks Advanced I WS 06/07 All perfect elimination sequences produce the same domain set, namely the set of cliques of the domain Any perfect elimination sequence ending with A is optimal with respect to computing P(A) Complexity of Elimination Sequence - Inference (Junction Tree)

27. Bayesian Networks Advanced I WS 06/07 Triangulated Graphs and Join Trees An undirected graph with complete elimination sequence is called a triangulated graph A1 A3 A4 A2 A5 A1 A3 A4 A2 A5 triangulated nontriangulated - Inference (Junction Tree)

28. Bayesian Networks Advanced I WS 06/07 Triangulated Graphs and Join Trees A1 A3 A4 A2 A5 Neighbours of A5 A1 A3 A4 A2 A5 Family of A5 Complete neighbour set = simplicial node X is simplicial iff family of X is a clique A1 A3 A4 A2 A5 Family of A3, i.e. A3 is simplicial - Inference (Junction Tree)

29. Bayesian Networks Advanced I WS 06/07 Let G be a triangulated graph, and let X be a simplicial node. Let G´be the graph resulting from eliminating X from G. Then G´is a triangulated graph A1 A3 A4 A2 A5 X Eliminating X A1 A3 A4 A2 A5 Triangulated Graphs and Join Trees - Inference (Junction Tree)

30. Bayesian Networks Advanced I WS 06/07 Triangulated Graphs and Join Trees A triangulated graph with at least two nodes has at least two simplicial nodes In a triangulated graph, each variable A has a perfect elimination sequence ending with A Not all domain graphs are triangulated: An undirected graph is triangulated iff all nodes can be eliminated by successively eliminating a simplicial node - Inference (Junction Tree)

31. Bayesian Networks Advanced I WS 06/07 Join Trees Let G be the set of cliques from an undirected graph, and let the cliques of G be organized in a tree. T is a join tree if for any pair of nodes V, W all nodes on the path between V an W contain the intersection V  W BCDE ABCDDEFI BCDG CHGJ BCDE ABCDDEFI BCDG CHGJ - Inference (Junction Tree)

32. Bayesian Networks Advanced I WS 06/07 Join Trees Let G be the set of cliques from an undirected graph, and let the cliques of G be organized in a tree. T is a join tree if for any pair of nodes V, W all nodes on the path between V an W contain the intersection V  W BCDE ABCDDEFI BCDG CHGJ BCDE ABCDDEFI BCDG CHGJ join tree not a join tree - Inference (Junction Tree)

33. Bayesian Networks Advanced I WS 06/07 Join Trees If the cliques of an undirected graph G can be organized into a join tree, then G is triangulated If the undirected graph is triangulated, then the cliques of G can be organizes into a join tree - Inference (Junction Tree)

34. Bayesian Networks Advanced I WS 06/07 Triangulated, undirected Graph -> Join trees AE F I D J GH B C Simplicial node X Family of X is a clique Eliminate nodes from family of X which have only neighbours in the family of X Give family of X is number according to the number of nodes eliminated Denote the set of remaining nodes S i - Inference (Junction Tree)

35. Bayesian Networks Advanced I WS 06/07 Triangulated, undirected Graph -> Join trees A E F I D J GH B C X ABCD V 1 BCD S 1 separator Simplicial node X Family of X is a clique Eliminate nodes from family of X which have only neighbours in the family of X Give family of X a number i according to the number of nodes eliminated Denote the set of remaining nodes S i - Inference (Junction Tree)

36. Bayesian Networks Advanced I WS 06/07 Simplicial node X Family of X is a clique Eliminate nodes from family of X which have only neighbours in the family of X Give family of X a number i according to the number of nodes eliminated Denote the set of remaining nodes S i Triangulated, undirected Graph -> Join trees A E F I D J GH B C X ABCD V 1 BCD S 1 separator A {A,B,C,D} {B,C,D} 1 S1S1 - Inference (Junction Tree)

37. Bayesian Networks Advanced I WS 06/07 Triangulated, undirected Graph -> Join trees E F I D J GH B C ABCD V 1 BCD S 1 X DEFI V 3 DE S 3 X Simplicial node X Family of X is a clique Eliminate nodes from family of X which have only neighbours in the family of X Give family of X is number according to the number of nodes eliminated Denote the set of remaining nodes S i - Inference (Junction Tree)

38. Bayesian Networks Advanced I WS 06/07 Simplicial node X Family of X is a clique Eliminate nodes from family of X which have only neighbours in the family of X Give family of X is number according to the number of nodes eliminated Denote the set of remaining nodes S i Triangulated, undirected Graph -> Join trees E F I D J GH B C ABCD V 1 BCD S 1 X DEFI V 3 DE S 3 X {D,E,F,I} E {D,E} 3 S3S3 - Inference (Junction Tree)

39. Bayesian Networks Advanced I WS 06/07 Triangulated, undirected Graph -> Join trees ABCD V 1 BCD S 1 DEFI V 3 DE S 3 CGHJ V 5 CG S 5 BCDG V 6 BCD S 6 BCDE V 10 Simplicial node X Family of X is a clique Eliminate nodes from family of X which have only neighbours in the family of X Give family of X is number according to the number of nodes eliminated Denote the set of remaining nodes S i - Inference (Junction Tree)

40. Bayesian Networks Advanced I WS 06/07 Triangulated, undirected Graph -> Join trees Connect each separator S i to a clique V j, j>i, such that S i is a subset of V j ABCD V 1 BCD S 1 DEFI V 3 DE S 3 CGHJ V 5 CG S 5 BCDG V 6 BCD S 6 BCDE V 10 - Inference (Junction Tree)

41. Bayesian Networks Advanced I WS 06/07 Join Tree ABCD V 1 BCD S 1 DEFI V 3 DE S 3 CGHJ V 5 CG S 5 BCDG V 6 BCD S 6 BCDE V 10 - Inference (Junction Tree) The potential representation of a join tree (aka clique tree) is the product of the clique potentials, dived by the product of the separator potentials.

42. Bayesian Networks Advanced I WS 06/07 Yet Another Example A1 A3 A4 A2 A5 A6 A1 A3 A4 A2 A5 A6 A3,A6 V 1 A3 S 1 A2,A3,A5 V 2 A2,A3 S 2 A2,A4 V 6 A2 S 4 A1,A2,A3 V 4 - Inference (Junction Tree)

43. Bayesian Networks Advanced I WS 06/07 Junction Trees Let F be a set of potentials with a triangulated domain graph. A junction tree for F is join tree for G with –Each potential f in F is associated to a clique containing dom( f) –Each link has separator attached containing two mailboxes, one for each direction A1 A3 A4 A2 A5 A6 - Inference (Junction Tree)

44. Bayesian Networks Advanced I WS 06/07 Propagation on a Junction Tree Node V can send exactly one message to a neighbour W, and it may only be sent when V has received a message from all of its other neighbours Choose one clique (arbitrarily) as a root of the tree; collect message to this node and then distribute messages away from it. After collection and distribution phases, we have all we need in each clique to compute potential for variables. - Inference (Junction Tree)

45. Bayesian Networks Advanced I WS 06/07 Junction Trees - Collect Evidence - Inference (Junction Tree) P(A4) ? Find clique containing A4 V6 temporary root Send messages from leaves to root V4 assembles the incoming messages, potential

46. Bayesian Networks Advanced I WS 06/07 Propagation/message passing between two adjacent cliques C 1, C 2 (S 0 is their seperator) –Marginalize C 1 ’s potential to get new potential for S 0 –Update C 2 ’s potential –Update S 0 ’s potential to its new potential Junction Tree - Messages

47. Bayesian Networks Advanced I WS 06/07 Junction Trees - Collect Evidence - Inference (Junction Tree) P(A4) ? Find clique containing A4 V6 temporary root Send messages from leaves to root V4 assembles the incoming messages, potential

48. Bayesian Networks Advanced I WS 06/07 Junction Trees - Collect Evidence - Inference (Junction Tree) P(A4) ? All marginals?

49. Bayesian Networks Advanced I WS 06/07 Junction Trees - Distribute Evidence - Inference (Junction Tree) All marginals?

50. Bayesian Networks Advanced I WS 06/07 Nontriangulated Domain Graphs Embed domain graph in a traingulated graph Use ist junction tree Simple idea: –Eliminate variables in some order –If you wish to eliminate a node with non-complete neighbour set, make it complete by adding fill-ins A E F I D J G H B C A E F I D J G H B C A,C,H,I,J,B,G,D,E,F - Inference (Junction Tree)

51. Bayesian Networks Advanced I WS 06/07 Summary JTA Convert Bayesian network into JT Initialize potentials and separators Incorporate Evidence (set potentials accordingly) Collect and distribute evidence Obtain clique marginals by marginalization/normalization - Inference (Junction Tree) Image from Cecil Huang

52. Bayesian Networks Advanced I WS 06/07 Inference Engines (Commercial) HUGIN : http://www.hugin.com http://www.hugin.com (Commercial) NETICA: http://www.norsys.com http://www.norsys.com Bayesian Network Toolbox for Matlab www.cs.ubc.ca/~murphyk/Software/BNT/bnt.html www.cs.ubc.ca/~murphyk/Software/BNT/bnt.html GENIE/SMILE (JAVA) http://www2.sis.pitt.edu/~genie/ http://www2.sis.pitt.edu/~genie/ MSBNx, Microsoft, http://research.microsoft.com/adapt/MSBNx/ http://research.microsoft.com/adapt/MSBNx/ Profile Toolbox http://www.informatik.uni-freiburg.de/~kersting/profile/ - Inference (Junction Tree)