0. Lecture 22: Inference in Graphical Models
Machine Learning
CUNY Graduate Center

1. Today Graphical Models Representing conditional dependence graphically
Inference
Junction Tree Algorithm

2. Undirected Graphical Models
In an undirected graphical model, there is no trigger/response relationship.
Represent slightly different conditional independence relationships
Conditional independence determined by graph separability. D B C A

3. Undirected Graphical Models
Different relationships can be described with directed and undirected graphs.
Cannot represent:

4. Undirected Graphical Models
Different relationships can be described with directed and undirected graphs.

5. Probabilities in Undirected Graphs
Clique: a set of nodes such that there is an edge between every pair of nodes that are members of the set
We will defined the joint probability as a relationship between functions defined over cliques in the graphical model

6. Probabilities in Undirected Graphs
Guanrantees a sum of 1
Potential Functions: positive functions over groups of connected variables (represented by maximal cliques of graphical model nodes)
Maximal cliques: if a clique of nodes A are not a proper subset of a clique B, then A is a maximal clique.

7. Logical Inference
NOT
AND
XOR
In logical inference, nodes are binary, edges represent gates.
AND, OR, XOR, NAND, NOR, NOT, etc.
Inference: given observed variables, predict others
Problems: uncertainty, conflicts, inconsistency

8. Probabilistic Inference
NOT
AND
XOR
Rather than a logic network, use a Bayesian Network
Probabilistic Inference: given observed variables, calculate marginals over others.
Logic networks are generalized by Bayesian Networks
Conditional Probability Table
B=TRUE
B=FALSE
A=TRUE 1
A=FALSE
NOT

9. Probabilistic Inference
AND
NOT
XOR
Rather than a logic network, use a Bayesian Network
Probabilistic Inference: given observed variables, calculate marginals over others.
Logic networks are generalized by Bayesian Networks
B=TRUE
B=FALSE
A=TRUE
0.1
0.9
A=FALSE
NOT-ish

10. Inference in Graphical Models
General Problem: Given a graphical model, for any subsets of observed and expected variables find
Direct approach can be quite inefficient if there are many irrelevant variables

11. Marginal Computation
Graphical models provide efficient storage by decomposing p(x) into conditional probabilities and a simple MLE result.
Now look for efficient calculation of marginals which will lead to efficient inference.

12. Brute Force Marginal Calculation
First approach: have CPTs and graphical model. We can compute arbitrary joints.
Assume 6 variables

13. Computation of Marginals
Pass messages (small tables) around the graph
The messages are small functions that propagate potentials around and undirected graphical model.
The inference technique is the Junction Tree Algorithm

14. Junction Tree Algorithm
Efficient Message Passing for Undirected Graphs.
For Directed Graphs, first convert to undirected.
Goal: Efficient Inference in Graphical Models

15. Junction Tree Algorithm
Moralization
Introduce Evidence
Triangulate
Construct Junction Tree
Propagate Probabilities

16. Moralization Converts a directed graph to an undirected graph.
Moralization “marries” the parents.
Insert an undirected edge between every pair of nodes that have a child in common.
Replace all directed edges with undirected edges.

17. Moralization Examples

18. Moralization Examples

19. Junction Tree Algorithm
Moralization
Introduce Evidence
Triangulate
Construct Junction Tree
Propagate Probabilities

20. Introduce Evidence
Given a moral graph, identify the observed variables.
Reduce probability functions since we know some are fixed.
Only keep probability functions over remaining nodes.

21. Slices .4 .1 .12 .15 Differentiate potential functions from slices
Potential Functions are related to joint probabilities over groups of nodes, but aren’t necessarily correctly normalized, and can even be initialized to conditionals.
A slice of a potential function is a row or column of the underlying table (in the discrete case) or unnormalized marginal (in the continuous case) .4 .1
.12
.15

22. Separation from Introducing Evidence
Observing nodes separates conditionally independent sets of variables
Normalization Calculation. Don’t bother until the end when we want to determine an individual marginal.

23. Junction Trees Construction of junction trees.
Each node represents a clique of variables.
Edges connect cliques
There is a unique path from node to root
Between each clique node is a separator node.
Separators contain intersections of variables
TODO DRAW JUNCTION TREE

24. Triangulation Constructing a junction tree.B C D A E B C D A E
ABD B D BC DE C E CE
Constructing a junction tree.
Need to guarantee that a Junction Graph, made up of cliques and separators of an undirected graph is a Tree.
Eliminate any chordless cycles of four or more nodes.

25. Junction Tree Algorithm
Moralization
Introduce Evidence
Triangulate
Construct Junction Tree
Propagate Probabilities

26. Triangulation
When eliminating cycles there may be many choices about which edge to add.
Want to keep the largest clique size small – small potential functions
Triangulation that minimizes the largest clique size is NP-complete.
Suboptimal triangulation is acceptable (poly-time) and doesn’t introduce many extra dimensions.

27. Triangulation
When eliminating cycles there may be many choices about which edge to add.
Want to keep the largest clique size small – small potential functions
Triangulation that minimizes the largest clique size is NP-complete.
Suboptimal triangulation is acceptable (poly-time) and doesn’t introduce many extra dimensions.

28. Triangulation ExamplesB D A C E A A F

29. Junction Tree Algorithm
Moralization
Introduce Evidence
Triangulate
Construct Junction Tree
Propagate Probabilities

30. Constructing Junction Trees
CDE
BCD
ABD BD CD B C D A E
CDE
BCD
ABD D CD
Junction trees must satisfy the Running Intersection Property:
All nodes on a path between a clique node V and clique node W must include all nodes in V ∩ W
Junction trees will have maximal separator cardinality.

31. Forming a Junction Tree
Given a set of cliques, connect the nodes, s.t. the Running Intersection Property holds.
Maximize the cardinality of the separators.
Maximum Spanning Tree (Kruskal’s algorithm)
Initialize a tree with no edges.
Calculate the size of separators between all pairs
O(N2)
Connect two cliques with the largest separator cardinality without creating a loop.
Repeat until all nodes are connected.

32. Junction Tree Algorithm
Moralization
Introduce Evidence
Triangulate
Construct Junction Tree
Propagate Probabilities

33. Propagating Probabilities
We have a valid junction tree.
What can we do with it?
Probabilities in Junction Trees:
De-absorb smaller cliques from maximal cliques.
Doesn’t change anything, but is a less compact description.

34. Conversion from Directed Graph
Example conversion.
Represent CPTs as potential and separator functions (with a normalizer) X1 X2 X3 X4 X1 X2 X3 X4
X1 X2 X2
X2 X3 X3
X3 X4

35. Junction Tree Algorithm
Goal: Make marginals consistent
Junction Tree Algorithm sends messages between cliques and separators until this consistency is reached.

36. Message passing Send a message from a clique to a separator
The message is what the clique thinks the marginal should be.
Normalize the clique by each message from the separators s.t. agreement is reached
A B B
B C
If they agree, finished. Otherwise, iterate.

37. Message Passing
A B B
B C

38. Junction Tree Algorithm
When convergence is reached – clique potentials are marginals and separator potentials are submarginals.
p(x) is consistent across all of the message passing.
This implies that, so long as p(x) is correctly represented in the potential functions, the JTA can be used to make each potential function correspond to an appropriate marginal without impacting the overall probability function.

39. Converting a DAG to a Junction Tree
X3 X4 X1 X2 X3 X4
X2 X3 X5
X3 X5
X1 X2 X6 X7
X5 X6
X5 X7
Initialize separators to 1 and clique tables to CPTs
Run JTA to convert potential functions (CPTs) to marginals.

40. Evidence in a Junction Tree
Initialize as usual.
Update with a slice rather that the whole table
Conditional

41. Efficiency of the Junction Tree Algorithm
Construct CPTs
Polynomial in # of data points
Moralization
Polynomial in # of nodes
Introduce Evidence
Triangulate
Suboptimal = polynomial. Optimal = NP
Construct Junction Tree
Polynomial in the number of cliques
Identifying cliques = polynomial in the number of nodes
Propagate Probabilities
Exponential in the size of cliques

42. Hidden Markov Models Q1 Q2 Q3 Q4 X1 X2 X3 X4
Powerful graphical model to describe sequential information.