Inference in Gaussian and Hybrid Bayesian Networks ICS 275B

Gaussian Distribution

gaussian(x,0,1) gaussian(x,1,1) N( ,  )

gaussian(x,0,1) gaussian(x,0,2) N( ,  )

Multivariate Gaussian Definition: Let X 1,…,X n. Be a set of random variables. A multivariate Gaussian distribution over X 1,…,X n is a parameterized by an n-dimensional mean vector  and an n x n positive definitive covariance matrix . It defines a joint density via:

Multivariate Gaussian

Linear Gaussian Distribution Definition: Let Y be a continuous node with continuous parents X 1,…,X k. We say that Y has a linear Gaussian model if it can be described using parameters  0, …,  k and  2 such that: P(y| x 1,…,x k )=N (μ y +  1 x 1 +…,  k x k ;  ) =N([μ y,  1,…,  k ],  )

AB ABAB

Linear Gaussian Network Definition Linear Gaussian Bayesian network is a Bayesian network all of whose variables are continuous and where all of the CPTs are linear Gaussians. Linear Gaussian BN  Multivariate Gaussian =>Linear Gaussian BN has a compact representation

Inference in Continuous Networks AB

Marginalization

Problems: When we Multiply two arbitrary Gaussians! Inverse of K and M is always well defined. However, this inverse is not!

Theoretical explanation: Why this is the case ? Inverse of a matrix of size n x n exists when the matrix is of rank n. If all sigmas and w’s are assumed to be 1. (K -1 +M -1 ) has rank 2 and so is not invertible.

Density vs conditional However,  Theorem: If the product of the gaussians represents a multi-variate gaussian density, then the inverse always exists. For example, For P(A|B)*P(B)=P(A,B) = N(c,C) then inverse of C always exists. P(A,B) is a multi-variate gaussian (density). But P(A|B)*P(B|X)=P(A,B|X) = N(c,C) then inverse of C may not exist. P(A,B|X) is a conditional gaussian.

Inference: A general algorithm Computing marginal of a given variable, say Z. Step 1: Convert all conditional gaussians to canonical form

Inference: A general algorithm Computing marginal of a given variable, say Z. Step 2:  Extend all g’s,h’s and k’s to the same domain by adding 0’s.

Inference: A general algorithm Computing marginal of a given variable, say Z. Step 3: Add all g’s, all h’s and all k’s. Step 4: Let the variables involved in the computation be: P(X1,X2,…,Xk,Z)= N(μ,∑)

Inference: A general algorithm Computing marginal of a given variable, say Z. Step 5: Extract the marginal

Inference: Computing marginal of a given variable For a continuous Gaussian Bayesian Network, inference is polynomial O(N 3 ).  Complexity of matrix inversion So algorithms like belief propagation are not generally used when all variables are Gaussian. Can we do better than N^3?  Use Bucket elimination.

Bucket elimination Algorithm elim-bel (Dechter 1996) Multiplication operator P(a|e=0) W*=4 ”induced width” (max clique size) bucket B: P(a) P(c|a) P(b|a) P(d|b,a) P(e|b,c) bucket C: bucket D: bucket E: bucket A: e=0 B C D E A Marginalization operator

Multiplication Operator Convert all functions to canonical form if necessary. Extend all functions to the same variables (g1,h1,k1)*(g2,h2,k2) =(g1+g2,h1+h2,k1+k2)

Again our problem! Multiplication operator P(a) W*=4 ”induced width” (max clique size) bucket B: P(a) P(c|a) P(b|a) P(d|b,a) P(e|b,c) bucket C: bucket D: bucket E: bucket A: P(e) B C D E A Marginalization operator h(a,d,c,e) does not represent a density and so cannot be computed in our usual form N(μ,σ)

Solution: Marginalize in canonical form Although intermediate functions computed in bucket elimination are conditional, we can marginalize in canonical form, so we can eliminate the problem of non-existence of inverse completely.

Algorithm In each bucket, convert all functions in canonical form if necessary, multiply them and marginalize out the variable in the bucket as shown in the previous slide. Theorem: P(A) is a density and is correct. Complexity: Time and space: O((w+1)^3) where w is the width of the ordering used.

Continuous Node, Discrete Parents Definition: Let X be a continuous node, and let U={U 1,U 2,…,U n } be its discrete parents and Y={Y 1,Y 2,…,Y k } be its continuous parents. We say that X has a conditional linear Gaussian (CLG) CPT if, for every value u  D(U), we have a a set of (k+1) coefficients a u,0, a u,1, …, a u,k+1 and a variance  u 2 such that:

CLG Network Definition: A Bayesian network is called a CLG network if every discrete node has only discrete parents, and every continuous node has a CLG CPT.

Inference in CLGs Can we use the same algorithm?  Yes, but the algorithm is unbounded if we are not careful. Reason:  Marginalizing out discrete variables from any arbitrary function in CLGs is not bounded. If we marginalize out y and k from f(x,y,i,k), the result is a mixture of 4 gaussians instead of 2.  X and y are continuous variables  I and k are discrete binary variables.

Solution: Approximate the mixture of Gaussians by a single gaussian

Multiplication and Marginalization Convert all functions to canonical form if necessary. Extend all functions to the same variables (g1,h1,k1)*(g2,h2,k2) =(g1+g2,h1+h2,k1+k2) Multiplication Strong marginal when marginalizing continuous variables Weak marginal when marginalizing discrete variables

Problem while using this marginalization in bucket elimination Requires computing ∑ and μ which is not possible due to non-existence of inverse. Solution: Use an ordering such that you never have to marginalize out discrete variables from a function that has both discrete and continuous gaussian variables. Special case: Compute marginal at a discrete node Homework: Derive a bucket elimination algorithm for computing marginal of a continuous variable.

Multiplication operator P(a) W*=4 ”induced width” (max clique size) bucket B: P(a) P(c|a) P(b|a,e) P(d|b,a) P(d|b,c) bucket C: bucket D: bucket E: bucket A: P(e) Marginalization operator Special Case: A marginal on a discrete variable in a CLG is to be computed. B,C and D are continuous variables and A and E is discrete

Complexity of the special case Discrete-width (wd): Maximum number of discrete variables in a clique Continuous-width (wc): Maximum number of continuous variables in a clique Time: O(exp(wd)+wc^3) Space: O(exp(wd)+wc^3)

Algorithm for the general case:Computing Belief at a continuous node of a CLG Convert all functions to canonical form. Create a special tree-decomposition Assign functions to appropriate cliques (Same as assigning functions to buckets) Select a Strong Root Perform message passing

Creating a Special-tree decomposition Moralize the Bayesian Network. Select an ordering such that all continuous variables are ordered before discrete variables (Increases induced width).

Elimination order w y x z Strong elimination order: First eliminate continuous variables Eliminate discrete variable when no available continuous variables Moralized graph has this edge W and X are discrete variables and Y and Z are continuous.

Elimination order (1) w y x z dim: 2 1

Elimination order (2) w y x z dim: 2 2 1

Elimination order (3) w y x z 3dim: 2 2 1

Elimination order (4) w y x z w y z w y x 34 2 w y 3 2 Cliques 1 Cliques 2 separator

Bucket tree or Junction tree (1) w y z w y x w y Cliques 1 Cliques 2: root separator

Algorithm for the general case:Computing Belief at a continuous node of a CLG Convert all functions to canonical form. Create a special tree-decomposition Assign functions to appropriate cliques (Same as assigning functions to buckets) Select a Strong Root Perform message passing

Assigning Functions to cliques Select a function and place it in an arbitrary clique that mentions all variables in the function.

Algorithm for the general case:Computing Belief at a continuous node of a CLG Convert all functions to canonical form. Create a special tree-decomposition Assign functions to appropriate cliques (Same as assigning functions to buckets) Select a Strong Root Perform message passing

Strong Root We define a strong root as any node R in the bucket-tree which satisfies the following property: for any pair (V,W) which are neighbors on the tree with W closer to R than V, we have

Example Strong root Strong Root

Algorithm for the general case:Computing Belief at a continuous node of a CLG Create a special tree-decomposition Assign functions to appropriate cliques (Same as assigning functions to buckets) Select a Strong Root Perform message passing

Message passing at a typical node x2 oNode “a” contains functions assigned to it according to the tree-decomposition scheme denoted by p j (a) a b x1

Message Passing root Collect root Distribute Figure from P. Green Two pass algorithm: Bucket-tree propagation

Lets look at the messages Collect Evidence ∫C∫C ∫ L ∫ Mout ∫ Min ∫ D ∫D∫D Strong Root

Distribute Evidence ∫ E ∑ W,B ∑W∑W ∫E∑B∫E∑B ∑F∑F Strong Root

Lauritzens theorem When you perform message passing such that collect evidence contains only strong marginals and distribute evidence may contain weak marginals, the junction-tree algorithm in exact in the sense that:  The first (mean) and second moments (variance) computed are true moments

Complexity Polynomial in #of continuous variables in a clique (n 3 ) Exponential in #of discrete variables in a clique Possible options for approximation  Ignore the strong root assumption and use approximation like MBTE, IJGP, Sampling  Respect the strong root assumption and use approximation like MBTE, IJGP, Sampling Inaccuracies only due to discrete variables if done in one pass of MBTE.

W=0W=1 X=0X=1 Initialization (1) w y x z dim: 2 w= w= x=00.4 x=10.6

Initialization (2) wyzwyzwxy wy Cliques 1 Cliques 2 (root) w=0g=log(0.5),h=[],K=[] w=1g=log(0.5),h=[],K=[] x=0g=log(0.4),h=[],K=[] x=1g=log(0.6),h=[],K=[] X=0X=1 g = h = [ ]’ K = [0.1 0; 0 0.1] g = h = [ ]’ K = [ ; ] W=0W=1 g = h = [ ] K = g = h = [ ] K =

W=0W=1 g = h = K = g = h = K = Initialization (3) wyzwyzwxy wy Cliques 1 Cliques 2 (root) wx=00wx=10 g = h = [ ]’ K = [0.1 0; 0 0.1] g = h = [ ]’ K = [0.1 0; 0 0.1] wx=01wx=11 g = h = [ ]’ K = [ ; ] g = h = [ ]’ K = [ ; ] empty

Message Passing wyzwyzwxy wy Cliques 1 Cliques 2 (root) empty Collect evidence Distribute evidence

Collect evidence (1) wyzwyzwxy wy Cliques 1 Cliques 2 (root) empty y2y3y2y3 y1y2y1y2 y2y2 (y 1,y 2 ) (y2)

Collect evidence (2) wyzwyzwxy wy Cliques 1 Cliques 2 (root) empty W=0W=1 g = h = K = g = h = K = W=0W=1 g = h = [ ]’ *1.0e-16 K = [ ; ]*1.0e-16 g = h = [0 0]’ K = [ ] marginalization

Collect evidence (3) wyzwyzwxy wy Cliques 1 Cliques 2 (root) empty W=0W=1 g = h = [ ]’ *1.0e-16 K = [ ; ]*1.0e-16 g = h = [0 0]’ K = [ ] wx=00wx=10 g = h = [ ]’ K = [0.1 0; 0 0.1] g = h = [ ]’ K = [0.1 0; 0 0.1] wx=01wx=11 g = h = [ ]’ K = [ ; ] g = h = [ ]’ K = [ ; ] multiplication wx=00wx=10 g = h = [ ]’ K = [0.1 0; 0 0.1] g = h = [ ]’ K = [0.1 0; 0 0.1] wx=01wx=11 g = h = [ ]’ K = [ ; ] g = h = [ ]’ K = [ ; ]

Distribute evidence (1) wyzwyzwxy wy Cliques 1 Cliques 2 (root) W=0W=1 g = h = K = g = h = K = W=0W=1 g = h = [ ]’ *1.0e-16 K = [ ; ]*1.0e-16 g = h = [0 0]’ K = [ ] division

Distribute evidence (2) wyzwyzwxy wy Cliques 1 Cliques 2 (root) W=0W=1 g = h = K = g = h = K =

Distribute evidence (3) wyzwyzwxy wy Cliques 1 Cliques 2 (root) wx=00wx=10 g = h = [ ]’ K = [0.1 0; 0 0.1] g = h = [ ]’ K = [0.1 0; 0 0.1] wx=01wx=11 g = h = [ ]’ K = [ ; ] g = h = [ ]’ K = [ ; ] Marginalize over x w=0w=1 logp = mu = [ ]’ Sigma = logp = mu = [ ]’ Sigma =

Distribute evidence (4) wyzwyzwxy wy Cliques 1 Cliques 2 (root) W=0W=1 g = h = K = g = h = K = w=0w=1 logp = mu = [ ]’ Sigma = logp = mu = [ ]’ Sigma = multiplication w=0w=1 g = h = [ ]’ K = g = h = [ ]’ K = Canonical form

Distribute evidence (5) wyzwyzwxy wy Cliques 1 Cliques 2 (root) W=0W=1 g = h = K = g = h = K =

After Message Passing p(wyz)p(wxy) p(wy) Cliques 1 Cliques 2 (root) Local marginal distributions