Approximating the Partition Function by Deleting and then Correcting for Model Edges Arthur Choi and Adnan Darwiche University of California, Los Angeles.

Slides:

Advertisements

Similar presentations

Exact Inference. Inference Basic task for inference: – Compute a posterior distribution for some query variables given some observed evidence – Sum out.

Advertisements

Join-graph based cost-shifting Alexander Ihler, Natalia Flerova, Rina Dechter and Lars Otten University of California Irvine Introduction Mini-Bucket Elimination.

Exact Inference in Bayes Nets

Online Performance Guarantees for Sparse Recovery Raja Giryes ICASSP 2011 Volkan Cevher.

Loopy Belief Propagation a summary. What is inference? Given: –Observabled variables Y –Hidden variables X –Some model of P(X,Y) We want to make some.

Convergent Message-Passing Algorithms for Inference over General Graphs with Convex Free Energies Tamir Hazan, Amnon Shashua School of Computer Science.

Introduction to Belief Propagation and its Generalizations. Max Welling Donald Bren School of Information and Computer and Science University of California.

Belief Propagation by Jakob Metzler. Outline Motivation Pearl’s BP Algorithm Turbo Codes Generalized Belief Propagation Free Energies.

CS774. Markov Random Field : Theory and Application Lecture 04 Kyomin Jung KAIST Sep

What is the next line of the proof? a). Let G be a graph with k vertices. b). Assume the theorem holds for all graphs with k+1 vertices. c). Let G be a.

Global Approximate Inference Eran Segal Weizmann Institute.

Belief Propagation, Junction Trees, and Factor Graphs

A Constraint Satisfaction Problem (CSP) is a combinatorial decision problem defined by a set of variables, a set of domain values for these variables,

Genome evolution: a sequence-centric approach Lecture 6: Belief propagation.

Bioinformatics and Graphical Models: Computation, approximation, and their value MSR: Nebojsa Jojic, Vladimir Jojic, Chris Meek, David Heckerman UW: Jim.

On the Power of Belief Propagation: A Constraint Propagation Perspective Rina Dechter Bozhena Bidyuk Robert Mateescu Emma Rollon.

A Trainable Graph Combination Scheme for Belief Propagation Kai Ju Liu New York University.

Genome Evolution. Amos Tanay 2009 Genome evolution: Lecture 8: Belief propagation.

Lec 18: May 31st, 2006EE512 - Graphical Models - J. BilmesPage 1 Jeff A. Bilmes University of Washington Department of Electrical Engineering EE512 Spring,

Daphne Koller Message Passing Loopy BP and Message Decoding Probabilistic Graphical Models Inference.

Adaptive CSMA under the SINR Model: Fast convergence using the Bethe Approximation Krishna Jagannathan IIT Madras (Joint work with) Peruru Subrahmanya.

1 Structured Region Graphs: Morphing EP into GBP Max Welling Tom Minka Yee Whye Teh.

Belief Propagation. What is Belief Propagation (BP)? BP is a specific instance of a general class of methods that exist for approximate inference in Bayes.

EBAS: An Energy-Efficient Event Boundary Approximated Suppression Algorithm in Wireless Sensor Networks Longjiang Guo Heilongjiang University

Belief Propagation Revisited Adnan Darwiche. Graphical Models Battery Age Alternator Fan Belt Battery Charge Delivered Battery Power Starter Radio LightsEngine.

Survey Propagation. Outline Survey Propagation: an algorithm for satisfiability 1 – Warning Propagation – Belief Propagation – Survey Propagation Survey.

Tokyo Institute of Technology, Japan Yu Nishiyama and Sumio Watanabe Theoretical Analysis of Accuracy of Gaussian Belief Propagation.

Readings: K&F: 11.3, 11.5 Yedidia et al. paper from the class website

Two Approximate Algorithms for Belief Updating Mini-Clustering - MC Robert Mateescu, Rina Dechter, Kalev Kask. "Tree Approximation for Belief Updating",

What is a Confidence Interval?. Sampling Distribution of the Sample Mean The statistic estimates the population mean We want the sampling distribution.

1 Mean Field and Variational Methods finishing off Graphical Models – Carlos Guestrin Carnegie Mellon University November 5 th, 2008 Readings: K&F:

Belief Propagation and its Generalizations Shane Oldenburger.

Join-graph based cost-shifting Alexander Ihler, Natalia Flerova, Rina Dechter and Lars Otten University of California Irvine Introduction Mini-Bucket Elimination.

Structured learning: overview Sunita Sarawagi IIT Bombay TexPoint fonts used in EMF. Read the TexPoint manual before.

ED-BP: Belief Propagation via Edge Deletion UCLA Automated Reasoning Group Arthur Choi, Adnan Darwiche, Glen Lenker, Knot Pipatsrisawat Last updated 07/15/2010:

Many-Pairs Mutual Information for Adding Structure to Belief Propagation Approximations Arthur Choi and Adnan Darwiche University of California, Los Angeles.

ICPR2004 (24 July, 2004, Cambridge) 1 Probabilistic image processing based on the Q-Ising model by means of the mean- field method and loopy belief propagation.

ベーテ自由エネルギーに対するCCCPアルゴリズムの拡張

Provable Learning of Noisy-OR Networks

Belief Propagation and Approximate Inference: Compensating for Relaxations Arthur Choi.

Cui Di Supervisor: Andrzej Lingas Lund University

Graduate School of Information Sciences, Tohoku University, Japan

Computability and Complexity

Bucket Renormalization for Approximate Inference

Encoding CNFs to Enhance Component Analysis

Markov Networks.

The Power of Two in Consistent Network Updates: Hard Loop Freedom, Easy Flow Migration Klaus-Tycho Förster and Roger Wattenhofer.

Arthur Choi and Adnan Darwiche UCLA

Comparing Fractions.

Generalized Belief Propagation

一般化された確率伝搬法の数学的構造東北大学大学院情報科学研究科田中和之

Markov Random Fields Presented by: Vladan Radosavljevic.

Graduate School of Information Sciences, Tohoku University

Arthur Choi and Adnan Darwiche UCLA

Physical Fluctuomatics 7th~10th Belief propagation

Expectation-Maximization & Belief Propagation

Probabilistic image processing and Bayesian network

Lecture 3: Exact Inference in GMs

3. 3 Properties of Rational Numbers Identity and Multiplication Prop

Unifying Variational and GBP Learning Parameters of MNs EM for BNs

Junction Trees 3 Undirected Graphical Models

Readings: K&F: 11.3, 11.5 Yedidia et al. paper from the class website

Ch5 Initial-Value Problems for ODE

Markov Networks.

Minimum Spanning Trees

Mean Field and Variational Methods Loopy Belief Propagation

Generalized Belief Propagation

Iterative Join Graph Propagation

Presentation transcript:

Approximating the Partition Function by Deleting and then Correcting for Model Edges Arthur Choi and Adnan Darwiche University of California, Los Angeles {aychoi,darwiche}@cs.ucla.edu

Edge Deletion: Idea Delete an edge Model M Model M'

Deleting an Equivalence Edge j

Deleting an Equivalence Edge j j

Deleting an Equivalence Edge j

Deleting an Equivalence Edge j

Deleting an Equivalence Edge j

Deleting an Equivalence Edge j

Edge Parameters: ED-BP i j

Edge Parameters: ED-BP i j Conditions on edge parameters imply an iterative algorithm: ED-BP

Edge Parameters: ED-BP i j Yields a weaker notion of equivalence:

A Spectrum of Approximations ED-BP networks: [CD06]

A Spectrum of Approximations ED-BP networks: [CD06] Exact Inference

A Spectrum of Approximations ED-BP networks: [CD06] Loopy BP marginals Exact Inference

A Spectrum of Approximations ED-BP networks: [CD06] Loopy BP marginals Exact Inference partition function?

A Partition Function i j

An Easy Case: Delete a Single Edge Prop.: If MI(Xi,Xj) = 0 in ED-BP network M', then: where i j

An Easy Case: Delete a Single Edge Prop.: If MI(Xi,Xj) = 0 in ED-BP network M', then: where i With multiple edges deleted (ZERO-EC): j

Bethe Free Energy and ZERO-EC Bethe free energy approximation: as a partition function approximation:

Bethe Free Energy is ZERO-EC Bethe free energy approximation: as a partition function approximation: Theorem: The Bethe approximation is ZERO-EC when M' is a tree :

An Easy Case: Delete a Single Edge Prop.: If MI(Xi,Xj) = 0 in ED-BP network M', then: where i With multiple edges deleted (ZERO-EC): j

An Easy Case: Delete a Single Edge Prop.: For any edge in an ED-BP network M', then where i With multiple edges deleted (GENERAL-EC): j

Overview tree exact exact marginals marginals LBP IJGP

joingraph free energies Overview tree exact exact marginals marginals LBP IJGP joingraph free energies zero-EC Bethe exact Z

Overview tree exact exact marginals marginals LBP IJGP joingraph free energies zero-EC Bethe exact Z improved approximations (higher order EP/GBP energies) general-EC

joingraph free energies Overview tree exact exact marginals marginals LBP IJGP joingraph free energies zero-EC Bethe exact Z recover edges general-EC

Edge Recovery: ZERO-EC i j Recover edges with largest MI(Xi;Xj)

Edge Recovery: GENERAL-EC j i t s Recover edges with largest MI(Xi,Xj; Xs,Xt)

Edge Recovery Bethe exact Z 6x6 grid EC-Z,rand 0.07 0.06 0.05 relative error 0.04 0.03 0.02 exact Z 0.01 edges recovered 25

Edge Recovery Bethe exact Z 6x6 grid EC-Z,rand EC-G,rand 0.07 0.06 0.05 relative error 0.04 0.03 0.02 exact Z 0.01 edges recovered 25

Edge Recovery Bethe exact Z 6x6 grid EC-Z,rand EC-G,rand 0.07 EC-Z,MI 0.06 0.05 relative error 0.04 0.03 0.02 exact Z 0.01 edges recovered 25

Edge Recovery Bethe exact Z 6x6 grid EC-Z,rand EC-G,rand 0.07 EC-Z,MI EC-G,MI 0.06 0.05 relative error 0.04 0.03 0.02 exact Z 0.01 edges recovered 25

Edge Recovery Bethe exact Z 6x6 grid EC-Z,rand EC-G,rand 0.07 EC-Z,MI EC-G,MI 0.06 EC-G,MI2 0.05 relative error 0.04 0.03 0.02 exact Z 0.01 edges recovered 25

Edge Recovery

joingraph free energies Overview tree exact exact marginals marginals LBP IJGP/GBP joingraph free energies zero-EC Bethe exact Z recover edges general-EC

joingraph free energies Overview tree exact exact marginals marginals LBP IJGP/GBP joingraph free energies zero-EC Bethe exact Z partial corrections: general-EC

Partial Correction pigs 1 ED-BP (Bethe) 0.8 0.6 relative error 0.4 general-EC 0.2 500 1000 1500 2000 2500 3000 time (ms)

Partial Correction pigs 1 ED-BP (Bethe) 0.8 0.6 relative error soft-sep see AAAI’08 0.4 general-EC 0.2 500 1000 1500 2000 2500 3000 time (ms)

Partial Correction

joingraph free energies Overview tree exact exact marginals marginals LBP IJGP/GBP joingraph free energies zero-EC Bethe exact Z general-EC

joingraph free energies Overview tree exact exact marginals marginals LBP IJGP/GBP marginal corrections (AAAI'08) joingraph free energies zero-EC Bethe exact Z general-EC

Thanks!