1. Mean Field Inference in Dependency Networks: An Empirical Study Daniel Lowd and Arash Shamaei University of Oregon

2. Learning and Inference in Graphical Models We want to learn a probability distribution from data and use it to answer queries. Applications: medical diagnosis, fault diagnosis, web usage analysis, bioinformatics, collaborative filtering, etc. A BC Answers! Data Model LearningInference 2

3. One-Slide Summary 1.In dependency networks, mean field inference is faster than Gibbs sampling, with similar accuracy. 2.Dependency networks are competitive with Bayesian networks. A BC Answers! Data Model LearningInference 3

4. Outline Graphical models: Dependency networks vs. others – Representation – Learning – Inference Mean field inference in dependency networks Experiments 4

5. Dependency Networks Represents a probability distribution over {X 1, …, X n } as a set of conditional probability distributions. Example: X1X1 X2X2 X3X3 [Heckerman et al., 2000] 5

6. Comparison of Graphical Models Bayesian Network Markov Network Dependency Network Allow cycles?NYY Easy to learn?YNY Consistent distribution? YYN Inference algorithms …lots… Gibbs, MF (new!) 6

7. Learning Dependency Networks For each variable X i, learn conditional distribution, B=? false C=? true C=? false true [Heckerman et al., 2000] 7

8. Approximate Inference Methods Gibbs sampling: Slow but effective Mean field: Fast and usually accurate Belief propagation: Fast and usually accurate A BC Answers!Model 8

9. Gibbs Sampling Resample each variable in turn, given its neighbors: Use set of samples to answer queries. e.g., Converges to true distribution, given enough samples (assuming positive distribution). Previously, the only method used to compute probabilities in DNs. 9

10. Mean Field Approximate P with simpler distribution Q: To find best Q, optimize reverse K-L divergence: Mean field updates converge to local optimum: Works for DNs! Never before tested! 10

11. Mean Field in Dependency Networks 1.Initialize each Q(X i ) to a uniform distribution. 2.Update each Q(X i ) in turn: 3.Stop when marginals Q(X i ) converge. If consistent, this is guaranteed to converge. If inconsistent, this always seems to converge in practice. 11

12. Empirical Questions Q1. In DNs, how does MF compare to Gibbs sampling in speed and accuracy? Q2. How do DNs compare to BNs in inference speed and accuracy? 12

13. Experiments Learned DNs and BNs on 12 datasets Generated queries from test data – Varied evidence variables from 10% to 90% – Score using average CMLL per variable (conditional marginal log-likelihood): 13

14. Results: Accuracy in DNs 14 Negative CMLL

15. Results: Timing in DNs (log scale) 15

16. MF vs. Gibbs in DNs, run for equal time Evidence# of MF wins% wins 10%975% 20%1083% 30%1083% 40%975% 50%1083% 60%1083% 70%1192% 80%1192% 90%12100% Average10.285% In DNs, MF usually more accurate, given equal time. 16

17. Results: Accuracy 17

18. Gibbs: DN vs. BN EvidenceDN winsPercent wins 10%325% 20%18% 30%18% 40%325% 50%542% 60%758% 70%1083% 80%1083% 90%1192% Average5.747% With more evidence, DNs are more accurate. 18

19. Experimental Results Q1. In DNs, how does MF compare to Gibbs sampling in speed and accuracy? A1. MF is consistently faster with similar accuracy, or more accurate with similar speed. Q2. How do DNs compare to BNs in inference speed and accuracy? A2. DNs are competitive with BNs – better with more evidence, worse with less evidence. 19

20. Conclusion MF inference in DNs is fast and accurate, especially with more evidence. Future work: – Relational dependency networks (Neville & Jensen, 2007) – More powerful approximations Source code available: http://libra.cs.uoregon.edu/http://libra.cs.uoregon.edu/ Source code available: http://libra.cs.uoregon.edu/http://libra.cs.uoregon.edu/ 20

21. Results: Timing (log scale) 21

22. Learned Models 1.Learning time is comparable. 2.DNs usually have higher pseudo-likelihood (PLL) 3.DNs sometimes have higher log-likelihood (LL) 22