1. B AYESIAN N ETWORKS

2. S OME A PPLICATIONS OF BN Medical diagnosis Troubleshooting of hardware/software systems Fraud/uncollectible debt detection Data mining Analysis of genetic sequences Data interpretation, computer vision, image understanding

3. M ORE C OMPLICATED S INGLY -C ONNECTED B ELIEF N ET Radio Battery SparkPlugs Starts Gas Moves

4. Region = {Sky, Tree, Grass, Rock} R2 R4 R3 R1 Above

6. C ALCULATION OF J OINT P ROBABILITY BEP(A| … ) TTFFTTFF TFTFTFTF 0.95 0.94 0.29 0.001 BurglaryEarthquake Alarm MaryCallsJohnCalls P(B) 0.001 P(E) 0.002 AP(J|…) TFTF 0.90 0.05 AP(M|…) TFTF 0.70 0.01 P(J  M  A   B   E) = P(J|A)P(M|A)P(A|  B,  E)P(  B)P(  E) = 0.9 x 0.7 x 0.001 x 0.999 x 0.998 = 0.00062 P(x 1  x 2  …  x n ) =  i=1,…,n P(x i |parents(X i ))  full joint distribution table

7. W HAT DOES THE BN ENCODE ? Burglary  Earthquake JohnCalls  MaryCalls | Alarm JohnCalls  Burglary | Alarm JohnCalls  Earthquake | Alarm MaryCalls  Burglary | Alarm MaryCalls  Earthquake | Alarm BurglaryEarthquake Alarm MaryCallsJohnCalls A node is independent of its non-descendents, given its parents

8. P ROBABILISTIC I NFERENCE Is the following problem…. Given: A belief state P(X 1,…,X n ) in some form (e.g., a Bayes net or a joint probability table) A query variable indexed by q Some subset of evidence variables indexed by e 1,…,e k Find: P(X q | X e1,…, X ek )

9. BEP(A| … ) TTFFTTFF TFTFTFTF 0.95 0.94 0.29 0.001 BurglaryEarthquake Alarm MaryCallsJohnCalls P(B) 0.001 P(E) 0.002 AP(J|…) TFTF 0.90 0.05 AP(M|…) TFTF 0.70 0.01 T OP -D OWN INFERENCE : RECURSIVE COMPUTATION OF A LL M ARGINALS D OWNSTREAM OF E VIDENCE P(A|E) = P(A|B,E)P(B) + P(A|  B,E)P(  B) P(J|E) = P(J|A,E)P(A) + P(J|  A,E)P(  A) P(M|E) = P(M|A,E)P(A) + P(M|  A,E)P(  A)

10. T OP -D OWN INFERENCE Only works if the graph of ancestors of a variable is a polytree Evidence given on ancestor(s) of the query variable Efficient: O(d 2 k ) time, where d is the number of ancestors of a variable, with k a bound on # of parents Evidence on an ancestor cuts off influence of portion of graph above evidence node

11. Q UERYING THE BN The BN gives P(T|C) P(C|T) can be computed using Bayes rule: P(A|B) = P(B|A) P(A) / P(B) Cavity Toothache P(C) 0.1 CP(T|C) TFTF 0.4 0.01111

12. Q UERYING THE BN The BN gives P(T|C) What about P(C|T)? P(Cavity|Toothache) = P(Toothache|Cavity) P(Cavity) P(Toothache) [Bayes’ rule] Querying a BN is just applying Bayes’ rule on a larger scale… Cavity Toothache P(C) 0.1 CP(T|C) TFTF 0.4 0.01111 Denominator computed by summing out numerator over Cavity and  Cavity

13. N AÏVE B AYES M ODELS P(Cause,Effect 1,…,Effect n ) = P(Cause)  i P(Effect i | Cause) Cause Effect 1 Effect 2 Effect n

14. N AÏVE B AYES C LASSIFIER P(Class,Feature 1,…,Feature n ) = P(Class)  i P(Feature i | Class) Class Feature 1 Feature 2 Feature n P(C|F 1,….,F k ) = P(C,F 1,….,F k )/P(F 1,….,F k ) = 1/Z P(C)  i P(Fi|C) Given features, what class? Spam / Not Spam English / French/ Latin … Word occurrences

15. C OMMENTS ON N AÏVE B AYES MODELS Very scalable (thousands or millions of features!), easy to implement Easily handles missing data: just ignore the feature Conditional independence of features is main weakness. What if two features were actually correlated? Many features?

16. V ARIABLE E LIMINATION : P ROBABILISTIC I NFERENCE IN G ENERAL N ETWORKS Coherence DifficultyIntelligence Happy GradeSAT Letter Job Basic idea: Eliminate “nuisance” variables one at a time via marginalization Example: P(J) Elimination order: C,D,I,H,G,S,L

17. Coherence DifficultyIntelligence Happy GradeSAT Letter Job P(D|C) P(C) P(I) P(G|I,D) P(H|G,J) P(J|S,L) P(S|I)

18. Coherence DifficultyIntelligence Happy GradeSAT Letter Job P(D|C) P(C) E LIMINATING C

19. DifficultyIntelligence Happy GradeSAT Letter Job P(D)=  c P(D|C)P(C) C IS E LIMINATED, GIVING A NEW FACTOR OVER D

20. DifficultyIntelligence Happy GradeSAT Letter Job P(D) E LIMINATING D P(G|I,D)

21. Intelligence Happy GradeSAT Letter Job D IS E LIMINATED, GIVING A NEW F ACTOR OVER G, I P(G|I)=  d P(G|I,d)P(d)

22. Intelligence Happy GradeSAT Letter Job E LIMINATING I P(G|I)P(S|I) P(I)

23. Happy GradeSAT Letter Job I IS E LIMINATED, PRODUCING A NEW F ILL E DGE AND F ACTOR OVER G AND S P(G,S)=  i P(i)P(G|i)P(S|i) New undirected fill edge

24. Happy GradeSAT Letter Job E LIMINATING H P(H|G,J)

25. Happy GradeSAT Letter Job E LIMINATING H P(H|G,J) f GJ (G,J)=  h P(h|G,J)=1

26. GradeSAT Letter Job H IS E LIMINATED, PRODUCING A NEW F ILL E DGE AND F ACTOR OVER G, J f GJ (G,J)

27. GradeSAT Letter Job E LIMINATING G f GJ (G,J) P(G,S) P(L|G)

28. GradeSAT Letter Job G IS E LIMINATED, M AKING A NEW T RINARY F ACTOR OVER S,L,J AND A NEW F ILL E DGE f GJ (G,J) P(G,S) P(L|G) f SLJ (S,L,J) =  g P(g,S) P(L|g) f GJ (g,J)

29. SAT Letter Job E LIMINATING S f SLJ (S,L,J) P(J|S,L)

30. SAT Letter Job S IS E LIMINATED, CREATING A NEW FACTOR OVER L, J f SLJ (S,L,J) P(J|S,L) f LJ (L,J) =  s f SLJ (s,L,J) P(J|s,L)

31. Letter Job E LIMINATING L f LJ (L,J)

32. Letter Job L IS ELIMINATED, GIVING A NEW FACTOR OVER J ( WHICH TURNS OUT TO BE P(J)) f LJ (L,J) P(J)=  l f LJ (l,J)

33. Job L IS ELIMINATED, GIVING A NEW FACTOR OVER J ( WHICH TURNS OUT TO BE P(J)) P(J)

34. J OINT DISTRIBUTION P( X ) = P(C)P(D|C)P(I)P(G|I,D)P(S|I)P(L|G) P(J|L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L

35. G OING THROUGH VE P( X ) = P(C)P(D|C)P(I)P(G|I,D)P(S|I)P(L|G) P(J|L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L f D (D)=  C P(C)P(D|C)

36. G OING THROUGH VE  C P( X ) = f D (D)P(I)P(G|I,D)P(S|I)P(L|G) P(J|L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L f D (D)=  C P(C)P(D|C)

37. G OING THROUGH VE  C P( X ) = f D (D)P(I)P(G|I,D)P(S|I)P(L|G) P(J|L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L f GI (G,I)=  D f D (D)P(G|I,D)

38. G OING THROUGH VE  C,D P( X ) = f GI (G,I)P(I)P(S|I)P(L|G) P(J|L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L f GI (G,I)=  D f D (D)P(G|I,D)

39. G OING THROUGH VE  C,D P( X ) = f GI (G,I)P(I)P(S|I)P(L|G) P(J|L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L f GS (G,S)=  I f GI (G,I)P(I)P(S|I)

40. G OING THROUGH VE  C,D,I P( X ) = f GS (G,S)P(L|G)P(J|L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L f GS (G,S)=  I f GI (G,I)P(I)P(S|I)

41. G OING THROUGH VE  C,D,I P( X ) = f GS (G,S)P(L|G)P(J|L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L f GJ (G,J)=  H P(H|G,J) What values does this factor store?

42. G OING THROUGH VE  C,D,I,H P( X ) = f GS (G,S)P(L|G)P(J|L,S)f GJ (G,J) Apply elimination ordering C,D,I,H,G,S,L f GJ (G,J)=  H P(H|G,J)

43. G OING THROUGH VE  C,D,I,H P( X ) = f GS (G,S)P(L|G)P(J|L,S)f GJ (G,J) Apply elimination ordering C,D,I,H,G,S,L f SLJ (S,L,J)=  G f GS (G,S)P(L|G)f GJ (G,J)

44. G OING THROUGH VE  C,D,I,H,G P( X ) = f SLJ (S,L,J)P(J|L,S) Apply elimination ordering C,D,I,H,G,S,L f SLJ (S,L,J)=  G f GS (G,S)P(L|G)f GJ (G,J)

45. G OING THROUGH VE  C,D,I,H,G P( X ) = f SLJ (S,L,J)P(J|L,S) Apply elimination ordering C,D,I,H,G,S,L f LJ (L,J)=  S f SLJ (S,L,J)P(J|L,S)

46. G OING THROUGH VE  C,D,I,H,G,S P( X ) = f LJ (L,J) Apply elimination ordering C,D,I,H,G,S,L f LJ (L,J)=  S f SLJ (S,L,J)

47. G OING THROUGH VE  C,D,I,H,G,S P( X ) = f LJ (L,J) Apply elimination ordering C,D,I,H,G,S,L f J (J)=  L f LJ (L,J)

48. G OING THROUGH VE  C,D,I,H,G,S,L P( X ) = f J (J) Apply elimination ordering C,D,I,H,G,S,L f J (J)=  L f LJ (L,J)

49. O RDER -D EPENDENCE

50. O RDER MATTERS Coherence DifficultyIntelligence Happy GradeSAT Letter Job If we were to eliminate G first, we’d create a factor over D, I, L, and H (their distribution becomes coupled)

51. E LIMINATION O RDER M ATTERS Coherence DifficultyIntelligence Happy SAT Letter Job If we were to eliminate G first, we’d create a factor over D, I, L, and H (their distribution becomes coupled) f DILH (D,I,L,H) =  g P(g|D,I) *P(L|g)*P(H|g)

52. C OMPLEXITY In polytree networks where each node has at most k parents, O(n2 k ) with top-down ordering In other networks, intermediate factors may involve more than k terms Worst case O(n) Good ordering heuristics exist, e.g. min-neighbors, min-fill Exact inference on non-polytree networks is NP- hard!

53. V ARIABLE E LIMINATION WITH E VIDENCE Coherence DifficultyIntelligence Happy GradeSAT Letter Job Two-step process: 1. Find P(X,e) with VE 2. Normalize by P(e) Example: P(J|H) 1.Run VE, enforcing H=T when H is eliminated. 2.This produces P(J,H=T) (a factor over J) 3.P(J=T|H=T) = P(J=T,H=T) / (P(J=T,H=T)+P(J=F,H=T))

54. R ECAP Exact inference techniques Top-down inference: linear time when ancestors of query variable are polytree, evidence is on ancestors Bottom-up inference in Naïve Bayes models General inference using Variable Elimination (We’ll come back to approximation techniques in a week.)

55. N EXT TIME Learning Bayes nets R&N 20.1-2