1. Grids
A1,1
A1,2
A1,3
A1,4
A2,1
A2,2
A2,3
A2,4
A3,1
A3,2
A3,3
A3,4
A4,1
A4,2
A4,3
A4,4

2. 1: A,B 4: A,D A B D 2: B,C 3: C,D C 1,4 = 1 4,1 = 1 1,2 = 1 1
3,4 = B D
2,1 = 1
4,3 = 1
2: B,C
3: C,D C
2,3 = 1
3,2 = 1

3. 1: A,B 4: A,D A B D 2: B,C 3: C,D C 1,4 = B 2,1(B) y1(A,B) 4,1 =
D 3,4(D) y4(A,D)
1: A,B
4: A,D A
1,2 =
A 4,1(A) y1 (A,B)
3,4 =
C 2,3(C) y3(C,D) B D
2,1 =
C 3,2(C) y2(B,C)
4,3 =
A 1,4(A) y4(A,D)
2: B,C
3: C,D C
2,3 =
B 1,2(B) y2(B,C)
3,2 =
D 4,3(D) y3(C,D)

4. Loopy BP Run P(a1) Iteration # True posterior 0.85 0.8 0.75 0.7 0.65
0.55 5 10 15 20
Iteration #

5. Grid Cluster Graph A1,1 A1,1 , A1,2 A1,2 A1,2 , A1,3 A1,3 A1,1 , A2,1

6. Cluster Graphs 1: A, B, C C 3: B,D,F B D E 2: B, C, D 5: D, E 4: B, E

8. Loopy in Practice Synchronous BP: all messages are updated in parallel
asynchronous order 1
Time (seconds) 2 4 6 8 10 12 14
# messages converged
Ising Grid
x 100
asynchronous order 2
synchronous
11
11
12
12
13
13
21
21
22
22
23
23
31
31
32
32
33
33
Asynchronous is faster than synchronous
Order of messages has a significant effect on behaviour

10. Shannon’s Theorem Goal: Transmit bits over a noisy channel
How efficient can we make our transmission, for arbitrarily low probability of making an error
Shannon’s result:
Define Channel Capacity = bound on code rate for a given signal to noise ratio
all rates under this are achievable, for arbitrarily low error rate (simply make messages long enough)
no rate above that is achievable
# bits per message
Rate of a code =
# bits sent

11. Y1 Y2 Y3 Y4 X1 X2 X3 X4 U1 U2 U3 U4 X5 X6 X7 Y5 Y6 Y7

12. U1 U2 Un Z1 Z2 Zn X1 X2 Xn Y1 Y2 Yn

13. Permuter Y4 Y8 X4 X8 W1 W2 W3 W4 U1 U2 U3 U4 Y1 Y3 Y5 Y7 Z1 Z2 Z3 Z4

14. Coding: Post 1993
Shannon limit = -0.79