1. ARTIFICIAL INTELLIGENCE
CS 621 Artificial Intelligence Lecture /09/05 Prof. Pushpak Bhattacharyya Noisy Channel
Prof. Pushpak Bhattacharyya, IIT Bombay

2. Noisy Channel Metaphor
Source Receiver
{s1, s2, {t1, t2,..
… ss} tr}
Noise
Prof. Pushpak Bhattacharyya, IIT Bombay

3. Prof. Pushpak Bhattacharyya, IIT Bombay
Example 1
Speech
Source Receiver
(Speech (Words in
Signal) the signal)
Noisy Channel
Prof. Pushpak Bhattacharyya, IIT Bombay

4. Prof. Pushpak Bhattacharyya, IIT Bombay
Example 2
Vision
Source Receiver
(Visual (observes
Signal) objects)
Noisy Channel
Prof. Pushpak Bhattacharyya, IIT Bombay

5. Prof. Pushpak Bhattacharyya, IIT Bombay
Example 3
Spell Checking
Source Receiver
W W’
W’ – misspelt W
Noisy Channel
Prof. Pushpak Bhattacharyya, IIT Bombay

6. Prof. Pushpak Bhattacharyya, IIT Bombay
Example 4
Information Retrieval
Source Receiver
Query Q Retrieved Doc D
Noisy Channel
Prof. Pushpak Bhattacharyya, IIT Bombay

7. Prof. Pushpak Bhattacharyya, IIT Bombay
Example 5
Machine Translation
Source Receiver
(Sentences (Sentences
in language in language
L1) L2)
Noisy Channel
Prof. Pushpak Bhattacharyya, IIT Bombay

8. Prof. Pushpak Bhattacharyya, IIT Bombay
Example 6
Encryption Decryption
Source Receiver
Information decoded
encoded
Prof. Pushpak Bhattacharyya, IIT Bombay

9. Prof. Pushpak Bhattacharyya, IIT Bombay
Explanation
Noise
S R
{s1, s2, {t1, t2,..
… ss} tr}
pi = Pr(si) ∑ri=1 pi = 1
qj = Pr(tj) ∑rj=1 qj = 1
Prof. Pushpak Bhattacharyya, IIT Bombay

10. Prof. Pushpak Bhattacharyya, IIT Bombay
Notation of “E”
E(S) = ∑ri=1 pi log 1/pi
E(R) = ∑rj=1 qj log 1/qj
E(S) = measure of information transmitted by S.
S -> newspaper
S prints one day “Sun arose in the east” (No information is present)
Prints another day “Sun today arose in the west” (Information is present now)
Prof. Pushpak Bhattacharyya, IIT Bombay

11. Prof. Pushpak Bhattacharyya, IIT Bombay
Channel Matrix
s x r matrix, M
Where M = P11 P12 ….. P1r
P21 P22 ….. P2r
Ps1 Ps2 ….. Psr
Pij = Pr(tj is received when si is transmitted)
= Pr(t = tj | s = si)
Prof. Pushpak Bhattacharyya, IIT Bombay

12. Prof. Pushpak Bhattacharyya, IIT Bombay
Example 1
Binary Symmetric Channel
0 1
M = 0 P P’
1 P’ P p
p’ = 1-p
p’ = 1-p 1 1 p
Prof. Pushpak Bhattacharyya, IIT Bombay

13. Prof. Pushpak Bhattacharyya, IIT Bombay
Example 2
Binary Erasure Channel
0 1 ?(no symbol)
M = 0 P 0 P’
1 0 P P’
Prof. Pushpak Bhattacharyya, IIT Bombay

14. Forward & Backward Probability
Pij = Pr(t = tj | s = si)
= Forward Probability
Qij = Pr(s = si | t = tj)
= Backward Probability
Rij = Pr(t = tj, s = si)
= Joint Probability
Pij = Sender’s world view
Qij = Receiver’s world view
Rij = Observer’s world view
Prof. Pushpak Bhattacharyya, IIT Bombay

15. Bayes Theorem
Pij, Qij, Rij are related by a very very important theorem
BAYES THEOREM
(explained in later slides)
Prof. Pushpak Bhattacharyya, IIT Bombay

16. Prof. Pushpak Bhattacharyya, IIT Bombay
Relevance of Pij & Qij
Pij is relevant for
Speech to word recognition
Visual recognition
Qij is relevant for
Spell checking
Prof. Pushpak Bhattacharyya, IIT Bombay

17. Prof. Pushpak Bhattacharyya, IIT Bombay
Bayes Theorem
The probability that si is transmitted and tj is received given si
= Pi Pij = Qj Qij
= Prob that tj is received and tj is received given si is transmitted.
Thus, Qij = (Pi Pij) / Qi
Pi = prior probabilities
Pij = forward probabilities
= likelihood
Bayesian decision drops the denominator
Prof. Pushpak Bhattacharyya, IIT Bombay

18. Prof. Pushpak Bhattacharyya, IIT Bombay
Example
English French
E1,E2,..,Es F2,F2,…,Fr
Pij = Pr(F = Fj | E = Ei)
Pr(F = Fj | E = Ei) = Pr(F = Fj) . Pr(E = Ei | F = Fj)
Pr(E = Ei)
Prof. Pushpak Bhattacharyya, IIT Bombay

19. Prof. Pushpak Bhattacharyya, IIT Bombay
Example (contd)
Most likely French sentence is the one for which
Pr(F = Fj | E = Ei) is maximum.
One can drop Pr(E = Ej)
F* = Required French sentence
F* = argmax Pr(F = Fj) . Pr(E = Ei | F = Fj)
(Language model) (Translation model)
faithfulness fluency
Prof. Pushpak Bhattacharyya, IIT Bombay

20. Prof. Pushpak Bhattacharyya, IIT Bombay
Exercise
Use backward probability over noisy channel to model the spell checking problem.
Assumption: At most one mistake per misspelt word.
Prof. Pushpak Bhattacharyya, IIT Bombay

21. Prof. Pushpak Bhattacharyya, IIT Bombay
Kinds of Mistakes
Replacement . e.g p↔q
Insertion after
Insertion before
Deletion after
Deletion before
Prof. Pushpak Bhattacharyya, IIT Bombay

22. Prof. Pushpak Bhattacharyya, IIT Bombay
Summary
Two information agents
Concept of noisy channel
Model key AI problems as noisy channel problems
Channel matrix
Forward, backward & joint probabilities
Bayes theorem
Application to MT & spell checking
Prof. Pushpak Bhattacharyya, IIT Bombay