1. CS621/CS449 Artificial Intelligence Lecture Notes
Set 7: 08/10/2004, 15/10/2004
13/08/2004
CS-621/CS-449 Lecture Notes

2. Outline Probability, Statistics & AI The Speech Recognition problem
Bayesian Decision Theory
A Probabilistic Spell Checker
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

3. Probability, Statistics & AI
Importance of knowledge understood from 1950’s research on Machine Translation.
Field of AI – characterized by the need for representation & processing of enormous amount of knowledge.
Manual input of knowledge difficult.
So, consider possibility of acquiring the knowledge or training
E.g. Web data  knowledge - not easy by any means!!
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

4. Probabilistic approaches?
Fundamental question : Can probabilistic approaches be used?
“Next-Word Guessing Program” – a speech recognition problem :
Given w1, w2….wn , we need to guess wn+1
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

5. The Speech Recognition Problem
Text T1:
I like Artificial Intelligence and Cognitive Sciences. They excite my imagination.
Text T2:
I hate communal riots. They disrupt life and work.
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

6. Sample Word Frequencies
Word-list w
Frequency
P(w) I 2
2/20
like 1
1/20
Artificial
Intelligence
and
Cognitive
Sciences
They … ….
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

7. Guessing the next word Word-guesses or Predictions
= Next-Word (Model of domain, speaker, hearer)
Having seen w1 = “I”
taking next word as word with the highest probability, we have:
Highly unlikely sequence!! w1 w2 I
I/and/they???
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

8. A Better Solution Next word =
word with highest conditional probability
P(w2 | w1)
P(like | I) = 0.5
P(hate | I) = 0.5
But, “disrupt”, “work”, “excite” are also possibilities not covered by this rule.
 Language Model not adequate!
Ideally, “I” is followed by not only “hate” and “like” but by verbs in general
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

9. Bayesian Decision Theory
Bayes Theorem : Given the random variables A & B,
Where
is the posterior probability (the conditional probability of A given that B has occurred),
is the prior probability and
is the likelihood of B given A
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

10. Bayes Theorem Derivation
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

11. Example
It is known that in a population, 1 in has meningitis and 1 in 20 has stiff neck. It is also observed that 50% of the meningitis patients have stiff neck.
A doctor observes that a patient has stiff neck. What is the probability that the patient has meningitis?
Solution: Record the event probabilities P(s) & P(m)
s – stiff neck, m - meningitis
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

12. Probabilities P(m) = P(an individual has meningitis) = , P(s) =
Prior probability = P(s | m) = 0.5
Posterior probability is l
It is most likely that the person does not have meningitis since
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

13. Some Issues
Important question : Is P(m|s) greater or lesser than P(s |m)?
P(m|s) could have been found as
Some questions:
Which is more reliable to compute, P(s|m) or P(m|s)?
Which evidence is more sparse , P(s|m) or P(m|s)?
Test of significance : The counts are always on a sample of population. Which probability count has sufficient statistics?
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

14. A Probabilistic Spell Checker
3 broad problems:
Non-word deletion : Apple  Aple
Isolated word correction :
Apple
Aple Applet
Maple
Detection & correction from context (very difficult and needs full NLP) : piece  peace
E.g. : The piece for which he struggled long and hard was stolen at night one day. The peace for which he struggled long and hard was destroyed in a single day with the outbreak of sect rivalries.
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

15. Types of Errors Isolated word errors: Errors which come from typing
Deletion : Apple  Aple
Insertion : Apple  Applet
Substitution : Apple  Apqle
Transposition : Apple  Aplpe
Errors from OCR (Optical Character Recognizer)
Blank Deletion : I watchcricket
Blank Insertion : I wa tch cricket
Mangling : I wdch cricket at
08/10/ /10/2004
CS-621/CS-449 Lecture Notes

16. Noisy Channel Model
The problem formulation for spell checker is based on the Noisy Channel Model
s t
(sn, sn-1, … , s1) (tm, tm-1, … , t1)
Given t, find the most probable s : Find that ŝ for which P(s|t) is maximum, where s, t and ŝ are strings
Is this a Bayesian problem case??
Noisy Channel ŝ
08/10/ /10/2004
CS-621/CS-449 Lecture Notes