Download presentation

Presentation is loading. Please wait.

1
LING 438/538 Computational Linguistics Sandiway Fong Lecture 17: 10/24

2
Administrivia Reminder –Homework 4 due Thursday

3
Today’s Topic Introduction to Statistical Language Models –Background: General Introduction to Probability Concepts Sample Space Events Counting Event Probability Entropy

4
Introduction to Probability some definitions –sample space the set of all possible outcomes of a statistical experiment is called the sample space (S) finite or infinite (also discrete or continuous) –example coin toss experiment possible outcomes: {heads, tails} –example die toss experiment possible outcomes: {1,2,3,4,5,6}

5
Introduction to Probability some definitions –sample space the set of all possible outcomes of a statistical experiment is called the sample space (S) finite or infinite (also discrete or continuous) –example die toss experiment for whether the number is even or odd possible outcomes: {even,odd} not {1,2,3,4,5,6}

6
Introduction to Probability some definitions –events an event is a subset of sample space simple and compound events –example die toss experiment let A represent the event such that the outcome of the die toss experiment is divisible by 3 A = {3,6} a subset of the sample space {1,2,3,4,5,6}

7
Introduction to Probability some definitions –events an event is a subset of sample space simple and compound events –example deck of cards draw experiment suppose sample space S = {heart,spade,club,diamond} (four suits) let A represent the event of drawing a heart let B represent the event of drawing a red card A = {heart} (simple event) B = {heart} ∪ {diamond} = {heart,diamond} (compound event) –a compound event can be expressed as a set union of simple events –example alternative sample space S = set of 52 cards A and B would both be compound events

8
Introduction to Probability some definitions –events an event is a subset of sample space null space {} (or ) intersection of two events A and B is the event containing all elements common to A and B union of two events A and B is the event containing all elements belonging to A or B or both –example die toss experiment, sample space S = {1,2,3,4,5,6} let A represent the event such that the outcome of the experiment is divisible by 3 let B represent the event such that the outcome of the experiment is divisible by 2 intersection of events A and B is {6} (simple event) union of events A and B is the compound event {2,3,4,6}

9
Introduction to Probability some definitions –rule of counting suppose operation o i can be performed in n i ways, a sequence of k operations o 1 o 2...o k can be performed in n 1 n 2 ... n k ways –example die toss experiment, 6 possible outcomes two dice are thrown at the same time number of sample points in sample space = 6 6 = 36

10
Introduction to Probability some definitions –permutations a permutation is an arrangement of all or part of a set of objects the number of permutations of n distinct objects is n! (n! is read as n factorial) –Definition: n! = n x (n-1) ... x 2 x 1 n! = n x (n-1)! 1!=1 0!=1 1st 2nd 3rd 3 ways 2 ways 1 way –example suppose there are 3 students: adam, bill and carol how many ways are there of lining up the students? Answer: 6 3! permutations

11
Introduction to Probability some definitions –permutations a permutation is an arrangement of all or part of a set of objects the number of permutations of n distinct objects taken r at a time is n!/(n-r)! –example a first and a second prize raffle ticket is drawn from a book of 425 tickets Total number of sample points = 425!/(425-2)! = 425!/423! = 425 x 424 = 180,200 possibilities instance of sample space calculation

12
Introduction to Probability some definitions –combinations the number of combinations of n distinct objects taken r at a time is n!/(r!(n-r)!) combinations differ from permutations in that in the former case the selection is taken without regard for order –example given 5 linguists and 4 computer scientists what is the number of three- person committees that can be formed consisting of two linguists and one computer scientist? note: order does not matter here select 2 from 5: 5!/(2!3!) = (5 x 4)/2 = 10 select 1 from 4: 4!(1!3!) = 4 answer = 10 x 4 = 40 (rule of counting)

13
Introduction to Probability some definitions –probability probability are weights associated with sample points a sample point with relatively low weight is unlikely to occur a sample point with relatively high weight is likely to occur weights are in the range zero to 1 sum of all the weights in the sample space must be 1 (see smoothing) probability of an event is the sum of all the weights for the sample points of the event –example unbiased coin tossed twice sample space = {hh, ht, th, tt} (h = heads, t = tails) coin is unbiased => each outcome in the sample space is equally likely weight = 0.25 (0.25 x 4 = 1) What is the probability that at least one head occurs? sample points/probability for the event: hh 0.25 th 0.25 ht 0.25 Answer: 0.75 (sum of weights)

14
Introduction to Probability some definitions –probability probability are weights associated with sample points a sample point with relatively low weight is unlikely to occur a sample point with relatively high weight is likely to occur weights are in the range zero to 1 sum of all the weights in the sample space must be 1 probability of an event is the sum of all the weights for the sample points of the event heads and tailstails 1/3 2/3 –example a biased coin, twice as likely to come up tails as heads, is tossed twice What is the probability that at least one head occurs? sample space = {hh, ht, th, tt} (h = heads, t = tails) sample points/probability for the event: –ht 1/3 x 2/3 = 2/9 –hh 1/3 x 1/3= 1/9 –th 2/3 x 1/3 = 2/9 –tt 2/3 x 2/3 = 4/9 Answer: 0.56 (sum of weights in bold) cf. probability of event for the unbiased coin = 0.75 > 50% chance or < 50% chance?

15
S Introduction to Probability some definitions –probability let p(A) and p(B) be the probability of events A and B, respectively. additive rule: p(A B) = p(A) + p(B) - p(A B) if A and B are mutually exclusive events: p(A B) = p(A) + p(B) –since p(A B) = p( ) = 0 A B –example suppose probability of a student getting an A in linguistics is 2/3 ( 0.66) suppose probability of a student getting an A in computer science is 4/9 ( 0.44) suppose probability of a student getting at least one A is 4/5 (= 0.8) What is the probability a student will get an A in both? p(A B) = p(A) + p(B) - p(A B) 4/5 = 2/3 + 4/9 - p(A B) p(A B) = 2/3 + 4/9 - 4/5 = 14/45 0.31

16
S Introduction to Probability some definitions –conditional probability let A and B be events p(B|A) = the probability of event B occurring given event A occurs definition: p(B|A) = p(A B) / p(A)provided p(A)>0 –used an awful lot in language processing (context-independent) probability of a word occurring in a corpus (context-dependent) probability of a word occurring given the previous word

17
Entropy concept of uncertainty –example biased coin –0.8 heads –0.2 tails unbiased coin –0.5 heads –0.5 tails log conversion formula: lg = log 2 coin toss uncertainty vs. probability uncertainty measure (Shannon) –also mentioned a lot in corpus statistics r =2, p i = probability the event is i –biased coin -0.8 * lg 0.8 + -0.2 * lg 0.2 = 0.258 + 0.464 = 0.722 –unbiased coin: - 2* 0.5 * lg 0.5 = 1 –it’s a measure of the sample space as a whole uncertainty 50-50

18
Entropy uncertainty measure (Shannon) –given a random variable x r =2, p i = probability the event is i –biased coin: 0.722, unbiased coin: 1 –entropy = H(x) = Shannon uncertainty perplexity –a measure of branching factor –2 H –biased coin: 2 0.722 = 0.52 –unbiased coin: 2 1 = 2

19
Next Time today we have seen fundamental concepts so far next –apply probability theory to language –Bayes’ Rule –N-grams –Peirera’s Experiment with “Colorless green ideas sleep furiously”

Similar presentations

© 2019 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google