1. CPSC 531: Probability Review1 CPSC 531:Probability & Statistics: Review Instructor: Anirban Mahanti Office: ICT 745 Email: mahanti@cpsc.ucalgary.ca Class Location: TRB 101 Lectures: TR 15:30 – 16:45 hours Class web page: http://pages.cpsc.ucalgary.ca/~mahanti/teaching/F05/CPSC531 http://pages.cpsc.ucalgary.ca/~mahanti/teaching/F05/CPSC531 Notes derived from “Probability and Statistics” by M. DeGroot and M. Schervish, Third edition, Addison Wesley, 2002.

2. CPSC 531: Probability Review2 Outline r Experiments, sample space, and events r Review of set theory r Probability: definition, property, interpretation r Finite sample space r Counting methods r Conditional probability r Independent events r Law of total probability r Bayes’ Theorem

3. CPSC 531: Probability Review3 Experiments, sample space, and events r Experiment – in probability theory refers to a process whose outcome is not known in advance with certainty m E.g., An experiment consists of tossing a coin 10 times. What is the probability that at least four heads are obtained in a row? r Sample space (S) – is the collection of all possible outcomes of an experiment. Each outcome is a sample point. m E.g., Rolling a six-sided dice: S = {1, 2, 3, 4, 5, 6} r Event – is a subset of the sample space m E.g., A is an event that an even number is rolled: A = {2, 4, 6}

4. CPSC 531: Probability Review4 Set Theory (1) r Some sets contain only finite number of elements, while others contain infinitely many elements. An infinite set can be classified as: m Countable - if there is a one-to-one correspondence between elements of the set and the set of natural numbers m Uncountable – converse of countable; e.g., set of real numbers, numbers in the interval [0, 1] r Empty set: Ø is the “null” set r Union: A  B = is the event containing all outcomes that are in A or B, or both

5. CPSC 531: Probability Review5 Set Theory (2) r Intersection: A  B = event contained in both A and B r Complement: A C is the complement of event A and contains all outcomes in the sample space S that do not belong to A r Idempotent laws m A  A = A m A  A = A r Associative laws m (A  B)  C = A  (B  C) = (A  B)  C m (A  B)  C = A  (B  C) = (A  B)  C

6. CPSC 531: Probability Review6 Set Theory (3) r Commutative laws m A  B = B  A m A  B = B  A r Distributive laws m A  (B  C) = (A  B)  (A  C) m A  (B  C) = (A  B)  (A  C) r Identity laws m A  Ø = A m A  U = A m A  U = U m A  Ø = Ø

7. CPSC 531: Probability Review7 Probability - definition r A probability distribution, or simply a probability, on a sample space S is a specification of numbers P(.) that satisfies the following axioms: 1) For every event A, P(A) ≥ 0 2) P(S) = 1 3) For every infinite sequence of mutually exclusive events A 1, A 2, …

8. CPSC 531: Probability Review8 Properties of Probability r For any event A, P(A C ) = 1 – P(A) r Since Ø = S C, P(Ø) = 0 r If A  B, P(A) ≤ P(B) r For every event A, 0 ≤ P(A) ≤ 1 r Home work: Recall Venn diagrams … they can help in proving the above properties.

9. CPSC 531: Probability Review9 Example (1) r Select one ball from a box containing white (W), red (R), blue (B), and green (G) balls. Suppose that P(R) = 0.1 and P(B) = 0.5. What is the probability of selecting a white or a green ball? r Solution: W U R U B U G = S From axiom 3, we have P(S) = P(W) + P(R) + P(B) + P(G) Also, P(WUG) = P(W)+P(G) = 1 – P(R) – P(B) = 0.4

10. CPSC 531: Probability Review10 Example (2) r Rolling a fair dice: P(1) = P(2) = P(3) = P(4) = P(5) = P(6) = 1/6, where P(i) is the probability of rolling a face with i dots. r Question: m P({1, 3}) = ? m P({2, 4, 6}) = ?

11. CPSC 531: Probability Review11 Interpretations of probability r Classical view: Probability measures the long- term likelihood of outcomes. This is, if an experiment is repeated “many” times, event A should occur roughly P(A) fraction of the time. m Can “many” be quantified? r Bayesian view: Probability represents a personal estimate of how likely an event is going to occur. r Note: the theory of probability does not depend on the interpretation. Good!

12. CPSC 531: Probability Review12 Finite Sample Spaces r A common assumption for a finite sample space S containing n outcomes s 1, s 2, … s n is that each outcome is equally likely. r If an event A in this sample space contains exactly m outcomes, then P(A) = m/n r Example: Toss three fair coins simultaneously. What is the probability of obtaining exactly two heads? S = {(HHH), (HHT), (HTH), (THH), (TTH), (HTT), (THT), (TTT)} P(2 H’s) = 3/8

13. CPSC 531: Probability Review13 Counting Methods r Multiplication rule r Permutations r Combinations

14. CPSC 531: Probability Review14 Multiplication rule r If an experiment has k “parts” (k ≥ 2), and the i th part has n i possible outcomes, then the total number of possible outcomes is n 1 x n 2 … x n n r Example: 6 coins are tossed m 6 parts to the experiments: toss 1, 2, …, 6 m 2 possible outcomes per part (i.e., per toss) m Possible outcomes = 2 6 = 64

15. CPSC 531: Probability Review15 Permutations (1) r Consider an experiment that involves sampling without replacement. E.g., arrange 5 different books on a shelf. How many ways can the books be arranged? r Think of the shelf as having 5 slots r 5 choices for filling the first slot r 4 choices for filling the second slot r 3 choices for filling the third slot r 2 choices for filling the fourth slot …. r So, we have 5x4x3x2x1 = 120 possibilities

16. CPSC 531: Probability Review16 Permutations (2) r Given n items, the number of permutations (or arrangements) of all n items equals n! m n! = n x (n-1) x (n-2) x … x 2 x 1 m Definition for 0! - It equals 1 since there is only 1 way to arrange zero items. r When not all items are ordered: E.g., a club has 25 members and 2 officers (a president and a secretary). These officers are to be chosen from the members of the club. How many ways can the officers be selected? m 25 ways to select the first officer and 24 ways to select the second officer. Answer = 25x24 = 600

17. CPSC 531: Probability Review17 Permutations (3) r Generalization: The number of ordered ways k items can be selected from a pool of n items is: P n,k = n x (n-1) x … X (n-k+1) = n!/(n-k)! r Home work problem: What is the probability that at least two people share the same birthday? You may assume the following: m 365 possible birthdays (ignore leap years) m Our group consists of 2 ≤ k ≤ 365 people who are unrelated (no twins) m Each birthday is equally likely (birth rate independent of the time of year)

18. CPSC 531: Probability Review18 Combinations (1) r Consider choosing a subset containing k items from a set of n distinct items. How many different subsets are possible? m The arrangement of elements in the subset is irrelevant. We treat each subset as an unit. r Such unordered samples are called combinations. The number of combinations possible when selecting k items from a set of n items is denoted by C n,k. r We will use the multiplication rule to find C n,k.

19. CPSC 531: Probability Review19 Combinations (2) r Note that one way to find P n,k is to compute C n,k (i.e., unordered selection of k items) and multiply by the number of ways of ordering k items (i.e., k!). r Therefore, P n,k = C n,k x k! r That is,

20. CPSC 531: Probability Review20 Combinations (3) r Example: A fair coin is tossed 10 times. Determine the probability, p, of obtaining exactly 3 heads. Solution m Each experiment has two possible outcome, heads or tails. Since the coins are fair, we can assume that each outcome is equally likely. Therefore, the total number of outcomes in this experiment is = 2 10 m The number of different arrangements possible with 3 heads and 7 tails is = C 10,3 m Therefore,

21. CPSC 531: Probability Review21 Conditional Probability (1) r The conditional probability of event A given that event B has occurred is r The conditional probability P(A|B) is undefined if P(B)=0 r The frequency interpretation: If an experiment is repeated a large number of times, then event B occurs approximately P(B) fraction of the time and the event A and B both occur approximately P(A  B) portion of time. Therefore, among the repetitions in which B occurs, the proportion of events in which A also occurs is P(A  B) /P(B). P(A  B) also denoted as P(AB).

22. CPSC 531: Probability Review22 Conditional Probability (2) r Example: Two balls are selected at random, without replacement, from a bag containing r red balls and b blue balls. Determine probability p that the first selected ball is red and the second selected ball is blue. Solution A = event that first ball is red; B = event that second ball is blue P(A) = prob. first ball is red = r/(r+b) P(B|A) = prob. blue in second draw = b/(r+b-1) P(B|A) = P(AB)/P(A) P(AB) = (r.b)/((r+b)(r+b-1))

23. CPSC 531: Probability Review23 Conditional Probability (3) r P(AB) = P(A) P(A|B); extended, as shown next r Suppose that A 1, A 2, A 3, …, A n are events such that P(B) > 0 and P(A 1 A 2 A 3 …A n-1 ) > 0. Then, P(A 1 A 2 A 3 …A n ) = P(A 1 )P(A 2 |A 1 ) P(A 3 |A 1 A 2 )… P(A n |A 1 A 2 A 3 …A n-1 ) r Example: Draw 4 balls, one by one, without replacement, from a box containing r red balls and b blue balls. Probability of sequence red, red, blue, blue? Solution: Let R j and B j denote event that a red and blue ball is drawn, respectively. P(R 1 R 2 B 3 B 4 ) = P(R 1 )P(R 2 |R 1 )P(B 3 |R 1 R 2 ) P(B 4 |R 1 R 2 B 3 ) complete the calculations on your own …

24. CPSC 531: Probability Review24 Independent Events (1) r Independent events are those that don’t have any effect on each other. That is, knowing one of them occurs does not provide any information about the occurrence of the other event. Mathematically, A and B are independent if P(A|B) = P(A) and P(B|A) = P(B) r From conditional probability definition, we have P(AB) = P(A|B)P(B). Therefore, if A and B are independent events, P(AB) = P(A)P(B)

25. CPSC 531: Probability Review25 Independent Events (2) r Example: Determine the probability p n that exactly n tosses are required for a head to appear for the first time. Solution We need to determine the probability of a sequence of (n-1) tails followed by a head. The probability of obtaining a heads or tails (for a fair coin) is ½. Therefore, p n = (½) n.

26. CPSC 531: Probability Review26 Law of Total Probability r Let B 1, B 2, B 3, …, B k be mutually disjoint events such that B 1 U B 2 U B 3 U … B k equal the sample space S. Then, for any event A in S, we have Explanation A = (B 1 A)U (B 2 A)U… (B k A). The (B j A)’s are disjoint events. Therefore, using laws of conditional probability we get: B1B1 B2B2 B3B3 B4B4 A

27. CPSC 531: Probability Review27 Bayes’ Theorem r Partition: The events B 1, B 2, B 3, …, B k form a partition of a set S if they are mutually disjoint and r Bayes’ Theorem: Suppose that B 1, B 2, B 3, …, B k form a partition of sample space S such that P(B j )> 0 for j = 1, …, k. Let A be an event in S such that P(A)>0. Then, for i = 1, …, k,

28. CPSC 531: Probability Review28 Summary r A review of basic probability and statistics was presented. You probably enjoyed it! r Next, we will talk about discrete and continuous distributions.