1. Introduction to Probability Theory
Fangrong Yan

2. Outline Introduction Compound Events
Basic concepts in probability theory
Addition Rule
Multiplication Rule for Independent Events
Conditional Probability
Bayes’ rule

3. Introduction
Probability is a measure of the likelihood of a random phenomenon or chance behavior. Probability describes the long-term proportion with which a certain outcome will occur in situations with short-term uncertainty.
EXAMPLE
Simulate flipping a coin 100 times. Plot the proportion of heads against the number of flips. Repeat the simulation.

4. Probability deals with experiments that yield random short-term results or outcomes, yet reveal long-term predictability.
The long-term proportion with which a certain outcome is observed is the probability of that outcome.

5. In probability, an experiment is any process that can be repeated in which the results are uncertain.
A simple event is any single outcome from a probability experiment. Each simple event is denoted ei.

6. The sample space, S, of a probability experiment is the collection of all possible simple events. In other words, the sample space is a list of all possible outcomes of a probability experiment.
An event is any collection of outcomes from a probability experiment. An event may consist of one or more simple events. Events are denoted using capital letters such as E.

7. 2-7
Basic Definitions
Set - a collection of elements or objects of interest
Empty set (denoted by )
a set containing no elements
Universal set (denoted by S)
a set containing all possible elements
Complement (Not). The complement of A is
a set containing all elements of S not in A

8. 2-8
Complement of a Set S A
Venn Diagram illustrating the Complement of an event

9. Basic Concepts (from Set Theory)
The union of two events A and B, A  B, is the event consisting of all outcomes that are either in A or in B or in both events.
The intersection of two events A and B, A  B, is the event consisting of all outcomes that are in both events.
When two events A and B have no outcomes in common, they are said to be mutually exclusive, or disjoint, events.

10. Sets: A Intersecting with B
2-10
Sets: A Intersecting with B S A B

11. 2-11
Sets: A Union B S A B

12. Mutually Exclusive or Disjoint Sets
2-12
Mutually Exclusive or Disjoint Sets
Sets have nothing in common S B A

13. 2-13
Experiment
Process that leads to one of several possible outcomes *, e.g.:
Coin toss
Heads, Tails
Rolling a die
1, 2, 3, 4, 5, 6
Pick a card
AH, KH, QH, ...
Introduce a new product
Each trial of an experiment has a single observed outcome.
The precise outcome of a random experiment is unknown before a trial.
* Also called a basic outcome, elementary event, or simple event

14. Events : Definition Sample Space or Event Set Event
2-14
Events : Definition
Sample Space or Event Set
Set of all possible outcomes (universal set) for a given experiment
E.g.: Roll a regular six-sided die
S = {1,2,3,4,5,6}
Event
Collection of outcomes having a common characteristic
E.g.: Even number
A = {2,4,6}
Event A occurs if an outcome in the set A occurs

15. Example
Experiment: toss a coin 10 times and the number of heads is observed.
Let A = { 0, 2, 4, 6, 8, 10}.
B = { 1, 3, 5, 7, 9}, C = {0, 1, 2, 3, 4, 5}.
A  B= {0, 1, …, 10} = .
A  B contains no outcomes. So A and B are mutually exclusive.
Cc = {6, 7, 8, 9, 10}, A  C = {0, 2, 4}.

16. Rules DeMorgan’s Laws: Commutative Laws: Associative Laws:
A  B = B  A, A  B = B  A
Associative Laws:
(A  B)  C = A  (B  C )
(A  B)  C = A  (B  C) .
Distributive Laws:
(A  B)  C = (A  C)  (B  C)
(A  B)  C = (A  C)  (B  C)
DeMorgan’s Laws:

17. Probability (cont.)
Probability theory provides a basis for the science of statistical inference from data
Figure 6.1
The role of probability in inferential statistics. Probability is used to predict what kind of samples are likely to be obtained from a population. Thus, probability establishes a connection between samples and populations. Inferential statistics rely on this connection when they use sample data as the basis for making conclusions about populations.

18. Probability (cont.)
Whenever the scores in a population are variable it is impossible to predict with perfect accuracy exactly which score or scores will be obtained when you take a sample from the population.
In this situation, researchers rely on probability to determine the relative likelihood for specific samples.
Thus, although you may not be able to predict exactly which value(s) will be obtained for a sample, it is possible to determine which outcomes have high probability and which have low probability.

19. Probability (cont.)
Probability is determined by a fraction or proportion.
When a population of scores is represented by a frequency distribution, probabilities can be defined by proportions of the distribution.
In graphs, probability can be defined as a proportion of area under the curve.

20. Probability of an Event
Probability of an event is a measure of the likelihood that the event will occur.
Remember probability is a number not a set.
Mathematically speaking the probability of an event E denoted by P(E) is:
P(E) = n(E)/n(S).
Recall that that n(E) is the cardinal number of set E and n(S) is the cardinal number of set S.

21. Definition of Probability
Experiment: toss a coin twice
Sample space: possible outcomes of an experiment
S = {HH, HT, TH, TT}
Event: a subset of possible outcomes
A={HH}, B={HT, TH}
Probability of an event : an number assigned to an event Pr(A)
Axiom 1: Pr(A)  0
Axiom 2: Pr(S) = 1
Axiom 3: For every sequence of disjoint events
Example: Pr(A) = n(A)/N: frequentist statistics
P(A),P(B)??

22. Combinations of Events
The complement Ac of an event A is the event that A does not occur
Probability Rule 3:
P(Ac) = 1 - P(A)
The union of two events A and B is the event that either A or B or both occurs
The intersection of two events A and B is the event that both A and B occur
Event A
Complement of A
Union of A and B
Intersection of A and B

23. Disjoint Events
Two events are called disjoint if they can not happen at the same time
Events A and B are disjoint means that the intersection of A and B is zero
Example: coin is tossed twice
S = {HH,TH,HT,TT}
Events A={HH} and B={TT} are disjoint
Events A={HH,HT} and B = {HH} are not disjoint
Probability Rule 4: If A and B are disjoint events then
P(A or B) = P(A) + P(B)

24. Example
Hypertension :Let A be the event that a person has normotensive diastolic bloodpressure(DBP) readings (DBP < 90), and let B be the event that a person has borderline DBP readings (90 ≤ DBP < 95). Suppose that Pr(A) = .7, and Pr(B) = .1. Let Z be the event that a person has a DBP < 95.
Then Pr (Z) = Pr (A) + Pr (B) = .8 because the events A and B cannot occur at the same time.

25. Mutually Exclusive
Two events A and B are mutually exclusive if and only if:
In a Venn diagram this means that event A is disjoint from event B.
A and B are M.E.
A and B are not M.E.

26. The Addition Rule
The probability that at least one of the events A or B will occur, P(A or B), is given by:
If events A and B are mutually exclusive, then the addition rule is simplified to:
This simplified rule can be extended to any number of mutually exclusive events.

27. Example
In a large city, two newspapers are published, the Sun and the Post. The circulation departments report that 22% of the city’s households have a subscription to the Sun and 35% subscribe to the Post. A survey reveals that 6% of all households subscribe to both newspapers. What proportion of the city’s households subscribe to either newspaper?
That is, what is the probability of selecting a household at random that subscribes to the Sun or the Post or both?
P(Sun or Post) = P(Sun) + P(Post) – P(Sun and Post)
= – .06 = .51

28. Joint Probability: Multiplication Rule
For events A and B, joint probability Pr(AB) stands for the probability that both events happen.
Example: A={HH}, B={HT, TH}, what is the joint probability Pr(AB)?

29. Independence Two events A and B are independent in case
Pr(AB) = Pr(A)Pr(B)
A set of events {Ai} is independent in case

30. Independence Two events A and B are independent in case
Pr(AB) = Pr(A)Pr(B)
A set of events {Ai} is independent in case
Example: Drug test
A = {A patient is a Women}
B = {Drug fails}
Will event A be independent from event B ?
Women
Men
Success
200
1800
Failure

31. Independent events
Events A and B are independent if knowing that A occurs does not affect the probability that B occurs
Example: tossing two coins
Event A = first coin is a head
Event B = second coin is a head
Disjoint events cannot be independent!
If A and B can not occur together (disjoint), then knowing that A occurs does change probability that B occurs
Probability Rule 5: If A and B are independent
P(A and B) = P(A) x P(B)
Independent
multiplication rule for independent events

32. Independence Consider the experiment of tossing a coin twice
Example I:
A = {HT, HH}, B = {HT}
Will event A independent from event B?
Example II:
A = {HT}, B = {TH}
Disjoint  Independence
If A is independent from B, B is independent from C, will A be independent from C?

33. Conditional Probability…
Conditional probability is used to determine how two events are related; that is, we can determine the probability of one event given the occurrence of another related event.
Experiment: random select one student in class.
P(randomly selected student is male) =
P(randomly selected student is male/student is on 3rd row) =
Conditional probabilities are written as P(A | B) and read as “the probability of A given B” and is calculated as:

34. Conditioning
If A and B are events with Pr(A) > 0, the conditional probability of B given A is

35. Conditional Probability…
Again, the probability of an event given that another event has occurred is called a conditional probability…
P( A and B) = P(A)*P(B/A) = P(B)*P(A/B) both are true
Keep this in mind!

36. Conditioning
If A and B are events with Pr(A) > 0, the conditional probability of B given A is
Example: Drug test
A = {Patient is a Women}
B = {Drug fails}
Pr(B|A) = ?
Pr(A|B) = ?
Women
Men
Success
200
1800
Failure

37. Conditioning
If A and B are events with Pr(A) > 0, the conditional probability of B given A is
Example: Drug test
Given A is independent from B, what is the relationship between Pr(A|B) and Pr(A)?
A = {Patient is a Women}
B = {Drug fails}
Pr(B|A) = ?
Pr(A|B) = ?
Women
Men
Success
200
1800
Failure

38. Independence…
One of the objectives of calculating conditional probability is to determine whether two events are related.
In particular, we would like to know whether they are independent, that is, if the probability of one event is not affected by the occurrence of the other event.
Two events A and B are said to be independent if
P(A|B) = P(A)
and
P(B|A) = P(B)
P(you have a flat tire going home/radio quits working)

39. Product Rules for Independent Events
2-39
Product Rules for Independent Events
The probability of the intersection of several independent events
is the product of their separate individual probabilities:
The probability of the union of several independent events
is 1 minus the product of probabilities of their complements:
Example 2-7:

40. Independence of Events
2-40
Independence of Events
Conditions for the statistical independence of events A and B:

41. Which Drug is Better ?

42. Simpson’s Paradox: View I
Drug II is better than Drug I
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 10%
Pr(C|B) ~ 50%
Drug I
Drug II
Success
219
1010
Failure
1801
1190

43. Simpson’s Paradox: View II
Female Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 20%
Pr(C|B) ~ 5%

44. Simpson’s Paradox: View II
Female Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 20%
Pr(C|B) ~ 5%
Male Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 100%
Pr(C|B) ~ 50%

45. Simpson’s Paradox: View II
Drug I is better than Drug II
Female Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 20%
Pr(C|B) ~ 5%
Male Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 100%
Pr(C|B) ~ 50%

46. Total probability rule
Definition: A collection of events {S1, S2, …, Sn} is a partition of a sample space S if
1. S = S1  S2  …  Sn
2. S1, S2, …, Sn are mutually exclusive events.
Example:
Recall the example of rolling two dice.
Define S2={sum = 2}, S3={sum = 3}, …, S12={sum = 12}.
Then {S2, S3, …, S12} is a partition of the sample space.

47. Total probability rule
Assume the set of events {S1, S2, …, Sn} is a partition of a sample space S. Assume P(Si) > 0 for every i, 1 ≤ i ≤ n.
Then for any event A, S1 S2 S3 S4
AS1
AS2
AS3
AS4

48. Example: A diagnostic test for a certain disease is known to be 95% accurate. It is also known from previous data that only 1% of the population has the disease. What is the probability that a person chosen at random will be tested positive?
Solution: Let
T+ denote the event that a person is tested positive;
T- denote the event that a person is tested negative;
D denote the event that a person has the disease.
Then
Applying the formula of total probability:

49. The Law of Total Probability- Example
2-49
The Law of Total Probability- Example
Event U: Stock market will go up in the next year
Event W: Economy will do well in the next year

50. Bayes’ Rule
Given two events A and B and suppose that Pr(A) > 0. Then
Example:
Pr(R) = 0.8
R: It is a rainy day
W: The grass is wet
Pr(R|W) = ?
Pr(W|R) R R W
0.7
0.4 W
0.3
0.6

51. Bayes’ Rule R: It rains W: The grass is wet R R W 0.7 0.4 W 0.3 0.6
Information
Pr(W|R) R W
Inference
Pr(R|W)

52. Bayes’ Rule Posterior Likelihood Prior R R W 0.7 0.4 W 0.3 0.6
R: It rains
W: The grass is wet
Information: Pr(E|H)
Hypothesis H
Evidence E
Posterior
Likelihood
Prior
Inference: Pr(H|E)

53. Bayes’ Rule: More Complicated
Suppose that B1, B2, … Bk form a partition of S:
Suppose that Pr(Bi) > 0 and Pr(A) > 0. Then

54. Bayes’ Rule: More Complicated
Suppose that B1, B2, … Bk form a partition of S:
Suppose that Pr(Bi) > 0 and Pr(A) > 0. Then

55. Bayes’ Rule: More Complicated
Suppose that B1, B2, … Bk form a partition of S:
Suppose that Pr(Bi) > 0 and Pr(A) > 0. Then

56. Bayes’ rule
Example (cont.): If a person tested positive then what is the probability he has the disease?

57. Bayes’ Theorem Extended - Example 2
2-57
Bayes’ Theorem Extended - Example 2
An economist believes that during periods of high economic growth, the U.S. dollar appreciates with probability 0.70; in periods of moderate economic growth, the dollar appreciates with probability 0.40; and during periods of low economic growth, the dollar appreciates with probability 0.20.
During any period of time, the probability of high economic growth is 0.30, the probability of moderate economic growth is 0.50, and the probability of low economic growth is 0.5.
Suppose the dollar has been appreciating during the present period. What is the probability we are experiencing a period of high economic growth?
Partition:
H - High growth P(H) = 0.30
M - Moderate growth P(M) = 0.50
L - Low growth P(L) = 0.20

58. 2-58
Example 2 (continued)

59. Example 3
Pulmonary Disease: Suppose a 60-year-old man who has never smoked cigarettes presents to a physician with symptoms of a chronic cough and occasional breathlessness.The physician becomes concerned and orders the patient admitted to the hospital for a lung biopsy. Suppose the results of the lung biopsy are consistent either with lung cancer or with sarcoidosis, a fairly common, nonfatal lung disease.
In this case
A = {chronic cough, results of lung biopsy}
Disease state B1 = normal B2 = lung cancer B3 = sarcoidosis
Suppose that Pr(A|B1) = .001 Pr(A|B2) = .9 Pr(A|B3) = .9
and that in 60-year-old, never-smoking men
Pr(B1) = .99 Pr(B2 ) = .001 Pr(B3 ) = .009

62. Homework
PP.61: