1. Basic Concepts of Discrete Probability (Theory of Sets: Continuation) 1

2. Functions If X is a set and Y is a set, and there is a sequence of well-specified operations for assigning a well-defined object to every element, and by applying these sequence of operations to every member of a set X we obtain a set Y, then this sequence of operations forms a rule, which is commonly known as function. 2

3. Functions The set X is called the domain of the function, and the set Y is called the range. Functions that have numerical values ere called numerical functions. A numerical function establishing the correspondence between the elements of a set having a finite number of elements and a set of natural numbers will be of our particular interest: 3

4. The number of elements in a set The number of elements in a set may or may not be finite. If the elements of the set can be placed in one-to-one correspondence with the set of natural numbers {1, 2, 3,…}, we say that the set has a denumerable or countable number of elements; otherwise the set is nondenumerable (e.g., a set of points on any line or the set of points in [0,1]). 4

5. The number of elements in a set If a set is denumerable and is a finite number, we say that the power of the set X (the cardinality of the set X) is |X| (the number of elements in the set X). If a set is equivalent to the set of points in [0,1] that is, it is denumerable, we say that the set has the power of continuum. 5

6. The number of elements in a set The following properties hold for finite sets:: If A and B are two disjoint sets, then For any finite sets A, B, and C: 6

7. Basic Concepts of Discrete Probability Elements of the Probability Theory 7

8. Sample Space When “probability” is applied to something, we usually mean an experiment with certain outcomes. An outcome is any one of the possibilities that may be expected from the experiment. The totality of all these outcomes forms a universal set which is called the sample space. 8

9. Sample Space For example, if we checked occasionally the number of people in this classroom on Tuesday from 7pm to 9-45pm, we should consider this an experiment having 15 possible outcomes {0,1,2,…,13,14} that form a universal set. 0 – nobody is in the classroom, … 14 – all students taking the Information Theory Class and the instructor are in the classroom 9

10. Sample Space A sample space containing at most a denumerable number of elements is called discrete. A sample space containing a nondenumerable number of elements is called continuous. 10

11. Sample Space A subset of a sample space containing any number of elements (outcomes) is called an event. Null event is an empty subset. It represents an event that is impossible. An event containing all sample points is an event that is certain to occur. 11

12. Probability Measure The probability measure is a specific type of function which can be associated with sets. Since an experiment is defined so that to each possible outcome of this experiment there corresponds a point in the sample space, the number of outcomes of the experiment is assumed to be at most denumerable. 12

13. Probability Measure Let us consider an event of interest A as the set of outcomes a k. Let a real function m ( a k ) be the probability measure of the outcome a k. The probability measure of an event is defined as the sum of the probability measures associated with all the outcomes a k of that event. It is also referred to as the “additive probability measure”: 13

14. Probability Measure The additive probability measure has the following important property: the measure associated with the union of two disjoint sets is equal to the sum of their individual measures. 14

15. Probability Measure Two events A and B are called disjoint if they contain no outcome in common: two disjoint events cannot happen simultaneously. The probability measure has the following assumed properties: 15

16. Probability Measure If with an additive probability measure, then the following relations are valid: 16

17. Probability Measure For three disjoint sets : For three sets in general: 17

18. Frequency of events Let us consider a specific event X k among all the possible events of the experiment under consideration. If the basic experiment is repeated N times among which the event X k has appeared times, the ratio is defined as the relative frequency of the occurrence of the event X k. 18

19. Frequency of events. The Probability If N is increased indefinitely, that is if then, intuitively speaking, is the probability of the event X k. 19

20. The Probability The classical definition of Laplace says that the probability is the ratio of the number of favorable events to the total number of possible events. All events in this definition are considered to be equally likely: e.g., throwing of a true die by an honest person under prescribed circumstances… …but not checking of the number of people in the classroom. 20

21. The Probability The following properties are important: 21

22. The Probability To verify whether our “empiric” definition of probability satisfies the properties of the probability measure, we have to consider the probability of two events. Let A and B be the events among the ones resulting from the experiment. Let the experiment be repeated n times. 22

23. The Probability Each observation can belong to only one of the four following categories: 1) A has occurred, but not B  AB’; 2) B has occurred, but not A  BA’; 3) Both A and B have occurred  AB; 4) Neither A nor B has occurred  A’B’ 23

24. The Probability If the number of events of each category from the mentioned four ones is denoted by then relative frequency of A independent of B relative frequency of B independent of A 24

25. The Probability relative frequency of either A, B or both relative frequency of A and B occurring together relative frequency of A under condition that B has occurred relative frequency of B under condition that A has occurred 25

26. The Probability When the number of experiments tends to infinity, these relations lead to: If A and B are mutually exclusive events, then P{AB}=0; P{A+B}=P{A}+P{B} 26

27. The Probability Thus, we have proved that Let us compare the latter with the properties of the probability measure: 27

28. The Probability We have just shown that our empirical definitions of the frequency and probability, respectively, satisfy all the properties of the probability measure. 28

29. Theorem of addition For two events A and B of the sample space: The additive property of the probability measure suggests that If A and B are mutually exclusive events, then 29

30. Theorem of addition For two opposite events A and A’: A+A’=U; AA’=0, then 30

31. Theorem of addition For the three events A, B, and C: In general, for a number of events 31

32. Theorem of addition If the events are mutually exclusive, then It is also clear now that 32

33. Conditional Probability Let A and B be two events. The conditional probability of event A based on the hypothesis that event B has occurred is defined by the following relation: 33

34. Conditional Probability Returning to the example of two events 1) A has occurred, but not B  AB’  n 1 ; 2) B has occurred, but not A  BA’  n 2 ; 3) Both A and B have occurred  AB  n 3 ; 4) Neither A nor B has occurred  A’B’  n 4 : 34

35. Conditional Probability The two events A and B are called mutually independent if P{A|B}=P{A} and P{B|A}=P{B}. For mutually independent events P{AB}=P{A}P{B}. 35

36. Theorem of multiplication The multiplication rule for the case of two events A and B can be obtained through the definition of the conditional probability: P{AB}=P{A}P{B|A}=P{B}P{A|B}. For three events A,B, and C: P{ABC}=P{AB}P{C|AB}=P{A}P{B|A}P{C|AB} 36

37. Theorem of multiplication For n events: For a countable infinite number of the mutually independent events 37