1. Information Theory and Security

2. Lecture Motivation Up to this point we have seen:
Classical Crypto
Symmetric Crypto
Asymmetric Crypto
These systems have focused on issues of confidentiality: Ensuring that an adversary cannot infer the original plaintext message, or cannot learn any information about the original plaintext from the ciphertext.
But what does “information” mean?
In this lecture and the next we will put a more formal framework around the notion of what information is, and use this to provide a definition of security from an information-theoretic point of view.

3. Lecture Outline Probability Review: Conditional Probability and Bayes
Entropy:
Desired properties and definition
Chain Rule and conditioning
Coding and Information Theory
Huffman codes
General source coding results
Secrecy and Information Theory
Probabilistic definitions of a cryptosystem
Perfect Secrecy

4. The Basic Idea Suppose we roll a 6-sided dice.
Let A be the event that the number of dots is odd.
Let B be the event that the number of dots is at least 3.
A = {1, 3, 5}
B = {3, 4, 5, 6}
I tell you: the roll belongs to both A and B then you know there are only two possibilities: {3, 5}
In this sense tells you more than just A or just B.
That is, there is less uncertainty in than in A or B.
Information is closely linked with this idea of uncertainty: Information increases when uncertainty decreases.

5. Probability Review, pg. 1
A random variable (event) is an experiment whose outcomes are mapped to real numbers.
For our discussion we will deal with discrete-valued random variables.
Probability: We denote pX(x) = Pr(X = x).
For a subset A,
Joint Probability: Sometimes we want to consider more than two events at the same time, in which we case we lump them together into a joint random variable, e.g. Z = (X,Y).
Independence: We say that two events are independent if

6. Probability Review, pg. 2
Conditional Probability: We will often ask questions about the probability of events Y given that we have observed X=x. In particular, we define the conditional probability of Y=y given X=x by
Independence: We immediately get
Bayes’s Theorem: If pX(x)>0 and pY(y)>0 then

7. Example
Example: Suppose we draw a card from a standard deck. Let X be the random variable describing the suit (e.g. clubs, diamonds, hearts, spades). Let Y be the value of the card (e.g. two, three, …, ace). Then Z=(X,Y) gives the 52 possibilities for the card.
P( (X,Y) = (x,y) ) = P(X=x, Y=y) = 1/52
P(X=“clubs”) = 13/52 = ¼
P(Y=“3”) = 4/52 = 1/13

8. Entropy and Uncertainty
We are concerned with how much uncertainty a random event has, but how do we define or measure uncertainty?
We want our measure to have the following properties:
To each set of nonnegative numbers with , we define the uncertainty by
should be a continuous function: A slight change in p should not drastically change
for all n>0. Uncertainty increases when there are more outcomes.
If 0<q<1, then

9. Entropy, pg. 2 We define the entropy of a random variable by
Example: Consider a fair coin toss. There are two outcomes, with probability ½ each. The entropy is
Example: Consider a non-fair coin toss X with probability p of getting heads and 1-p of getting tails. The entropy is
The entropy is maximum when p= ½.

10. Entropy, pg. 3
Entropy may be thought of as the number of yes-no questions needed to accurately determine the outcome of a random event.
Example: Flip two coins, and let X be the number of heads. The possibilities are {0,1,2} and the probabilities are {1/4, 1/2, 1/4}. The Entropy is
So how can we relate this to questions?
First, ask “Is there exactly one head?” You will half the time get the right answer…
Next, ask “Are there two heads?”
Half the time you needed one question, half you needed two

11. Entropy, pg. 4
Suppose we have two random variables X and Y, the joint entropy H(X,Y) is given by
Conditional Entropy: In security, we ask questions of whether an observation reduces the uncertainty in something else. In particular, we want a notion of conditional entropy. Given that we observe event X, how much uncertainty is left in Y?

12. Entropy, pg. 5
Chain Rule: The Chain Rule allows us to relate joint entropy to conditional entropy via H(X,Y) = H(Y|X)+H(X).
(Remaining details will be provided on the white board)
Meaning: Uncertainty in (X,Y) is the uncertainty of X plus whatever uncertainty remains in Y given we observe X.

13. Entropy, pg. 6 Main Theorem: Entropy is non-negative.
where denotes the number of elements in the sample space of X.
(Conditioning reduces entropy)
with equality if and only if X and Y are independent.

14. Entropy and Source Coding Theory
There is a close relationship between entropy and representing information.
Entropy captures the notion of how many “Yes-No” questions are needed to accurately identify a piece of information… that is, how many bits are needed!
One of the main focus areas in the field of information theory is on the issue of source-coding:
How to efficiently (“Compress”) information into as few bits as possible.
We will talk about one such technique, Huffman Coding.
Huffman coding is for a simple scenario, where the source is a stationary stochastic process with independence between successive source symbols

15. Huffman Coding, pg. 1
Suppose we have an alphabet with four letters A, B, C, D with frequencies:
We could represent this with A=00, B=01, C=10, D=11. This would mean we use an average of 2 bits per letter.
On the other hand, we could use the following representation: A=1, B=01, C=001, D=000. Then the average number of bits per letter becomes
(0.5)*1+(0.3)*2+(0.1)*3+(0.1)*3 = 1.7
Hence, this representation, on average, is more efficient.
A B C D

16. Huffman Coding, pg. 2
Huffman Coding is an algorithm that produces a representation for a source.
The Algorithm:
List all outputs and their probabilities
Assign a 1 and 0 to smallest two, and combine to form an output with probability equal to the sum
Sort List according to probabilities and repeat the process
The binary strings are then obtained by reading backwards through the procedure A
0.5 1
1.0 1 B
0.3
0.5 1 C
0.1
0.2 D
0.1
Symbol Representations
A: 1
B: 01
C: 001
D: 000

17. Huffman Coding, pg. 3
In the previous example, we used probabilities. We may directly use event counts.
Example: Consider 8 symbols, and suppose we have counted how many times they have occurred in an output sample.
We may derive the Huffman Tree
The corresponding length vector is (2,2,3,3,3,4,5,5)
The average codelength is If we had used a full-balanced tree representation (i.e. the straight-forward representation) we would have had an average codelength of 3.
S1 S2 S3 S4 S5 S6 S7 S8

18. Huffman Coding, pg. 4
We would like to quantify the average amount of bits needed in terms of entropy.
Theorem: Let L be the average number of bits per output for Huffman encoding of a random variable X, then
Here, lx =length of codeword assigned to symbol x.
Example: Let’s look back at the 4 symbol example
Our average codelength was 1.7 bits.

19. Next Time We will look at how entropy is related to security
Generalized definition of encryption
Perfect Secrecy
Manipulating entropy relationships
The next computer project will also be handed out