2. CSC 350 - Dr. Gary Locklair Exam #4 …

3. CSC 350 - Dr. Gary Locklair update date on slides 5, 6, 7

4. CSC 350 - Dr. Gary Locklair III. Information Theory Computers are informational tools Information (“I”) comes from Intelligence

5. CSC 350 - Dr. Gary Locklair A. Shannon (Info) Theory Claude Shannon (1916-2001) American computer scientist of the 20th century

6. CSC 350 - Dr. Gary Locklair Two Statements: “It is Thursday, December 8, 2005, 1:15pm, S118B, CUW …” “Plasma beings from the planet Threa have infiltrated CUW. One of them has taken the shape of Dr. Ferry and assumed his role as president.”

7. CSC 350 - Dr. Gary Locklair Two Statements: One is dull: “It is Thursday, December 8, 2005, 1:15pm, S118B, CUW …” The other bizarre: “Plasma beings from the planet Threa have infiltrated CUW. One of them has taken the shape of Dr. Ferry and assumed his role as president.”

8. CSC 350 - Dr. Gary Locklair Two Statements: One doesn’t convey any (Shannon) information “It is Thursday, December 8, 2005, 1:15pm, S118B, CUW …” Why? Because its probability is high - it is indeed Thursday, etc

9. CSC 350 - Dr. Gary Locklair Two Statements: The other conveys lots of (Shannon) information: “Plasma beings from the planet Threa have infiltrated CUW. One of them has taken the shape of Dr. Ferry and assumed his role as president.” Why? Because its probability is low.

10. CSC 350 - Dr. Gary Locklair A. Shannon (Info) Theory deals only with syntax; it does not deal with semantics.

11. CSC 350 - Dr. Gary Locklair A. Shannon (Info) Theory As one example, the truth or falsity of the statement isn’t considered. For the moment, we won’t care if Plasma beings have really taken over Dr. Ferry or not! :-)

12. CSC 350 - Dr. Gary Locklair Dr. A. E. Wilder-Smith called Information the “surprise effect”

13. CSC 350 - Dr. Gary Locklair Shannon (Info) Theory attempts to: 1. Quantify, or measure, I 2. Set theoretical limits (define what’s possible) for conveying (transmitting) I

14. CSC 350 - Dr. Gary Locklair Shannon was concerned with transmitting I over phone lines. He wanted to reliably convey I from source to destination. In other words, we don’t want the I corrupted during transmission.

15. CSC 350 - Dr. Gary Locklair What would corrupt the I? [student responses here :-] How might you deal with a noisy channel? [student responses here :-] Notice that all require more effort

16. CSC 350 - Dr. Gary Locklair Shannon asked 1. Is it possible to detect and correct a corrupted message (within what limits)? 2. How can a garbled message be recovered?

17. CSC 350 - Dr. Gary Locklair Deal with noise by 1. Recognize the problem 2. Compensate for it to begin with

18. CSC 350 - Dr. Gary Locklair Situation I must be conveyed from A  B over a noisy channel 1. Reliability – message at B should be identical to message at A

19. CSC 350 - Dr. Gary Locklair Situation I must be conveyed from A  B over a noisy channel 2. Maximize Rate – effective transmission time {These are the real world tradeoffs}

20. CSC 350 - Dr. Gary Locklair We don’t want to give up channel capacity …eg, we don’t want to repeat message 5 times to ensure it arrives since there is no new I conveyed during times 2-5

21. CSC 350 - Dr. Gary Locklair Shannon (Info) Theory shows that we don’t have to give up rate to gain reliability however, there is a price: delay – time to recover or decode the message increases due to (perhaps) longer messages

22. CSC 350 - Dr. Gary Locklair Actually two subjects Info Theory – what is possible Coding Theory – how to do it

23. CSC 350 - Dr. Gary Locklair

25. Theory (fun!) Mutual Information – I provided about event X by occurrence of event Y I(X;Y) = LOG P(X|Y) / P(X)

26. CSC 350 - Dr. Gary Locklair Theory (fun!) Self Info – I(X) = LOG 1 / P(X) Entropy – “average” I = H(X) =  P(X) * LOG 1 / P(X)

27. CSC 350 - Dr. Gary Locklair remember 1. don’t consider if message is T or F (yet) 2. Shannon (Info) Theory depends upon probability of the message

28. CSC 350 - Dr. Gary Locklair Example Self Info – I(X) = LOG 1 / P(X)

29. CSC 350 - Dr. Gary Locklair If message is certain, Info should be …? I(X) = LOG 1 / P(X) … P(X) = 1 (100%) we usually use LOG 2 and unit of I is the bit LOG 2 1 = 0 … makes sense, there is no “info” in a certain message

30. CSC 350 - Dr. Gary Locklair Practical Example Using Shannon (Info) Theory along with Coding Theory to see how to efficiently (and reliably) transmit I …

31. CSC 350 - Dr. Gary Locklair What if we know some messages are more likely than others? Ex: a weather forecaster with 4 possible forecasts:

32. CSC 350 - Dr. Gary Locklair Ex: a weather forecaster with 4 possible forecasts: Cloudy = 50% - ½ Sunny = 25% - ¼ Rainy = 12.5% - 1 / 8 Tornado = 12.5% - 1 / 8

33. CSC 350 - Dr. Gary Locklair Normally, for 4 different messages we’d need at least … how many bits? 2 bits (encode 4 possibilities as 00, 01, 10, 11)

34. CSC 350 - Dr. Gary Locklair Shannon (Info) Theory says “average info” (entropy) of this “alphabet” (set of messages) is:  P(X) * LOG 1 / P(X) sum up (probability * self I)

35. CSC 350 - Dr. Gary Locklair (½ * LOG 2 1/(½)) + (¼ * LOG 2 1/(¼)) + ( 1 / 8 * LOG 2 1/( 1 / 8 )) + ( 1 / 8 * LOG 2 1/( 1 / 8 )) = 1/2+ 2/4 + 3/8+ 3/8 = 1¾

36. CSC 350 - Dr. Gary Locklair What? How can we transmit in less than 2 bits? Shannon says only 1¾ bits! On average, one message is much more likely, therefore encode it in a shorter bit string than the others.

37. CSC 350 - Dr. Gary Locklair Huffman encoding – variable length codes Coding Theory (prefix code) 0 = cloudy 10 = sunny 110 = rainy 111 = tornado

38. CSC 350 - Dr. Gary Locklair Zeb/Zeke Joke Zeb and Zeke were sadly returning from an expensive fishing trip which only produced one fish.

39. CSC 350 - Dr. Gary Locklair Zeb/Zeke Joke “The way I figure it,” said Zeke, “that lousy fish cost us $400!” “Wow,” replied Zeb, “it’s a good thing we didn’t catch more!”

40. CSC 350 - Dr. Gary Locklair B. Information Theory Gary Locklair - American computer scientist of the 21st century Who? Information only comes from Intelligence

41. CSC 350 - Dr. Gary Locklair Dr A. E. Wilder-Smith British scientist of the 20th century Life is matter + teleonomy (Information Content) Ultimate source of teleonomy is an omnipotent God

42. CSC 350 - Dr. Gary Locklair Dr. Werner Gitt German computer scientist of the 21st century “Laws of Information” Information consists of syntax and semantics

43. CSC 350 - Dr. Gary Locklair Dr. Werner Gitt “According to Shannon’s Theory any random sequence of symbols is regarded as information, without regard to its origin, nor whether it is meaningful or not.”

44. CSC 350 - Dr. Gary Locklair Gitt’s Levels of Information 1 - Statistics (transmitted signal) 2 - Syntax (coding method) 3 - Semantics (meaning) 4 - Pragmatics (action) 5 - Apobetics (purpose)

45. CSC 350 - Dr. Gary Locklair Information Theory is more than just statistics (Shannon) There must be an associated meaning for information to be present Example: computer program …

46. CSC 350 - Dr. Gary Locklair In Shannon (Info) Theory the truth or falsity of the statement isn’t considered! Although the “Plasma being” message has high Shannon (Info) Content It’s a bloomin’ lie! :-)

47. CSC 350 - Dr. Gary Locklair In Shannon (Info) Theory the truth or falsity of the statement isn’t considered! Although the “Today is …” message has low Shannon (Info) Content It’s the truth! :-)

48. CSC 350 - Dr. Gary Locklair Information Theory implies a sender and a receiver Sender: I have a purpose in mind, that will require some action, so I will communicate my idea using a particular code and then transmit it.

49. CSC 350 - Dr. Gary Locklair Information Theory implies a sender and a receiver Receiver: I received a signal, now I must decode it. I can now understand the idea and implement some action to achieve the desired result.

50. CSC 350 - Dr. Gary Locklair This is Information “For God so loved the world that he gave his one and only Son, that whoever believes in him shall not perish but have eternal life.”

51. CSC 350 - Dr. Gary Locklair From Computer Scientist Don Knuth