1. Communication, data transfer
1. Selected parts of information theory
coding, transfer rates, enthropy, compression, encryption
under continuous construction...

2. Units of information
b, bit („binary digit”): basic unit of information in binary number system, with value of 0 or 1; can be used equivalently in digital technics as True (1) or False (0 or -1 or any other number if not in binary representation)
B, byte: 1B=8b
also called „octet” (eg. in French), partly because byte is not trivially 8b (historically other formats existed)
Modern computers, processors and controllers are typically based on n*8b representation of data (eg. memory cell size, ALU size or data bus size etc.)

3. Binary prefixes
Binary storage devices typically have sizes that are powers of two (question: why?)
Thus was the K=1024 prefix developed long ago
They agreed that k=1000 and K=1024, but it was not universal of course
Larger prefixes (M,G,T) are already capital letters so problems arose
Especially with hard disks (1TB=?B)

4. Binary prefixes 1999: IEC 60027-2 Amendment 2
to make prefixes unambigous
traditional SI prefixes are decimal
new binary prefixes:
Ki,Mi,Gi,Ti etc.
KiB: kibibyte, MiB: mebibyet etc.
(„kilo binary byte”)

5. ASCII American Standard Code for Information Interchange
Original version: 1963, actual: 1986
7 bit symbols, 128 characters:
lowercase and uppercase letters,numbers, special characters, control characters
control chars: eg. new line, carriage return, tabulator, end-of-file etc. – some were made for printers
Numbers’ format: to (notice the code contains the number)
A: , a: (A+32)
Originally the 8.bit was used as parity
Later it was extended. Using the 8.bit another 128 characters were added, but these are not standard. More than 200 „code pages” exist.

6. Units of information transfer
Bit rate: number of bits transferred in unit time [b/s]
informal unit notations:
bps (bit per second)
Bps (byte per second)
CASE SENSITIVE! 1Bps=8bps

7. Units of information transfer
Gross bit rate (raw bit rate, data signaling rate):
Counts all ("physical") bits
Net bit rate (information rate):
counts only bits containing the message (information), discounts bits for header, error correction, etc.
net bit rate <= gross bit rate
eg. 4B5B coding (4bit over 5bit) (Ethernet):
100Mb/s net, 125Mb/s gross bit rate

8. Units of information transfer
Symbol rate:
symbols transferred per unit time
unit: baud (Bd) = symbol per second
only in binary coding!: 1baud=1b/s
non binary:
one symbol can be decoded to binary with more than one bits
used eg. in modulations such as QAM,QPSK
coding efficiency: bit/s per baud

9. Units of information transfer

10. Channel capacity (C)
the upper limit of the gross bit rate of a noisy channel, while keeping an arbitrarily small bit error rate (without using error correction codes)
in principle, a noiseless channel could have infinite bit rate, but as noise increases, the useful bit rate decreases
see: Shannon-Hartley (later)

11. Bandwidth More meanings in vernacular:
actual transfer rate of information (bitrate R)
channel capacity C (maximum of R)
spectral bandwidth (B) (we'll use this)
the bandwidth of the frequency spectrum of the signal transmitted
can be -3dB bandwidth, but also can be measured btw zero points..
These are related, but not the same!
Eg. for same bit rate, B will be dependent on coding/modulation

12. Bandwidth
Nyquist-Hartley-Shannon

13. Bandwidth Eg. binary baseband signal
N=1; max 2 (b/s) / Hz7
For double side band (DSB) modulations, only half of bandwidth counts
eg. 16QAM: N=4; max 4 (b/s) / Hz

14. Examples for baseband coding spectra
(more about it in next chapter

15. Noise, interference, error
often: additive white gaussian noise (AWGN)
interference
EMI (electromagnetic int.)
crosstalk (between wires/signals)
ISI (inter-symbol int.)
distortion
SNR: Signal to Noise Ratio
SINR: Signal to Noise and Interference Ratio

16. Distortion "linear distortion" "nonlinear distortion":
doesn't introduce new frequency (spectral) components
modifies existing spectral components
(of course the relation of input and output signal is still nonlinear, but it can be achieved by linear circuits, eg. filters)
"nonlinear distortion":
introduces new spectral components
made by circuits with nonlinear characteristics
eg. diode/transistor, limiting circuit

17. Bit Error Ratio (BER)
Bit Error Probability (theoretical probability of a bit error)
BER is converging to BEP (see prob.calculus)
BER used in practice is max 50%
for a totally random channel
question: why?

18. Bit Error Ratio (BER) BER is made worse by: Modify by (possibly):
noise, interference, distortion
Modify by (possibly):
amplitude, bit time, baseband coding, modulation
error correcting codes

19. Eb/N0: energy per bit to noise power spectral density ratio
additív gaussi fehérzajjal definiálva
Eb/N0: energy per bit to noise power spectral density ratio
Ratio of the power in one bit/s, divided by the noise power in 1Hz bandwidth
for different modulations

20. Number of symbols
Number of symbols available (in "alphabet"): M (which is often a power of two, ie. M=2^N)
remember: one symbol is transmitted in a "unit time"
If M (N) is increased:
spectral efficiency (R/B) is increased,
ie. a larger bit rate can be achieved on the same spectral bandwidth
the distance between levels (be it amplitude or frequency or phase) will be smaller, thus it will be more sensitive to noise
see the BER graph: if M is smaller, a lower Eb/N0 is needed to achieve the same BER

21. Eye diagram (In this case):
a binary signal is seen on the oscilloscope, with the different bit values superimposed
if the noise is small, the "eye" is open, the levels can be easily distinguished

22. Constellation diagram
Complex plane representation of a QAM (amplitude and phase modulation)
Each spot is a level or symbol– ie. we have 4 symbols (M=4)
If there is no noise, the spots are point-like.
If the noise is too large, the spots will merge/overlap

23. Shannon-Hartley theorem
Relationship between channel capacity,bandwidth and signal-to-noise ratio

24. Shannon-Hartley theorem
If no noise: S/N is infinite => capacity is infinite even if bandwidth is finite
remember: if B constant, bit rate can be increased by using larger M (alphabet)
If noise increased: capacity decreased
as in earlier, we mean that the bit error rate is to be kept arbitrarily low

25. Shannon-Hartley theorem
In noisy channel:
Combined with Hartley's theorem, we can get an estimation on the max number of levels (symbol) usable (ideally):

26. Entropy Entropy according to Shannon:
X: stochastic variable with possible outcomes {x1...xn}, with probability distribution P(X) with values P(xi).
Here the xi are the codewords (symbols) with P(xi) probability of occuring in the message.
Entropy is practically about the information content of a message.
If all symbols have the same probability, then entropy is maximal.

27. Entropy Entropy tells the randomness, the unexpectedness of a message.
Eg. N=1, (M=2 symbols), with equal probability (eg. generated by coin flipping).

28. Entropy Eg. N=2, (M=4 symbols), equal probability.
Entropy is 2 shannon (bit). This is correlating with the fact that in an alphabet of 4 symbols, each symbol can be converted to a 2bit binary representation.
In a binary code with equal prob., the entropy is 1 (see prev.page).
With equal probabilities: H=N

29. Entropy
Eg. N=2, (M=4 symbols), not equal probability (one symbol more probable than the others).
The entropy is 1,36 shannon (bit), which is smaller than the previous case. This can interpreted as saying, that if symbol x1 arrives, it is the most "expected", thus conveys less information than the others.
For example if one symbol has probability=1 (certain outcome) then thus the others have to be 0, then entropy is 0, because we are certain that we'll always receive the one symbol with P=1, thus no news is gained.

30. Question?
Randomness as a measure of information can be problematic in practice
Eg. how random is the following series? How much can it be compressed?

31. Lossless compression
If entropy is smaller than N, there is a possibility of lossless compression. The new code created can not have an average code word length longer than the entropy of the message. (Shannon's source coding theorem, 1948)
There exists no such compression algorithm, that would work on all messages (ie. make them smaller)
a simple proof: try compressing an already compressed message
There are different methods used for different message types, eg. text, image, executable files, video, music etc.

32. Lossless compression using binary representation of codes:
variable length code: we associate target symbols (codewords) of different length (number of bits) to each source symbol
prefix-free: no codeword starts with another codeword
eg. {0;01;10} set is not prefix-free, because 01 starts with 0
but {0;10;110;111} is prefix-free

33. Compression - Huffman coding
prefix-free, variable length code
it has the minimal average code word length
given: the source symbols with their probability (or relative frequency of occurance) (this is also called weight here)

34. Compression - Huffman coding
Write down the source symbols and their weigths, in order of weigths
Pair the two smallest weigths and connect them to a new node in a tree structure
Repeat until there is only one node in the top row (the root)
Starting from the root, new codewords are assigned to the nodes (see next page)

35. Compression - Huffman coding

36. Question For example we have a binary code with: P(0)=0,9 P(1)=0,1
Entropy is 0,47 so we feel we could achieve good compression
Huffman coding, as shown previously, can not be directly applied
What to do?

37. Compression
LZ algorithm

38. Sound/voice coding base: lossless: lossy: PCM, ADPCM FLAC, MLP
MPEG Layer III (MP3)
GSM

39. Voice coding FLAC: Free Lossless Audio Codec Linear predictor based
The next sample is approximated as a linear function of previous samples
Transmits the coefficients of the linear functions and the error (difference of approx. and real sample)
Compression ratio of 40-50%
from: dr. Tóth Zoltán: Médiakommunikáció: Digitális rádiós információ és műsorszórás diasor

40. Voice coding Lossy compression:
utilizes the knowledge about human hearing (psychoacoustics)
eg. a louder sound can "mask" another one
eg. two slightly different frequencies can not always be perceived to be separate
also the stereo channels are often very similar, allowing to encode the difference only for one channel

41. Redundancy Eg. human languages are redundant
Hwo stnarge we cna raed tihs.
Evn tis txt cn b undrstod, if hrdly.
Redundancy also means possibility of compression.
If we keep redundancy, it gives:
some protection against noise, interference / the ability to correct errors

42. Error detection and correction
Error detecting codes
message has to be sent again
need a two-way channel!
Error correcting codes (ECC)
message doesn't have to be sent again
Forward Error Correction (FEC): called forward, because no talk-back (re-send) is needed

43. Error detection and correction
Generally, methods for detecting or correcting errors will increase the total amount of data transmitted for the same amount of actual information. That is, the gross bit rate is greater than the net bit rate.

44. Error detection: parity
Simplest error detection method: parity bit (for binary codes)
Plus one bit at the end of the message (or frame, usually a byte) will make the total number of ones either odd or even.
We can detect the error (in binary: inversion) of an odd number of bits.
Can not correct, the message/frame/byte has to be re-sent.
Eg. ASCII was originally 7b code and 8.bit was the parity.
Other apps: SCSI,PCI,cache, UART, etc.

45. Error correction: repetition code
Send each bit (or message/frame) n times.
If not all received are the same, make a majority decision (it helps if n is odd).
Net bit rate is 1/n times the gross bit rate.
Eg. 5b version: FlexRay automobile bus system.
Also used in space probes.
Similar method for hardware: Triple Modular Redundancy, TMR
storages (eg. some ECC RAM) – total storage capacity changes
on board computer and other hw multiplied (eg. spacecraft)
homework: create a majority logic gate (it creates the majority decision)

46. Error correction: Hamming code
For every n bit of message, create an m+n bit long code word
eg. (7,4) code: for every 4 bits, add +3 bits
detects 1 or 2 bits of error
or corrects 1 bit of error
used eg. in ECC RAM

47. Error correction – burst error
Burst error: if a larger group of neighbouring bits are involved
esp. in radio or optical comms.
Most methods can only correct a few bits of error in a larger frame.
=>interleaving: the message is broken into smaller parts, which are re-ordered before transmission. After reception, order is restored, thus groups errors are broken into more but smaller error groups.

48. Encryption
Coding can be thought of as a one-to-one relation btw two sets of symbols
If someone doesn’t know the second set of symbols or the relationship, then it is also encryption.

49. Encryption terms Encryption: the act of making a message secret
Cryptography: the science of encryption
Plaintext: the source message, not encrypted
Cleartext: info stored/sent without encryption
Ciphertext: encrypted message

50. Encryption Possible methods Encrypting the contents of the message
this will presented first
Hiding the message or the act of communication

51. Encryption
In practice, we have to keep in mind that most methods can be decyphered if given enough time.
We have to choose a method such that the decyphering time is longer than the actuality of the message, or the cost of decyphering is larger than it is worth.
For electronic communication, usage of special characters, symbols is not applicable, so we use methods that relate eg. latin characters to latin characters, or more correctly, binary numbers to binary numbers.

52. Encryption Suppose encrypting human text.
Simple relation of latin-to-latin characters:
shift all characters by same number: A->C; B->D; Z->B, etc
almost trivial to decypher
relate randomly
just a little bit harder

53. Encryption Decyphering simple codes:
Find out the language (eg. from knowing the sender and receiver, or trying – it’s a finite set)
Need a table of character relative frequency of that language
Need a dictionary of that language
Often start with 1..2 letter words.

54. Encryption Make the previous methods a little bit more complicated:
code the „space” character also into something else
use multi-language text, make some spelling errors

55. Encryption
If we use the same method for a longer time (for many messages), or use longer messages, the chance of decyphering will be close to 1.
Therefore we need a method where the encryption process is split into two: a fix algorithm and a changing part (the cryptographic key)

56. Encryption
Security through obscurity: when we trust the algorithm to be unknown – don’t!
With keyed techniques, the knowledge of the algorithm without the key is not enough to decypher.

57. Encryption To defeat frequency analysis:
Instead of 1-to-1 relation, each letter (symbol) in the source will be related to another based on its position in the text.
Thus the same letter can be coded to different letters each time.
This relation is accomplished using the key, which can be a series of letters or numbers.

58. Encryption
Example:
Write the message in a row, under it the key, then use some algorithm on letters (or numbers) in one column to form a third one, write these in the third row to create the cyphertext.
If the key is shorter than the message, copy it enough times after itself. This of course weakens the method.

59. Encryption Example: Source: „ez a szoveg titkositando”
Key: „a kulcsmondat”
Method: sum of letters modulo 27 (space=1, a=2, b=3, etc)
Result: „g lwmwsbktlbnkuwjemmocst”

60. Encryption One time pad (OTP): key used only onces
G.S.Vernam : XOR method one time pad, 1919.
check: XOR can be used both ways (to encrypt and decrypt)
OTP can be theoretically unbreakable if:
key is really only used once (for one message)
key is at least as long as the message
key is random
Please note that purely with software you can’t create true random series
you need some hw source (eg. thermal noise, tunnel effect, radioactive decay)

61. Part of a key book from Germany WWII, using Enigma

62. Encryption Decyphering OTP code book is acquired keys are re-used
key is not random
see: Venona project, Lorenz, Enigma

63. Public key cryptography
A big problem with OTP is the sending of the pair of the code book to the recipient. Especially hard if you want to send message to someone you didn’t meet before.
To solve this, Public-Key Cryptography was invented.

64. Public key cryptography
Everyone has two keys: a public and a private.
Aims:
to identify the sender (digital signature)
to check if message was tampered with
to encrypt message
these two can be done alone or together
solve problem of sending keys
public key can be sent on unencrypted channel

65. Public key cryptography
Method of sending message:
Use receiver’s public key to encrypt message.
Receiver can decrypt it using her private key.
This is called asymmetric encryption (different key need to encrypt and decrypt)
hybrid cryptosystems: in practice often a traditional symmetric key is encrypted using the above method and rest of message is traditional (because asymm. encryption needs more calc.resources)
Keep your private key secret. Public key is found in „phone books”.

66. Public key cryptography
To check identity of sender or whether message was altered by third party:
sender creates a „hash” (a short block made from the message using special algorithm), which is appended to the end of the message (encrypted of course)
receiver also calculates hash and compares it to the sender’s (decrypted using the sender’s public key)

67. Public key cryptography
Problem:
Identity: we can only know that it was sent by someone with a given public key; but we can’t be sure of her real identity
solution trials:
public-key infrastructure (PKI), web of trust (decentralization)

68. Public key cryptography
Mathematical foundation (most usual): calculation of prime factors
It is very easy (quick) to multiply two very large prime numbers.
It is very hard (very slow) to find the prime factors from the result.
Ie. it is an asymmetric algorithm

69. Public key cryptography
Calculation resources needed
Let „n” be the „size” of the input. (Think about the number of cities in the travelling salesman problem.)
If the algorithm to solve the problem needs running time proportional to a polinomial function of n then this problem is member of P class of problems.
It is supposed that there exists a class of problems that can only be solved much slower (eg. exponential – polinomial is very slow, but exp is much slower). If n is large, these problems can take millions of years with the best computers. This class is the NP class.
Such a problem is prime factorization. (Interesting fact: if one NP full problem is solved in P time, then all can be solved.)
If a faster way of solving NP problems is found, the world’s cryptography (think banking, military, company data etc) would crumble? Quantum computers might give a solution...
Millennium Prize Problems
- 1millio dollár jutalom, közte a NP=P probléma

70. Public key cryptography
a method :RSA
m: message; n,e: public key of receiver, d: private key of receiver; c: encrypted message
coding:
decoding:

71. Encryption Other methods using code words hiding of message
steganography
communication below noise level (spread spectrum)
misguiding – hide the true message by sending a more visible false message

72. Encryption code words – these have previously agreed meaning
ie. Navajo codespeakers in WWII: language is known by only few people, and used codewords (ie. names of animals for ships etc)
hard to decode if sparingly used, esp. if more codewords have similar meaning
Often the codewords are put into a sentence, the rest of the sentence having no real meaning, just for misguiding.