1. Complexity & the O-Notation

2. Computability
So far we talked about Turing Machines that decide languages (solve yes-no problems).
We only cared about making the machine decide the language.
We didn’t care about the machine’s performance.

3. Time Complexity
The time complexity of a machine is the number of transitions it takes on input x in order accept or reject the input x.
The time is counted in terms of the length of the input. The time complexity function is a function t: N → N.

4. Worst and average cases
We say that a Turing machine has worst case time complexity t(n), if for every possible input x of length n the machine needs at most t(n) transitions in order to compute f(x)- decide if x is in the language.
The average case analysis is to consider all the running times on every possible input of length n and take the average.

5. Space Complexity
The space complexity of a Turing machine is very similar to the time complexity.
The idea is exactly the same. Instead of number of transitions we count the number of explored cells.
In the following examples the unexplored cells are shown in white and the explored ones are shown in blue.

6. Example: Put a $ before the input
One solution:
Repeat
Erase the first symbol from the input. Write it last in the output.
Until there is no symbol left in the input. Place a $. 1 1 1

7. Example: Put a $ before the input
One solution:
Repeat
Erase the first symbol from the input. Write it last in the output.
Until there is no symbol left in the input. Place a $. 1 1 1

8. Example: Put a $ before the input
One solution:
Repeat
Erase the first symbol from the input. Write it last in the output.
Until there is no symbol left in the input. Place a $. 1 1 1

9. Example: Put a $ before the input
One solution:
Repeat
Erase the first symbol from the input. Write it last in the output.
Until there is no symbol left in the input. Place a $. 1 1 1

10. Example: Put a $ before the input
One solution:
Repeat
Erase the first symbol from the input. Write it last in the output.
Until there is no symbol left in the input. Place a $. 1 1 1

11. Example: Put a $ before the input
One solution:
Repeat
Erase the first symbol from the input. Write it last in the output.
Until there is no symbol left in the input. Place a $. $ 1 1 1

12. Example: Put a $ before the input
The number of transitions you need for this solution is:
You repeat n times the following procedure (since n is the input length):
Erase the first symbol of the input
Move across the input and the output (about n transitions)
Paste this symbol in the last position of the output
Move across the output and the input (about n more transitions.
So the number of transitions in total is about 2n2

13. Example: Put a $ before the input
The number of explored cells is exactly 2n+1

14. Example: Put a $ before the input
Another solution:
Erase the first symbol. Remember it. Replace it with a $.
Repeat:
Move one cell right. Replace this symbol with the one you remember from the left.
Until the input is consumed (you see a blank space). Put there the last symbol. 1 1

15. Example: Put a $ before the input
Another solution:
Erase the first symbol. Remember it. Replace it with a $.
Repeat:
Move one cell right. Replace this symbol with the one you remember from the left.
Until the input is consumed (you see a blank space). Put there the last symbol. $ 1
remember 1

16. Example: Put a $ before the input
Another solution:
Erase the first symbol. Remember it. Replace it with a $.
Repeat:
Move one cell right. Replace this symbol with the one you remember from the left.
Until the input is consumed (you see a blank space). Put there the last symbol. $ 1 1
remember 0

17. Example: Put a $ before the input
Another solution:
Erase the first symbol. Remember it. Replace it with a $.
Repeat:
Move one cell right. Replace this symbol with the one you remember from the left.
Until the input is consumed (you see a blank space). Put there the last symbol. $ 1 1
remember 0

18. Example: Put a $ before the input
Another solution:
Erase the first symbol. Remember it. Replace it with a $.
Repeat:
Move one cell right. Replace this symbol with the one you remember from the left.
Until the input is consumed (you see a blank space). Put there the last symbol. $ 1
remember 1

19. Example: Put a $ before the input
Another solution:
Erase the first symbol. Remember it. Replace it with a $.
Repeat:
Move one cell right. Replace this symbol with the one you remember from the left.
Until the input is consumed (you see a blank space). Put there the last symbol. $ 1 1

20. Example: Put a $ before the input
The number of transitions you need for this solution is:
Replace the first symbol with a $.
You repeat n times the following procedure (since n is the input length):
Replace the symbol we see with the one that was written in the left cell.
So the number of transitions in total is n+1

21. Example: Put a $ before the input
The number of explored cells is exactly n+1

22. Example: Put a $ before the input
The most efficient solution:
Just move left and place a $ 1 1

23. Example: Put a $ before the input
The most efficient solution:
Just move left and place a $ $ 1 1

24. Example: Put a $ before the input
The number of transitions is 2.
The number of explored cells is 2.
The time and space complexity of this machine doesn’t depend on the input (it is as we say a “constant”). This is a very rare phenomenon! Most of the times we need at least to read all the input (so we need time and space at least n).

25. O-Notation
It is not always easy to count the exact complexity of a Turing Machine.
Furthermore sometimes we are just not interested in finding the exact number of transitions made or cells explored.
In those cases we perform as we say “asymptotic analysis”.

26. Asymptotic Analysis
In asymptotic analysis we completely ignore additive and multiplicative constants.
We also don’t care about small additive terms.
We want to see how the machine performs on large input.

27. Asymptotic Analysis
Imagine that a TM runs in time 2n2+5n steps. For really large n, the term 5n becomes negligible.
For example, for n=1000:
5n steps is equal to 5000 steps while
2n2 steps is as large as steps!!!

28. Asymptotic Analysis

29. Asymptotic Analysis
Also, making a machine run in time n2 rather than 2n2 (twice as fast) might be a good improvement but it is not compared with making it run in time n, which is considered much more efficient!

30. Asymptotic Analysis

31. O-Notation
We say that a function f is O(g(n)) (or that f is upper-bounded by g) if there is a constant c>0 and an integer n0 such that: n0
c∙g f

32. O-Notation
We say that a function f is Ω(g(n)) (or that f is lower-bounded by g) if there is a constant c>0 and an integer n0 such that: n0
c∙g f

33. O-Notation
We say that a function f is Θ(g(n)) (or that f is upper- and lower-bounded by g) if there are constants c1 ,c2 >0 and an integer n0 such that: n0
c1∙g f
c2∙g

34. O-Notation - Properties
If f is upper-bounded by g then g is lower-bounded by f.
f = O(g) then g = Ω(f)
If f is both upper- and lower- bounded by g then it is upper- and lower- bounded by g.
f = O(g) and f = Ω(g) then f = Θ(g).

35. The o- and ω- symbols
We say that a function f is o(g(n)) if for every constant c>0 there is an integer n0 such that:
We say that a function f is ω(g(n)) if for every constant c>0 there is an integer n0 such that:

36. The o- and ω- symbols
Another way to prove that f is o(g) or ω(g) is by using limits.
A function f is o(g(n)) if:
A function f is ω(g(n)) if:

37. The o- and ω- symbols
o and ω have the same relation with O and Ω as < and > have with ≤ and ≥. Θ stands for ≈
If f = O(g) but f ≠ Θ(g) then we say that f = o(g)
Similarly, if f = Ω(g) but f ≠ Θ(g) then f = ω(g)
It is important to understand that if a function f is o(g) this doesn’t mean that f is going to be less than g for every input but that there is some input after which f is always less than g.

38. Prove it formally
Example 1: 10n = Θ(2n) because 10n and 2n are only a constant factor apart. In other words , for c1=1/10, c2=1 and n0 =1, for all n≥1, n ≤ 2n ≤ 10n.
Example 2: 10n = O(n2) because for c=1 and n0=10, for all n≥n0 10n ≤ n2 .

39. Prove it formally
Furthermore 10n is o(n2) because for all c>0 there is a n0 (= 10/c) such that for all n≥n0 , 10n ≤ cn2.
An other way to see this is by taking the limit
The function n2 is considered greater than 10n despite the fact that for some small inputs 10n > n2 (for example for n=7, 70>49)

40. Examples
Example 3: 1000n2 = o(2n). That is because:

41. Input representation, reminder
If the input is just a natural number n, we can represent it in unary, in binary or in decimal… … … 1 1 1 1 1 1
In unary … … 1 1
In binary … …
In decimal 6

42. Input length
If the number is in unary then the space it consumes to represent it is n. … … 1 1 1 1 1 1
In unary

43. Input length
If the number is in binary then the space it consumes to represent it is log2 n. … … 1 1
In binary

44. Input length
If the number is in decimal then the space it consumes to represent it is log10 n. … … 6
In binary

45. Input length
Binary is much more efficient than unary.
log2n = o(n)

46. Input length Decimal is not much more efficient than binary
log2n = Θ(log10n).
There is a property of the logarithms:
logxn = logyn ∙ logxy.
So log2n = log210 ∙ log10n
But log210 is a constant, 3 ≤ log210 ≤ 4.
So for all n≥1, 3∙log10n ≤ log2n ≤ 4∙log10n

47. Input representation for graphs
As we said, we can give a graph as input to a Turing Machine.
To represent a graph we have to list all of its vertices and all of its edges.
If the number of vertices is n and the number of edges is m, the input size will be O(n+m).
The maximum number of edges a graph can have is n choose 2 = n(n-1)/2 = O(n2).
Thus the input size is O(n2).

48. Complexity Classes
The class DTIME(t(n)) contains all those languages L for which there is a DTM that decides L in time O(t(n)) (i.e. performing O(t(n)) steps)
The class DSPACE(t(n)) contains all those languages L for which there is a DTM that decides L exploring O(t(n)) cells in total