1. Algorithms: Design and Analysis
, Semester 2,
1. Running Time of Programs
Objective
to introduce the Big-Oh notation for estimating the worst case running time of programs

2. Overview Running Time: T() Big-Oh and Approximate Running Time
Calculating Big-Oh Directly
Analyzing Function Calls
Analyzing Recursive Functions

3. 1. Running Time: T() What is the running time of this program?
void main() { int i, n; scanf("%d", &n); for(i=0; i<n; i++) printf("%d"\n", i); }
continued

4. Counting Instructions
Assume 1 instruction takes 1 ms
n value
no. of loops
Time (ms) 1 3
1,000
3,000
1,000,000
3,000,000

5. There is no single answer!
the running time depends on the size of the n value
Instead of a time answer in seconds, we want a time answer which is related to the size of the input.
A simple way of writing the running time is:
T(n) = 3n
continued

6. More generally: running time constant * n size of n
running time T(n) = constant * n
running time is linearly related to the input size
running
time
constant * n
size of n

7. Running Time Theory A program/algorithm has a running time T(n)
n is some measure of the input size
T(n) is the largest amount of time the program takes on any input of size n
T(n) is the worst running time
not always accurate, but easy to calculate
Time units are left unspecified.

8. 1.1. Kinds of Running Time Worst-case: (we use this usually)
• T(n) = maximum time of algorithm on any input of size n.
- one possible value
Average-case: (we sometimes use this)
• T(n) = expected time of algorithm over all inputs of size n.
- this approach needs info about the statistical distribution (probability) of the inputs.
- e.g. uniform spread of data (i.e. all data is equally likely)
Best-case: (don't use this, it's misleading)
• e.g. write a slow algorithm that works fast on specially selected input.

9. 1.2. T(n) Example
Loop fragment for finding the product of all the positive numbers in the A[] array of size n:
(2) int prod = 1; (3) for(j = 0; j < n; j++) (4) if (A[j] > 0) (5) prod = prod * A[j];
Count each assignment and test as 1 “time unit”.

10. Convert 'for' to 'while'
The while-loop is easier to count (and equivalent to the for-loop):
int prod = 1; // 1
int j = 0; // 1
while (j < n) { // 1 for the test
if (A[j] > 0) // 1
prod = prod*A[j]; // 1 + 1
j++; // 1 }
What about
counting the
loop?
We assume that 1 instruction
takes 1 "time unit"

11. Calculation The for loop executes n times Total time T(n) = 5n + 3
each loop carries out (in the worse case) 5 ops
test of j < n, if test, multiply, assign, j increment
total loop time = 5n
plus 3 ops at start and end
small assign (line 2), init of j (line 3), final j < n test
Total time T(n) = 5n + 3
running time is linear with the size of the array

12. 1.3. Comparing Different T()’s
TB(n) = 2n2
T(n)
TA(n) = 100n
input size n
If input size < 50, program B takes less time.
But for large n’s, which are more common in real code, program B is worse and worse (slower).

13. 1.4. Common Growth Formulae & Names
Formula (n = input size) Name
n linear n2 quadratic n3 cubic nm polynomial, e.g. n10 mn ( m >= 2) exponential, e.g. 5n n! factorial 1 constant log n logarithmic n log n log log n

14. Growth Examples 2n n3 n2
n log n n
log n

15. 1.5. Execution Times
Assume 1 instruction takes 1 microsec (10-6 secs) to execute.
How long will n instructions take?
n (no. of instructions)
n ms 1sec n ms 10ms 1sec 12 days n ms 1sec 16.7 min 31,710yr 2n yr 4*1016 yr 3*10287 yr 3* yr log n
growth formula T()
if n is 50, you will
wait 36 years for an answer!

16. Notes Logarithmic running times are best.
Polynomial running times are acceptable, if the power isn’t too big
e.g. n2 is ok, n100 is terrible
Exponential times mean sloooooooow code.
some size problems may take longer to finish than the lifetime of the universe!

17. 1.6. Why use T(n)?
T() can guide our choice of which algorithm to implement, or program to use
e.g. selection sort or merge sort?
T() helps us look for better algorithms in our own code, without expensive implementation, testing, and measurement.

18. (Wrong) Arguments against T(n)
Some people say:
“Who cares about running time? In a few years, machines will be so fast that even bad algorithms will be fast.”
History shows this argument to be wrong. As machines get faster, problem sizes get bigger.
Most interesting problems (e.g. computer vision, natural language processing) always require more resources
fast algorithms will always be needed
continued

19. Some people say:
"Benchmarking (running programs on a standard set of test cases) is easier."
This is sometimes true, but the benchmarks only give numbers for that particular machine, OS, compiler, computer language.

20. T() is too Machine Dependent
We want T() to be the same for an algorithm independent of the machine where it is running.
This is not true since different machines (and OSes) execute instructions at different speeds.
Consider the loop example (slide 10)
on machine A, every instruction takes 1 "time unit"
the result is TA(n) = 5n + 3

21. The for loop executes n times
On machine B, every instruction takes 1 "time unit" except for multiplication, which takes 5 "time units".
The for loop executes n times
each loop carries out (in the worse case) 5 ops
test of j < n, if test, multiply, assign, j increment
total loop time = 9n
plus 3 ops at start and end
small assign (line 2), init of j (line 3), final j < n test
Total time TB(n) = 9n + 3
running time is linear with the size of the array

22. TA() = 5n + 3 and TB() = 9n +3
These are both linear equations (which is good), but the constants are different (which is bad)
We want a T() notation that is independent of machines.

23. 2. Big-Oh and Running Time
Big-Oh notation for T(n) ignores constant factors which depend on compiler/machine behaviour
that's good
Big-Oh simplifies the process of estimating the running time of programs
we don't need to count every code line
that's also good

24. The Big-Oh value specifies running time independent of:
machine architecture
e.g. don’t consider the running speed of individual machine operations
machine load (usage)
e.g. time delays due to other users
compiler design effects
e.g. gcc versus Visual C

25. Example When we counted instructions for the loop example, we found:
TA() = 5n + 3
TB() = 9n + 3
The Big-Oh equation, O(), is based on the T(n) equation but ignores constants (which vary from machine to machine). This means for both machine A and B:
T(n) is O(n) we say "T(n) is order n"

26. More Examples 10n2+ 50n+100 O(n2) 5n3 + 1 O(n3)
T(n) value Big Oh : O()
10n2+ 50n O(n2)
(n+1) O(n2)
n O(2n)
5n O(n3)
These simplifications have a mathematical reason, which is explained in section 2.2.
hard to
understand

27. 2.1. Is Big-Oh Useful?
O() ignores constant factors, which means it is a more general measure of running time for algorithms across different platforms/compilers.
It can be compared with Big-Oh values for other algorithms.
i.e. linear is better than polynomial and exponential, but worse than logarithmic
i.e. O(log n) < O(n) < O(n2) < O(2n)

28. 2.2. Definition of Big-Oh The connection between T() and O() is:
the T() = f(n) can be written as T(n) is O( g(n) )
this means that g(n) is the most important thing in T()'s f(n) function when n is large
Example 1:
T(n) = 5n + 3 // the f() function is 5n + 3
write as T(n) is O(n) // the g() function is n
Example 2:
T(n) = 9n + 3 // the f() function is 9n + 3
continued

29. More Formally
We write T(n) = f(n) as T(n) is O(g(n)) if there exist constants c > 0, n0 > 0 such that 0  f(n)  c*g(n) for all n  n0.
n0 and c are called witnesses to the relationship: T(n) = f(n) and T(n) is O(g(n) )
In some textbooks this is written as:
f(n) is O(g(n)) // leave out the T(n)

30. What is g(n)?
g(n) is a simple function of n without constants and additional smaller terms
e.g. n, n2, 2n, n log n
Not 5n2, 2n + 4, n3 + n2
Use n2, 2n , n3

31. O-notation as a Graph
O-notation gives an upper bound for a function to within a constant factor. We write T(n)=f(n) as T(n) is O(g(n)) if there are positive constants n0 and c such that at and to the right of n0, the value of f(n) always lies on or below c*g(n).
simple to calculate
and analyse and
bigger
above
complex is

32. Informally, the n2 part is the most important thing in
Example 1
Informally, the n2 part is the most important thing in
the T() function
T(n) = 10n2 + 50n + 100
can be written as T(n) is O(n2)
Why?
f(n) = 10n2 + 50n + 100; g(n) = n2
Witnesses: n0 = 1, c = 160
then f(n) <= c*g(n), n >= 1 so 10n2 + 50n <= 160 n2 since 10n2 + 50n <= 10n2 + 50n n2 <= 160 n2

33. T() and O() Graphed T(n) T(n) is O(n2) (c g(n) == 160n2) above
graphcalc/graphcalc.html
T(n)
T(n) is O(n2)
(c g(n) == 160n2)
above
T(n) = 10n2 + 50n + 10
( f(n) == 10n2 + 50n + 10)
n0 == 1 n

34. Example 2 T(n) = (n+1)2 Why? can be wriiten as T(n) is O(n2)
f(n) = (n+1)2; g(n) = n2
Witnesses: n0 = 1, c = 4
then f(n) <= c*g(n), n >= 1 so (n+1)2 <= 4n2 since n2 + 2n + 1 <= n2 + 2n2 + n2 <= 4n2

35. T() and O() Graphed T(n) T(n) is O(n2) (c g(n) == 4n2) above
T(n) = (n+1)2
(f(n) == (n+1)2
n0 == 1 n

36. Example 3 T(n) = n10 Why? can be written as T(n) is O(2n)
f(n) = n10 ; g(n) = 2n
Witnesses: n0 = 64, c = 1
then f(n) <= c*g(n), n >= 64 so n10 <= 2n since 10*log2 n <= n (by taking logs of both sides) which is true when n >= 64 (10*log2 64 == 10*6; 60 <= 64)

37. n10 and 2n Graphed T(n) T(n) is O(2n) (c g(n) == 2n above T(n) = n10
f(n) == n10
(58.770,
4.915E17) n

38. Is n10 slow? Is a running tgime of n10 slow?
not sure
But the maths shows that T(n) = n10 is about the same as 2n
We know 2n is very slow, and that means that n10 is very slow also

39. 2.4. Some Observations about O()
When choosing an O() approximation to T(), remember that:
constant factors do not matter
e.g. T(n) = (n+1)2 is O(n2)
low-order terms do not matter
e.g. T(n) = 10n2 + 50n is O(n2)
there are many possible witnesses because there are usually many O() graphs that are above the T() equation

40. 2.5. Simplifying O() Expressions
Inside an O() expressions, always drop constant factors and low-order terms.
For example:
T(n) = 3n5+ 10n4 + n
T(n) is O(3n5)
but, T(n) is O(n5) is simpler and tighter
this means that the O() is closer to the T() curve

41. 3. Calculating Big-Oh Directly
Up to now, I have calculated T(n) = f(n) by counting instructions (e.g. see the loop example), and then simplified T(n) to becomes T(n) is O(g(n))
We can calculate the big-oh function, g(), directly, without counting instructions
easier and faster

42. 3. Big-Oh for Programs Data Type Possible Size Measure
First decide on a size measure for the data in the program. This will become the n.
Data Type Possible Size Measure
integer its value string its length array its length

43. 3.1. Building a Big-Oh Result
The Big-Oh value for a program is built up in stages by:
1) Calculate the Big-Oh’s for all the simple statements in the program
e.g. assignment, arithmetic
2) Then use those value to obtain the Big-Oh’s for the complex statements
e.g. blocks, for loops, if-statements

44. Simple Statements (in C)
We assume that simple statements always take a constant amount of time to execute
written as O(1)
this is not a time unit (not 1 ms, not 1 microsec)
O(1) means a running time independent of the input size n
Kinds of simple statements:
assignment, break, continue, return, all library functions (e.g. putchar(),scanf()), arithmetic, boolean tests, array indexing

45. Complex Statements
The Big-Oh value for a complex statement is a combination of the Big-Oh values of its component simple statements.
Kinds of complex statements:
blocks { ... }
conditionals: if-then-else, switch
loops: for, while, do-while
continued

46. 3.2. Structure Trees
The easiest way to see how complex statement timings are based on simple statements (and other complex statements) is by drawing a structure tree for the program.

47. Example: binary conversion
void main() { int i; (1) scanf(“%d”, &i); (2) while (i > 0) { (3) putchar(‘0’ + i%2); (4) i = i/2; } (5) putchar(‘\n’); }

48. Structure Tree for Example
block 1-5 1 5
while 2-4
the time for this
is the time for
(3) + (4)
block 3-4 3 4

49. 3.3. Details for Complex Statements
Blocks: Running time bound = summation of the bounds of its parts.
The summation rule means that only the largest Big-Oh value is considered.
"summation" means 'add'

50. Block Calculation Graphically
O( f1(n) )
summation rule
O( f2(n) )
O( f1(n) + f2(n) fk(n))
In other words:
O( largest fi(n) )
The cost of a block is the cost of the biggest statement.
O( fk(n) )

51. Block Summation Rule Example
First block's time T1(n) = O(n2)
Second block's time T2(n) = O(n)
Total running time = O(n2 + n) = O(n2) the largest part

52. e.g. if statements, switches
Conditionals
e.g. if statements, switches
Conditionals: Running time bound = the cost of the if-test + larger of the bounds for the if- and else- parts
When the if-test is a simple statement (a boolean test), it is O(1).

53. Conditional Graphically
Test
O( max( f1(n), f2(n)) +1 )
which is the same as
O( max( f1(n), f2(n)) )
If Part
Else Part
O( f1(n) )
O( f2(n) )

54. If Example Code fragment:
if (x < y) // O(1) foo(x); // O(n) else bar(y); // O(n2)
Total running time = O( max(n,n2) + 1) = O(n2 + 1) = O(n2)
Usually the cost of
an if-statement is
the cost of the biggest
branch.

55. Loops
Loops: Running time bound is usually = the max. number of times round the loop * the time to execute the loop body once
But we must include O(1) for the increment and test each time around the loop.
Must also include the initialization and final test costs (both O(1)).

56. While Graphically Altogether this is: O( lp(n)*(f(n)+1) + 1 )
which can be simplified to:
O( lp(n)*f(n) )
Test
O(1)
At most lp(n) times around
O( lp(n)*f(n) )
Body
O( f(n) )

57. While Loop Example Code fragment:
x = 0; while (x < n) { // O(1) for test foo(x, n); // O(n2) x++; // O(1) }
Total running time of loop:
= O( n*( 1 + n2 + 1) + 1 )
= O(n3 + 2n + 1) = O(n3)
Usually the cost of
a loop is the cost of the body * the number of loops.
lp(n) = n
f(n) = 1 + n2 + 1

58. Do-While Graphically Altogether this is: O( lp(n)*(f(n)+1) + 1 )
which can be simplified to:
O( g(n)*f(n) )
Body
O( f(n) )
At most lp(n) times around
O( lp(n)*f(n) )
Test
O(1)
Usually the cost of
a loop is the cost of the body * the number of loops.

59. For-loop Graphically O(1) Initialize Test O( lp(n)*(f(n)+1+1) + 1)
which can be simplified to:
O( lp(n)*f(n) )
O(1)
At most lp(n) times around
Body
O( f(n) )
Usually the cost of
a loop is the cost of the body * the number of loops.
Increment
O(1)

60. For Loop Example Code Fragment:
for (i=0; i < n; i++) foo(i, n); // O(n2)
It helps to rewrite this as a while loop:
i=0; // O(1) while (i < n) { // O(1) for test foo(i, n); // O(n2) i++; // O(1) }
continued

61. Running time for the for loop:
= O( 1 + n*( 1 + n2 + 1) + 1 )
= O( 2 + n3 + 2n )
= O(n3)
Usually the cost of
a loop is the cost of the body * the number of loops.
lp(n) = n
f(n) = 1 + n2 + 1

62. Example: nested loops
(1) for(i=0; i < n; i++)_ (2) for (j = 0; j < n; j++) (3) A[i][j] = 0;
line (3) is a simple op - takes O(1)
line (2) is a loop carried out n times
takes O(n *1) = O(n)
line (1) is a loop carried out n times
takes O(n * n) = O(n2)

63. Example: if statement
(1) if (A[0][0] == 0) { (2) for(i=0; i < n; i++)_ (3) for (j = 0; j < n; j++) (4) A[i][j] = 0; } (5) else { (6) for (i=0; i < n; i++) (7) A[i][i] = 1; }
continued

64. The if-test takes O(1); the if block takes O(n2); the else block takes O(n).
Total running time:
= O(1) + O( max(n2, n) )
= O(1) + O(n2)
= O(n2) // using the summation rule

65. 3.4.3. Time for a Binary Conversion
void main() { int i; (1) scanf(“%d”, &i); (2) while (i > 0) { (3) putchar(‘0’ + i%2); (4) i = i/2; } (5) putchar(‘\n’); }
continued

66. Block of 3-4 is O(1) + O(1) = O(1)
Lines 1, 2, 3, 4, 5: each O(1)
Block of 3-4 is O(1) + O(1) = O(1)
While of 2-4 loops at most (log2 i)+1 times
total running time = O(1 * log2 i+1) = O(log2 i)
Block of 1-5:
= O(1) + O(log2 i) + O(1)
= O(log2 i)
why?

67. Why (log2 i)+1 ?
Assume i = 2k
Start 1st iteration, i = 2k Start 2nd iteration, i = 2k-1 Start 3rd iteration, i = 2k-2 Start kth iteration, i = 2k-(k-1) = 21 = 2 Start k+1th iteration, i = 2k-k = 20 = 1
the while will terminate after this iteration
Since 2k = i, so k = log2 i
So k+1, the no. of iterations, = (log2 i)+1

68. Using a Structure Tree block 1-5 O(log2 i) 1 5 while 2-4 O(1) O(1)3 4
O(1)
O(1)

69. Selection Sort Code
void selectionSort(int A[], int n) { int i, j, small, temp; (1) for (i=0; i < n-1; i++) { //outer loop (2) current = i; (3) for(j=i+1; j < n; j++) //inner loop (4) if (A[j] < A[small]) (5) current= j; (6) temp = A[current]; // exchange (7) A[current] = A[i]; (8) A[i] = temp; }}

70. Execution Example inner loops Outer Loop #1 Outer Loop #2
when n-1 elems
are sorted
we can stop!
Outer Loop #1
Outer Loop #2
Outer Loop #3
Outer Loop #4
sorted

71. Selection Sort Structure Tree
for 1-8
block 2-8
for 3-5 2 6 7 8
if 4-5 5

72. If of 4-5 is O(max(1,0)+1) = O(1)
if part
else part
Lines 2, 5, 6, 7, 8: each is O(1)
If of 4-5 is O(max(1,0)+1) = O(1)
For of 3-5 is O( (n-(i+1))*1) = O(n-i-1) = O(n), simplified
Block of 2-8
= O(1) + O(n) + O(1) + O(1) + O(1) = O(n)
For of 1-8 is:
= O( (n-1) * n) = O(n2 - n)
= O(n2), simplified

73. 4. Analyzing Function calls
In this section, we assume that the functions are not recursive
we add recursion in section (5)
Size measures for all the functions must be similar, so they can be combined to give the program’s Big-Oh value.

74. Example Program
#include <stdio.h> int bar(int x, int n); int foo(int x, int n): void main() { int a, n; (1) scanf(“%d”, &n); (2) a = foo(0, n); (3) printf(“%d\n”, bar(a,n)); }
continued

75. int bar(int x, int n) { int i; (4) for(i = 1; i <= n; i++) (5) x += i; (6) return x; } int foo(int x, int n) { int i; (7) for(i = 1; i <= n; i++) (8) x += bar(i, n); (9) return x; }

76. Calling Graph
main
foo
bar

77. Calculating Times with a Calling Graph
1. Calculate times for Group 0 functions
those that call no other user functions
2. Calculate times for Group 1 functions
those that call Group 0 functions only
3. Calculate times for Group 2 functions
those that call Group 0 and Group 1 functions only
4. Continue until the time for main() is obtained.

78. Example Program Analysis
Group 0: bar() is O(n)
Group 1: foo() is O( n * n) = O(n2)
Group 2: main() is
= O(1) + O(n2) + O(1) + O(n)
= O(n2)
bar() in body

79. 5. Analyzing Recursive Functions
Recursive functions call themselves with a “smaller size” argument, and terminate by calling a base case.
int factorial(int n) { if (n <= 1) return 1; else return n*factorial(n-1); }

80. Running Time for a Recursive Function
1. Develop basis and inductive statements for the running time.
2. Solve the corresponding recurrence relation.
this usually requires the Big-Oh notation to be rewritten as constants and multiples of n
e.g. O(1) becomes a, O(n) becomes b*n, O(n2) becomes c*n2, etc.
continued

81. 3. Translate the solved relation back into Big-Oh notation
rewrite the remaining constants back into Big-Oh form
e.g. a becomes O(1), b*n becomes O(n)

82. 5.1. Factorial Running Time
Step 1.
Basis: T(1) = O(1)
Induction: T(n) = O(1) + T(n-1), for n > 1
Step 2.
Simplify the relation by replacing the O() notation with constants.
Basis: T(1) = a
Induction: T(n) = b + T(n-1), for n > 1

83. “Obviously”, the general form is:
The simplest way to solve T(n) is to calculate it for some values of n, and then guess the general expression.
T(1) = a T(2) = b + T(1) = b + a T(3) = b + T(2) = 2b + a T(4) = b + T(3) = 3b + a
“Obviously”, the general form is:
T(n) = ((n-1)*b) + a = bn + (a-b)
continued

84. Replace constants by Big-Oh notation:
Step 3. Translate back:
T(n) = bn + (a-b)
Replace constants by Big-Oh notation:
T(n) = O(n) + O(1) = O(n)
The running time for recursive factorial is O(n). That is fast.

85. 5.2. Recursive Selection Sort
void rSSort(int A[], int n) { int imax, i; if (n == 1) return; else { imax = 0; /* A[0] is biggest */ for (i=1; i < n; i++) if (A[i] > A[imax]) imax = i; swap(A, n-1, imax); rSSort(A, n-1); } }

86. Assume swap() is O(1), so ignore
Running Time
Assume swap() is O(1), so ignore
n == the size
of the array
the
loop
call to
rSSort()
Step 1.
Basis: T(1) = O(1)
Induction: T(n) = O(n-1) + T(n-1), for n > 1
Step 2.
Basis: T(1) = a
Induction: T(n) = b(n-1) + T(n-1), for n > 1
multiple of n-1
continued

87. Solve the relation: General Form: continued
T(1) = a T(2) = b + T(1) = b + a T(3) = 2b + T(2) = 2b + b + a T(4) = 3b + T(3) = 3b + 2b + b + a
General Form:
T(n) = (n-1)b b + a = a + b(n-1)n/2
continued

88. Replace constants by Big-Oh notation:
Step 3. Translate back:
T(n) = a + b(n-1)n/2
Replace constants by Big-Oh notation:
T(n) = O(1) + O(n2) + O(n) = O(n2)
The running time for recursive selection sort is O(n2). That is slow for large arrays.

89. 5.3. Binary Search
int binSrch(char A[], int i,int j, char key) { int k; if (i > j) /* key not found */ return -1; k = (i+j)/2; if (key == A[k]) /* key found */ return k; if (key < A[k]) j = k-1; /* search left half */ else i = k+1; /* search right half */ return binSrch(A, i, j, key); }

90. Execution Example a d f g h w x y 1 2 3 4 5 6 7 A1 2 3 4 5 6 7 a d f g h w x y A
/* find 'h' in A[] */ binSrch(A, 0, 7, ‘h’);
you use binary search
to search for a number in a phone book

91. Another Example 1 2 3 4 5 6
// find '6' in B[] binSrch(B, 0, 6, '6') B

92. n == the range of the array being looked at
Running Time
n == the range of the array being looked at
Step 1.
Basis: T(1) = O(1)
Induction: T(n) = O(1) + T(< n/2 >), for n > 1
Step 2.
Basis: T(1) = a
Induction: T(n) = b + T(< n/2 >), for n > 1

93. Solve the Relation
This time the relation is harder to solve since the precise value of < n/2 > depends on n.
This affects how many calls it takes to get to T(1).
Assume the simple case:
n starts as a power of 2: 2k
continued

94. Evaluate T(n), where n is 2k:
Notice we are not
starting at T(1) in this
example; that's ok.
Evaluate T(n), where n is 2k:
T(2k) = b + T(2k-1) = b + b + T(2k-2) = b + b + b + T(2k-3) = b + b + b b + T(1)
General Form:
T(n) = bk +a
k of these
e.g. T(32) = b + b + b + b + b + 1
continued

95. We assumed that 2k = n, so k = log2 n This means that:
T(n) = b*log2 n + a
Step 3. Replace constants by Big-Oh notation:
T(n) = O(log2 n) + O(1) = O(log2 n)
Running time for binary search is O(log2 n).