1. Foundations of Algorithm
算法基础
Foundations of Algorithm

2. Outline
What is algorithm?
Time complexity analysis
Divide and Conquer

3. What is Algorithm?

4. What is Algorithm?

5. What is Algorithm?

6. What is Algorithm?
Pseudocode

7. Problem : A question to which we seek an answer. Answer Definition:
Algorithm_Lecture
Problem : A question to which we seek an answer.
Answer
Definition:
A step-by-step procedure → Algorithm
(Eg)
problem : Determine whether
instance :
answer → yes
Algorithm1 : Sequential Search
Procedure seqsearch ( n : integer; S : array[1..n] of keytype
x : keytype; var location : index);
begin
location := 1;
while location ≤ n and S[location] ≠ x do
location := location + 1;
if location > n then location := 0;
end; 5 7 8 10 11 13
Digital Media Lab.

8. Algorithm2 : Binary Search
Procedure binsearch ( n : integer; S : array[1..n] of keytype;
x : keytype; var location : index);
var low, high, mid : index;
begin
low = 1; high = n; location = 0;
while low ≤ high and location = 0 do
mid = ( low + high ) / 2;
if x = S[mid] then location = mid;
else if x < S[mid] then high = mid – 1;
else low = mid + 1;
end;

9. Algorithm2 : Binary Search5 7 8 10 11 13
middle 8
11>8 10 11
middle 11
Find!

10. Algorithm2 : Binary Search
Procedure binsearch ( n : integer; S : array[1..n] of keytype;
x : keytype; var location : index);
var low, high, mid : index;
begin
low = 1; high = n; location = 0;
while low ≤ high and location = 0 do
mid = ( low + high ) / 2;
if x = S[mid] then location = mid;
else if x < S[mid] then high = mid – 1;
else low = mid + 1;
end;
2 different techniques for same problem → Efficiency?
time :
space :

11. Fibonacci Sequence Algorithm 1 (Recursion)
function fib (n : integer) : integer;
begin
if n ≤ 1 then fib=n
else fib = fib(n-1) + fib(n-2);
end; n
#comp(Seq. Srch)
#comp(Bin. Srch)
128 8
1,024 11
1,048,576 21
4,294,967,296 33

12. proof : Induction on n Theorem <Induction Basis> n = 2 & n = 3
<Induction Hypothesis>
Suppose for all m such that 2 ≤ m < n
T(m) > 2m/2
<Induction Step>
T(n) = T(n-1) + T(n-2) + 1
> 2(n-1)/2 + 2(n-2)/2 + 1
> 2(n-2)/2 + 2(n-2)/2
= 2·2(n-2)/2 = 2n/2

13. Algorithm 2 (Iteration)
function fib2 (n : integer) : integer;
var f : array[0..n] of integer;
i : index;
begin (n+1) terms for fib2(n)
f[0] = 0;
if n > 0 then
f[1] = 1;
for i = 2 to n do
f[i] = f[I – 1] + f[I – 2]
fib2 = f[n]
end; n
(n+1)
Iterative Alg
Recursive Alg 40
41 ns
1048 micro s. 60
61 ns
1 s 80
81 ns
18 min
100
101 ns
13 days
120
121 ns
36 years
160
161 ns
3.8 ×107 years
200
201 ns
4 ×1013 years

14. Algorithm Design : techniques
(D-&-C, Dynamic Programming, Greedy, Backtracking,
Branch-&-Bound, … )
Several algorithms for the same problem.
“Which is the most Efficient algorithm?”
=> Algorithm Analysis
♧ Note : problem analysis
(solvable, unsolvable, NP-Complete, … )
=𝑎×𝑏?
3607×3803= ?

15. Time complexity Analysis
Actual CPU time ×
# instructions × (programmers, languages, …)
→ some independent measure !
Condition: a function of input size n
n+1, 2n/2, log n …
Time complexity analysis
Determine input size
Choose the basic operation
Determine how many time the B.O. is done for each value of the input size

16. Time complexity
Space complexity (memory)
Complexity function
Note : 2 algorithms for the same prob.
1000·n , n2
Which one is faster? (or better? )
n2 > 1000·n if n > 1000
Threshold : 1000
But, theoretically 1000·n is better
Correctness analysis, Correctness proof

17. Order 1000·n is better than 0.01·n2 if n > 100,000
Linear-time algorithm quadratic-time algorithm
Quadratic term eventually dominates
“Throw away” low-order terms
“ algorithm” or “quadratic algorithm.” n
0.1n2
0.1n2+n+100 10
120 20 40
160 50
250
400
100
1000
1200
100000
101100

18. Digital Media Lab.

19. Digital Media Lab.

20. Digital Media Lab.

21. Def) (“Big-Oh of ƒ of n”)
but
“asymptotic upper bound” : big-Oh
O(n2) “at least
as good as”
3lgn+8, 4n2
5n+7, 6n2+9
2nlgn, 5n2+2

22. Def) (“Omega of ƒ(n)”) but Ω(n2) “at least as bad as” 4n2 4n3+3n2
5n2+2n 2n+4n .

23. Def) (“Theta of ƒ(n)”, “order of ƒ(n)”)
(Eg) Prove that
(proof by contradiction) Assume that
Then
This implies
Since is a constant, this can never be True. → Contradiction
3lgn n2
5n n2+9
2nlgn n2+2n
4n3+3n2
6n6+n4
2n+4n

24. Def) (Small Oh of ƒ(n)) (Eg) Let be given Choose Then
(Proof by contradiction) Assume
Let Then must exist

25. o(n2) “asymptotically negligible”
Theorem : if , then
(proof) Because
Next, we prove
by contradiction.
if , then
Because
4n3+3n2
6n6+n4
2n+4n
3lgn n2
5n n2+9
2nlgn n2+2n
o(n2) “asymptotically negligible”

26. In practice, exact time comp. → not easy or : enough Note : instead of
Theorem
(Eg) , because

27. (Eg) , because
(Eg)
(i) , Trivial
(ii) , Let be big enough

28. Theorem (L’Hôpital Rule)
(Eg)

29. Divide-and-Conquer (i) Divide into ≥ 2 smaller instances
(ii) Solve each instance (Recursively)
(iii) Combine the subsolutions.
(Eg) merge sort, Quick sort, …
Top-down : more natural
iteration : faster

30. Binary Search : locate x in a sorted array.
Strategy (or approach)
If x equals S[mid], return with mid.
1. Divide into 2 subarrays.
If x < s[mid], left subarray. Right subarray otherwise.
2. Conquer(Solve) the chosen subarray.
Unless sufficiently small, use recursion.
3. Obtain the solution from the subsolution.
(Eg) x= S :
Choose L.S.A
Because x<20
mid
Choose R.S.A
Because x>13
mid
14 18
mid 18
return (mid)
mid

31. function location (low, high : index) : index;
var mid : index;
begin
if low > high then location = 0;
else mid = (low + high) / 2;
if x = S[mid] then location = mid;
else if x < S[mid] then
location = location(low, mid – 1);
else location = location(mid + 1, high);
end;
n, S, x : global variables. Why? (ugly)
In implementation of recursive routine,
call-by-value in each rec. call.
unchanging variables : parameters ×

32. Recursion : more natural, concise, clear, …
Call-by-address? No. Confusing.
Another technique – good!
procedure binsrch2 (n : integer;
S : array [1..n] of keytype;
x : keytype) : index;
var locationout : index;
{ function location is defined here }
begin
locationout = location( 1, n );
end;
Recursion : more natural, concise, clear, …
iteration : faster, save memory space.
(because of stack manipulation)
stack depth (recursion depth) :

33. No every-case time complexity
Best-case :
Avg-case, worst-case :
Worst-case time complexity analysis
Basic operation : element comparison.
( x : S[mid] ) why?
Input size : n (size of array S)
(case 1) n is a power of 2

34. (case 2) n is not a power of 2

35. Merge Sort : 2-way mergesort
Note : k-way mergesort (k > 2)
Strategy (or approach)
1. Divide into 2 subarrays of same size
2. Conquer (solve) each subarray. (sort)
Unless sufficiently small, use recursion.
3. Combine the subsolution : merge
(Example)
1. Divide the array :
2. Sort each subarray :
3. Merge the subarrays :

37. procedure mergesort ( n : integer;
var S : array [1..n] of keytype);
const h=n/2;
m=n-h;
var u : array [1..h] of keytype;
v : array [1..m] of keytype;
begin
if n > 1 then {
copy S[1] ~ S[h] to u;
copy S[h+1] ~ S[n] to v;
mergesort ( h, u );
mergesort ( m, v );
merge ( h, m, u, v, S ); }
end;

38. procedure merge ( h, m, u, v, S ) var i, j, k ; index; begin
while i ≤ h and j ≤ m do {
if u[ i ] < v[ j ] then {
S[ k ] = u[ i ]; i++ }
else {
S[ k ] = v[ j ]; j++
k++
if i > h then copy v[ j..m ] to S[ k..h+m ];
else copy u[ i..h ] to S[ k..h+m ];
end;
u v
i j
S : k

39. worst-case time comp. Analysis of mergesort.
Basic op. : element comp. In merge
Input size : n (size of S)
(i)
(ii)

40. In-Place Sort “Only constant extra space.” extra memory space (why?)
Reduce to n ? Yes!
produce mergesort2 ( low, high : index);
var mid : index;
begin
if low < high then {
mid = ( low + high ) / 2;
mergesort2( low, mid );
mergesort2( mid + 1, high );
merge2( low, mid, high ); }
end;

41. procedure merge2 ( low, mid, high : index ) ;
begin
i = low; j = mid + 1; k = low;
while i ≤ mid and j ≤ high do {
if S[ i ] < S[ j ] then {
U[ k ] = S[ i ]; i++ }
else {
U[ k ] = S[ j ]; j++
k++
if i > mid then copy S[ j .. high ] to U[ k .. high ];
else copy S[ i .. mid ] to U[ k .. high ];
move U[ low .. high ] to S[ low .. high ];
end;

42. Quick Sort – Hoare (1962) Strategy (approach) (Eg)
Choose a pivot item (randomly)
Partition into 2 subarrays.
Sort each subarray recursively.
(Eg)
1. Partition.
2. Sort subarrays recursively.
pivot item

43. Quick Sort – Hoare (1962)

44. procedure quicksort ( low, high : index) ;
var pivotpoint : index;
begin
if high > low then {
partition( low, high, pivotpoint );
quicksort( low, pivotpoint – 1 );
quicksort( pivotpoint + 1, high ); }
end;
n, S : parameters ×
main : quicksort (1, n );

45. Procedure partition ( low, high : index;
var pivotpoint : index );
var i, j : index;
pivotitem : keytype;
begin
pivotitem = S[ low ]; j = low;
for i = low + 1 to high do
if S[ i ] < pivotitem then {
j++; exchange S[ i ] & S[ j ]; }
pivotpoint = j;
exchange S[ low ] and S[ pivotpoint ]
end;

46. In-place? Yes. If we ignore stack space.
Stable? No. Why?
In-place? Yes. If we ignore stack space.
(Every-case) Time complexity Analysis of Partition
Basic op. : element comp.(S[i]: pivotitem)
Input size : n=high-low+1
T(n)=n-1
Worst-case t.c. Analysis of Quicksort
T(n)=T(0) + T(n-1) + n-1
T(n)=T(n-1)+(n-1) for n>0
T(0)=0
T(n)= T(n-1)+n-1=T(n-2)+(n-2)+(n-1)=…

47. Strassen’s Matrix Multiplication ㅡ1969
# multiplication + # additions :
Better algorithm? Asymptotically yes.

48. Generally, A, B, C : n×n matrices

49. Summary What is algorithm? Time complexity analysis Divide and conquer
A step-by-step procedure
Time complexity analysis
Divide and conquer

50. Coin Change Problem Usable coin 968? G r e e d y 500 100 10 50 5 1 5004 1 1 1 3
G r e e d y

51. X Coin Change Problem Problem 12? Dynamic Programming Greedy (3)
Best : 8 6 2 1 X 8 2 2 6 6
Dynamic Programming

52. { { } Divide and Conquer N[k] : #min coins for $k 0 if k == 0
N[k] = ∞ if k < 0
min if k > 0 { { }
N[k - 1],
N[k - 2],
N[k - 6],
N[k - 8]
+ 1