2. J. Elder COSC 3101N Thinking about Algorithms Abstractly Lecture 1. Introduction

3. J. Elder COSC 3101N So you want to be a computer scientist?

4. J. Elder COSC 3101N Is your goal to be a mundane programmer?

5. J. Elder COSC 3101N Or a great leader and thinker?

6. J. Elder COSC 3101N Original Thinking

7. J. Elder COSC 3101N Boss assigns task: Given today’s prices of pork, grain, sawdust, … Given constraints on what constitutes a hotdog. Make the cheapest hotdog. Everyday industry asks these questions.

8. J. Elder COSC 3101N Um? Tell me what to code. With more suffocated software engineering systems, the demand for mundane programmers will diminish. Your answer:

9. J. Elder COSC 3101N Your answer: I learned this great algorithm that will work. Soon all known algorithms will be available in libraries.

10. J. Elder COSC 3101N Your answer: I can develop a new algorithm for you. Great thinkers will always be needed.

11. J. Elder COSC 3101N Content: An up to date grasp of fundamental problems and solutions Method: Principles and techniques to solve the vast array of unfamiliar problems that arise in a rapidly changing field The future belongs to the computer scientist who has Rudich www.discretemath.com

12. J. Elder COSC 3101N Course Content A list of algorithms. Learn their code. Trace them until you are convinced that they work. Implement them. class InsertionSortAlgorithm extends SortAlgorithm { void sort(int a[]) throws Exception { for (int i = 1; i < a.length; i++) { int j = i; int B = a[i]; while ((j > 0) && (a[j-1] > B)) { a[j] = a[j-1]; j--; } a[j] = B; }}

13. J. Elder COSC 3101N Course Content A survey of algorithmic design techniques. Abstract thinking. How to develop new algorithms for any problem that may arise.

14. J. Elder COSC 3101N Study: Many experienced programmers were asked to code up binary search.

15. J. Elder COSC 3101N Study: Many experienced programmers were asked to code up binary search. 80% got it wrong Good thing is was not for a nuclear power plant.

16. J. Elder COSC 3101N What did they lack?

17. J. Elder COSC 3101N What did they lack? Formal proof methods?

18. J. Elder COSC 3101N What did they lack? Formal proof methods? Yes, likely Industry is starting to realize that formal methods are important. But even without formal methods …. ?

19. J. Elder COSC 3101N What did they lack? Fundamental understanding of algorithmic design techniques. Abstract thinking.

20. J. Elder COSC 3101N Course Content Notations, analogies, and abstractions for developing, thinking about, and describing algorithms so correctness is transparent

21. J. Elder COSC 3101N Slide Credits Some materials also due to Prof. A. Mirzaian (York). These in turn include some slides created by Prof. S. Rudich of Carnegie Mellon University. The slides for these 12 lectures have been adapted from slides created by Prof. J. Edmonds (York).

22. J. Elder COSC 3101N Please feel free to ask questions!

23. J. Elder COSC 3101N Useful Learning Techniques

24. J. Elder COSC 3101N Read Ahead You are expected to read the textbook and lecture notes before the lecture. This will facilitate more productive discussion during class.

25. J. Elder COSC 3101N Explaining We are going to test you on your ability to explain the material. Hence, the best way of studying is to explain the material over and over again out loud to yourself, to each other, and to your stuffed bear.

26. J. Elder COSC 3101N Day Dream Mathematics is not all linear thinking. Allow the essence of the material to seep into your subconscious Pursue ideas that percolate up and flashes of inspiration that appear. While going along with your day

27. J. Elder COSC 3101N Be Creative Ask questions. Why is it done this way and not that way?

28. J. Elder COSC 3101N Guesses and Counter Examples Guess at potential algorithms for solving a problem. Look for input instances for which your algorithm gives the wrong answer.

29. J. Elder COSC 3101N Refinement: Rudich www.discretemath.com The best solution comes from a process of repeatedly refining and inventing alternative solutions

30. J. Elder COSC 3101N Grade School Revisited: How To Multiply Two Numbers 2 X 2 = 5 A Few Example Algorithms Rudich

31. J. Elder COSC 3101N Complex Numbers Remember how to multiply 2 complex numbers? (a+bi)(c+di) = [ac –bd] + [ad + bc] i Input: a,b,c,d Output: ac-bd, ad+bc If a real multiplication costs $1 and an addition cost a penny. What is the cheapest way to obtain the output from the input? Can you do better than $4.02?

32. J. Elder COSC 3101N Input: a,b,c,d Output: ac-bd, ad+bc m 1 = ac m 2 = bd A 1 = m 1 – m 2 = ac-bd m 3 = (a+b)(c+d) = ac + ad + bc + bd A 2 = m 3 – m 1 – m 2 = ad+bc Gauss’ $3.05 Method:

33. J. Elder COSC 3101N Question: The Gauss method saves one multiplication out of four. It requires 25% less work. Could there be a context where performing 3 multiplications for every 4 provides a more dramatic savings?

34. J. Elder COSC 3101N Odette Bonzo

35. J. Elder COSC 3101N How to add 2 n-bit numbers. ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** +

36. J. Elder COSC 3101N How to add 2 n-bit numbers. *** *** ** ** ** ** ** ** ** ** ** ** *** *** ** ** ** ** ** ** ** ** +

37. J. Elder COSC 3101N How to add 2 n-bit numbers. *** *** ** ** ** ** ** ** ** ** *** *** **** **** ** ** ** ** ** ** ** ** +

38. J. Elder COSC 3101N How to add 2 n-bit numbers. *** *** ** ** ** ** ** ** *** *** **** **** **** **** ** ** ** ** ** ** ** ** +

39. J. Elder COSC 3101N How to add 2 n-bit numbers. *** *** ** ** ** ** *** *** **** **** **** **** **** **** ** ** ** ** ** ** ** ** +

40. J. Elder COSC 3101N How to add 2 n-bit numbers. *** *** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** * * * * + * ** *

41. J. Elder COSC 3101N Time complexity of grade school addition *** *** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** + * ** * **** **** T(n) = The amount of time grade school addition uses to add two n-bit numbers = θ(n) = linear time. On any reasonable computer adding 3 bits can be done in constant time.

42. J. Elder COSC 3101N # of bits in numbers time f = θ (n) means that f can be sandwiched between two lines

43. J. Elder COSC 3101N Is there a faster way to add? QUESTION: Is there an algorithm to add two n-bit numbers whose time grows sub-linearly in n?

44. J. Elder COSC 3101N Any algorithm for addition must read all of the input bits Suppose there is a mystery algorithm that does not examine each bit Give the algorithm a pair of numbers. There must be some unexamined bit position i in one of the numbers If the algorithm is not correct on the numbers, we found a bug If the algorithm is correct, flip the bit at position i and give the algorithm the new pair of numbers. It give the same answer as before so it must be wrong since the sum has changed

45. J. Elder COSC 3101N So any algorithm for addition must use time at least linear in the size of the numbers. Grade school addition is essentially as good as it can be.

46. J. Elder COSC 3101N How to multiply 2 n-bit numbers. X * * * * * * * * * * * * * * * * * * * * * * * * * * * * n2n2

47. J. Elder COSC 3101N I get it! The total time is bounded by cn 2.

48. J. Elder COSC 3101N Grade School Addition: Linear time Grade School Multiplication: Quadratic time No matter how dramatic the difference in the constants the quadratic curve will eventually dominate the linear curve # of bits in numbers time

49. J. Elder COSC 3101N Neat! We have demonstrated that as things scale multiplication is a harder problem than addition. Mathematical confirmation of our common sense.

50. J. Elder COSC 3101N Don’t jump to conclusions! We have argued that grade school multiplication uses more time than grade school addition. This is a comparison of the complexity of two algorithms. To argue that multiplication is an inherently harder problem than addition we would have to show that no possible multiplication algorithm runs in linear time.

51. J. Elder COSC 3101N Grade School Addition: θ (n) time Grade School Multiplication: θ (n 2 ) time Is there a clever algorithm to multiply two numbers in linear time?

52. J. Elder COSC 3101N Despite years of research, no one knows! If you resolve this question, York will give you a PhD!

53. J. Elder COSC 3101N Is there a faster way to multiply two numbers than the way you learned in grade school?

54. J. Elder COSC 3101N Divide And Conquer (an approach to faster algorithms) DIVIDE a problem into smaller subproblems CONQUER them recursively GLUE the answers together so as to obtain the answer to the larger problem

55. J. Elder COSC 3101N Multiplication of 2 n-bit numbers X = Y = X = a 2 n/2 + b Y = c 2 n/2 + d XY = ac 2 n + (ad+bc) 2 n/2 + bd ab cd

56. J. Elder COSC 3101N Multiplication of 2 n-bit numbers X = Y = XY = ac 2 n + (ad+bc) 2 n/2 + bd ab cd MULT(X,Y): If |X| = |Y| = 1 then RETURN XY Break X into a;b and Y into c;d RETURN MULT(a,c) 2 n + (MULT(a,d) + MULT(b,c)) 2 n/2 + MULT(b,d)

57. J. Elder COSC 3101N Time required by MULT T(n) = time taken by MULT on two n-bit numbers What is T(n)? What is its growth rate? Is it θ(n 2 )?

58. J. Elder COSC 3101N Recurrence Relation T(1) = k for some constant k T(n) = 4 T(n/2) + k’ n + k’’ for some constants k’ and k’’ MULT(X,Y): If |X| = |Y| = 1 then RETURN XY Break X into a;b and Y into c;d RETURN MULT(a,c) 2 n + (MULT(a,d) + MULT(b,c)) 2 n/2 + MULT(b,d)

59. J. Elder COSC 3101N Let’s be concrete T(1) = 1 T(n) = 4 T(n/2) + n How do we unravel T(n) so that we can determine its growth rate?

60. J. Elder COSC 3101N Technique 1: (Substitution) Recurrence Relation: T(1) = 1 & T(n) = 4T(n/2) + n Guess: G(n) = 2n 2 – n Verify: Left Hand SideRight Hand Side T(1) = 2(1) 2 – 1 T(n) = 2n 2 – n 1 4T(n/2) + n = 4 [2( n / 2 ) 2 – ( n / 2 )] + n = 2n 2 – n

61. J. Elder COSC 3101N Technique 2: Recursion Tree T(n) = n + 4 T(n/2) T(1) = 1 n T(n/2) = n = T(n) T(1) 1 =

62. J. Elder COSC 3101N n T(n/2) T(n) =

63. J. Elder COSC 3101N n T(n/2) T(n) = n/2 T(n/4)

64. J. Elder COSC 3101N n T(n) = n/2 T(n/4) n/2 T(n/4) n/2 T(n/4) n/2 T(n/4)

65. J. Elder COSC 3101N n T(n) = n/2 11111111111111111111111111111111...... 111111111111111111111111111111111 n/4

66. J. Elder COSC 3101N Level i is the sum of 4 i copies of n/2 i 1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1............. n/2 + n/2 + n/2 + n/2 n n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 0 1 2 i

67. J. Elder COSC 3101N =1  n = 4  n/2 = 16  n/4 = 4 i  n/2 i Total: θ( n log4 ) = θ( n 2 ) = 4 logn  n/2 logn =n log4  1

68. J. Elder COSC 3101N Divide and Conquer MULT: θ (n 2 ) time Grade School Multiplication: θ (n 2 ) time All that work for nothing!

69. J. Elder COSC 3101N MULT revisited MULT calls itself 4 times. Can you see a way to reduce the number of calls? MULT(X,Y): If |X| = |Y| = 1 then RETURN XY Break X into a;b and Y into c;d RETURN MULT(a,c) 2 n + (MULT(a,d) + MULT(b,c)) 2 n/2 + MULT(b,d)

70. J. Elder COSC 3101N Gauss’ Idea: Input: a,b,c,d Output: ac, ad+bc, bd A 1 = ac A 3 = bd m 3 = (a+b)(c+d) = ac + ad + bc + bd A 2 = m 3 – A 1 - A 3 = ad + bc

71. J. Elder COSC 3101N Gaussified MULT (Karatsuba 1962) T(n) = 3 T(n/2) + n Actually: T(n) = 2 T(n/2) + T(n/2 + 1) + kn MULT(X,Y): If |X| = |Y| = 1 then RETURN XY Break X into a;b and Y into c;d e = MULT(a,c) and f =MULT(b,d) RETURN e2 n + (MULT(a+b, c+d) – e - f) 2 n/2 + f

72. J. Elder COSC 3101N n T(n) = n/2 T(n/4) n/2 T(n/4) n/2 T(n/4) n/2 T(n/4)

73. J. Elder COSC 3101N n T(n/2) T(n) =

74. J. Elder COSC 3101N n T(n/2) T(n) = n/2 T(n/4)

75. J. Elder COSC 3101N n T(n) = n/2 T(n/4) n/2 T(n/4) n/2 T(n/4)

76. J. Elder COSC 3101N Level i is the sum of 3 i copies of n/2 i 1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1............. n/2 + n/2 + n/2 n n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 0 1 2 i

77. J. Elder COSC 3101N =1  n = 3  n/2 = 9  n/4 = 3 i  n/2 i Total: θ( n log3 ) = θ( n 1.58.. ) = 3 logn  n/2 logn =n log3  1

78. J. Elder COSC 3101N Dramatic improvement for large n Not just a 25% savings! θ( n 2 ) vs θ( n 1.58.. )

79. J. Elder COSC 3101N Multiplication Algorithms Kindergartenn2 n Grade Schooln2n2 Karatsuban 1.58… Fastest Knownn logn loglogn

80. J. Elder COSC 3101N End