Theory of Algorithms: Introduction James Gain and Edwin Blake {jgain | Department of Computer Science University of Cape Town August - October 2004

Objectives lTo define an algorithm lTo introduce:  Problem types  The Process of Algorithm Design  Solution strategies  Ways of Analysing Algorithms lTo cover the structure of the course, including practicals

Definitions lAn algorithm is a sequence of unambiguous instructions for solving a problem  For obtaining a required output for any legitimate input in a finite amount of time  Does not require implementation in software  Not an answer but a method for deriving an answer lHistorical Perspective:  Named after Muhammad ibn Musa al-Khwarizmi – 9 th century mathematician 

Notion of algorithm “computer” problem algorithm input output Each step of the algorithm must be unambiguous The range of inputs must be specified carefully The same algorithm can be represented in different ways Several algorithms for solving the same problem may exist - with different properties

What is an algorithm? lRecipe, process, method, technique, procedure, routine,… with following requirements: 1.Finiteness bTerminates after a finite number of steps 2.Definiteness bRigorously and unambiguously specified 3.Input bValid inputs are clearly specified 4.Output bCan be proved to produce the correct output given a valid input 5.Effectiveness bSteps are sufficiently simple and basic

Example: Sorting lStatement of problem:  Input: A sequence of n numbers  Output: A reordering of the input sequence so that a ´ i ≤ a ´ j whenever i < j lInstance: The sequence lAlgorithms: -Selection sort -Insertion sort -Merge sort -(many others)

Selection Sort Input: array a[1],..,a[n] Output: array a sorted in non-decreasing order lAlgorithm: for i=1 to n swap a[i] with smallest of a[i],…,a[n] for i  1 to n do min  i for j  i+1 to n do if a[j] < a[min] min  j swap a[i] and a[min]

Exercise: Bridge Puzzle lProblem:  4 People want to cross a bridge. You have 17 minutes to get them across lConstraints:  It is night and you have 1 flashlight. Max of 2 on the bridge at one time. All start on the same side  Those crossing must have the flashlight with them. The flashlight must be walked back and forth (no throwing)  People walk at different speeds: person A = 1 minute to cross, person B = 2 minutes, person C = 5 minutes, person D = 10 minutes  A pair walks at the speed of the slower person’s pace lRumour: this problem is given to Microsoft interviewees

Solution: Bridge Puzzle lStart (0 min): A B C D lAB Across (2 min): A B C D lA Back (1 min): B A C D lCD Across (10 min): B C D A lB Back (2 min): C D A B lAB Across (2 min): A B C D lTotal Time = 17 minutes

Extension Exercise lThis is an instance of a problem. How would you generalise it? lCan you derive an algorithm to solve this generalised problem?  Must show the sequence of moves  Must output the minimum time required for crossing  Are there any special cases to watch out for?  Are there any constraints on the input?

Extension Solution lInput: a list a of crossing times for n people, numbered 1, …, n lOutput: total time to cross lStrategy: use 1 & 2 as shuttles and send the others across in pairs for i  2 to n/2 do t  a[2] // 1 & 2 across t  t + a[1] // 1 back t  t + a[i*2] // i*2 & (i*2)-1 across t  t + a[2] // 2 back t  a[2] // 1 & 2 across return t

Extension Problems lThis is an inadequate solution lIt falsely assumes certain inputs lList may not be sorted in ascending order  Sort a ln may not be even numbered  Alter final iteration of loop ln > 3 not guaranteed  Special case for n = 1, 2, 3 lIs not optimal for all inputs, e.g. 1, 20, 21, 22  Can you quantify the nature of these inputs? Suggest an alternative. lFinal solution is left as an exercise. Attempt to make your solution elegant

Fundamentals of Algorithmic Problem Solving lUnderstanding the Problem  Make sure you are solving the correct problem and for all legitimate inputs lAscertaining the Capabilities of a Computational Device  Sequential vs. Parallel.  What are the speed and memory limits? lChoosing between exact and approximate Problem Solving  Is absolute precision required? Sometimes this may not be possible lDeciding on Appropriate Data Structures  Algorithms often rely on carefully structuring the data  Fundamental Data Structures: array, linked list, stacks, queues, heaps, graphs, trees, sets

Fundamentals of Algorithm Design lApplying an Algorithm Design Technique  Using a general approach to problem solving that is applicable to a variety of problems lSpecifying the Algorithm  Pseudocode is a mixture of natural language and programming constructs that has replaced flowcharts lProving an Algorithms Correctness  Prove that an algorithm yields a required result for legitimate inputs in finite time lAnalyzing an Algorithm  Consider time efficiency, space efficiency, simplicity, generality, optimality  Analysis can be empirical or theoretical lCoding an Algorithm

Well known Computational Problems lSorting lSearching lString Processing  String Matching lGraph Problems  Graph Traversal, Shortest Path, Graph Colouring lCombinatorial Problems  Find a combinatorial object - permutation, combination, subset - subject to constraints lGeometric Problems  Closest-Pair, Convex-Hull lNumerical Problems  Solving systems of equations, computing definite integrals, evaluating functions, etc.

Algorithm Design Strategies lBrute force  A straightforward approach to solving a problem, usually directly based on the problem’s statement lDivide and conquer  Divide a problem into smaller instances, solve smaller instances (perhaps recursively), combine lDecrease and conquer  Exploit relationship between the problem and a smaller instance reduced by some factor (often 1) lTransform and conquer  Transform the problem to a simpler instance, another representation or an instance with a known solution

More Algorithm Design Strategies lGreedy approach  Make locally optimal steps which (hopefully) lead to a globally optimal solution for an optimization problem lDynamic programming  Technique for solving problems with overlapping sub- domains lBacktracking and Branch and bound  A way of tackling difficult optimization and combinatorial problems without exploring all state-space lSpace and time tradeoffs  Preprocess the input and store additional information to accelerate solving the problem

How to Solve It: Understanding the Problem lTaken from G. Polya, “How to Solve It”, 2nd edition. A classic textbook on problem solving for mathematics 1.You have to understand the problem.  What is the unknown? What are the data? Is the problem statement sufficient, redundant, contradictory  Draw a figure. Introduce suitable notation  Separate the various parts of the problem. Can you write them down?

Devising a Plan lFind the connection between the data and the unknown. You may be obliged to consider auxiliary problems if an immediate connection cannot be found. You should obtain eventually a plan of the solution. lHave you seen it before? Or have you seen the same problem in a slightly different form? lDo you know a related problem? Do you know a theorem that could be useful? lLook at the unknown! And try to think of a familiar problem having the same or a similar unknown. lCould you restate the problem? Could you restate it still differently? Go back to definitions. lIf you cannot solve the proposed problem try to solve first some related problem. Are the unknown and the new data nearer to each other? lDid you use all the data? Did you use the whole condition? Have you taken into account all essential notions involved in the problem?

Carrying it Through 3.Carry out the Plan  Carrying out your plan of the solution, check each step. Can you see clearly that the step is correct? Can you prove that it is correct? 4.Looking Back  Can you check the result? Can you check the argument?  Can you derive the solution differently? Can you see it at a glance?  Can you use the result, or the method, for some other problem?

Analysis of Algorithms lHow good is the algorithm?  Correctness  Time efficiency  Space efficiency  Simplicity lDoes there exist a better algorithm?  Lower bounds  Optimality

Why Study Algorithms? lTheoretical importance  The core of computer science lPractical importance  A practitioner’s toolkit of known algorithms  Framework for designing and analyzing algorithms for new problems  Useful mindset

Course Structure lFundamentals of the Analysis of Algorithms (Ch. 2)  Asymptotic notations, analysis of recursive and non-recursive algorithms, empirical analysis lAlgorithmic Strategies (Ch. 3-9)  Brute force, Divide-and-Conquer, Decrease-and-Conquer, Transform-and-Conquer, Space and Time Tradeoffs, Greedy Techniques, Biologically-inspired techniques, Dynamic Programming lLimitations of Algorithms (Ch handouts)  Turing Machines, Computability, Problem Classification lCoping with Limitations on Algorithms (Ch. 11)  Backtracking and Branch and Bound lAnany Levitin, “Introduction to the Design and Analysis of Algorithms”, International Edition, Addison-Wesley, 2003

Practicals lWeekly mini prac exams lGiven a problem specification that is solvable using the algorithm design strategies presented in the course  Design Algorithm  Code it in C++  Submit it for automatic marking lAfter the 3-hour lab session will be asked to do a short analysis of the solution