Algorithm Analysis An algorithm is a clearly specified set of simple instructions to be followed to solve a problem. Three questions for algorithm analysis.

Slides:



Advertisements
Similar presentations
Analysis of Algorithms
Advertisements

CSE 373: Data Structures and Algorithms Lecture 5: Math Review/Asymptotic Analysis III 1.
Razdan with contribution from others 1 Algorithm Analysis What is the Big ‘O Bout? Anshuman Razdan Div of Computing.
Estimating Running Time Algorithm arrayMax executes 3n  1 primitive operations in the worst case Define a Time taken by the fastest primitive operation.
Fall 2006CENG 7071 Algorithm Analysis. Fall 2006CENG 7072 Algorithmic Performance There are two aspects of algorithmic performance: Time Instructions.
CS Section 600 CS Section 002 Dr. Angela Guercio Spring 2010.
Asymptotic Growth Rate
Cutler/HeadGrowth of Functions 1 Asymptotic Growth Rate.
Introduction to Analysis of Algorithms
Complexity Analysis (Part I)
Eleg667/2001-f/Topic-1a 1 A Brief Review of Algorithm Design and Analysis.
Analysis of Algorithms (Chapter 4)
1 Data Structures A program solves a problem. A program solves a problem. A solution consists of: A solution consists of:  a way to organize the data.
Algorithm Analysis. Math Review – 1.2 Exponents –X A X B = X A+B –X A /X B =X A-B –(X A ) B = X AB –X N +X N = 2X N ≠ X 2N –2 N+ 2 N = 2 N+1 Logarithms.
CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 1 Analysis.
Algorithm Analysis CS 201 Fundamental Structures of Computer Science.
CS Main Questions Given that the computer is the Great Symbol Manipulator, there are three main questions in the field of computer science: What kinds.
Analysis of Algorithms 7/2/2015CS202 - Fundamentals of Computer Science II1.
Cpt S 223 – Advanced Data Structures
Analysis of Algorithms COMP171 Fall Analysis of Algorithms / Slide 2 Introduction * What is Algorithm? n a clearly specified set of simple instructions.
Introduction CIS 606 Spring The sorting problem Input: A sequence of n numbers 〈 a 1, a 2, …, a n 〉. Output: A permutation (reordering) 〈 a’ 1,
Analysis of Algorithms Spring 2015CS202 - Fundamentals of Computer Science II1.
David Luebke 1 8/17/2015 CS 332: Algorithms Asymptotic Performance.
Instructor Neelima Gupta
Program Performance & Asymptotic Notations CSE, POSTECH.
Week 2 CS 361: Advanced Data Structures and Algorithms
Data Structures and Algorithms Lecture 5 and 6 Instructor: Quratulain Date: 15 th and 18 th September, 2009 Faculty of Computer Science, IBA.
Lecture 2 Computational Complexity
Algorithm Efficiency CS 110: Data Structures and Algorithms First Semester,
Mathematics Review and Asymptotic Notation
CSC 201 Analysis and Design of Algorithms Lecture 04: CSC 201 Analysis and Design of Algorithms Lecture 04: Time complexity analysis in form of Big-Oh.
1 Computer Algorithms Lecture 3 Asymptotic Notation Some of these slides are courtesy of D. Plaisted, UNC and M. Nicolescu, UNR.
Iterative Algorithm Analysis & Asymptotic Notations
Analysis of Algorithms
2IL50 Data Structures Fall 2015 Lecture 2: Analysis of Algorithms.
CS 221 Analysis of Algorithms Instructor: Don McLaughlin.
1 COMP3040 Tutorial 1 Analysis of algorithms. 2 Outline Motivation Analysis of algorithms Examples Practice questions.
CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,
ALGORITHMS THIRD YEAR BANHA UNIVERSITY FACULTY OF COMPUTERS AND INFORMATIC Lecture three Dr. Hamdy M. Mousa.
Introduction to Analysis of Algorithms COMP171 Fall 2005.
Algorithm Analysis (Algorithm Complexity). Correctness is Not Enough It isn’t sufficient that our algorithms perform the required tasks. We want them.
Program Efficiency & Complexity Analysis. Algorithm Review An algorithm is a definite procedure for solving a problem in finite number of steps Algorithm.
Growth of Functions. 2 Analysis of Bubble Sort 3 Time Analysis – Best Case The array is already sorted – no swap operations are required.
Asymptotic Growth Rates  Themes  Analyzing the cost of programs  Ignoring constants and Big-Oh  Recurrence Relations & Sums  Divide and Conquer 
Algorithmic Analysis Charl du Plessis and Robert Ketteringham.
CE 221 Data Structures and Algorithms Chapter 2: Algorithm Analysis - I Text: Read Weiss, §2.1 – Izmir University of Economics.
Algorithm Analysis Part of slides are borrowed from UST.
Analysis of algorithms. What are we going to learn? Need to say that some algorithms are “better” than others Criteria for evaluation Structure of programs.
David Luebke 1 1/6/2016 CS 332: Algorithms Asymptotic Performance.
1/6/20161 CS 3343: Analysis of Algorithms Lecture 2: Asymptotic Notations.
Asymptotic Performance. Review: Asymptotic Performance Asymptotic performance: How does algorithm behave as the problem size gets very large? Running.
ADVANCED ALGORITHMS REVIEW OF ANALYSIS TECHNIQUES (UNIT-1)
Spring 2015 Lecture 2: Analysis of Algorithms
E.G.M. PetrakisAlgorithm Analysis1  Algorithms that are equally correct can vary in their utilization of computational resources  time and memory  a.
BITS Pilani Pilani Campus Data Structure and Algorithms Design Dr. Maheswari Karthikeyan Lecture1.
1 Chapter 2 Algorithm Analysis Reading: Chapter 2.
Complexity of Algorithms Fundamental Data Structures and Algorithms Ananda Guna January 13, 2005.
Analysis of Algorithms Spring 2016CS202 - Fundamentals of Computer Science II1.
Chapter 15 Running Time Analysis. Topics Orders of Magnitude and Big-Oh Notation Running Time Analysis of Algorithms –Counting Statements –Evaluating.
Algorithm Analysis 1.
CMPT 438 Algorithms.
Introduction to Algorithms
CS 3343: Analysis of Algorithms
Algorithm Analysis (not included in any exams!)
CSC 413/513: Intro to Algorithms
Introduction to Algorithms Analysis
CS 201 Fundamental Structures of Computer Science
CE 221 Data Structures and Algorithms
CE 221 Data Structures and Algorithms
Introduction To Algorithms
Presentation transcript:

Algorithm Analysis An algorithm is a clearly specified set of simple instructions to be followed to solve a problem. Three questions for algorithm analysis What: What to analyze? How: How to analyze? Why: Why we need to do algorithm analysis? Judge an algorithm is “good” or “bad”. An algorithm with one year running time is hardly useful. An algorithm that requires tens gigabytes of main memory is not (currently) useful on most machines.

What to Analyze Analyzing an algorithm has come to mean predicting the resources that the algorithm requires, such as running time, memory, communication bandwidth, and logic gates. The most important resource to analyze is running time.

Model of Computation Our model is basically a normal computer, in which instructions are executed sequentially. Our model has the standard set of simple instructions. Each of these instructions takes a constant amount of time. Arithmetic: add, subtract, multiply, divide, remainder. Also, shift-left/shift-right (good for multiplying/dividing by 2k). Data movement: load, store, copy Control: conditional/unconditional branch, subroutine call and return Our model has fixed-size (say, 32-bit) integers and there are no fancy operations, such as matrix inversion or sorting. We assume infinite memory

Running Time The time taken by an algorithm depends on the input We use function T(n) with respect to the size of the input n, to define the running time of an algorithm. On a particular input, it is the number of primitive operations (steps) executed. Input size: depends on the problem being studied. Usually, the number of items in the input. But could be something else. If multiplying two integers, could be the total number of bits in the two integers. Could be described by more than one number.

More on Running Time Want to define steps to be machine-independent Figure that each line of pseudo-code requires a constant amount of time One line may take a different amount of time than another, but each execution of line i takes the same amount of time ci. This is assuming that the line consists only of primitive operations. If the line is a subroutine call, then the actual call takes constant time, but the execution of the subroutine being called might not. If the line specifies operations other than primitive ones, then it might take more than constant time. Ex. “sort the points by x-coordinate.”

Analysis of Insertion Sort Assume that the i th line takes time ci , which is a constant. For j = 2, 3, …, n, let tj be the number of times that the while loop test is executed for that value of j. Note that when a for or while loop exits in the usual way  due to the test in the loop header  the test is executed one time more than the loop body. The total running time:

Best Case of Insertion Sort The array is already sorted Always find that A[i] <= key upon the first time the while loop test is run (when i = j-1). All tj are 1. Running time is Can express T(n) as an+b for constants a and b (that depend on the statement costs ci)  T(n) is a linear function of n.

Worst Case of Insertion Sort The array is in reverse sorted order. Always find that A[i] > key in while loop test. Have to compare key with all elements to the left of the j th position  compare with j -1 elements. Since the while loop exits because i reaches 0, there’s one additional test after the j -1 tests  tj = j. Running time is Can express T(n) as an2+bn+c for constants a, b, and c (that depend on the statement costs ci)  T(n) is a quadratic function of n.

Worst-case Analysis Three kinds of running time The worst-case running time: the longest running time for any input of size n. The average running time: the average running time over all possible inputs of size n. The best-case running time: the shortest running time for any input of size n. We usually concentrate on finding the worst-cast running time. The worst-case running time of an algorithm is an upper bound on the running time for any input. Knowing it gives us a guarantee that the algorithms will never take any longer. For some algorithms, the worst case occurs fairly often. The average case is often roughly as bad as the worst case, e.g., insertion sort.

Order of Growth Another abstraction to ease analysis and focus on the important features. Look only at the leading term of the formula for running time. Drop lower-order terms. Ignore the constant coefficient in the leading term. Ex. For insertion sort, we already abstracted away the actual statement costs to conclude that the worst-case running time is an2+bn+c. Drop lower-order terms  an2. Ignore constant coefficient n2. We say the worst-case running time T(n) grows like n2, but it does not equal n2. We say that the running time is (n2) to capture the notion that the order of growth is n2. We usually consider one algorithm to be more efficient than another if its worst-case running time has a smaller order of growth.

Growth of Functions A way to describe behavior of functions in the limit, as the size of the input increases without bound. We are studying asymptotic efficiency, that is, we look at the running time of an algorithm when the input sizes are large enough to make only the order of growth of the running time relevant. Describe growth of functions. Focus on what’s important by abstracting away low-order terms and constant factors.

Big-Oh Notation f(n) = O(g(n)): if there are positive constants c and n0 such that f(n) <= c g(n) when n >= n0 g(n) is an asymptotic upper bound for f(n). Think “at most”. cg(n) dominates f(n) to right of n0. Example: 1000n2 = O(n3), with c = 1 and n0 = 1000 Examples of functions in O(n2): n2, n2+n, 1000n2+1000n n, n/1000, n1.9999, n2/logloglogn cg(n) f(n) n0

Big-Oh Notation Big-Oh is an upper bound estimate, so it is good to say “running time is O(n2)”. But, we cannot say “running time is at least O(n2)”. Instead, we use  notation.

Big Omega() Notation f(n) = (g(n)): if there are positive constants c and n0 such that f(n) >= cg(n) when n >= n0 g(n) is an asymptotic lower bound for f(n). Think “at least” Example: n2 = (nlogn), with c = 1 and n0 =1. Examples of functions in (n2): n2, n2 – n, 1000n2+1000n n3, n2.00001, n2loglogn, 22n f(n) cg(n) n0

Big Theta() Notation f(n) = (g(n)): if there exist positive constants c1, c2, and n0 such that c1 g(n) <= f(n) <= c2 g(n) when n >= n0 g(n) is an asymptotically tight bound for f(n). Example: n2/2-2n = (n2), with c1 = ¼, c2 = ½, and n0 =8. c2g(n) f(n) c1g(n) n0

Asymptotic Notations L’Hopital’s rule: if and , then f(n) = (g(n)) means f(n) and g(n) grow at the same rate.   = f(n) = O(g(n)) means f(n) grows at a rate no faster than g(n). O   f(n) = (g(n)) means f(n) grows at a rate no slower than g(n).    Another way to determine the relative growth rates of two functions f(n) and g(n) is to compute The limit is 0: f(n) = O(g(n)) The limit is c  0: f(n) = (g(n)) The limit is : f(n) = (g(n)) The limit oscillates: no relation L’Hopital’s rule: if and , then where f’(n) and g’(n) are the derivatives of f(n) and g(n), respectively.

Properties of Asymptotic Notations Three rules: If T1(n) = O(f(n)) and T2(n) = O(g(n)), then (a) T1(n) + T2(n) = max(O(f(n)), O(g(n))), (b) T1(n)*T2(n) = O(f(n)*g(n)). If T(n) is a polynomial of degree k, then T(n) = (nk). logkn = O(n) for any constant k. This tells us that logarithms grow very slowly.

General Rules for Computing Running Time Rule 1: FOR loops The running time of a for loop is at most the running time of the statements inside the for loop (including tests) times the number of iterations. Rule 2: Nested loops Analyze these inside out. The total running time of a statement inside a group of nested loops is the running time of the statement multiplied by the product of the sizes of all the loops. example: for (i=0; i < n; i ++) for (j=0; j < n; j++) k++; Running time: T(N) = O(N2)

General rules (cont.) Rule 3: Consecutive statements These just add ( which means that the maximum is the one that counts) example: for (i=0; i < n; i++) a[i] = 0; for (i=0; i < n; i++) for (j=0; j < n; j++) a[i] += a[j] + i + j; Running Time: T(N) = O(N) + O(N2) = O(N2) Rule 4: IF/ELSE For the fragment: if (condition) S1 else S2 The running time of an if/else statement is never more than the running time of the test plus the larger of the running times of S1 and S2. Takes O(N) Takes O(N2)

Recurrences Definition Methods for solving recurrences Substitution The master method

Definitions When an algorithm contains a recursive call to itself, its running time can often be described by a recurrence. A recurrence is an equation or inequality that describes a function in terms of its value on smaller inputs. T(n) = T(m1) + T(m2) + … + T(mk) + f(n) T(a) = b where m1, m2, …, mk < n, and a and b are constants.

Example Recurrences Solution: T(n) = n Solution: T(n) = nlogn + n Solution: T(n) = loglogn Solution: T(n) = (nlogn)

Substitution Basic Steps Guess the solution. Use induction to find the constants and show that the solution works

Substitution Examples 1: T(n) = 2T(n/2) + O(n) Write T(n) <= 2T(n/2)+cn for some positive constant c. Guess: T(n) <= d nlogn for some positive constant d. Note: we need to choose an appropriate d and d may depends on c. Substitution: Therefore, T(n) = O(nlogn)

Substitution Comments on examples 1 Generally, we use asymptotic notation. Thus, we would write T(n) = 2T(n/2)+O(n). We assume T(n) = O(1) for sufficiently small n. We express the solution by asymptotic notation: T(n) = O(nlogn) We do not worry about boundary cases, nor do we show base cases in the substitution proof. T(n) is always constant for any constant n. Since we are ultimately interested in an asymptotic solution to a recurrence, it will always be possible to choose base cases that work. When we want an asymptotic solution to a recurrence, we do not worry about the base cases in our proofs.

Substitution Examples 2: T(n) = 2T(n/2) + (n) Write T(n) >= 2T(n/2)+cn for some positive constant c. Guess: T(n) >= d nlogn for some positive constant d. Note: we need to choose an appropriate d, and d may depends on c. Substitution: Therefore, T(n) = (nlogn) Therefore, T(n) = (nlogn)

Substitution Subtleties: There are times even when you can correctly guess at an asymptotic bound on the solution of a recurrence, but somehow the math doesn’t seem to work out in the induction. Example: T(n) = 8T(n/2)+O(n2) Let T(n) <= 8T(n/2)+cn2. Guess: T(n) <= dn3. Substitution: The problem is that the inductive assumption isn’t strong enough to prove the detailed bound.

Subtleties Example: T(n) = 8T(n/2)+O(n2) Let T(n) <= 8T(n/2)+cn2. Remedy: Subtract off a lower-order term. Guess: T(n) <= dn3 – d’n2 Substitution:

The Master Method The master method provides a “cookbook” method for solving recurrences of the form T(n) = aT(n/b) + f(n) where a >= 1 and b > 1 are constants and f(n) is an asymptotically positive function. T(n) can be bounded asymptotically as follows. If f(n) = O(nlogba-) for some constant  > 0, then T(n) = (nlogba) If f(n) = (nlogba), then T(n) = (nlogbalogn) If f(n) =  (nlogba+) for some constant  > 0, and if af(n/b) <= cf(n) for some constant c < 1 and all sufficiently large n, then T(n) = (f(n))

Examples T(n) = 9T(n/3) + n T(n) = T(2n/3) + 1 T(n) = 3T(n/4) + nlogn For this recurrence, we have a = 9, b = 3, f(n) = n, and thus nlogba= nlog39 = (n2). Since f(n) = O(nlog39-), where  = 1, we can apply case 1 of the master method and conclude that the solution is T(n) = (n2). T(n) = T(2n/3) + 1 a = 1, b = 3/2, f(n) = 1, and nlogba= nlog3/21= n0 = 1. Case 2 applies, since f(n) = (nlogba) = (1), and thus the solution to the recurrence is T(n) = (logn) T(n) = 3T(n/4) + nlogn We have a = 3, b = 4, f(n) = nlogn, and nlogba= nlog43=O(n0.793). Since f(n) = (nlog43+), where   0.2, and for sufficiently large n, af(n/b) = 3(n/4)log(n/4) <= (3/4)nlogn = cf(n) for c = ¾, by case 3, the solution to the recurrence is T(n) = (nlogn)