1. Software Design Analysis of Algorithms i206 Fall 2010 John Chuang Some slides adapted from Glenn Brookshear, Marti Hearst, or Goodrich & Tamassia

2. John Chuang2 Analysis of Algorithms Bits & Bytes Binary Numbers Number Systems Gates Boolean Logic Circuits CPU Machine Instructions Assembly Instructions Program Algorithms Application Memory Data compression Compiler/ Interpreter Operating System Data Structures Analysis I/O Memory hierarchy Design Methodologies/ Tools Process Truth table Venn Diagram DeMorgan’s Law Numbers, text, audio, video, image, … Decimal, Hexadecimal, Binary AND, OR, NOT, XOR, NAND, NOR, etc. Register, Cache Main Memory, Secondary Storage Context switch Process vs. Thread Locks and deadlocks Op-code, operands Instruction set arch Lossless v. lossy Info entropy & Huffman code Adders, decoders, Memory latches, ALUs, etc. Data Representation Data storage Principles ALUs, Registers, Program Counter, Instruction Register Network Distributed Systems Security Cryptography Standards & Protocols Inter-process Communication Searching, sorting, Encryption, etc. Stacks, queues, maps, trees, graphs, … Big-O UML, CRC TCP/IP, RSA, … Confidentiality Integrity Authentication … C/S, P2P Caching sockets Formal models Finite automata regex

3. John Chuang3 Traveling Salesman Problem http://xkcd.com/399/

4. John Chuang4 Which Algorithm is Better? What do we mean by “Better”?  Sorting algorithms -Bubble sort -Insertion sort -Shell sort -Merge sort -Heapsort -Quicksort -Radix sort -…  Search algorithms -Linear search -Binary search -Breadth first search -Depth first search -…

5. John Chuang5 Analysis  Characterizing the running times of algorithms and data structure operations  Secondarily, characterizing the space usage of algorithms and data structures

6. John Chuang6 Running Time  In general, running time increases with input size  Running time also affected by: -Hardware environment (processor, clock rate, memory, disk, etc.) -Software environment (operating system, programming language, compiler, interpreter, etc.) (n)

7. John Chuang7 Quantifying Running Time  Experimentally measure running time -Need to fully implement and execute  Analysis of pseudo- code (n)

8. John Chuang8 Pseudo-code Analysis Example Algorithm: arrayMax(A,n): Input: An array A storing n >= 1 integers Output: the maximum element in A currentMax = A[0] for i=1 to (n-1) do if currentMax < A[i] then currentMax = A[i] Return currentMax

9. John Chuang9 The Seven Common Functions  Constant  Linear  Quadratic  Cubic  Exponential  Logarithmic  n-log-n Polynomial

10. John Chuang10 The Seven Common Functions  Constant: f(n) = c -E.g., adding two numbers, assigning a value to some variable, comparing two numbers, and other basic operations  Linear: f(n) = n -Do a single basic operation for each of n elements, e.g., reading n elements into computer memory, comparing a number x to each element of an array of size n

11. John Chuang11 The Seven Common Functions  Quadratic: f(n) = n 2 -E.g., nested loops with inner and outer loops -E.g., insertion sort algorithm  Cubic: f(n) = n 3  More generally, all of the above are polynomial functions: -f(n) = a 0 + a 1 n + a 2 n 2 + a 3 n 3 + … + a n n d -where the a i ’s are constants, called the coefficients of the polynomial Brookshear Figure 5.19

12. John Chuang12 The Seven Common Functions  Exponential: f(n) = b n -Where b is a positive constant, called the base, and the argument n is the exponent -In algorithm analysis, the most common base is b=2 -E.g., loop that starts with one operation and doubles the number of operations with each iteration -Similar to the geometric sum: -1+2+4+8+…+2 n-1 = 2 n – 1 -BTW, 2 n – 1 is the largest integer that can be represented in binary notation using n bits

13. John Chuang13 The Seven Common Functions  Logarithmic: f(n) = log b n -Intuition: number of times we can divide n by b until we get a number less than or equal to 1 -E.g., the binary search algorithm has a logarithmic running time -The base is omitted for the case of b=2, which is the most common in computing: -log n = log 2 n Brookshear Figure 5.20

14. John Chuang14 The Seven Common Functions  n-log-n function: f(n) = n log n -Product of linear and logarithmic -E.g., quicksort algorithm has n log n running time on average (but n 2 in worst case)

15. John Chuang15 Comparing Growth Rates  1 < log n < n 1/2 < n < n log n < n 2 < n 3 < b n http://www.cs.pomona.edu/~marshall/courses/2002/spring/cs50/BigO/

16. John Chuang16 Polynomials  Only the dominant terms of a polynomial matter in the long run. Lower-order terms fade to insignificance as the problem size increases. http://www.cs.pomona.edu/~marshall/courses/2002/spring/cs50/BigO/

17. John Chuang17 Example: Counting Primitive Operations Algorithm: arrayMax(A,n): Input: An array A storing n >= 1 integers Output: the maximum element in A currentMax = A[0] for (i=1; i<=n-1; i++) if currentMax < A[i] then currentMax = A[i] Return currentMax Two operations: indexing into array; assign value to variable Two operations repeated n times: subtract, compare One operation: assign value to variable Four or six operations repeated (n-1) times: - index and compare (for the if statement); - index and assign (for the then statement if necessary) - addition and assign (for the increment) One operation: return value of variable Total: 2 + 1 + 2n + 4(n-1) + 1 = 6n (best case) Total: 2 + 1 + 2n + 6(n-1) + 1 = 8n – 2 (worst case)

18. John Chuang18 Big-O Notation  Asymptotic analysis -As n becomes large and approaches infinity  Let f(n) and g(n) be functions mapping non- negative integers to real numbers  Then f(n) is O(g(n)) or “f(n) is big-Oh of g(n)”  If there is a real constant c > 0 and an integer constant n 0 >=1 such that -f(n) = n 0  Example: the function f(n)=8n–2 is O(n)  The running time of algorithm arrayMax is O(n)

19. John Chuang19 Example 1: Search  Given: -A physical phone book -Organized in alphabetical order -A name you want to look up -An algorithm in which you search through the book sequentially, from first page to last: Sequential Search or Linear Search -What is the order of: -The best case running time? -The worst case running time? -The average case running time? -What is: -A better algorithm? -The worst case running time for this algorithm?

20. John Chuang20 Example 1: Search  This better algorithm is called Binary Search  What is its running time? -First you look in the middle of n elements -Then you look in the middle of n/2 = ½*n elements -Then you look in the middle of ½ * ½*n elements -… -Continue until you find the target element, or until there is only 1 element left -Say you did this m times: ½ * ½ * ½* …*n -Then the number of repetitions is the smallest integer m such that

21. John Chuang21 Binary Search Pseudo-code Brookshear Figure 5.14

22. John Chuang22 Analyzing Binary Search -In the worst case, the number of repetitions is the smallest integer m such that -We can rewrite this as follows: Multiply both sides by Take the log of both sides Since m is the worst case time, the algorithm is O(log n)

23. John Chuang23 Example 2: Insertion Sort is O(n 2 ) Brookshear Figure 5.19 Brookshear Figure 5.10 Brookshear Figure 5.11

24. John Chuang24 Example 3: Prefix Average Compute running average of a sequence of numbers 5 10 15 20 25 30 5/1 15/2 30/3 50/4 75/5 105/6 There are two straightforward algorithms: The first is easy but wasteful. The second is more efficient, but requires insight into the problem.  j=0 X[j] i i+1 A[i] =

25. John Chuang25 1 st Algorithm Algorithm: prefixAverages1(X): Input: An n-element array X of numbers Output: An n-element array A of numbers such that A[i] is the average of elements X[0],…X[i] Let A be an array of n numbers For i  0 to n-1 do a  0 For j  0 to i do a  a+X[j] A[i]  a/(i+1) Return array A  j=0 X[j] i i+1 A[i] =

26. John Chuang26 2 nd Algorithm Algorithm: prefixAverages2(X): Input: An n-element array X of numbers Output: An n-element array A of numbers such that A[i] is the average of elements X[0],…X[i] Let A be an array of n numbers s  0 For i  0 to n-1 do s  s + X[i] A[i]  s/(i+1) Return array A A useful tool: store intermediate results in a variable! Uses space to save time. The key – don’t divide s. Eliminates one for loop – always a good thing to do.

27. John Chuang27 Summary: Analysis of Algorithms  A method for determining the asymptotic running time of an algorithm -Here asymptotic means “as n gets very large”  Useful for comparing algorithms  Useful also for determining tractability -E.g., Exponential time algorithms are usually intractable.

28. John Chuang28 Traveling Salesman Problem http://xkcd.com/399/