# M. G. Abbas Malik Assistant Professor Faculty of Computing and IT University of Jeddah, KSA Presented and Edited by : Dr. Muhammad Murtaza Khan CPCS 324.

## Presentation on theme: "M. G. Abbas Malik Assistant Professor Faculty of Computing and IT University of Jeddah, KSA Presented and Edited by : Dr. Muhammad Murtaza Khan CPCS 324."— Presentation transcript:

M. G. Abbas Malik Assistant Professor Faculty of Computing and IT University of Jeddah, KSA Presented and Edited by : Dr. Muhammad Murtaza Khan CPCS 324 – Algorithms and Data Structures

Course Course Book: Introduction to Design and Analysis of Algorithms, Anany Lenitin, Pearson 3 rd Ed. Marks Distribution: First Exam:15 Second Exam:15 Final Exam40 Quiz and class assessment10 Lab20 75% attendance is compulsory to sit in the final exam! M. G. Abbas Malik - FCIT, UoJ 2

Algorithms and Data Structures M. G. Abbas Malik - FCIT, UoJ 3 An algorithm is a sequence of unambiguous (clear) instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time. A Data Structure is a particular way of storing and organizing data in a computer so that it can be used efficiently.

M. G. Abbas Malik - FCIT, UoJ 4 What is an algorithm? Process, method, technique, procedure, routine,… with following requirements: 1. Finiteness b terminates after a finite number of steps 2. Definiteness b rigorously and unambiguously specified 3. Input b valid inputs are clearly specified 4. Output b can be proved to produce the correct output given a valid input 5. Effectiveness b steps are sufficiently simple and basic

M. G. Abbas Malik - FCIT, UoJ 5 Consider example of gcd (1424,3084) Step 1: Divide bigger number by smaller Step 2: Divide the divider by remainder Step 3: if (remainder ==0) divider is GCD Algorithm: gcd(m,n) m ← max(m,n) n ← min(m,n) while n ≠ 0 do r ← m mod n m← n n ← r return m

M. G. Abbas Malik - FCIT, UoJ 6 Two main issues related to algorithms How to design algorithms How to analyze algorithm efficiency How good is the algorithm? time efficiency space efficiency Does there exist a better algorithm? lower bounds optimality

M. G. Abbas Malik - FCIT, UoJ 7 How to “Dry Run” an Algorithm?

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 8

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30104020 1234 i =  j =  key =  A[j] =  A[j+1] =  9

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30104020 1234 i = 2 j =  key =  A[j] =  A[j+1] =  10

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30104020 1234 i = 2 j =  key = 10 A[j] =  A[j+1] =  11

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30104020 1234 i = 2 j = 1key = 10 A[j] = 30 A[j+1] = 10 12

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30104020 1234 i = 2 j = 1key = 10 A[j] = 30 A[j+1] = 10 13

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30104020 1234 i = 2j = 1key = 10 A[j] = 30 A[j+1] = 10 14

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30 4020 1234 i = 2j = 1key = 10 A[j] = 30 A[j+1] = 30 15

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30 4020 1234 i = 2j = 1key = 10 A[j] = 30 A[j+1] = 30 16

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30 4020 1234 i = 2j = 0key = 10 A[j] =  A[j+1] = 30 17

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30 4020 1234 i = 2j = 0key = 10 A[j] =  A[j+1] = 30 18

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30 4020 1234 i = 2j = 0key = 10 A[j] =  A[j+1] = 30 19

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 2j = 0key = 10 A[j] =  A[j+1] = 10 20

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 0key = 10 A[j] =  A[j+1] = 10 21

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 0key = 40 A[j] =  A[j+1] = 10 22

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 0key = 40 A[j] =  A[j+1] = 10 23

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40 24

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40 25

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40 26

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40 27

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 2key = 40 A[j] = 30 A[j+1] = 40 28

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40 29

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40 30

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 20 31

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 20 32

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 20 33

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40 34

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40 35

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40 36

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40 37

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 1030 40 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 30 38

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 1030 40 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 30 39

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 1030 40 1234 i = 4j = 1key = 20 A[j] = 10 A[j+1] = 30 40

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 1030 40 1234 i = 4j = 1key = 20 A[j] = 10 A[j+1] = 30 41

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 1030 40 1234 i = 4j = 1key = 20 A[j] = 10 A[j+1] = 30 42

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10203040 1234 i = 4j = 1key = 20 A[j] = 10 A[j+1] = 20 43

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1 while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10203040 1234 i = 4j = 1key = 20 A[j] = 10 A[j+1] = 20 Done! 44

Insertion Sort Statement Effort InsertionSort(A, n) { for i = 2 to n { c 1 n key = A[i] c 2 (n-1) j = i - 1; c 3 (n-1) while (j > 0) and (A[j] > key) { c 4 Σt i A[j+1] = A[j] c 5 Σ(t i -1) j = j - 1 c 6 Σ(t i -1) } A[j+1] = key c 7 (n-1) } 45 t i represents the number of times the loop will run for different values of i Best case scenario the inner loop never executes which means that the numbers are already in increasing order (sequence) Worst case scenario the inner loop executes as many number of times as the numbers in array before it thus t i = i

Analyzing Insertion Sort T(n) = c 1 n + c 2 (n-1) + c 3 (n-1) + c 4 Σ 2 ≤ i ≤ n t i + c 5 Σ 2 ≤ i ≤ n (t i -1) + c 6 Σ 2 ≤ i ≤ n (t i - 1) + c 7 (n-1) What can T be? Best case -- inner loop body never executed t i = 1  T(n) is a linear function Worst case -- inner loop body executed for all previous elements t i = i  T(n) is a quadratic function 46

Relation and Function - Review Function Growth nf(n)g(n) 1010,000 50250,0006,250,000 1001,000,000100,000,000 1502,250,000506,250,000 47

Order of growth O(g(n))is a set of all functions with a lower or same order of growth as g(n) to within a constant multiple, as n→∞ Example: Linear, Linear and Quadratic all are of same or smaller order than n 2 Cubic and higher order functions hence bigger than n 2 Ω(g(n))is set of all functions with a higher or same order of growth as g(n) to within a constant multiple, as n→∞ Example: Cubic and quadratic functions are of higher and same order as n 2 Linear function is of smaller order than n 2 48

Asymptotic Notations O – Notation (Upper Bound) g(n) is upper bound of f(n) for n ≥ 10. Definition A function f(n) is said to be in O(g(n)) i.e. f(n)εO(g(n)): if f(n) is bounded above by some constant multiple of g(n) for all large n such that f(n) ≤ cg(n) for all n≥n o } 49

Big-oh 50

Big-O Prove that Solving We know that Also we get Hence we can select c=105 n 0 =1 51

Asymptotic Notations Ω – Notation (lower Bound) Used for lower bounds. Think least. Asymptotic lower bound Definition A function f(n) is said to be in Ω(g(n)) i.e. f(n)εΩ(g(n)): if f(n) is bounded below by some constant multiple of g(n) for all large n such that cg(n) ≤ f(n) for all n≥n o } 52

Big-omega 53

Asymptotic Notations Θ – Notation (Bound) Used for upper and lower bounds. Tight bound Asymptotically tight bound Definition A function f(n) is said to be in Θ (g(n)) i.e. f(n)εΘ(g(n)): if f(n) is bounded both above and below by some constant multiple of g(n) for all large n such that c 1 g(n) ≤ f(n) ≤ c 2 g(n) for all n≥n o } 54

Big-theta 55

Big-Theta Prove that Solving From LHS we get Also we get Hence we can select c 1 =1/2 c 2 =1/4 and n 0 =2 56

Asymptotic Notations Relation between O, Ω and Θ – Notations  Θ(g(n)) = O(g(n)) ∩ Ω(g(n)) 57

Asymptotic Notations Relation between O, Ω and Θ – Notations  Θ(g(n)) = O(g(n)) ∩ Ω(g(n)) 58

Review of Algorithm Analysis Fundamentals of Algorithms Important Problems: Sorting Searching String processing Graph problems Analysis Framework: Input size Running Time Order of Growth M. G. Abbas Malik - FCIT, UoJ 59

Review of Algorithm Analysis Asymptotic Notation Big O notation (big O) Big Ω notation (big omega) Big θ notation (big theta) Brute Force Selection and Bubble sorts Depth First and Breadth First searches Divide and Conquer Decrease by a constant, by a constant factor and by a variable size Merge sort Quick sort Binary Tree Traversal M. G. Abbas Malik - FCIT, UoJ 60

Transfer and Conquer M. G. Abbas Malik - FCIT, UoJ 61 It consists of two stages: Transfer Conquer Three Variations: Transform to a simpler or more convenient instance of the same problem – Instance Simplification Transform to a different representation of the same instance – Representation change Transform to an instance of a different problem for which an algorithm is already available – Problem Reduction

Transfer and Conquer M. G. Abbas Malik - FCIT, UoJ 62 Problem’s Instance Solution Simpler Instance Or Another representation Or Another Problem’s Instance

Transfer and Conquer M. G. Abbas Malik - FCIT, UoJ 63 Instance Simplification Presorting Gaussian Elimination Representation Change Balanced Search Tree AVL Trees Heap and Heap Sort Problem Reduction Least Common Multiple Counting Path in a graph Linear Programming

Presorting M. G. Abbas Malik - FCIT, UoJ 64

Presorting M. G. Abbas Malik - FCIT, UoJ 65

Presorting M. G. Abbas Malik - FCIT, UoJ 66 Checking Element Uniqueness in a List The Brute-Force algorithm compared pairs of the list’s elements until either two equal elements were found or not more pairs were left. UniqueElements(A) for i 0 to n – 2 do for j i + 1 to n – 1 do if A[i] == A[j] then return False return True

Presorting M. G. Abbas Malik - FCIT, UoJ 67 Checking Element Uniqueness in a List The Brute-Force algorithm: What is the worst case? What is the efficiency in the worst case? UniqueElements(A) for i 0 to n – 2 do for j i + 1 to n – 1 do if A[i] == A[j] then return False return True

Presorting M. G. Abbas Malik - FCIT, UoJ 68 Checking Element Uniqueness in a List If the list is already sorted, then how we can check the element uniqueness? Sort the list first and then check only its consecutive elements. If the list is sorted and it has equal elements, then a pair of them must be next to each other Write an algorithm that check the element uniqueness in a sorted lest.

Presorting M. G. Abbas Malik - FCIT, UoJ 69 Checking Element Uniqueness in a List What is the complexity of this algorithm? PresortUniqueElements(A)\ // Input: An array A[0…n-1] – a sorted array // Output: determine uniqueness of elements for i 0 to n – 2 do if A[i] = A[i+1] then return False return True

Presorting M. G. Abbas Malik - FCIT, UoJ 70

Presorting M. G. Abbas Malik - FCIT, UoJ 71 Computing a Mode A mode is a value that occurs most often in a given list of numbers. Consider the list of numbers: 5, 1, 5, 7, 6, 5, 7

Presorting M. G. Abbas Malik - FCIT, UoJ 72 Computing a Mode A mode is a value that occurs most often in a given list of numbers. Consider the list of numbers: 5, 1, 5, 7, 6, 5, 7 – the mode is 5 The Brute-Force approach to compute a mode would scan the list and compute the frequencies of all its distinct values, then find the value with the largest frequency.

Presorting M. G. Abbas Malik - FCIT, UoJ 73 Computing a Mode PresortMode(A) // Input: An array A[0…n-1] – a sorted array; // Output: Array’s mode i 0 modeFrequency 0 while i ≤ n – 1 do runLength 1 runValue A[i] while i + runLength ≤ n – 1 and A[i+runLength] == runValue do runLength runLength + 1 if runLength > modeFrequency then modeFrequency runLength modeValue runValue i i + runLength return modeValue

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 74

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 75 In general, we need to solve a system of n equations in n-unknowns: Where n is a large number. Theoretically, we can solve such a system by generalizing the substitution method for solving systems of two linear equations.

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 76 Gaussian Elimination is an elegant algorithm for solving systems of linear equations. Carl Friedrich Gauss (1777 - 1855) The idea of Gaussian Elimination is to transform a system of n linear equations in n-unknowns to an equivalent system with an upper triangular coefficient matrix, a matrix with all ZEROS below its main diagonal

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 77 Consider a system of n equations with n unknowns In Matrix representation

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 78

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 79 Elementary operations of conversion 1. Exchanging two equations of the system 2. Replacing an equation with its non-zero multiple 3. Replacing an equation with a sum or different of this equation and some multiple of another equations Elementary operations can change the system of linear equations into another equivalent system of linear equations such that the solution of these systems remain the same.

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 80

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 81

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 82

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 83

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 84 Stage 1: Reduction to Upper Triangular Matrix ForwardElimination(A[1…n,1…n], b[1…n]) // Input: matrix A of a system’s coefficient and matrix b of the system’s right hand side // Output: equivalent upper-triangular matrices A and b For i 1 to n do A[i, n+1] b[i]// augment the matrix A For i 1 to n – 1 do For j i +1 to n do For k i to n+1 do A[j,k] A[j,k] – A[i,k]*A[j,i]/A[i,i] Two problems with algorithm? 1.If A[I,i]=0 or very small then there will be a problem. Solution, exchange rows. 2.In the innermost for loop is inefficient. Why calculate A[j,i]/A[I,i] repeatedly?

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 85 Stage 1: Reduction to Upper Triangular Matrix BetterForwardElimination(A[1…n,1…n], b[1…n]) // Input: matrix A of a system’s coefficient and matrix b of the system’s right hand side // Output: equivalent upper-triangular matrices A and b For i 1 to n do A[i, n+1] b[i]// augment the matrix A For i 1 to n – 1 do pivotRow i For j i +1 to n do if |A[j,i]|>|A[pivotRow,i]| then pivotRow j For k i to n+1 do Swap (A[i,k], A[pivotRow, k]) For j i+1 to n do temp = A[j,i]/A[i,i] For k i to n+1 do A[j,k] A[j,k] – A[i,k]*temp

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 86 Stage 1: Reduction to Upper Triangular Matrix

Gaussian Elimination M. G. Abbas Malik - FCIT, UoJ 87 Stage 2: Back Substitution The complexity of this stage is Θ(n 2 ) The over all complexity of Gaussian Elimination algorithm is Θ(n 2 ) + Θ(n 3 ) = Θ(n 3 )

Finding if a matrix is singular M. G. Abbas Malik - FCIT, UoJ 88 The inverse of an nxn square matrix A is an nxn square matrix, denoted by A -1, such that AA -1 =1Where I is identity nxn matrix Only inverse of a matrix can be calculated if its not singular i.e. not all zeros on diagonal Identity Matrix of 3x3

Transfer and Conquer M. G. Abbas Malik - FCIT, UoJ 89 Instance Simplification Presorting Gaussian Elimination Representation Change Balanced Search Tree AVL Trees Heap and Heap Sort Problem Reduction Least Common Multiple Counting Path in a graph Linear Programming

Heap M. G. Abbas Malik - FCIT, UoJ 90 A Specialized Tree-based Data Structure that satisfies the heap property: 1. if node B is a child of Node A, then key(A) ≥ key(B) – MAX–HEAP 2. if node B is a child of Node A, then key(A) ≤ key(B) – MIN–HEAP In MAX-HEAP, the largest element is always at the root of the tree In MIN-HEAP, the smallest element is always at the root of the tree

Heap M. G. Abbas Malik - FCIT, UoJ 91 Term coined in the context of Heap Sort Can be viewed as a nearly complete binary tree The tree is completely filled on all levels except possibly the lowest Suitable for implementing Priority Queue

Heap M. G. Abbas Malik - FCIT, UoJ 92 Definition A heap is a binary tree with keys assigned to its nodes, one key per node, provided the following two properties are hold: 1. Shape Property: binary tree is essentially complete 2. Heap Property: the key in each node is greater (MAX- HEAP) or smaller (MIN-HEAP) than or equal to the key in its children

Heap M. G. Abbas Malik - FCIT, UoJ 93 Heap Properties There exist exactly one essentially complete binary tree with n nodes. Its height is equal to log 2 n The root of a heap always contain the largest (for MAX-HEAP) or smallest (for MIN-HEAP) element. A node of a heap considered with all its descendants is also a heap A heap can be implemented as an ARRAY by recording its elements in the top-down, left-to-right fashion. Parental nodes will be in first n/2 positions of array, leaf keys occupy last n/2 positions Children of a key in arrays parental position I (1<i<n/2) will be in positions 2i and 2i+1 Correspondingly parent of a key in position I (2<i<n) will be in position i/2

Heap M. G. Abbas Malik - FCIT, UoJ 94 1614108793241 123456789 Array Representation 16 1410 87 24 1 93 Binary Tree Representation 1 23 4567 8 910

Heap M. G. Abbas Malik - FCIT, UoJ 95 Array Representation Root of tree is A[1] 16 1410 8 7 241 93 1 23 4 5 67 8 9 1614108793241 123456789

Heap M. G. Abbas Malik - FCIT, UoJ 96 Array Representation Root of tree is A[1] Parent(i)=i/2 16 1410 8 7 241 93 1 23 4 5 67 8 9 1614108793241 123456789

Heap M. G. Abbas Malik - FCIT, UoJ 97 Array Representation Root of tree is A[1] Parent(i)=i/2 16 1410 8 7 241 93 1 23 4 5 67 8 9 1614108793241 123456789

Heap M. G. Abbas Malik - FCIT, UoJ 98 Array Representation Root of tree is A[1] Parent(i)=i/2 16 1410 8 7 241 93 1 23 4 5 67 8 9 1614108793241 123456789

Heap M. G. Abbas Malik - FCIT, UoJ 99 Array Representation Root of tree is A[1] Parent(i)=i/2 LeftChild(i)=2i 16 1410 8 7 241 93 1 23 4 5 67 8 9 1614108793241 123456789

Heap M. G. Abbas Malik - FCIT, UoJ 100 Array Representation Root of tree is A[1] Parent(i)=i/2 LeftChild(i)=2i 16 1410 8 7 241 93 1 23 4 5 67 8 9 1614108793241 123456789

Heap M. G. Abbas Malik - FCIT, UoJ 101 Array Representation Root of tree is A[1] Parent(i)=i/2 LeftChild(i)=2i 16 1410 8 7 241 93 1 23 4 5 67 8 9 1614108793241 123456789

Heap M. G. Abbas Malik - FCIT, UoJ 102 Array Representation Root of tree is A[1] Parent(i)=i/2 LeftChild(i)=2i RightChild(i)=2i+1 16 1410 8 7 241 93 1 23 4 5 67 8 9 1614108793241 123456789

Heap M. G. Abbas Malik - FCIT, UoJ 103 Array Representation Root of tree is A[1] Parent(i)=i/2 LeftChild(i)=2i RightChild(i)=2i+1 Computing is fast with binary representation implementation 16 1410 8 7 241 93 1 23 4 5 67 8 9 1614108793241 123456789

Heap M. G. Abbas Malik - FCIT, UoJ 104 Binary Tree Representation Height of Node: No. of edges on the longest simple downward path from the node to a leaf 16 1410 87 24 1 93 1 23 4567 8 9

Heap M. G. Abbas Malik - FCIT, UoJ 105 Binary Tree Representation Height of Node: No. of edges on the longest simple downward path from the node to a leaf 16 1410 87 24 1 93 1 23 4567 8 9 Node Height = 1

Heap M. G. Abbas Malik - FCIT, UoJ 106 Binary Tree Representation Height of Node: No. of edges on the longest simple downward path from the node to a leaf 16 1410 87 24 1 93 1 23 4567 8 9 Node Height = 1 Node Height = 2

Heap M. G. Abbas Malik - FCIT, UoJ 107 Binary Tree Representation Heap Height: Height of root 16 1410 87 24 1 93 1 23 4567 8 9 Heap Height = 3

Maintaining Heap Property M. G. Abbas Malik - FCIT, UoJ 108 Max-Heapify is an important algorithm for maintaining the Heap Property Suppose A[i] may be smaller than its children Assumes the left and right sub-trees of the index i are MAX-HEAPs After Max-Heapify, sub-tree at the index i is a MAX-HEAP

Maintaining Heap Property M. G. Abbas Malik - FCIT, UoJ 109 Max-Heapify (A, i, RightIndex) 1L = Left-Child(i) 2R = Right-Child(i) 3If L ≤ RightIndex and A[L] > A[i] then 4Largest = L Else 5Largest = i 7If R ≤ RightIndex and A[R] > A[Largest] then 8Largest = R 9If Largest ≠ i then 10Swap A[i] A[Largest] 11MAX-Heapify(A, Largest, RightIndex)

Maintaining Heap Property M. G. Abbas Malik - FCIT, UoJ 110 16 410 147 28 1 93 1 23 4567 8 910 Max-heapify (A, 2, 10) i = 2 RightIndex = 10 L = 4 R = 5 Largest = 4 Swap A[i] with A[Largest] Recursive call: Max-Heapify(A, 4, 10)

Maintaining Heap Property M. G. Abbas Malik - FCIT, UoJ 111 16 1410 47 28 1 93 1 23 4567 8 9 Max-heapify (A, 4, 10) i = 4 RightIndex = 10 L = 8 R = 9 Largest = 9 Swap A[i] with A[Largest] Recursive call: Max-Heapify(A, 9, 10)

Maintaining Heap Property M. G. Abbas Malik - FCIT, UoJ 112 16 1410 87 24 1 93 1 23 4567 8 9 Max-heapify (A, 9, 10) i = 9 RightIndex = 10 L = 18 R = 19 Largest = 9

Analysis of Max-Heapify M. G. Abbas Malik - FCIT, UoJ 113 Max-Heapify (A, i, RightIndex) 1L = Left-Child(i)C 1 2R = Right-Child(i)C 2 3If L ≤ RightIndex and A[L] > A[i] thenC 3 4Largest = LC 4 Else 5Largest = iC 5 6If R ≤ RightIndex and A[R] > A[Largest] thenC 6 7Largest = RC 7 8If Largest ≠ i thenC 8 9Swap A[i] with A[Largest]C 9 10MAX-Heapify(A, Largest, RightIndex)C 10 Constant Time –  (1)

Analysis of Max-Heapify M. G. Abbas Malik - FCIT, UoJ 114 Max-Heapify takes  (1) time to fix the relationship among A[i], A[Left-Child] and A[Right- Child] + recursive call if any The complexity lies in the total number of recursive calls. Worst-Case Analysis: When the last row of the tree is exactly half-full. Left or Right sub-tree each has size at most By Case 2 of Master Theorem:

Building Heap M. G. Abbas Malik - FCIT, UoJ 115 A heap can be build from a list of numbers. The input is an ordinary list of numbers or an array of numbers The output is a heap containing all the numbers in the list or array. Build-Max-Heap (A, n) 1For i = n/2 to 1 2Max-Heapify(A, i, n)

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 116 4132169101487 12345678910 Array Representation 4 13 216 148 7 910 Binary Tree Representation 1 23 4567 8 910

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 117 4132169101487 12345678910 4 13 216 148 7 910 1 23 4567 8 9 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 118 4132169101487 12345678910 4 13 216 148 7 910 1 23 4567 8 9 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 119 4132169101487 12345678910 4 13 216 148 7 910 1 23 4567 8 9 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 120 4131416910287 123456789 4 13 1416 28 7 910 1 23 4567 8 9 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 121 4131416910287 123456789 4 13 1416 28 7 910 1 23 4567 8 9 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 122 4131416910287 123456789 4 13 1416 28 7 910 1 23 4567 8 9 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 123 4110141693287 12345678910 4 1 1416 28 7 93 1 23 4567 8 910 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 124 4110141693287 12345678910 4 1 1416 28 7 93 1 23 4567 8 910 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 125 4110141693287 12345678910 4 1 1416 28 7 93 1 23 4567 8 910 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 126 4161014193287 12345678910 4 1610 141 28 7 93 1 23 4567 8 910 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 127 4161014193287 12345678910 4 1610 141 28 7 93 1 23 4567 8 910 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 128 4161014793281 12345678910 4 1610 147 28 1 93 1 23 4567 8 910 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 129 4161014793281 12345678910 4 1610 147 28 1 93 1 23 4567 8 910 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 130 4161014793281 12345678910 4 1610 147 28 1 93 1 23 4567 8 910 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 131 1641014793281 12345678910 16 410 147 28 1 93 1 23 4567 8 910 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 132 1641014793281 12345678910 16 410 147 28 1 93 1 23 4567 8 910 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 133 1614104793281 123456789 16 1410 47 28 1 93 1 23 4567 8 9 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 134 1614104793281 123456789 16 1410 47 28 1 93 1 23 4567 8 9 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 135 1614108793241 123456789 16 1410 87 24 1 93 1 23 4567 8 9 i

Building Heap – Example M. G. Abbas Malik - FCIT, UoJ 136 1614108793241 123456789 16 1410 87 24 1 93 1 23 4567 8 9 i

M. G. Abbas Malik Assistant Professor Faculty of Computing and IT University of Jeddah, KSA CPCS 324 – Algorithms and Data Structures

Heap Abbas Malik; FCIT, UoJ 138 A Specialized Tree-based Data Structure that satisfies the heap property: 1. if node B is a child of Node A, then key(A)  key(B) – MAX–HEAP 2. if node B is a child of Node A, then key(A)  key(B) – MIN–HEAP In MAX-HEAP, the largest element is always in the root of the tree In MIN-HEAP, the smallest element is always in the root of the tree

Heap Abbas Malik; FCIT, UoJ 139 Term coined in the context of Heap Sort Not a garbage-collected storage – LISP and JAVA Can be viewed as a nearly complete binary tree The tree is completely filled on all levels except possibly the lowest Suitable for implementing Priority Queue

Heap Abbas Malik; FCIT, UoJ 140 Definition A heap is a binary tree with keys assigned to its nodes, one key per node, provided the following two properties are hold: 1. Shape Property: binary tree is essentially complete 2. Heap Property: the key in each node is greater (MAX-HEAP) or smaller (MIN-HEAP) than or equal to the key in its children

Heap Abbas Malik; FCIT, UoJ 141

Maintaining Heap Property Abbas Malik; FCIT, UoJ 142 Max-Heapify (A, i, RightIndex) 1L = Left-Child(i) 2R = Right-Child(i) 3If L  RightIndex and A[L] > A[i] then 4Largest = L Else 5Largest = i 7If R  RightIndex and A[R] > A[Largest] then 8Largest = R 9If Largest  i then 10Swap A[i]  A[Largest] 11MAX-Heapify(A, Largest, RightIndex)

Building Heap Abbas Malik; FCIT, UoJ 143 A heap can be build from a list of numbers. The input is an ordinary list of numbers or an array of numbers The output is a heap containing all the numbers in the list or array. Build-Max-Heap (A, n) 1For i =  n/2  to 1 2Max-Heapify(A, i, n)

Heap Sort Abbas Malik; FCIT, UoJ 144 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1)

Heap Sort Abbas Malik; FCIT, UoJ 145 4132169101487 12345678910 Input Array 4 13 216 148 7 910 Binary Tree Representation 1 23 4567 8 910 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1)

Heap Sort Abbas Malik; FCIT, UoJ 146 1614108793241 123456789 Array Representation 16 1410 87 24 1 93 Binary Tree Representation 1 23 4567 8 910 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1)

Heap Sort Abbas Malik; FCIT, UoJ 147 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 1614108793241 123456789 16 1410 87 24 1 93 1 23 4567 8 9 i Maximum No.

Heap Sort Abbas Malik; FCIT, UoJ 148 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 1141087932416 12345678910 1 1410 87 24 16 93 1 23 4567 8 910 i

Heap Sort Abbas Malik; FCIT, UoJ 149 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 1141087932416 12345678910 1 1410 87 24 16 93 1 23 4567 8 910 i

Heap Sort Abbas Malik; FCIT, UoJ 150 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 1141087932416 12345678910 1 1410 87 24 16 93 1 23 4567 8 910 i

Heap Sort Abbas Malik; FCIT, UoJ 151 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 10 i 1411087932416 12345678910 14 110 87 24 16 93 1 23 4567 8 910

Heap Sort Abbas Malik; FCIT, UoJ 152 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 10 i 1411087932416 12345678910 14 110 87 24 16 93 1 23 4567 8 910

Heap Sort Abbas Malik; FCIT, UoJ 153 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 10 i 1481017932416 12345678910 14 810 17 24 16 93 1 23 4567 8 910

Heap Sort Abbas Malik; FCIT, UoJ 154 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 10 i 1481017932416 12345678910 14 810 17 24 16 93 1 23 4567 8 910

Heap Sort Abbas Malik; FCIT, UoJ 155 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 10 i 1481047932116 12345678910 14 810 47 21 16 93 1 23 4567 8 910

Heap Sort Abbas Malik; FCIT, UoJ 156 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 1481047932116 12345678910 14 810 47 21 16 93 1 23 4567 8 910 9 i Maximum No.

Heap Sort Abbas Malik; FCIT, UoJ 157 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 1810479321416 12345678910 1 8 47 214 16 93 1 23 4567 8 910 9 i

Heap Sort Abbas Malik; FCIT, UoJ 158 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 1810479321416 12345678910 1 8 47 214 16 93 1 23 4567 8 910 9 i

Heap Sort Abbas Malik; FCIT, UoJ 159 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 1810479321416 12345678910 1 8 47 214 16 93 1 23 4567 8 910 9 i

Heap Sort Abbas Malik; FCIT, UoJ 160 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 9 i 1081479321416 12345678910 81 47 214 16 93 1 23 4567 8 910

Heap Sort Abbas Malik; FCIT, UoJ 161 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 9 i 1081479321416 12345678910 81 47 214 16 93 1 23 4567 8 910

Heap Sort Abbas Malik; FCIT, UoJ 162 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 9 i 1089471321416 12345678910 89 47 214 16 13 1 23 4567 8 910

Heap Sort Abbas Malik; FCIT, UoJ 163 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 1089471321416 12345678910 89 47 214 16 13 1 23 4567 8 910 8 i Maximum No.

Heap Sort Abbas Malik; FCIT, UoJ 164 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 2894713101416 12345678910 2 89 47 14 16 13 1 23 4567 8 910 8 i

Heap Sort Abbas Malik; FCIT, UoJ 165 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 2894713101416 12345678910 2 89 47 14 16 13 1 23 4567 8 910 8 i

Heap Sort Abbas Malik; FCIT, UoJ 166 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 2894713101416 12345678910 2 89 47 14 16 13 1 23 4567 8 910 8 i

Heap Sort Abbas Malik; FCIT, UoJ 167 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 8 i 9824713101416 12345678910 9 82 47 14 16 13 1 23 4567 8 910

Heap Sort Abbas Malik; FCIT, UoJ 168 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 8 i 9824713101416 12345678910 9 82 47 14 16 13 1 23 4567 8 910

Heap Sort Abbas Malik; FCIT, UoJ 169 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 8 i 9834712101416 12345678910 9 83 47 14 16 12 1 23 4567 8 910

Heap Sort Abbas Malik; FCIT, UoJ 170 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 9834712101416 12345678910 9 83 47 14 16 12 1 23 4567 8 910 7 i Maximum No.

Heap Sort Abbas Malik; FCIT, UoJ 171 Heap-Sort (A, n) 1Build-Max-Heap(A, n) 2For i = n to 2 3Exchange A[1] ↔ A[i] 4Max-Heapify(A, 1, i - 1) 1 i 1234789101416 12345678910 1 23 47 14 16 89 1 23 4567 8 910

Heap Sort – Analysis Abbas Malik; FCIT, UoJ 172 Heap-Sort (A, n) 1Build-Max-Heap(A, n)O(n) 2For i = n to 2 3Exchange A[1] ↔ A[i](n- 1)c 1 4Max-Heapify(A, 1, i - 1)(n-1)log 2 n

Priority Queue Abbas Malik; FCIT, UoJ 173 A data structure for maintaining a set of elements such that the next element in the queue is with the maximum/minimum priority. Hospital Emergency Printer Job Scheduling Processer Job Scheduling

Priority Queue Abbas Malik; FCIT, UoJ 174 Operations Make a priority queue Insert an element with given priority Return Max/Min element Return and delete Max/Min element Increase/Decrease the priority of an element Union of two priority queues Delete queue

Priority Queue ImplementationComparison Abbas Malik; FCIT, UoJ 175 Unsorted Array Sorted Array Heaps OperationsUnsortedSorted Build  (1) O(nlog 2 n) * Insert  (1) O(n) * Find Max/MinO(n) *  (1) Delete Max/MinO(n) *  (1) Union  (1) O(n) * Increase Priority  (1) Is-empty  (1) * Complexity depends on the algorithm used

Priority Queue Abbas Malik; FCIT, UoJ 176 OperationsUnsortedSortedBinary Heaps Build  (1) O(nlgn) *  (n) Insert  (1) O(n) *  (lgn) Find Max/MinO(n) *  (1) Delete Max/MinO(n) *  (1)  (lgn) Union  (1) O(n)*  (n) Increase Priority  (1)  (lgn) Is-empty  (1) * Complexity depends on the algorithm used Comparison

Priority Queue Abbas Malik; FCIT, UoJ 177 Find Max/Min Heap-Maximum (A) 1Return A[1] What is the worst case running time?

Priority Queue Abbas Malik; FCIT, UoJ 178 Extract (Find and Delete) Max/Min Heap-Extract-Max (A, n) 1If n > 0 then 2Max = A[1] 3A[1] = A[n] 4n = n-1 5Max-Heapify (A,1,n) 6Return Max 7Else 8Return “Error: Heap underflow!” Replacing root with last Element Reduce the size of Array by 1 Maintaining Heap property What is the worst case running time?

Priority Queue Abbas Malik; FCIT, UoJ 179 Increase Priority Heap-Increase-Key (A, i, Key) 1If Key > A[i] then 2A[i] = Key 3While i > 1 and A[Parent(i)] < A[i] 4Exchange A[i] ↔ A[Parent(i)] 5i = Parent(i) 6Else 7“Error: New key is smaller than current key” Maintaining Heap property What is the worst case running time?

Priority Queue Abbas Malik; FCIT, UoJ 180 Increase Priority 16 1410 87 24 1 93 1 23 4567 8 9 Heap-Increase-Key(A,10,15) 10 i

Priority Queue Abbas Malik; FCIT, UoJ 181 Increase Priority 16 1410 87 24 15 93 1 23 4567 8 910 i

Priority Queue Abbas Malik; FCIT, UoJ 182 Increase Priority 16 1410 815 24 7 93 1 23 4567 8 910 5 i

Priority Queue Abbas Malik; FCIT, UoJ 183 Increase Priority 16 1510 814 24 7 93 1 23 4567 8 910 2 i What is the worst case running time?

Priority Queue Abbas Malik; FCIT, UoJ 184 Insert a new element Max-Heap-Insert (A, Key, n) 1n = n + 1 2A[n] = -∞ 3Heap-Increase-Key(A, n, Key) Increase the size of Array by 1 What is the worst case running time? Initialize the value by a very small value

Priority Queue Abbas Malik; FCIT, UoJ 185 Union How to design an algorithm for union? What is the input of the algorithm? What is the output of the algorithm? 1. Input: Two Heaps 2. Output: one Heap that contain all the elements of two input heaps.

Priority Queue Abbas Malik; FCIT, UoJ 186 Union What steps we will follow in the algorithm? 1. Create an array of with size equal to the sum of sizes of two input heaps 2. Copy all elements of two input heaps in the new array 3. Build a heap from the new array

Priority Queue Abbas Malik; FCIT, UoJ 187 Union Max-Heaps-Union (A 1, A 2, n 1, n 2 ) 1Build an Array A of size n 1 + n 2 2For i = 1 to n 1 3A[i] = A 1 [i] 4For j = 1 to n 2 5A[i] = A 2 [j] 6i = i + 1 7Build-Max-Heap (A, n 1 + n 2 ) What is the worst case running time?

M. G. Abbas Malik Assistant Professor Faculty of Computing and IT University of Jeddah, KSA CPCS 324 – Algorithms and Data Structures

Trees Abbas Malik; FCIT, UoJ 189 A tree is a connected undirected graph with no simple circuits. Unique path between two vertices. A connected graph where each node has zero or more children and at most one parent. 1 2 3 4 5 6 Vertex

Trees Abbas Malik; FCIT, UoJ 190 1 23 4 56 Root o A vertex which does not have a parent Root

Trees Abbas Malik; FCIT, UoJ 191 1 23 4 56 Root o A vertex which does not have a parent Parent Root Parent

Trees Abbas Malik; FCIT, UoJ 192 1 23 4 56 Root o A vertex which does not have a parent Parent Child Root Parent Child

Trees Abbas Malik; FCIT, UoJ 193 1 23 4 56 Root o A vertex which does not have a parent Parent Child Internal vertex Root Parent Child

Trees Abbas Malik; FCIT, UoJ 194 1 23 4 56 Root o A vertex which does not have a parent Parent Child Internal vertex Leaf o A vertex with no child Root Parent Child

Trees Abbas Malik; FCIT, UoJ 195 1 23 4 56 Root o A vertex which does not have a parent Parent Child Internal vertex Leaf o A vertex with no child Sibling o Children of the same parent Root Parent Child

Trees Abbas Malik; FCIT, UoJ 196 1 23 4 56 Degree of a node o No. of children of a node Root Degree=2Degree=1

Trees Abbas Malik; FCIT, UoJ 197 1 23 4 56 Degree of a node o No. of children of a node Depth or Level of a node o No. of edges from the root Root Depth=1 Depth=2

Trees Abbas Malik; FCIT, UoJ 198 1 23 4 56 Degree of a node o No. of children of the node Depth or Level of a node o No. of edges from the root Height of Tree o No. of edges in the longest path from root to a leaf. Root Tree Height = 2

Binary Tree Abbas Malik; FCIT, UoJ 199 A tree with degree of every vertex not more than 2. o A vertex cannot have more than 2 children. A Binary Tree with i internal vertices contains how many vertices ? − ≤2i+1 What is the height h of a binary tree with vertices? − h  log   How many leaves l a binary tree of height h may have? − l≤2 h

Binary Search Tree Abbas Malik; FCIT, UoJ 200 An organized Binary Tree. Binary Search Tree Property o Key of a Parent will be greater than the key of left child (if any) o Key of a Parent will be smaller than the key of right child (if any) Let x be a node in Binary Search Tree. If y is node in the left sub-tree of x, then Key[y] ≤ Key[x]. If y is node in the right sub-tree of x, then Key[y] ≥ Key[x].

Binary Search Tree Abbas Malik; FCIT, UoJ 201 It is binary tree whose nodes contain elements of a set of orderable items, one element per node. All elements in the left sub-tree are smaller than the element in the right sub-tree’s root. All elements in the right sub-tree are greater than the element in the sub-tree’s root. We gain in the time efficiency of searching, insertion, and deletion – All operations are  (log 2 n), but only in an average case In Worst Case, they are O(n), because tree is severely unbalance.

AVL Tree Abbas Malik; FCIT, UoJ 202 AVL Tree is a balanced binary search tree Discovered in 1962 by Russian scientists G. M. Adelson-Velsky and E. M. Landis Definition An AVL tree is a binary search tree in which the balance factor of every node, which is defined as the difference between the heights of the node’s left and right sub-trees, is either 0 or +1 or –1

AVL Tree Abbas Malik; FCIT, UoJ 203 0 1 0 1 0 0 0 0 0 0 0 0 The Balance factor Height of Left sub-tree – Height of right sub-tree

AVL Tree Abbas Malik; FCIT, UoJ 204 Rotation If an insertion or deletion of a node makes an AVL tree unbalanced, we balanced the tree by a rotation A rotation is a local transformation of its sub-tree rooted at a node whose balance has become either +2 or –2 1. Single Right Rotation or R-rotation 2. Single Left Rotation or L-rotation 3. Double Left-Right Rotation or LR-rotation 4. Double Right-Left Rotation or RL-rotation

AVL Tree Abbas Malik; FCIT, UoJ 205 Simple R Rotation 3 2 1 2 1 0 3 2 1 0 0 0 Left child becomes the parent and parent becomes its right child

AVL Tree Abbas Malik; FCIT, UoJ 206 Simple L Rotation 1 -2 2 3 0 3 2 1 0 0 0 Right child becomes the parent and parent becomes its left child

AVL Tree Abbas Malik; FCIT, UoJ 207 LR Rotation 3 2 1 0 0 0 3 2 1 2 0 3 2 1 2 1 0 Left RotationRight Rotation

AVL Tree Abbas Malik; FCIT, UoJ 208 RL Rotation 3 2 1 0 0 0 1 -2 3 1 2 0 Right RotationLeft Rotation 1 -2 2 3 0

AVL Tree Abbas Malik; FCIT, UoJ 209 Construction of an AVL from a list of number 5683247

AVL Tree Abbas Malik; FCIT, UoJ 210 Construction of an AVL from a list of number 5 0 5683247

AVL Tree Abbas Malik; FCIT, UoJ 211 Construction of an AVL from a list of number 5 5683247 6 0

AVL Tree Abbas Malik; FCIT, UoJ 212 Construction of an AVL from a list of number 5 -2 5683247 6 8 0 8 0 6 0 5 0 Left Rotation L(5)

AVL Tree Abbas Malik; FCIT, UoJ 213 Construction of an AVL from a list of number 5683247 8 0 6 1 5 1 3 0

AVL Tree Abbas Malik; FCIT, UoJ 214 Construction of an AVL from a list of number 5683247 8 0 6 2 5 2 3 1 2 0 Right Rotation R(5) 8 0 6 1 3 0 2 0 5 0

AVL Tree Abbas Malik; FCIT, UoJ 215 Construction of an AVL from a list of number 5683247 Right Rotation R(6) 8 0 6 2 3 2 0 5 1 4 0 6 1 3 -2 2 0 5 1 4 0 8 0 Need Double Rotation

AVL Tree Abbas Malik; FCIT, UoJ 216 Construction of an AVL from a list of number 5683247 LR-Rotation(6) First Left rotation on left 8 0 6 2 3 2 0 5 1 4 0 8 0 6 2 5 2 3 0 2 0 4 0

AVL Tree Abbas Malik; FCIT, UoJ 217 Left Rotation – General Right child will become the parent and parent will become its left child. If there is a left child of the original right child, it will become the right child of the original parent. 1 -2 2 3 0 3 2 1 0 0 0

AVL Tree Abbas Malik; FCIT, UoJ 218 Left Rotation – General Right child will become the parent and parent will become its left child. If there is a left child of the original right child, it will become the right child of the original parent. A +2 B T3T3 h T2T2 h T1T1 h+1 A 0 B 0 T3T3 h T2T2 h T1T1

AVL Tree Abbas Malik; FCIT, UoJ 219 Construction of an AVL from a list of number 5683247 LR-Rotation(6) Second Right rotation 8 0 6 5 0 3 0 2 0 4 0 8 0 6 2 5 2 3 0 2 0 4 0

AVL Tree Abbas Malik; FCIT, UoJ 220 Construction of an AVL from a list of number 5683247 L-Rotation(6) 8 1 6 -2 5 3 0 2 0 4 0 7 0 6 1 8 2 5 3 0 2 0 4 0 7 0 Need Double Rotation

AVL Tree Abbas Malik; FCIT, UoJ 221 Construction of an AVL from a list of number 5683247 RL-Rotation(6) First Right rotation on right 8 1 6 -2 5 3 0 2 0 4 0 7 0 7 6 -2 5 3 0 2 0 4 0 8 0

AVL Tree Abbas Malik; FCIT, UoJ 222 Construction of an AVL from a list of number 5683247 RL-Rotation(6) Second Left rotation 7 6 -2 5 3 0 2 0 4 0 8 0 6 0 7 0 5 0 3 0 2 0 4 0 8 0

AVL Tree Abbas Malik; FCIT, UoJ 223 Right Rotation – General Left child will become the parent and parent will become its right child. If there is a right child of the original left child, it will become the left child of the original parent. 3 2 1 2 1 0 3 2 1 0 0 0

AVL Tree Abbas Malik; FCIT, UoJ 224 Right Rotation – General Left child will become the parent and parent will become its right child. If there is a right child of the original left child, it will become the left child of the original parent. B 1 A -2 T3T3 h T2T2 h T1T1 h+1 A 0 B 0 T2T2 h T3T3 h T1T1

Assignment Abbas Malik; FCIT, UoJ 225 Design an algorithm that takes a list of numbers as input; and build and return an AVL Tree as output. Give the algorithmic analysis of the above designed algorithm.

M. G. Abbas Malik Assistant Professor Faculty of Computing and IT King Abdulaziz University, KSA CPCS 324 – Algorithms and Data Structures

Comparison Sorting Algorithms Abbas Malik; FCIT, King Abdulaziz University 227 1.All sorting algorithms so far are Comparison sorting algorithms 2.The best worst-case running time is AlgorithmWorst Case Sorting Time Insertion Sortn2n2 Merge Sortn lgn Quick Sortn2n2 Heap Sortn lgn Is the best we can do?

Decision Tree for Comparison Sorting Abbas Malik; FCIT, King Abdulaziz University 228 1:2 2:3 1:3 ≤ ≤ > ≤ > > > ≤ ≤> Left sub-tree shows subsequent comparison if a i ≤ a j Right sub-tree shows subsequent comparison if a i > a j

Decision Tree for Comparison Sorting Abbas Malik; FCIT, King Abdulaziz University 229 = 1:2 2:3 1:3 ≤ > ≤ > > > ≤ ≤> 6≤14

Decision Tree for Comparison Sorting Abbas Malik; FCIT, King Abdulaziz University 230 = 1:2 2:3 1:3 ≤ > ≤ > > ≤ ≤> 14>3 6≤14

Decision Tree for Comparison Sorting Abbas Malik; FCIT, King Abdulaziz University 231 = 1:2 2:3 1:3 ≤ > ≤ > 6>3 ≤ ≤> 14>3 6≤14

Decision Tree for Comparison Sorting Abbas Malik; FCIT, King Abdulaziz University 232 1:2 2:3 1:3 6≤14 ≤ > ≤ 14>3 > 6>3 ≤ ≤> =  Each leaf contains a permutation to indicate that the ordering a π(1), a π(2), …, a π(n) 3≤6≤14

Decision Tree Model Abbas Malik; FCIT, King Abdulaziz University 233 A Decision Tree can model the execution of any Comparison Sort. One tree for each input size n. View the algorithm as splitting whenever it compare two elements. The Decision Tree contains the comparisons along its path. Running Time = The Length of the Path Worst-case Running Time = Height of Tree

Lower Bound for Decision Tree Sorting Abbas Malik; FCIT, King Abdulaziz University 234 Theorem: Any decision tree that can sort n elements must have height Ω(nlgn). Proof: Array with n elements has n! permutations (leaves) Binary tree with height ‘h’ has ≤ 2 h leaves. Where, n! ≤ 2 h => Taking log 2 of both sides log 2 n! ≤ log 2 2 h =>log 2 n! ≤ hlog 2 2 =>log 2 n! ≤ h h ≥ lg(n!)[lg is monotonicallyincreasing function] ≥ lg((n/e) n ) [Stirling’s formula] ≥ nlgn – nlge ≥ nlgn

Sorting in Linear Time Abbas Malik; FCIT, King Abdulaziz University 235 Counting Sort  No comparison between elements  Input: A[1…n], where A[j]ε{1,2,…,k}  Output: B[1…n], sorted  Auxiliary storage: C[1…k]

Counting Sort Abbas Malik; FCIT, King Abdulaziz University 236 Counting Sort(A, B, n, k)  For i = 1 to k  C[i] = 0  For j = 1 to n  C[A[j]] = C[A[j]] + 1  For i = 2 to k  C[i] = C[i] + C[i-1]  For j = n to 1  B[C[A[j]]] = A[j]  C[A[j]] = C[A[j]] – 1

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 237 61252 A C B

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 238 61252 A 000000 C B Loop 1 1For i = 1 to k 2C[i] = 0

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 239 61252 A 000000 C B Loop 2 1For j = 1 to n 2C[A[j]] = C[A[j]]+1 123456

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 240 61252 A 000000 C B Loop 2 1For j = 1 to n 2C[A[j]] = C[A[j]]+1 123456 1 j

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 241 61252 A 000001 C B Loop 2 1For j = 1 to n 2C[A[j]] = C[A[j]]+1 123456 1 j

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 242 61252 A 000001 C B Loop 2 1For j = 1 to n 2C[A[j]] = C[A[j]]+1 123456 2 j

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 243 61252 A 100001 C B Loop 2 1For j = 1 to n 2C[A[j]] = C[A[j]]+1 123456 2 j

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 244 61252 A 100001 C B Loop 2 1For j = 1 to n 2C[A[j]] = C[A[j]]+1 123456 3 j

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 245 61252 A 110001 C B Loop 2 1For j = 1 to n 2C[A[j]] = C[A[j]]+1 123456 3 j

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 246 61252 A 110001 C B Loop 2 1For j = 1 to n 2C[A[j]] = C[A[j]]+1 123456 4 j

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 247 61252 A 110011 C B Loop 2 1For j = 1 to n 2C[A[j]] = C[A[j]]+1 123456 4 j

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 248 61252 A 110011 C B Loop 2 1For j = 1 to n 2C[A[j]] = C[A[j]]+1 123456 5 j

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 249 61252 A 120011 C B Loop 2 1For j = 1 to n 2C[A[j]] = C[A[j]]+1 123456 5 j

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 250 61252 A 120011 C B Loop 3 1For i = 2 to k 2C[i] = C[i] + C[i-1] 123456

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 251 61252 A 120011 C B Loop 3 1For i = 2 to k 2C[i] = C[i] + C[i-1] 123456 2 i

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 252 61252 A 130011 C B Loop 3 1For i = 2 to k 2C[i] = C[i] + C[i-1] 123456 2 i

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 253 61252 A 130011 C B Loop 3 1For i = 2 to k 2C[i] = C[i] + C[i-1] 123456 3 i

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 254 61252 A 133011 C B Loop 3 1For i = 2 to k 2C[i] = C[i] + C[i-1] 123456 3 i

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 255 61252 A 133011 C B Loop 3 1For i = 2 to k 2C[i] = C[i] + C[i-1] 123456 4 i

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 256 61252 A 133311 C B Loop 3 1For i = 2 to k 2C[i] = C[i] + C[i-1] 123456 4 i

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 257 61252 A 133311 C B Loop 3 1For i = 2 to k 2C[i] = C[i] + C[i-1] 123456 5 i

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 258 61252 A 133341 C B Loop 3 1For i = 2 to k 2C[i] = C[i] + C[i-1] 123456 5 i

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 259 61252 A 133341 C B Loop 3 1For i = 2 to k 2C[i] = C[i] + C[i-1] 123456 6 i

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 260 61252 A 133345 C B Loop 3 1For i = 2 to k 2C[i] = C[i] + C[i-1] 123456 6 i

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 261 61252 A 133345 C B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 262 61252 A 133345 C B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 5 j A[j] = 2 C[A[j]] = 3

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 263 61252 A 133345 C 2 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 5 j A[j] = 2 C[A[j]] = 3

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 264 61252 A 123345 C 2 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 5 j A[j] = 2 C[A[j]] = 2

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 265 61252 A 123345 C 2 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 4 j A[j] = 5 C[A[j]] = 4

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 266 61252 A 123345 C 25 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 4 j A[j] = 5 C[A[j]] = 4

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 267 61252 A 123335 C 25 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 4 j A[j] = 5 C[A[j]] = 3

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 268 61252 A 123335 C 25 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 3 j A[j] = 2 C[A[j]] = 2

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 269 61252 A 123335 C 225 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 3 j A[j] = 2 C[A[j]] = 2

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 270 61252 A 113335 C 225 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 3 j A[j] = 2 C[A[j]] = 1

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 271 61252 A 113335 C 225 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 2 j A[j] = 1 C[A[j]] = 1

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 272 61252 A 113335 C 1225 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 2 j A[j] = 1 C[A[j]] = 1

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 273 61252 A 013335 C 1225 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 2 j A[j] = 1 C[A[j]] = 0

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 274 61252 A 013335 C 1225 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 1 j A[j] = 6 C[A[j]] = 5

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 275 61252 A 013335 C 12256 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 1 j A[j] = 6 C[A[j]] = 5

Counting Sort – Example Abbas Malik; FCIT, King Abdulaziz University 276 61252 A 013334 C 12256 B Loop 4 1For j = n to 1 2B[C[A[j]]] = A[j] 3C[A[j]] = C[A[j]] – 1 123456 1 j A[j] = 6 C[A[j]] = 4 Array is sorted!

Counting Sort(A, B, n, k) 1 For i = 1 to k 2 C[i] = 0 3 For j = 1 to n 4 C[A[j]] = C[A[j]] + 1 5 For i = 2 to k 6 C[i] = C[i] + C[i-1] 7 For j = n to 1 8 B[C[A[j]]] = A[j] 9 C[A[j]] = C[A[j]] – 1 Counting Sort – Analysis Abbas Malik; FCIT, King Abdulaziz University 277 O(n+k)

Algorithms and Data Structure-2 CPCS-324 By Aman Ullah Edited by Dr. Muhammad Murtaza Khan Aman Ullah Algorithms and Data Structure-2

Space and Time Trade-offs (Input Enhancement in String Matching) Aman Ullah Algorithms and Data Structure-2

In this lecture we will cover the following String Matching Brute-Force Algorithm Horspool’s Algorithm Boyer-Moore Algorithm Aman Ullah Algorithms and Data Structure-2 Today’s Topics

In string matching, we find an occurrence of a given string of ‘m’ characters (called the Pattern) in a longer string of ‘n’ characters (called the text). Pattern: Given string of ‘m’ characters which is to be matched/searched. Text: Longer string of ‘n’ characters from which pattern will be searched. Aman Ullah Algorithms and Data Structures-2 String Matching

Align the pattern of ‘m’ characters against the first ‘m’ characters of the text Match corresponding pairs of characters from left to right until either mismatch occurs or all characters match In case of mismatch, shift the pattern one position to the right and resume character comparisons Note that last position in the text which can still be the beginning of a matching substring is n-m (provided the text’s positions are indexed from 0 to n-1). For text with n=8 and pattern m=3 characters, last position at which this pattern can be searched will be 8-3 = 5 Aman Ullah Algorithms and Data Structures-2 Brute Force Algorithm (For String Matching)

The above situation can be explained by the following example: Aman Ullah Algorithms and Data Structures-2 Brute Force Algorithm (Contd...) If Not found, shift the pattern one character to right Matched/Found

Outer for loop runs for a maximum of n-m+1 (i.e. From 0 to n-m) times Inner while loop will run for a maximum of m (i.e. From 0 to m-1) times Hence worst-case number of character comparisons is m(n-m+1). This puts the worst-case performance of the brute-force algorithm in the class (nm). Aman Ullah Algorithms and Data Structures-2 Analysis of Brute Force Algorithm

Several better algorithms for string matching have been discovered e.g. Knuth-Morris-Pratt algorithm, Boyer-Moore algorithm, Horspool’s algorithm. Most of them use the input enhancement idea. In this approach, they preprocess the pattern to get some information about it, store this information in a table and then use this information during an actual search for the pattern in a given text. Algorithms based on Input Enhancement Idea Aman Ullah Algorithms and Data Structures-2

Suppose we want to search a pattern BARBER in some text s 0... c... s n-1 Start with the right most character of pattern and moving right to left, compare corresponding pairs of characters in the pattern and the text. If we encounter a mismatch, we need to shift the pattern to right, like the following case. We want to shift pattern to the right as much as possible without risking the possibility of missing the pattern in text. Aman Ullah Algorithms and Data Structures-2 Horspool’s Algorithm

Horspool’s algorithm determines the size of such a shift by looking at the character c of the text that was aligned against the last character of the pattern. –Case 1: If there are no c’s in the pattern. e.g. if c is letter S in, we can safely shift the pattern by its entire length –Case 2: Character c present in the pattern but has multiple instances, e.g. if c is the character B in our example, then shift should align the rightmost occurrence of c (i.e. B in our case) in the pattern with the c in the text (i.e. shift should be made equal to the number of characters to the right of B which is 2 in our case) as: Aman Ullah Algorithms and Data Structures-2 Horspool’s Algorithm (Contd…)

–Case 3: If c happens is the last character in the pattern but there are no c’s among its other m-1 characters, then the shift should be similar to that of case 1 i.e. pattern should be shifted by the entire pattern’s length m as: –Case 4: If c is the last character in pattern and there are other c’s among its first m-1 characters, the shift should be similar to that of case 2 Aman Ullah Algorithms and Data Structures-2 Horspool’s Algorithm (Contd…)

Before beginning the search, we can scan the pattern, precompute the shift sizes and store them in a table called “Shift table”. It contains indexes of all possible characters that can be encountered in a text, including punctuation symbols and other special characters (for natural language texts). if c is not among the first m-1 characters of the pattern, then t(c) = the pattern’s length m else t(c) = the distance from the rightmost c, among the first m-1 characters of the pattern to its last character Aman Ullah Algorithms and Data Structures-2 Shift Table

Suppose we want to search a pattern BARBER from the text that comprises English letters and spaces (denoted by underscores). The Shift table is filled based on the previous algorithm as follows: Aman Ullah Algorithms and Data Structures-2 Example (Constructing Shift Table) Character cABCDEF...R Z_ Shift t(c)42661663666

Step 1: For pattern of length m and the alphabets used in both the pattern and text, construct the shift table. Step 2: Align the pattern against the beginning of the text. Step 3: Compare the corresponding characters in the pattern and text until either all m characters are matched. Step 4: In case of mis-match, retrieve the entry t(c) from the c’s column of the shift table where c is the text’s character currently aligned against the last character of the pattern, and shift the pattern by t(c) characters to the right along the text. Aman Ullah Algorithms and Data Structures-2 Working of Horspool’s Algorithm

Then its shift table is as follows Suppose we want to search the above pattern from a particular text as: JIM SAW ME IN A BARBERSHOP Aman Ullah Algorithms and Data Structures-2 Example (Horspool’s Algorithms) A B C D E F G H I J K L M N O P Q R S T U V W X Y Z 4 2 6 6 1 6 6 6 6 6 6 6 6 6 6 6 6 3 6 6 6 6 6 6 6 6 _ 6 Example 1: Suppose Text: English Letters and spaces (denoted by _) and Pattern: BARBER

The working of Horspool’s algorithm will be as follows: J I M _ S A W _ M E _ I N _ A _ B A R B E R S H O P B A R B E R Aman Ullah Algorithms and Data Structures-2 Example (Horspool’s Algorithms) (Contd...) All characters matched

Aman Ullah Algorithms and Data Structures-2 Example (Horspool’s Algorithm) A B C D E F G H I J K L M N O P Q R S T U V W X Y Z 1 2 6 6 6 6 6 6 6 6 6 6 6 6 3 6 6 6 6 6 6 6 6 6 6 6 _ 6 Example 2: BARD LOVED BANANAS BAOBAB BAOBAB (unsuccessful search) Example 3: BARD LOVED BAOBAB BANANAS BAOBAB BAOBAB (successful search)

Aman Ullah Algorithms and Data Structures-2 Horspool’s Algorithm

BM determines shift size by considering t 1 (c)- k. t 1 (c) is the entry in the precomputed shift table (and is called Bad Symbol Shift) and k is the number of matched characters. Example, search for pattern BARBER in some text and match last two characters before failing on letter S, we can shift the pattern by t 1 (S) - 2 = 6 – 2 = 4 positions to the right. (Since letter S does not occur in the pattern, therefore t 1 (S) = 6 (length of pattern)). (In Horspool’s algorithm, we used t 1 (R)) Aman Ullah Algorithms and Data Structures-2 Boyer-Moore (BM) Algorithm

For example, if we search for the pattern BARBER in some text and match the last two characters before failing on letter A (which exists in the pattern), we can shift the pattern by t 1 (A)- k = 4 – 2 = 2 positions to the right. Note: If t 1 (c)- k ≤ 0, we do not want to shift the pattern by 0 or negative number of positions. So, we shall shift the pattern simply by one position to the right by using the formula: d 1 = max(t 1 (c) – k, 1) Aman Ullah Algorithms and Data Structures-2 Boyer-Moore Algorithm (Contd...)

Use good-suffix shift, (matched characters from right) of the pattern as its suffix of size k and denote it by suff(k). In order to fill in the good-suffix shift table, we consider the following cases: Case 1: Suppose there is another occurrence of suff(k) in the pattern. In this case, we can shift the pattern by the distance d 2 between such a second rightmost occurrence (not preceded Aman Ullah Algorithms and Data Structures-2 Good-Suffix Shift kPatternd2d2 1ABCBAB2 2 4

Case 2: To avoid any erroneous shift based on a suffix of size k, for which there is no other occurrence in the pattern, we need to find the longest prefix of size l < k that matches the suffix of the same size l. If such a prefix exists, the shift size d 2 is computed as the distance between this prefix and the corresponding suffix; otherwise d 2 is set to the pattern’s length m. A good-suffix table for the pattern ABCBAB is as follows: Good-Suffix Shift (Contd...) Aman Ullah Algorithms and Data Structures-2 kPatternd2d2 1ABCBAB2 2 4 3 4 4 4 5 4

Unfortunately, shifting the pattern by its entire length when there is no other occurrence of suff(k), not preceded by the same character as in its last occurrence, is not always correct. For example, for the pattern ABCBAB and k=3, shifting by 6 positions could miss a matching substring that starts with the text’s AB aligned with the last two characters of the pattern, as shown below: Aman Ullah Algorithms and Data Structures-2 Good-Suffix Shift (Contd…)

Here we note that a shift by 6 positions is correct for the pattern DBCBAB but not for ABCBAB, because the latter pattern has the same substring AB as its prefix (beginning part of the pattern) and as its suffix (ending part of the pattern). To avoid such an erroneous shift based on a suffix of size k, for which there is no other occurrence in the pattern not preceded by the same character as in its last occurrence, we need to find the longest prefix of size l < k that matches the suffix of the same size l. Aman Ullah Algorithms and Data Structures-2 Good-Suffix Shift (Contd…)

If such a prefix exists, the shift size d 2 is computed as the distance between this prefix and the corresponding suffix; otherwise d 2 is set to the pattern’s length m. A good-suffix table for the pattern ABCBAB is as follows: kPattern d 2 1ABCBAB 2 2ABCBAB 4 3ABCBAB 4 4ABCBAB 4 5ABCBAB 4 Aman Ullah Algorithms and Data Structures-2 Good-Suffix Shift (Contd…)

Step 1: For a given pattern and the alphabets used in both the pattern and the text, construct the bad-symbol shift table. Step 2: Using the pattern, construct the good-suffix shift table. Step 3: Align the pattern against the beginning of the text. Step 4: Repeat the following step until either a matching substring is found or the pattern reaches beyond the last character of the text. Starting with the last character in the pattern, compare the corresponding characters in the pattern and the text until either all m character pairs are matched (then stop) or a mismatching pair is encountered after k ≥ 0 character pairs are matched successfully. In the latter case, retrieve the entry t 1 (c) from the c’s column of the bad-symbol table where c is the text’s mismatched character. Aman Ullah Algorithms and Data Structures-2 Working of Boyer-Moore Algorithm

If k > 0, also retrieve the corresponding d 2 entry from the good-suffix table, shift the pattern to the right by the number of positions computed by the formula: d 1 if k = 0 max(d 1, d 2 ) if k > 0 where d 1 = max(t 1 (c) – k, 1) Aman Ullah Algorithms and Data Structures-2 Working of Boyer-Moore Algorithm (Contd…) d =

Pattern: BAOBAB Text: English letters and spaces (denoted by underscores). Bad Symbol Table: A good-suffix table for the pattern BAOBAB is as follows: kPattern d 2 1BAOBAB 2 2BAOBAB 5 3BAOBAB 5 4BAOBAB 5 5BAOBAB 5 Aman Ullah Algorithms and Data Structures-2 Example (Contd...) Character cABCDEF...O Z_ Shift t 1 (c) 12666663666

B E S S _ K N E W _ A B O U T _ B A O B A B S B A O B A B d 1 =t(k)-0=6 B A O B A B d 1 =t(_)-2=4 d 2 =5 Max(d 1,d 2 ) B A O B A B d 1 =t(_)-1=5 d 2 =2 Max(d 1,d 2 ) B A O B A B Aman Ullah Algorithms and Data Structures-2 Example (Contd...) All matched

Conclusion: Horspool's algorithm only used the character value of the text aligned with last character of the pattern to determine the shift. Boyer-Moore algorithm also uses the location and the character mismatch to calculate the shift. In addition it uses the occurrences of the suffixes in the pattern to determine the shift. Aman Ullah Algorithms and Data Structures-2 Conclusion

Aman Ullah Algorithms and Data Structures-2 Boyer-Moore Algorithm

Algorithms and Data Structure-2 CPCS-324 By Aman Ullah Aman Ullah Algorithms and Data Structure-2

Lecture No 5 Space and Time Tradeoffs (B-Trees) Aman Ullah Algorithms and Data Structure-2

In this lecture we will cover the following B-Trees Balanced B-Tree Rules of Insertion in B-Tree Construction of B-Tree (Inserting a Key) Example Balancing a B-Tree Combining a B-Tree Deleting a key from B-Tree Example B-Tree Traversal Exercises Aman Ullah Algorithms and Data Structure-2 Today’s Topics

A B-Tree of order m ≥ 2 must satisfy the following structural properties. –The root is either a leaf or has between 2 and m children. –Each node, except for the root and the leaves, has between ceil(m/2) and m children (and hence between ceil(m/2) -1 and m -1 keys) –The tree is perfectly balanced i.e. all its leaves are at the same level. A typical pre-structuring of B-Tree for searching is as follows, with keys of the data entries in order: Aman Ullah Algorithms and Data Structures-2 B-Trees (Contd...) m = 4 Here we have 4 children and 3 keys

An example of a B-Tree of order 4 is given as follows: Aman Ullah Algorithms and Data Structures-2 B-Tree (Example) It can have between 2 & 4 children. Each internal node has between ceil(m/2) i.e. 2 and m i.e. 4 children and between ceil(m/2)-1 i.e. 1 and m-1 i.e. 3 keys.

B-tree is a perfectly balanced search tree with the following additional properties: 1.The root is not a leaf : It has 2 to m subtrees, where m is the order. 2.All internal nodes have at least (ceil (m/2) non-null subtrees and at most m non-null subtrees. 3.All leaf nodes are at the same level; that is the tree is perfectly balanced. 4.A leaf node has at least (ceil (m/2) – 1) and at the most m-1 entries. The following table defines the number of subtrees for non-root nodes: Aman Ullah Algorithms and Data Structures-2 Balanced B-Tree mCeiling |m/2|m... 636 535 424 323 MaximumMinimum Number of Subtrees Order

Aman Ullah Algorithms and Data Structures-2 Balanced B-Tree (Example) A B-tree of order; m=5. Min entry : Ceiling|5/2| - 1 = 2 entries i.e. keys. Max entry: m – 1 = 5 – 1 = 4 entries i.e. keys. Min subtrees : Ceiling|5/2|=3 Max subtrees: m = 5 For internal nodes apart from root For internal nodes and leaves

Consider the following B-Tree of order 4: After inserting the key 65 into it, we get the following B-Tree: Aman Ullah Algorithms and Data Structures-2 Insertion in B-Tree

Aman Ullah Algorithms and Data Structures-2 Insertion in B-Tree (Contd...)

Following are the rules for inserting any key into the existing B-Tree: Aman Ullah Algorithms and Data Structures-2 Rules of Insertion in B-Tree  B-tree grows from the bottom up.  When the node is full, this condition is called as overflow.  Overflow requires that the node be split into two nodes and median entry is pushed up, the tree grows one level. m=5

In the following example, we construct a B-tree of order 5. This means that (other than the root node) all internal nodes will have at least ceil(5 / 2) = ceil(2.5) = 3 children (and hence at least 2 keys). Of course, the maximum number of children that a node can have is 5 (so that 4 is the maximum number of keys). Each leaf node must contain at least 2 keys. Suppose we want to insert the following letters into what is originally an empty B-tree of order 5: C N G A H E K Q M F W L T Z D P R X Y S The first 4 letters get inserted into the same node (as a node can have maximum of 4 keys), resulting in the following picture: Aman Ullah Algorithms and Data Structures-2 Construction of B-Tree

When we try to insert the H, we find no room in this node, so we split it into 2 nodes, moving the median item (i.e. central item in sorted letters A, C, G, H, N) G up into a new root node as. Construction of B-Tree (Contd...) Aman Ullah Algorithms and Data Structures-2

Inserting E, K, and Q proceeds without requiring any splits as follows: Now inserting M requires a split of the right subtree. Note that M happens to be the median key (H, K, M, N, Q) and so is moved up into the parent node. Aman Ullah Algorithms and Data Structures-2 Construction of B-Tree (Contd...)

Thus the B-Tree looks like the following: The letters F, W, L, and T are then added without needing any split. Aman Ullah Algorithms and Data Structures-2 Construction of B-Tree (Contd…)

When Z is added, the rightmost leaf must be split. The median item T is moved up into the parent node. Note that by moving up the median key, the tree is kept fairly balanced, with 2 keys in each of the resulting nodes. The insertion of D causes the leftmost leaf to be split (as number of keys will exceed 4 in this subtree). D happens to be the median key and so is the one moved up into the parent node. The letters P, R, X, and Y are then added without any need of splitting. So B-Tree will look like the following: Aman Ullah Algorithms and Data Structures-2 Construction of B-Tree (Contd…)

Finally, when S is added to the second last subtree, the node with N, P, Q, and R splits, sending the median key Q up to the parent. However, the parent node is full (as it will have 5 keys in it), so it splits, sending the median M up to form a new root node. Note how the 3 pointers from the old parent node stay in the revised node that contains D and G. The final B-Tree will be as follows: Aman Ullah Algorithms and Data Structures-2 Construction of B-Tree(Contd…)

Aman Ullah Algorithms and Data Structures-2 Construction of B-Tree (Contd…)

Aman Ullah Algorithms and Data Structures-2 Another Example of Constructing B-Tree m = 5

Aman Ullah Algorithms and Data Structures-2 Example (Contd…)

Aman Ullah Algorithms and Data Structures-2 Balancing a B-Tree Order m=5. Min entry = 2 entries. Max entry= 4 entries.

Aman Ullah Algorithms and Data Structures-2 Combining a B-Tree Order m=5

Algorithms and Data Structure-2 CPCS-324 By Aman Ullah Edited by Dr. Muhammad Murtaza Khan Aman Ullah Algorithms and Data Structure-2

Data Access by a Key 331 Student Grades: (Student_ID, Record) –Student record contain multiple information – a collection of data about a Student (Name, DOB, Session, Courses, Grades, etc.) identified by unique Student ID Employee Record: (Employee_ID, Record) –A collection of data about an employee (Name, DOB, address, nationality, salary, etc.) identified by unique employee ID Supermarket Products: (Product_ID, Record) –Collection of data about the product (Name, Manufacturer, Date of manufacturing, Expiry Date, Warranty) identified by unique product ID

Limitation of using Array 332 Array stores n values indexed 0, 1, …, (n-1) –Maximum size must be known in advance –Index is an integer Key in a (Key, value) pair might not be a number –Can be converted into a number: having separate number for all possible keys. Extremely large array is required –Large number of possible keys (all possible products) –Total number of unique keys might be unknown –Major parts of array may be empty (example: years of flood) 19141939 1965

Using Array of (Key, record) Abbas Malik; FCIT, King Abdulaziz University 333 An array of structure/class of (key, record) can be used Managing the array –Add an element to the end –Find an element: search the array from start –Remove an element: find the element and copy the last element over it. Problems –Allocating too small memory: run out of space –Allocating too large memory: Space inefficient 1914World War 1 0 1939World War 2 1 1965 War 2 of Indo-Pak 2

Using Linked List 334 Each element is structure/class –Key, value, pointer to next element Linked List –Pointer to the first element of the list –Functions for adding and removing elements –Function for searching an element with a specific key ValueKeyNextValueKeyNext First ValueKeyNextValueKeyNext Null

Using Linked List 336 Adding element to the list Adding new element to the front of the list – Adding at first element –Make a new structure/class and point the next to first new -> next = first –Now point the first to this new structure/class first = new Value Key Next first Null Value Key Next Value Key Next Value Key Next new

Using Linked List 340 Searching an element Start from first element and search an element with a specific key. Return the pointer/address of searched element Or NULL in case the element is not found. WW1 1914 Next first Null WW2 1939 Next w2IP 1965 Next

Searching an element in Linked List 341

Search an element in Linked List 342

Search an element in Linked List 343

Using Linked List 344 Searching and delete an element While searching, also keep track of the previous element –In case delete first, point the first to the 2 nd element WW1 1914 Next first Null WW2 1939 Next w2IP 1965 Next

Using Linked List 345 Searching and delete an element While searching, also keep track of the previous element –In case delete first, point the first to the 2 nd element first Null WW2 1939 Next w2IP 1965 Next

Using Linked List 346 Searching and delete an element While searching, also keep track of the previous element –In case delete first, point the first to the 2 nd element –Otherwise, point the previous element’s next to the next of deleting element WW1 1914 Next first Null WW2 1939 Next w2IP 1965 Next prevp

Delete an element in Linked List 347

Delete an element in Linked List 348

Using Linked List 349 Good place to start –Simple algorithms and data structure Problems Searching or deleting an element –Require walk trough elements of the list Could store elements in sorted order –Keeping them in sorted order takes time –Searching by key still takes time Need a better approach Memory efficient: adds extra memory as needed Time efficient: finds element with its key instantly

We want an efficient way to implement dictionaries. Dictionary is an abstract data type with operations like searching, insertion and deletion defined on its elements. Elements are of an arbitrary nature: numbers, characters of some alphabet, character string etc. Records comprise several fields, each responsible for keeping a particular type of information. Key is used to uniquely identify the records (e.g. the student’s ID). For a dictionary of n records we have keys K 1, K 2,.. K n. Aman Ullah Algorithms and Data Structures-2 Dictionary

Hashing is based on the idea of distributing keys among a one dimensional array H[0.. m-1] called a hash table. Hash function assigns an integer between 0 and m-1, called the hash address, to a key. For example, if keys are non-negative integers, a hash function can be of the form: h(K)= K (Problem: Size) Better solution is h(K)=K mod m (remainder of k by m is always between 0 and m-1). Here m is an arbitrarily selected size of the hash table. If keys are letters of some alphabet assign their position as key, (e.g. position of A in English alphabet is 1 and that of N is 14 etc.) Aman Ullah Algorithms and Data Structures-2 Hashing

If key i.e. K is a character string of the form c 0, c 1,.... c s-1, we can use a better option to compute h(K) as follows: h ← 0; for i ← 0 to s-1 do h ← ord(c i ) mod m h ← (h * C + ord(c i )) mod m (Better Alternate) where m is the arbitrary size of the hash table and C is a constant larger than every ord(c i ). Aman Ullah Algorithms and Data Structures-2 Hashing (Contd...) For example, if K is MONEY, then c 0 =M and ord(c 0 ) = 13, c 1 =O and ord(c 1 ) = 15, c 2 =N and ord(c 2 ) = 14, c 3 =E and ord(c 3 ) = 5, c 4 =Y and ord(c 4 ) = 25

Every hash function needs to satisfy two somewhat conflicting requirements: –A hash function needs to distribute keys among the cells of the hash table as evenly as possible. Due to this reason, the value of m (size of hash table) is usually chosen to be prime. –A hash function has to be easy to compute. If we choose a hash table’s size m to be smaller than the number of keys n, we shall get collisions. A collision is a phenomenon when two or more keys are hashed into the same cell of hash table as shown below: Aman Ullah Algorithms and Data Structures-2 Collision

Collisions should be expected even if m is considerably larger than n. In worst-case, all the keys could be hashed to the same cell of the hash table. Every hashing scheme must have a collision resolution mechanism. There are two commonly used collision resolution mechanisms, namely –Open Hashing (Separate Chaining) –Closed Hashing (Open Addressing) Collision Resolution Mechanism Aman Ullah Algorithms and Data Structures-2

In open hashing, keys are stored in linked lists (rather than an array) attached to the cells of a hash table. Each list contains all the keys hashed to its cell. Consider the following list of words (hash keys): A, FOOL, AND, HIS, MONEY, ARE, SOON, PARTED Use a simple function (add positions of word’s alphabets and compute the sum’s remainder after division by 13) (arbitrarily selected hash table size). We start with the empty hash table. Aman Ullah Algorithms and Data Structures-2 Open Hashing (Separate Chaining)

The first key is the word A. So, its hash value is h(A) = 1 mod 13 = 1 The second key is the word FOOL. So, its hash value is h(FOOL) = (6+15+15+12) mod 13 = 48 mod 13 = 9 Our final hash table will be as follows: Aman Ullah Algorithms and Data Structures-2 Open Hashing (Contd…)

Here we note that there is a collision of the keys ARE and SOON at position 11. Suppose we want to search for the key KID in the hash table. Since h(KID) = (11+9+4) mod 13 = 24 mod 13 = 11, we shall go to the location 11 of hash table. Since linked list attached to cell 11 is not empty, we shall look into its linked list one by one. After comparing the string KID first with the string ARE of the linked list of cell 11, and then with the string SOON, we end up with an unsuccessful search. Note: Insertions are normally done at the end of a linked list and deletion is performed by searching for a key to be deleted and then removing it from the linked list. Hence the efficiency of these operations is identical to that of searching and they are all θ(1) (Constant time). Aman Ullah Algorithms and Data Structures-2 Open Hashing(Contd…)

All keys are stored in the hash table itself (no linked list) For this, table size m must be at least as large as the number of keys n. During construction of hash table, collision is resolved by employing “linear probing” Linear probing, checks the following cell –If that cell is empty, this new key is installed there. –If next cell is already occupied, the availability of that cell’s immediate successor is checked. During this process, if end of hash table is reached, search is wrapped (started) from the beginning of the table (like circular array). Aman Ullah Algorithms and Data Structures-2 Closed Hashing (Open Addressing)

In order to understand the construction of hash table with linear probing technique, consider the following keys and hash addresses: While constructing the hash table, we first encounter the word ‘A” and write it in location 1 of the table (according to above hash addresses). When we encounter the second word ‘FOOL’, we write it at location 9 of the hash table in second row (and all its previous keys). When we encounter the third word ‘AND’, we write it at location 6 of the hash table in third row (and all its previous keys) and so on. Aman Ullah Algorithms and Data Structures-2 Example (Closed Hashing)

When we want to place the key SOON at the location 11 of hash table, we see that it is already occupied. So, we look at the next cell i.e. 12 which is empty. Thus we place the key SOON in cell 12. Now finally, when we want to place the last key PARTED at location 12 of hash table, we see that it has already been occupied by the key SOON. Since we have already reached the end of hash table, we go to the start of the hash table i.e. the location 0, which is empty. So, we place the key PARTED at location 0 of the hash table. The entire hash table looks like the following: Aman Ullah Algorithms and Data Structures-2 Example (Closed Hashing) (Contd...)

Aman Ullah Algorithms and Data Structures-2 Example (Closed Hashing) (Contd...) A AFOOL AANDFOOL AANDFOOLHIS AANDMONEYFOOLHIS AANDMONEYFOOLHISARE AANDMONEYFOOLHISARESOON PARTEDAANDMONEYFOOLHISARESOON 0 1 2 3 4 5 6 7 8 9 10 11 12

To search a given key K, compute its hash value h(K), by using the same hash function (i.e. h(K) = ord(K) mod 13), which we used in the hash table’s construction. If the cell h(K) is empty the search is unsuccessful. If the cell h(K) is not empty Compare the key K with the cell’s occupant. If equal We have found a matching key. If not We compare K with a key in the next cell and continue in this manner until we encounter either a matching key (a successful search) or an empty cell (unsuccessful search). For example, for word LIT in table, h(LIT) = (12+9+20) mod 13 = 2. Since cell 2 is empty, we stop immediately as unsuccessful search. Aman Ullah Algorithms and Data Structures-2 Searching a Hash Table

Now if we search for the word KID, we see that h(KID) = (11+9+4) mod 13 = 11. We shall compare KID with the content of the cell 11 which is ARE. Since KID is not matched with ARE, we move to its subsequent cell i.e. 12. Here we get SOON, which is also not matched with KID. Now, we wrap the search and go to cell 0 and then cell 1 and cell 2 where we announce an unsuccessful search when we meet with the empty entry of cell 2. Aman Ullah Algorithms and Data Structures-2 Explanation (Contd...)

In closed hashing technique, search and insertion operations are straightforward, but deletion is not. Suppose we delete the key ARE from the above hash table. As a result, we shall get a blank in cell 11. Now when we try to search the key SOON from the hash table (which is now at location 12) with h(SOON) = (19+15+15+14) mod 13 = 11, the algorithm would find this location empty and will report an unsuccessful search. A simple solution to such problem is to use “lazy deletion” i.e. mark previously occupied locations by a special symbol to distinguish them from locations that have not been occupied. Aman Ullah Algorithms and Data Structures-2 Problem with Delete Operation (Lazy Deletion)

M. G. Abbas Malik Assistant Professor Faculty of Computing and IT University of Jeddah, KSA CPCS 324 – Algorithms and Data Structures

Assembly Line Scheduling 366

Smallest Time through Assembly Line 367 Two different assembly lines Each assembly line contains n stations A partially assembled car can be transferred to the other assembly line in case of some special rush order Total 2 n possible paths through two assembly lines (Exponential Problem) Ω(2 n ) time is required to analyze 2 n possible paths of each containing n stations Exponential problems are usually practically impossible for sufficiently large n

Dynamic Programming 368 Solves problems by combining the solutions to sub-problems Its design techniques are similar to Divide & Conquer Normally it is used for Optimality Problems It allows to solve different types of problems in time O(n 2 ) or O(n 3 ) for which a naïve approach would take Exponential time Examples: Longest (shortest) path, Longest Comment Subsequence, Matrix-chain multiplication.

Dynamic Programming Divide & Conquer 369 Sub-problems are not independent It solves a sub- problem just once and uses the solution in future when same sub-problem is encountered (Memoization) Sub-problems are independent It solve a sub-problem every time when the same sub-problem is encountered

Design of Dynamic Programming 370 A Dynamic Programming Algorithm design can be broken into 4 steps 1. Characterize the Structure of an optimal solution 2. Recursively define value of an optimal solution 3. Compute the value of an optimal solution in a bottom-up fashion 4. Construct an optimal solution from computed information

Design of Assembly Line Algo. 371 Step 1: The Structure of the fastest way through the factory

Design of Assembly Line Algorithm 372 Step 1: The Structure of the fastest way through the factory If we look at the fastest way through station S 1,j, it must go through station (j – 1) on either line 1 or line 2. Thus fastest way through station S 1,j is either 1. Directly through station S 1,j-1 or 2. Through station S 2,j-1 : a transfer from line 2 to line 1

Design of Assembly Line Algorithm 373 Step 2: Recursive Solution To define the value of an optimal solution recursively in terms of the optimal solutions to sub- problems Sub-problem: finding the fastest way through station S i,j for i = 1, 2 and j = 1, 2, … n Let f i [j] -> the fastest possible time to get a chassis from the starting point through station S i,j Fastest time through entire factory

Design of Assembly Line Algorithm 374 Step 2: Recursive Solution Assembly Line 1

Design of Assembly Line Algorithm 375 Step 2: Recursive Solution Assembly Line 1 Assembly Line 2

Design of Assembly Line Algorithm 376 Step 3: Computing Fastest Times Write a recursive algorithm based on the following equations Assembly Line 1 Assembly Line 2

Design of Assembly Line Algorithm 377 Step 3: Computing Fastest Times The problem with such a recursive algorithm: its running time is exponential in n. Suppose r i (j) be the number of references made to F i [j] in a recursive algorithm For recurrence we have: r 1 (n)=r 2 (n)=1 Abbas Malik; FCIT, King Abdulaziz University

Design of Assembly Line Algorithm 378 Step 3: Computing Fastest Times If we compute the F i [j] values in a different order from the recursive way, then we can do much better For j -> 2, each value of F i [j] depends only on the values of F 1 [j-1] and F 2 [j-1] By computing the F i [j] values in order of increasing station numbers j – left-to-right We can compute the fastest way through the factory in Θ(n)

Assembly Line Scheduling Abbas Malik; FCIT, King Abdulaziz University 379 Step 3: Computing Fastest Times

Assembly Line Scheduling Abbas Malik; FCIT, King Abdulaziz University 380 Step 3: Computing Fastest Times Computes F i [j] for j=1

Assembly Line Scheduling Abbas Malik; FCIT, King Abdulaziz University 381 Step 3: Computing Fastest Times Computes F i [j] and l i [j] for j > 2 and i = 1, 2

Assembly Line Scheduling Abbas Malik; FCIT, King Abdulaziz University 382 Step 3: Computing Fastest Times Computes Smallest time

Assembly Line Scheduling Abbas Malik; FCIT, King Abdulaziz University 383 Analysis Fastest-Way Constant time

Assembly Line Scheduling Abbas Malik; FCIT, King Abdulaziz University 384 Analysis Fastest-Way Loop runs (n- 1) times, thus the complexity of this part is linear

Assembly Line Scheduling Abbas Malik; FCIT, King Abdulaziz University 385 Analysis Fastest-Way Constant time

Assembly Line Scheduling Abbas Malik; FCIT, King Abdulaziz University 386 Analysis Fastest-Way The algorithm is linear, i.e. Θ(n)

Design of Assembly Line Algorithm 387

Design of Assembly Line Algorithm 388 Step 4: Constructing the fastest way We have computed the values F i [j], F*, l i [j] and l *, from these values we need to construct the sequence of fastest way through the factory

Algorithms and Data Structure-2 CPCS-324 By Aman Ullah Aman Ullah Algorithms and Data Structures-2

Dynamic Programming (Warshall’s and Floyd’s Algorithms) Aman Ullah Algorithms and Data Structures-2

Warshall’s Algorithm Adjacency Matrix Transitive Closure Example Finding Transitive Closure using Warshall’s Algorithm Formula for generating the elements of R (k) Example (Warshall’s Algorithm) Floyd’s Algorithm for all pairs shortest path problem Example (Floyd’s Algorithm) Aman Ullah Algorithms and Data Structures-2 Today’s Topics

Warshall’s algorithm will be used for computing the transitive closure of a directed graph. Adjacency Matrix: The adjacency matrix A= {a ij } of a directed graph is the boolean matrix (i.e. containing only I’s and 0’s) that has 1 in its ith row and jth column if and only if there is a directed edge from the ith vertex to the jth vertex. Transitive Closure: The transitive closure of a directed graph with n vertices can be defined as the n-by-n boolean matrix T = { t ij }, in which the element in the ith row (1≤ i ≤ n) and the jth column (1≤ j ≤ n) is 1 if there exists a nontrivial directed path (i.e. a directed path of a positive length) from the ith vertex to the jth vertex; otherwise, t ij is 0. Aman Ullah Algorithms and Data Structures-2 Warshall’s Algorithm

Consider the following directed graph: Its adjacency matrix is as follows: Aman Ullah Algorithms and Data Structures-2 Example

Example (Contd…) Its transitive closure is as follows: Note: Since this method traverses the same digraph several times, we should hope that a better algorithm can be found. Indeed, such an algorithm exists. It is called Warshall's algorithm which constructs the transitive closure of a given digraph with n vertices through a series of n-by-n boolean matrices:. Aman Ullah Algorithms and Data Structures-2

Each of these matrices provides certain information about directed paths in the digraph. Specifically, the element in the ith row and jth column of matrix R (k) (k = 0, 1,..., n) is equal to 1 if and only if there exists a directed path (of a positive length) from the ith vertex to the jth vertex with each intermediate vertex, if any, numbered not higher than ‘k’. Thus, the series starts with R (0), which does not allow any intermediate vertices in its paths; hence, R (0) is nothing else but the adjacency matrix of the digraph. Finding the transitive closure Aman Ullah Algorithms and Data Structures-2

Finding the transitive closure (Contd…) Aman Ullah Algorithms and Data Structures-2 R (1) contains the information about paths that can use the first vertex as intermediate; thus, with more freedom, so to speak, it may contain more ones than R (0). In general, each subsequent matrix in above series has one more vertex to use as intermediate for its paths than its predecessor and hence may, but does not have to, contain more ones. The last matrix in the series, R (n), reflects paths that can use all n vertices of the digraph as intermediate and hence is nothing else but the digraph's transitive closure.

Formula for generating the elements of R (k) Aman Ullah Algorithms and Data Structures-2 The formula for generating the elements of matrix R (k) from the elements of matrix R (k-1) (i.e. its predecessor in the above series) is: If an element r ij is 1 in matrix R (k-1), it remains 1 in R (k) If an element r ij is 0 in R (k-1), it has to be changed to 1 in its successor matrix R (k) if and only if the element in its row i and column k and the element in its row k and column j are both 1's in R (k-1). Element in ith row and jth column in matrix R (k)

Examples For example, if we have the following situation: then we shall get Formula for generating the elements of R (k) (Contd…) Aman Ullah Algorithms and Data Structures-2

Examples Consider the following directed graph (i.e. digraph): In order to find the transitive closure of this digraph, we shall start with the matrix R (0) which is nothing but the adjacency matrix of the above graph. So, Example (Warshall’s Algorithm) Aman Ullah Algorithms and Data Structures-2

Examples Hence, we get R (1) as In order to find R (2), we shall use row 2 and column 2 of its predecessor matrix R (1). In order to find R (2) first of all we shall write 1’s in the same places as they were in matrix R (1). Then we shall consider column 2 and row 2 of R (1). Example (Warshall’s Algorithm) (Contd…) Aman Ullah Algorithms and Data Structures-2 0 at position (4,2) has been changed into 1 0 at position (1,4) has been changed into 1 0 at position (4,4) has been changed into 1

Examples In order to find R (3), we shall use row 3 and column 3 of its predecessor matrix R (2). In order to find R (3) first of all we shall write 1’s in the same places as they were in matrix R (2). Then we shall consider column 3 and row 3 of R (2). Hence, we get R (3) same as R (2) Example (Warshall’s Algorithm) (Contd…) Aman Ullah Algorithms and Data Structures-2

Examples In order to find R (4), we shall use row 4 and column 4 of its predecessor matrix R (3). In order to find R (4) first of all we shall write 1’s in the same places as they were in matrix R (3). Then we shall consider column 4 and row 4 of R (3). It is the required transitive closure of the given digraph. Example (Warshall’s Algorithm) (Contd…) Aman Ullah Algorithms and Data Structures-2 0’s at positions (1,1), (1,2), (1,3), (1,4), (2,1), (2,2), (2,3), (2,4), (4,1), (4,2), (4,3) and (4,4), have been changed into 1’s

Examples It can be clearly seen that time efficiency of this algorithm is Θ(n 3 ). Even then, we accept it because it is short and concise. To make this algorithm run faster, we should treat matrix rows as bit strings and employ the bitwise OR operation available in almost all modern computer languages. Warshall’s Algorithm Aman Ullah Algorithms and Data Structures-2

Floyd’s Algorithm for All Pairs Shortest Path Problem Aman Ullah Algorithms and Data Structures-2 Given a weighted connected graph (undirected or directed), the all-pairs shortest paths problem asks to find the distances (the lengths of the shortest paths) from each vertex to all other vertices. It is convenient to record the lengths of shortest paths in an n- by-n matrix D called the distance matrix. In this matrix, the element d ij in the ith row and the jth column indicates the length of the shortest path from the ith vertex to the jth vertex (1 ≤ i, j ≤ n ). Floyd’s algorithm will be used to generate distance matrix from the weight matrix W of a graph with no negative-length cycle. In the weight matrix, if there is no direct edge from a vertex i to vertex j, the entry will be filled with ∞, otherwise it will be filled with the weight of the edge, coming from vertex i to vertex j.

Example Aman Ullah Algorithms and Data Structures-2 Consider the following weighted connected graph: Its weight matrix is: Its distance matrix is:

Examples Floyd’s algorithm starts with the matrix D (0) which is nothing but the weight matrix of the graph. The last matrix in the series, D (n), contains the lengths of the shortest paths among all paths that can use all n vertices as intermediate and hence is nothing but the distance matrix being sought. The formula to calculate the elements of distance matrix is: The element in the ith row and the jth column of the current distance matrix D (k-1) is replaced by the sum of the elements in the same row i and the kth column and in the same column j and the kth row if and only if the latter sum is smaller than its current value. Floyd’s Algorithm Aman Ullah Algorithms and Data Structures-2

Examples The time efficiency of Floyd’s algorithm is also Θ(n 3 ). Floyd’s Algorithm Aman Ullah Algorithms and Data Structures-2

Example (Floyd’s Algorithm) Consider the following weighted connected graph. We shall find distance matrix using Floyd’s algorithm for all pairs. For this purpose, we shall start with the matrix D (0) which is the weight matrix of the graph. So, Aman Ullah Algorithms and Data Structures-2

Examples In order to find D (1), we shall use row 1 and column 1 of its predecessor matrix D (0). We see that in column 1 of D (0), we have some numbers (other than ∞ ) at positions (i.e. rows) 1, 2 and 4 and in row 1 of D (0), we have some numbers (other than ∞ ) at positions (i.e. columns) 1 and 3. We go to (1,1) and (1,3) and see that there are already some numbers at these positions. So, we don’t change them. Same is the case with the position (2,1). Now, we go to the cell (2,3). Since, it contains ∞,we need to change it by using the formula: D[i,j] = min(D[i,j], D[i,k] + D[k,j]) to get D[2,3] = min(D[2,3], D[2,1] + D[1,3]) = min( ∞, 2+3) = 5 Example (Floyd’s Algorithm) (Contd…) Aman Ullah Algorithms and Data Structures-2 Here k = 1 for D (1)

Examples Now, we go to (4,1) and see that it already contains some number. So, we don’t change it Now, we go to the cell (4,3) of D (0). Since, it contains ∞,we need to change it by using the formula: D[i,j] = min(D[i,j], D[i,k] + D[k,j]) to get D[4,3] = min(D[4,3], D[4,1] + D[1,3]) = min( ∞, 6+3) = 9 Hence, we get D (1) as : Example (Floyd’s Algorithm) (Contd…) Aman Ullah Algorithms and Data Structures-2 New values

Examples In order to find D (2), we shall use row 2 and column 2 of its predecessor matrix D (1). We need to change the values of the cells that are ∞ in this row and column by using the formula: D[i,j] = min(D[i,j], D[i,k] + D[k,j]) to get D[3,1] = min(D[3,1], D[3,2] + D[2,1]) = min( ∞, 7+2) = 9 Hence, we get D (2) as : Example (Floyd’s Algorithm) (Contd…) Aman Ullah Algorithms and Data Structures-2 New value Here k = 2 for D (2)

Examples In order to find D (3), we shall use row 3 and column 3 of its predecessor matrix D (2). We see that in column 3 of D (2), we have some numbers (other than ∞ ) at positions (i.e. rows) 1, 2, 3 and 4 and in row 3 of D (2), we have some numbers (other than ∞ ) at positions (i.e. columns) 1, 2, 3 and 4. We further note that there are already some numbers at the positions (1,1), (1,3), (2,1), (2,2), (2,3), (3,1), (3,2), (3,3), (3,4), (4,1), (4,3) and (4,4). So, we don’t change them. Only the values (i.e. ∞) at positions (1,2), (1,4), (2,4) and (4,2) need to be changed by using the formula: D[i,j] = min(D[i,j], D[i,k] + D[k,j]) to get Example (Floyd’s Algorithm) (Contd…) Aman Ullah Algorithms and Data Structures-2 Here k = 3 for D (3)

Examples In order to find D (4), we shall use row 4 and column 4 of its predecessor matrix D (3). We see that in column 4 of D (3), we have some numbers (other than ∞ ) at positions (i.e. rows) 1, 2, 3 and 4 and in row 4 of D (3), we have some numbers (other than ∞ ) at positions (i.e. columns) 1, 2, 3 and 4. Example (Floyd’s Algorithm) (Contd…) Aman Ullah Algorithms and Data Structures-2 New values

Examples We further note that there are already some numbers at all these positions. But in order to get the shortest path from all the vertices, we shall use the formula: D[i,j] = min(D[i,j], D[i,k] + D[k,j]) to get minimum values at all positions. When we apply this formula to all the cells, we see that every cell will retain its previous value except the cell (3,1). because, for the cell (3,1), we shall get D[i,j] = min(D[i,j], D[i,k] + D[k,j]) D[3,1] = min(D[3,1], D[3,4] + D[4,1]) = min(9, 1+6) = 7 Hence, we get D (4) which is the required distance matrix of shortest path’s length, as : Example (Floyd’s Algorithm) (Contd…) Aman Ullah Algorithms and Data Structures-2 Here k = 4 for D (4)

Examples Example (Floyd’s Algorithm) (Contd…) Aman Ullah Algorithms and Data Structures-2 New value

Dynamic Programming (Optimal Binary Search Trees) Aman Ullah Algorithms and Data Structures-2

Binary Search Trees Optimal Binary Search Trees Finding Dynamic Programming formula for OBST Example of finding minimum expected cost of search and constructing OBST Aman Ullah Algorithms and Data Structures-2 Today’s Topics

In computer science, a binary search tree (BST), sometimes also called an ordered or sorted binary tree, is a node-based binary tree data structure which has the following properties: 1.The left subtree of a node contains only nodes with keys less than that node's key. 2.The right subtree of a node contains only nodes with keys greater than that node's key. 3.The left and right subtree must each also be a binary search tree. 4.There must be no duplicate nodes. When we know the probabilities of searching each one of the keys, it is quite easy to compute the expected cost of accessing the tree. An Optimal Binary Search Tree (OBST) is a BST which has minimal expected cost. Aman Ullah Algorithms and Data Structures-2 Binary Search Tree

Following is a Binary Search Tree of size 9 and depth 3, with root 8 and leaves 1, 4, 7 and 13. Aman Ullah Algorithms and Data Structures-2 Binary Search Tree (Example)

A binary search tree is one of the most important data structures in computer science. One of its principal applications is to implement a dictionary, a set of elements with the operations of searching, insertion, and deletion. If probabilities of searching for elements of a set are known (e.g., from accumulated data about past searches), it is natural to pose a question about an optimal binary search tree (OBST) for which the average number of comparisons in a search is the smallest possible. (For simplicity, we limit our discussion to minimizing the average number of comparisons in a successful search. The method can be extended to include unsuccessful searches as well.) Aman Ullah Algorithms and Data Structures-2 Optimal Binary Search Trees

Consider four keys A, B, C, and D to be searched with probabilities 0.1, 0.2, 0.4, and 0.3, respectively. The following figures depict two out of 14 possible binary search trees containing these keys. The average number of comparisons (or expected cost of search) in a successful search in the first of these trees is 1*(0.1) + 2*(0.2)+3*(0.4) + 4*(0.3)= 2.9, while for the second one, it is 1*(0.2) + 2*(0.1+0.4) + 3*(0.3) = 2.1. Neither of these two trees is, in fact, optimal. Aman Ullah Algorithms and Data Structures-2 Optimal Binary Search Trees (Example 1)

Consider six keys -5, 1, 8, 7, 13 and 21 to be searched with probabilities 1/8, 1/32, 1/16, 1/32, 1/4 and 1/2 respectively. We consider the following figure which depicts one of the possible binary search trees containing these keys. The average number of comparisons (i.e. expected cost of search) in a successful search is 1*(1/16) + 2*(1/32+1/4) + 3*(1/8+1/32+1/2)= 2.59 It's clear that this tree is not optimal. Aman Ullah Algorithms and Data Structures-2 Optimal Binary Search Trees (Example 2)

In order to find the Optimal Binary Search Tree for example 1 above, we could have used exhaustive-search approach to construct all 14 possible binary search trees with these keys. The Binary Search Tree with lowest expected cost of search would have given us the OBST. But this approach is unrealistic because the total number of binary search trees with n keys is equal to the nth Catalan number: which grows to infinity as fast as 4 n I n 1.5 So, we shall try to find the dynamic programming solution for this problem by constructing some recurrence relation. Aman Ullah Algorithms and Data Structures-2 Finding Dynamic Programming Formula for Example 1

So let a 1, a 2, ……, a n be distinct keys ordered from the smallest to the largest and let p 1, p 2, ……, p n be the probabilities of searching for them, respectively. Let C[i, j] be the smallest average number of comparisons made in a successful search in a binary search tree made up of keys a i, a i+1, ……, a j, where i, j are some integer indices with 1 ≤ i ≤ j ≤ n. We shall follow the classic dynamic programming approach to find the values of C[i, j] for all smaller instances of the problem, although we are interested just in C[1, n ]. To derive a recurrence underlying the dynamic programming algorithm, we will consider all possible ways to choose a root a k among the keys a i, a i+1, ……, a j. Aman Ullah Algorithms and Data Structures-2 Finding Dynamic Programming Formula for Example 1 (Contd…)

Such a binary search tree is shown in the following figure: It contains the root key a k, the left subtree containing the optimally arranged keys a i, a i+1, ……, a k-1 and the right subtree containing the optimally arranged keys a k+1, a k+2, ……, a j. Aman Ullah Algorithms and Data Structures-2 Finding Dynamic Programming Formula for Example 1 (Contd…)

If we count tree levels starting with 1 (to make the comparison numbers equal to the keys' levels), the following recurrence relation is obtained: Aman Ullah Algorithms and Data Structures-2 Finding Dynamic Programming Formula for Example 1 (Contd…)

Thus we get the following recurrence relation for OBST: Here we assume that C[i, i - 1] = 0 for 1 ≤ i ≤ n + 1, which can be interpreted as the number of comparisons in the empty tree. Moreover, we see that C[i, i] = P i for 1 ≤ i ≤ n, as it should be for a one-node binary search tree containing a i. In the dynamic programming approach, we shall construct a two-dimensional table which will show the values needed for computing C[i, j] by using the above mentioned formula. Aman Ullah Algorithms and Data Structures-2 Finding Dynamic Programming Formula for Example 1 (Contd…)

These values will be in row i and the columns to the left of column j and in column j and the rows below row i. The arrows point to the pairs of entries whose sums are computed in order to find the smallest one to be recorded as the value of C[i, j]. This suggests filling the table along its diagonals, starting with all zeros on the main diagonal and given probabilities Pi, 1 ≤ i ≤ n, right above it and moving toward the upper right corner. Aman Ullah Algorithms and Data Structures-2 Finding Dynamic Programming Formula for Example 1 (Contd…) C[1, n]

The algorithm for OBST will compute the value for C[1, n] (i.e. upper right corner of the main table), which will give us the average number of comparisons for successful searches in the OBST. If we also want to construct the optimal tree itself, we need to maintain another two-dimensional table (i.e root table) to record the value of k for which the minimum value from the above formula has been achieved. This root table has the same shape as the main table and is filled in the same manner, starting with entries R[i, i] = i for 1 ≤ i ≤ n. When the root table is filled, its entries indicate indices of the roots of the optimal subtrees, which makes it possible to reconstruct an optimal tree for the entire set given. Aman Ullah Algorithms and Data Structures-2 Finding Dynamic Programming Formula for Example 1 (Contd…)

Consider the four keys A, B, C and D with their probabilities given as follows: We are to find the minimum expected cost of searching a key by constructing Optimal Binary Search Tree. We shall construct the main table for finding the minimum expected cost of search and the root table for constructing the Optimal Binary Search Tree. Here n = 4 and a 1 = A, a 2 = B, a 3 = C, a 4 = D and P 1 = 0.1, P 2 = 0.2, P 3 = 0.4 and P 4 = 0.3 The entire procedure is shown in the following slides. Aman Ullah Algorithms and Data Structures-2 Example

Aman Ullah Algorithms and Data Structures-2 Example 00.1 00.2 00.4 00.3 0 Initially the main table and the root table have the following entries: Root Table for Computing the values of K MainTable for Computing the values of C[i, j] Value of K for which C[i, j] gives the minimum value 1 2 3 4 1234512345 1234512345 0 1 2 3 4

Aman Ullah Algorithms and Data Structures-2 Computing C[1, 2], C[2, 3] and C[3, 4] C[1,2] = min{ 0 + 0.2 + 0.1 + 0.2, 0.1 + 0 + 0.1 + 0.2} = min{0.5, 0.4} = 0.4 (for K=2)

Aman Ullah Algorithms and Data Structures-2 Computing C[1, 2], C[2, 3] and C[3, 4] C[2,3] = min{ 0 + 0.4 + 0.2 + 0.4, 0.2 + 0 + 0.2 + 0.4} = min{1.0, 0.8} = 0.8 (for K=3)

Aman Ullah Algorithms and Data Structures-2 Computing C[1, 2], C[2, 3] and C[3, 4] C[3,4] = min{ 0 + 0.3 + 0.4 + 0.3, 0.4 + 0 + 0.4 + 0.3} = min{1.0, 1.1} = 1.0 (for K=3)

Aman Ullah Algorithms and Data Structures-2 Updated Tables 00.10.4 00.20.8 00.41.0 00.3 0 So, after finding the above new diagonal values for both the tables, they will get the following form: Root Table for Computing the values of K MainTable for Computing the values of C[i, j] Value of K for which C[i, j] gives the minimum value 12 23 33 4 1234512345 1234512345 0 1 2 3 4

Aman Ullah Algorithms and Data Structures-2 Computing C[1, 3] and C[2, 4] C[1,3] = min{ 0 + 0.8 +0.1 + 0.2 + 0.4, 0.1 + 0.4 + 0.1 + 0.2 + 0.4, 0.4 + 0 + 0.1 + 0.2 + 0.4} = min{1.5, 1.2, 1.1} = 1.1 (for K=3)

Aman Ullah Algorithms and Data Structures-2 Computing C[1, 3] and C[2, 4] C[2,4] = min{ 0 + 1.0 + 0.2 + 0.4 + 0.3, 0.2 + 0.3 + 0.2 + 0.4 + 0.3, 0.8 + 0 + 0.2 + 0.4 + 0.3} = min{1.9, 1.4, 1.7} = 1.4 (for K=3)

Aman Ullah Algorithms and Data Structures-2 Updated Tables 00.10.41.1 00.20.81.4 00.41.0 00.3 0 After finding the above new diagonal values for both the tables, they will get the following form: Root Table for Computing the values of K MainTable for Computing the values of C[i, j] Value of K for which C[i, j] gives the minimum value 123 233 33 4 1234512345 1234512345 0 1 2 3 4

Aman Ullah Algorithms and Data Structures-2 Computing C[1, 4] C[1,4] = min{ 0 + 1.4 + 0.1 + 0.2 + 0.4 + 0.3, 0.1 + 1.0 + 0.1 + 0.2 + 0.4 + 0.3, 0.4 + 0.3 + 0.1 + 0.2 + 0.4 + 0.3, 1.1 + 0 + 0.1 + 0.2 + 0.4 + 0.3 } = min{2.4, 2.1, 1.7, 2.1} = 1.7 (for K=3)

Aman Ullah Algorithms and Data Structures-2 Final Tables 00.10.41.11.7 00.20.81.4 00.41.0 00.3 0 Final tables for finding expected costs of searching a key and for constructing the OBST are as follows: Root Table for Computing the values of K MainTable for Computing the values of C[i, j] Value of K for which C[i, j] gives the minimum value 1233 233 33 4 1234512345 1234512345 0 1 2 3 4

Aman Ullah Algorithms and Data Structures-2 Minimum Expected Cost of Search and Constructing OBST Thus, the average number of key comparisons in the optimal tree is equal to 1.7 (given by C[1, 4]. Now for constructing OBST, we see that since R[1, 4] = 3, so, the root of the optimal tree contains the third key, i.e., C. Its left subtree will be made up of keys A and B, and its right subtree will contain just key D because it is BST’s characteristic. Now, to find the specific structure of these subtrees, we find first their roots by consulting the root table again as follows.

Since for the left subtree, we have only two keys A and B, so we consult the entries R[l, 2] and R[4,4]. So R[1,2] gives us 2 (which is B). Thus the root of the optimal subtree containing A and B is B, with A being its left child. Similarly R[4, 4] = 4 (which is D), the root of this one-node optimal tree is its only key D. Following figure presents the optimal tree in its entirety. Constructing OBST (Contd…) Aman Ullah Algorithms and Data Structures-2

Pseudocode of dynamic programming algorithm for OBST is as follows: Algorithm for OBST Aman Ullah Algorithms and Data Structures-2

The first for loop is executed n times. The number of times the innermost loop is executed is given by the sum: So, time efficiency of this algorithm is (n 3 ). Time Efficiency of OBST Algorithm Aman Ullah Algorithms and Data Structures-2

In order to find the space efficiency of this algorithm, we see that it generates two tables: one table C of size (n+1) X (n+1) and the other table R of size n X n (as its first column and last row is unused). About one half of each of these tables is used. So space efficiency of this algorithm is calculated as: (n+1) X (n+1) /2 + n X n / 2 So, space efficiency of this algorithm is (n 2 ). Space Efficiency of OBST Algorithm Aman Ullah Algorithms and Data Structures-2

Algorithms and Data Structures-2 CPCS-324 By Aman Ullah

Aman Ullah Algorithms and Data Structures-2 Knapsack Problem (using Dynamic Programming)

0-1 Knapsack Problem Problem Analysis –Divide and Conquer –Dynamic Solution Algorithm using Dynamic Programming Time Complexity Generalization, Variations and Applications Aman Ullah Algorithms and Data Structures-2 Today’s Topics

Given a set of items, each with a cost and a value, then determine the items to include in a collection so that the total cost is less than some given cost and the total value is as large as possible. Knapsack problem is of combinatorial optimization It derives its name from the maximization problem of choosing possible essentials that can fit into one bag, of maximum weight, to be carried on a trip. A knapsack problem can be extended to business (gaining maximum profit out of various existing businesses) and cryptography (for generating public key). General Knapsack Problem Aman Ullah Algorithms and Data Structures-2

Given n items of known weights w 1, w 2,......., w n and values v 1, v 2,......., v n and a knapsack of capacity W, find the most valuable subset of the items that can fit into the knapsack. Brute force (Exhaustive Search) Approach Generate all subsets from given n items, computing the total weight of each subset to identify feasible subsets (i.e., the ones with the total weight not exceeding the knapsack's capacity), and finally finding a subset of the largest value among them. Since the number of subsets of an n-element set is 2 n, the exhaustive search leads to a (2 n ) algorithm, no matter how efficiently individual subsets are generated. (Inefficient) General Knapsack Problem Aman Ullah Algorithms and Data Structures-2

The knapsack problem arises whenever there is resource allocation with financial constraints Problem Statement You are in Japan on an official visit and want to shop. A list of required items is available at the store You are given a bag (knapsack), of fixed capacity, and only you can fill this bag with the selected items from the list. Every item has a value (cost) and weight, and your objective is to seek most valuable set of items which you can buy not exceeding bag limit. 0-1 Knapsack Problem Statement Aman Ullah Algorithms and Data Structures-2

Assumption Each item must be put entirely in the knapsack or not included at all. That is why the problem is called 0-1 knapsack problem Remarks Because an item cannot be broken up arbitrarily, so it is its 0-1 property that makes the knapsack problem hard. If an item can be broken and allowed to take part of it (fractional knapsack problem) then algorithm can be solved using greedy approach optimally. 0-1 Knapsack Problem: Remarks Aman Ullah Algorithms and Data Structures-2

Problem Construction You have prepared a list of n objects which you are interested to buy. The items are numbered as i 1, i 2,..., i n Capacity of bag is W Each item i has value v i, and weight w i We want to select a set of items among i 1, i 2,..., i n which do not exceed (in total weight) capacity W of the bag Total value of selected items must be maximum How should we select the items? Notations: 0-1 Knapsack Problem Construction Aman Ullah Algorithms and Data Structures-2

0-1 Knapsack Problem Construction Formal Construction of Problem Given a list of items: i 1, i 2,..., i n, with values: v 1, v 2,..., v n and weights: w 1, w 2,..., w n respectively Of course, capacity of knapsack, W > 0, and we wish to find a set of items S ε {i 1, i 2,..., i n } such that maximizes subject to Aman Ullah Algorithms and Data Structures-2 Model: 0-1 Knapsack Problem Construction i.e. we want to maximize the sum of values of all items selected from the set S i.e. sum of weights of all items selected should be less than or equal to the capacity of knapsack A kind of Linear Programming problem

Compute all the subsets of {i 1, i 2,..., i n }. There will be 2 n number of subsets. For example, for 3 items, we shall have 2 3 =8 subsets; Ø, {1}, {2}, {3}, {1,2}, {1,3}, {2,3}, {1,2,3} Find sum of the weights of total items in each set and list only those sets whose sum does not increase by W (capacity of knapsack) Compute sum of values of items in each selected list and find the highest one This highest value is the required solution The computational cost of Brute Force Approach is exponential (i.e. θ (2 n )). Hence it is not economical So we shall have to find some other way! Brute Force Solution Aman Ullah Algorithms and Data Structures-2

Approach Partition the knapsack problem into sub-problems Find the solutions of the sub-problems Combine these solutions to solve original problem Comments In this case the sub-problems are not independent And the sub-problems share sub-sub-problems Algorithm repeatedly solves common sub-sub- problems and takes more effort than required Because this is an optimization problem and hence dynamic approach is another solution if we are able to construct the problem dynamically Divide and Conquer Approach Aman Ullah Algorithms and Data Structures-2

Step1 (Structure): Characterize the structure of an optimal solution Next decompose the problem into sub-problems Step 2 (Principal of Optimality) Then express solution of the main problem in terms of optimal solutions of sub-problems. Simply, in dynamic programming, we first find base cases. Then we compute the objective function. Steps in Dynamic Programming Aman Ullah Algorithms and Data Structures-2

Step3 (Bottom-up Computation): In this step, compute the value of an optimal solution in a bottom-up fashion by using structure of the table already constructed. Step 4 (Construction of an Optimal Solution) Construct an optimal solution from the computed information based on Steps 1-3. Steps in Dynamic Programming Aman Ullah Algorithms and Data Structures-2

Step1 (Structure): Decompose problem into smaller problems Construct an array or matrix V[0..n, 0..W] Compute V[i, j] to store maximum value of items selected from {1, 2,..., i}, that can fit into a bag with capacity j, where 1 ≤ i ≤ n, 1 ≤ j ≤ W (i.e. we have reduced the number of items from ‘n’ (original) to ‘i’ and reduced the capacity of knapsack from ‘W’ to ‘j’) V[n, W] (Right bottom most entry) will contain the maximum value of the items selected from {1,2,…,n} that can fit into the bag with capacity W storage Hence V[n, W] is the required solution for our knapsack problem Mathematical Model: Dynamic Programming Aman Ullah Algorithms and Data Structures-2

Step 2 (Principal of Optimality) Recursively define value of an optimal solution in terms of solutions to sub-problems Base Case: Since V[0, j] = 0, 0 ≤ j ≤ W, (no items are available) Similarly V[i, 0] = 0, 0 ≤ i ≤ n, (Capacity of knapsack is zero) V[i, j] = -1, (Since we need maximum value in V[i,w], therefore we have stored very small value i.e. -1 in it) Aman Ullah Algorithms and Data Structures-2 Mathematical Model: Dynamic Programming

Recursive Formulation: Leave Object ‘i’: If we decide not to take object ‘i’, then the optimal value will come about by considering how to fill the knapsack of size ‘j’ with the remaining items {1, 2, 3, …… i-1} So we have V[i,j] = V[i-1, j] if w i >j (i.e. we leave object i because weight of item ‘i’ exceeds the capacity of knapsack) Mathematical Model: Dynamic Programming Aman Ullah Algorithms and Data Structures-2

Take Object ‘i’: If we decide to take object ‘i’, then we gain the value (i.e. v i ) of that object. But we have used up the weight w i of that object as well. So remaining capacity of knapsack is j – w i,from which we can fill our knapsack in the best possible way with remaining objects {1, 2, 3, …….., i-1} V[i, j] = v i + V[i-1, j - w i ] if w i ≤ j Thus our recursive formula becomes V[i-1, j] if w i > j max(V[i-1,j], v i + V[i-1,j – w i ] if w i ≤ j Mathematical Model: Dynamic Programming Aman Ullah Algorithms and Data Structures-2 V[i, j] =

Consider the following data We shall solve this knapsack problem using Dynamic Programming algorithm Base Cases: V[0, j] = 0, 0 ≤ j ≤ 5 V[i, 0] = 0, 0 ≤ i ≤ 4 Example Aman Ullah Algorithms and Data Structures-2 i1234 vivi 12\$10\$20\$15\$ wiwi 2132 Capacity, W = 5 Items Values Weights

V[1, 1] = 0, OR V[i, j] = V[i-1, j] if w i > j V[1,1] = V[0,1] = 0 V[i, j] = max(V[i-1, j], v i + V[i-1, j – w i ]); V[1, 2] = max(V[0, 2], v 1 + V[0, 2 – w 1 ]); = max(V[0, 2], 12 + V[0, 2 - 2]) = max(V[0, 2], 12 + V[0, 0]) = max(0, 12 + 0) = max(0, 12) = 12 Keep(1, 2) = 1 (Here 1 means item 1 has been chosen because maximum value has been taken from second formula) Example Aman Ullah Algorithms and Data Structures-2 i1234 vivi 12\$10\$20\$15\$ wiwi 2132 Capacity, W = 5 Items Values Weights

V[i, j] = max(V[i-1, j], v i + V[i-1, j – w i ]); V[1, 3] = max(V[0, 3], v 1 + V[0, 3 – w 1 ]); = max(V[0, 3], 12 + V[0, 3 - 2]) = max(V[0, 3], 12 + V[0, 1]) = max(0, 12 + 0) = max(0, 12) = 12 Keep(1, 3) = 1 V[1, 4] = max(V[0, 4], v 1 + V[0, 4 – w 1 ]); = max(V[0, 4], 12 + V[0, 4 - 2]) = max(V[0, 4], 12 + V[0, 2]) = max(0, 12 + 0) = max(0, 12) = 12 Keep(1, 4) = 1 Aman Ullah Algorithms and Data Structures-2 Example (Contd…)

V[i, j] = max(V[i-1, j], v i + V[i-1, j – w i ]); V[1, 5] = max(V[0, 5], v 1 + V[0, 5 – w 1 ]); = max(V[0, 5], 12 + V[0, 5 - 2]) = max(V[0, 5], 12 + V[0, 3]) = max(0, 12 + 0) = max(0, 12) = 12 Keep(1, 5) = 1 Example (Contd…) Aman Ullah Algorithms and Data Structures-2

V[i, j] = max(V[i-1, j], v i + V[i-1, j – w i ]); V[2, 1] = max(V[1, 1], v 2 + V[1, 1 – w 2 ]); = max(V[1, 1], 10 + V[1, 1 - 1]) = max(V[1, 1], 10 + V[1, 0]) = max(0, 10 + 0) = max(0, 10) = 10 Keep(2, 1) = 1 Example (Contd…) Aman Ullah Algorithms and Data Structures-2 i1234 vivi 12\$10\$20\$15\$ wiwi 2132 Capacity, W = 5 Items Values Weights

V[i, j] = max(V[i-1, j], v i + V[i-1, j – w i ]); V[2, 2] = max(V[1, 2], v 2 + V[1, 2 – w 2 ]); = max(V[1, 2], 10 + V[1, 2 - 1]) = max(V[1, 2], 10 + V[1, 1]) = max(12, 10 + 0) = max(12, 10) = 12 Keep(2, 2) = 0 V[2, 3] = max(V[1, 3], v 2 + V[1, 3 – w 2 ]); = max(V[1, 3], 10 + V[1, 3 - 1]) = max(V[1, 3], 10 + V[1, 2]) = max(12, 10 + 12) = max(12, 22) = 22 Keep(2, 3) = 1 Example (Contd…) Aman Ullah Algorithms and Data Structures-2

V[i, j] = max(V[i-1, j], v i + V[i-1, j – w i ]); V[2, 4] = max(V[1, 4], v 2 + V[1, 4 – w 2 ]); = max(V[1, 4], 10 + V[1, 4 - 1]) = max(V[1, 4], 10 + V[1, 3]) = max(12, 10 + 12) = max(12, 22) = 22 Keep(2, 4) = 1 V[2, 5] = max(V[1, 5], v 2 + V[1, 5 – w 2 ]); = max(V[1, 5], 10 + V[1, 5 - 1]) = max(V[1, 5], 10 + V[1, 4]) = max(12, 10 + 12) = max(12, 22) = 22 Keep(2, 5) = 1 Example (Contd…) Aman Ullah Algorithms and Data Structures-2

V[i, j] = V[i-1, j] if w i > j V[3, 1] = V[2, 1] = 10 Keep(3, 1) = 0 Example (Contd…) Aman Ullah Algorithms and Data Structures-2 i1234 vivi 12\$10\$20\$15\$ wiwi 2132 Capacity, W = 5 Items Values Weights

V[i, j] = V[i-1, j] if w i > j V[3, 2] = V[2, 2] = 12 Keep(3, 2) = 0 V[i, j] = max(V[i-1, j], v i + V[i-1, j – w i ]); V[3, 3] = max(V[2, 3], v 3 + V[2, 3 – w 3 ]); = max(V[2, 3], 20 + V[2, 3 - 3]) = max(V[2, 3], 20 + V[2, 0]) = max(22, 20 + 0) = max(22, 20) = 22 Keep(3, 3) = 0 Example (Contd…) Aman Ullah Algorithms and Data Structures-2

V[i, j] = max(V[i-1, j], v i + V[i-1, j – w i ]); V[3, 4] = max(V[2, 4], v 3 + V[2, 4 – w 3 ]); = max(V[2, 4], 20 + V[2, 4 - 3]) = max(V[2, 4], 20 + V[2, 1]) = max(22, 20 + 10) = max(22, 30) = 30 Keep(3, 4) = 1 V[3, 5] = max(V[2, 5], v 3 + V[2, 5 – w 3 ]); = max(V[2, 5], 20 + V[2, 5 - 3]) = max(V[2, 5], 20 + V[2, 2]) = max(22, 20 + 12) = max(22, 32) = 32 Keep(3, 5) = 1 Example (Contd…) Aman Ullah Algorithms and Data Structures-2

V[i, j] = V[i-1, j] if w i > j V[4, 1] = V[3, 1] = 10 Keep(4, 1) = 0 Example (Contd…) Aman Ullah Algorithms and Data Structures-2 i1234 vivi 12\$10\$20\$15\$ wiwi 2132 Capacity, W = 5 Items Values Weights

V[i, j] = max(V[i-1, j], v i + V[i-1, j – w i ]); V[4, 2] = max(V[3, 2], v 4 + V[3, 2 – w 4 ]); = max(V[3, 2], 15 + V[3, 2 - 2]) = max(V[3, 2], 15 + V[3, 0]) = max(12, 15 + 0) = max(12, 15) = 15 Keep(4, 2) = 1 V[i, j] = max(V[i-1, j], v i + V[i-1, j – w i ]); V[4, 3] = max(V[3, 3], v 4 + V[3, 3 – w 4 ]); = max(V[3, 3], 15 + V[3, 3 - 2]) = max(V[3, 3], 15 + V[3, 1]) = max(22, 15 + 10) = max(22, 25) = 25 Keep(4, 3) = 1 Example (Contd…) Aman Ullah Algorithms and Data Structures-2

V[i, j] = max(V[i-1, j], v i + V[i-1, j – w i ]); V[4, 4] = max(V[3, 4], v 4 + V[3, 4 – w 4 ]); = max(V[3, 4], 15 + V[3, 4 - 2]) = max(V[3, 4], 15 + V[3, 2]) = max(30, 15 + 12) = max(30, 27) = 30 Keep(4, 4) = 0 V[4, 5] = max(V[3, 5], v 4 + V[3, 5 – w 4 ]); = max(V[3, 5], 15 + V[3, 5 - 2]) = max(V[3, 5], 15 + V[3, 3]) = max(32, 15 + 22) = max(32, 37) = 37 Keep(4, 5) = 1 Example (Contd…) Aman Ullah Algorithms and Data Structures-2

V[i, j] j = 0 12345 i = 0 000000 i = 1 0012 i = 2 0101222 i = 3 01012223032 i = 4 01015253037 Aman Ullah Algorithms and Data Structures-2 Optimal Value: Entire Solution Capacity, W = 5 Final Solution: V[4, 5] = 37 Items selected = {1, 2, 4} i1234 vivi 12\$10\$20\$15\$ wiwi 2132

KnapSack (v, w, n, W) for (i = 0 to n), V[i, 0] = 0; (n+1) times for (j = 0 to W), V[0, j] = 0; (W+1) times for (i = 1 to n) (n) times //V[i,0] = 0 for (j = 1 to W) (W) times if (w(i) ≤ j) V[i, j] = max(V[i-1, j], v i + V[i-1, j – w i ]); else V[i, j] = V[i-1, j]; Return V[n, W] Algorithm : Dynamic Programming Aman Ullah Algorithms and Data Structures-2

KnapSack (v, w, n, W) for (i = 0 to n), V[i, 0] = 0; (n+1) times for (j = 0 to W), V[0, j] = 0; (W+1) times for (i = 1 to n) (n) times for (j = 1 to W) (W) times if (w(i) ≤ j AND v i + V[i-1, j – w i ] > V[i-1, j]) V[i, j] = v i + V[i-1, j – w i ]; else V[i, j] = V[i-1, j]; Return V[n, W] Algorithm : Dynamic Programming (Another form) Aman Ullah Algorithms and Data Structures-2

Analysis of 0-1 Knapsack Algorithm Aman Ullah Algorithms and Data Structures-2 The first two ‘for’ loops (for base cases) run for n+1 and W+1 times respectively. The outer ‘for’ loop runs for ‘n’ times and the inner ‘for’ loop runs for ‘W’ times. So, total iterations of these nested ‘for’ loops are (n*W). So, total running time of the above 0-1 knapsack algorithm, T(n) = n+1 + W + 1 + n*W T(n) = O(n*W) (We are not using θ here as in both cases, we choose maximum of the two values).

KnapSack(v, w, n, W) for (w = 0 to W), V[0, w] = 0; for (i = 1 to n), V[i, 0] = 0; //Base Cases for (i = 1 to n) for (w = 1 to W) if ((w(i) ≤ w) AND (v i + V[i-1,w – w i ] > V[i-1,w])) V[i, w] = (v i + V[i-1,w – w i ]; keep[i, w] = 1; else V[i, w] = V[i-1,w]; keep[i, w] = 0; K = W; for (i = n down to 1) if keep[i, K] = = 1 output i K = K – w i Return V[n, W] Complete: Dynamic Programming Algorithm Aman Ullah Algorithms and Data Structures-2

1. Generalization of Knapsack Problems #include int main() { int w[10],p[10],i,m,n; printf("ENTER THE NUMBER OF ITEMS:"); scanf("%d",&n); printf("ENTER THE WEIGHTS OF THE ITEMS:"); for(i=1;i<=n;i++) scanf("%d",&w[i]); printf("ENTER THE PROFITS OF THE ITEMS:"); for(i=1;i<=n;i++) scanf("%d",&p[i]); printf("ENTER THE CAPACITY OF KNAPSACK:"); scanf("%d",&m); knapsack(m,n,w,p); return 0; } Aman Ullah Algorithms and Data Structures-2 0-1 Knapsack Problem (Complete Code)

1. Generalization of Knapsack Problems void knapsack(int m,int n,int w[],int p[]) { int v[10][10],x[10],i,j; for(i=0;i<=n;i++) { for(j=0;j<=m;j++) { if(j==0||i==0) v[i][j]=0; if(j-w[i]<0) v[i][j]=v[i-1][j]; else v[i][j]=max(v[i-1][j],v[i-1][j-w[i]]+p[i]); } } Aman Ullah Algorithms and Data Structures-2 0-1 Knapsack Problem (Complete Code) (Contd…)

1. Generalization of Knapsack Problems for(i=0;i<=n;i++) { for(j=0;j<=m;j++) printf("%d\t",v[i][j]); printf("\n"); } printf("THE OPTIMAL SOLUTION IS:%d",v[n][m]); for(i=1;i<=n;i++) x[i]=0; i=n; j=m; Aman Ullah Algorithms and Data Structures-2 0-1 Knapsack Problem (Complete Code) (Contd…)

1. Generalization of Knapsack Problems while(i>0 && j>0) { if(v[i][j]!=v[i-1][j]) { x[i]=1; j=j-w[i]; } i--; } printf("THE OPTIMAL SET OF WEIGHTS IS:"); for(i=1;i<=n;i++) if(x[i]==1) printf("%d\t",i); printf("\n"); } //End of knapsack function Aman Ullah Algorithms and Data Structures-2 0-1 Knapsack Problem (Complete Code) (Contd…)

1. Generalization of Knapsack Problems int max(int a,int b) { if(a>b) return a; else return b; } Aman Ullah Algorithms and Data Structures-2 0-1 Knapsack Problem (Complete Code) (Contd…)

Dynamic programming deals with problems whose solutions satisfy a recurrence relation with overlapping (dependent) subproblems. The direct top-down approach to finding a solution to such a recurrence leads to an algorithm that solves common subproblems more than once and hence is very inefficient (typically, exponential or worse). The classic dynamic programming approach, on the other hand, works bottom-up: it fills a table with solutions to all smaller subproblems, but each of them is solved only once. An unsatisfying aspect of this approach is that solutions to some of these smaller subproblems are often not necessary for getting a solution to the bigger problem given. Memory Functions Aman Ullah Algorithms and Data Structures-2

Since this drawback is not present in the top-down approach, it is natural to try to combine the strengths of the top-down and bottom-up approaches. The goal is to get a method that solves only subproblems that are necessary and does it only once. Such a method exists; it is based on using memory functions. This method solves a given problem in the top-down manner but, in addition, maintains a table of the kind that would have been used by a bottom-up dynamic programming algorithm. Initially, all the table's entries are initialized with a special "null" symbol to indicate that they have not yet been calculated. Thereafter, whenever a new value needs to be calculated, the method checks the corresponding entry in the table first: if this entry is not "null," it is simply retrieved from the table; otherwise, it is computed by the recursive call whose result is then recorded in the table. Memory Functions (Contd…) Aman Ullah Algorithms and Data Structures-2

The following algorithm implements this idea for the knapsack problem. After initializing the table, the recursive function needs to be called with i = n (the number of items) and j = W (the capacity of the knapsack). Memory Functions (Contd…) Aman Ullah Algorithms and Data Structures-2

Knapsack Algorithm using Memory Function Aman Ullah Algorithms and Data Structures-2 MFKnapsack(i, j) if F[i,j] < 0 if j < Weights[i] value = MFKnapsack(i-1, j) else value = max(MFKnapsack(i-1, j), Values[i] + MFKnapsack(i-1, j-Weights[i])) F[i,j] = value return F[i,j]

Example Aman Ullah Algorithms and Data Structures-2 Let us apply the memory function method to the instance considered in above example. The next slide shows the results. In this case only 11 out of 20 nontrivial values (i.e., not those in row 0 or in column 0) have been computed. Just one nontrivial entry, V[1, 2], is retrieved rather than being recomputed. For larger instances, the proportion of such entries can be significantly larger.

Example (Contd…) Aman Ullah Algorithms and Data Structures-2

Change-Making Problem (Greedy Technique)

Greedy Technique Aman Ullah Algorithms and Data Structures-2 Today’s Topics

Optimization Problems: An optimization problem is one in which you want to find, not just a solution, but the best one. A “greedy” algorithm sometimes works well for optimization problems. It works in phases. At each phase, you take the best you can get right now, without regarding the future consequences. You hope that by choosing a local optimum value at each step, you will end up in a global optimal solution. For some problems, greedy approach always gets optimal solution but for others, it may find good solutions but not always the best solution. If so, it is called a greedy heuristic or approximation. Greedy Technique Aman Ullah Algorithms and Data Structures-2

Greedy algorithms are typically used to solve optimization problems. Most of these problems have n inputs and require us to obtain a subset that satisfies some constraints. Any subset that satisfies these constraints is called a feasible solution. We are required to find a feasible solution that either minimizes or maximizes a given objective function. Greedy Technique (Contd…) Aman Ullah Algorithms and Data Structures-2

Change-Making Problem: Suppose you want to give change for a certain amount of money, using the fewest possible bills (currency notes) and coins. A greedy algorithm would do this as: –At each step take the largest possible bill (currency note) or coin that does not overshoot. –Apply the following code: while ( N > 0) { //Here N is the currency note for which you want the change Give largest denomination coin <= N Reduce N by the value of coin } Change-Making Problem Aman Ullah Algorithms and Data Structures-2

Formally, the change making problem can be stated as: Given k coins (dimensions) d 1, d 2, d 3,....., d k (e.g. d 1 = 25(quarter), d 2 = 10(dime), d 3 = 5(nickel), d 4 = 1(cent) and given N(any currency for which you want to get change in coins), find a way of writing N = i 1 d 1, i 2 d 2, i 3 d 3,....., i k d k such that i 1 + i 2 + i 3 +....... i k is minimized. For example, we want to make \$6.39 in change of currency notes and coins, then we can choose –a \$5 note –a \$1 note to make a total of \$6 Change-Making Problem (Contd…) Aman Ullah Algorithms and Data Structures-2

–a 25 cents coin (quarter), to make \$6.25 –a 10 cents coin (dime), to make \$6.35 –Four 1 cent coins, to make \$6.39 Here greedy algorithm works well by choosing the largest currency note or coin at a stage. We now consider the problem where greedy algorithm cannot work well. In some monetary system, “Krons” come in 1 kron, 7 kron and 10 kron coins. Now if we use greedy algorithm to count out 15 krons amount, then we can get: –a 10 kron coin –Five 1 krone coins to make a total of 15 krons. Change-Making Problem (Contd…) Aman Ullah Algorithms and Data Structures-2

From above, we have used total of 6 coins to make an amount of 15 krons (using greedy approach). But a better solution would be to use two 7 kron coins and one 1 kron coin (a total of 3 coins). The greedy algorithm results in a solution above but not an optimal solution. The greedy algorithm approach gives us an optimal solution when the coins are all powers of a fixed denominations i.e. N = i 0 D 0, i 1 D 1, i 2 D 2,....., i k D k For example, USA coins are multiples of 5 i.e. –5 Cents (Nickle) = 5 1 –10 Cents (Dime) = 2X5 1 –25 Cents (Quarter) = 5 2 –1 Cent = 5 0 Change-Making Problem (Contd…) Aman Ullah Algorithms and Data Structures-2

Since krons were not written in powers or multiples of some digit, therefore we could not solve it by greedy approach. Dynamic Programming Approach: Set up a table, C[1.. K, 0.. N], where rows denote available denominations (i.e. categories of coins or currency notes), d i (1 ≤ i ≤ k) and columns denote the amount from 0 to N units (0 ≤ j ≤ N). C[i, j] denotes the minimum number of coins required to pay the amount ‘j’ using only coins of denominations 1 to i (somewhat like 0-1 knapsack problem). C[k, N] is the required solution. To pay an amount ‘j’ units, using coins of denominations 1 to i, we have two choices: Change-Making Problem (Contd…) Aman Ullah Algorithms and Data Structures-2

–Either choose NOT to use any coins of denominations –Or choose at least one coin of denomination i and also pay the amount (j - d i ). To pay j - d i units, it takes C[i, j - d i ] coins. Thus C[i, j] = 1 + C[i, j – d i ] Since we want to minimize the number of coins used, therefore we can write C[i, j] = min(C[i - 1, j], 1 + C[i, j - d i ]) So, we get the following dynamic programming algorithm for change making problem: Change-Making Problem (Contd…) Aman Ullah Algorithms and Data Structures-2

Coins(N) Given d= {1, 4, 6} //Here d[1] = 1, d[2] = 4 & d[3] = 6 for i ← 1 to k C[i, 0] ← 0 for j ← 1 to N C[0, j] ← ∞ for i ← 1 to k for j ← 1 to N if (j ≥ d[i]) C[i,j] ← min(C[i-1,j], 1 + C[i, j-d[i]]) else C[i,j] ← C[i-1,j] Change-Making Problem (Dynamic Programming Aman Ullah Algorithms and Data Structures-2

0-1 Knapsack Problem Construction Aman Ullah Algorithms and Data Structures-2 Change-Making Problem (Contd…) Note: This algorithm is not telling us how many coins of any denomination are being returned. This can be found by making an algorithm (with ‘Keep’ variable) like the one we made in 0-1 knapsack problem. The above dynamic programming algorithm takes O(k X N) time. Here N can be as large as 2 k, so it is actually an exponential algorithm in k. Hence it will take more time to execute.

Coins(N) Given d= {1, 4, 6} //Here d[1] = 1, d[2] = 4 & d[3] = 6 for i ← 1 to 3 i=1C[1, 0] ← 0 i=2C[2, 0] ← 0 i=3C[3, 0] ← 0 Change-Making Problem (Dynamic Programming Aman Ullah Algorithms and Data Structures-2 j i 0123456789101112 0 10 20 30

Coins(N) Given d= {1, 4, 6} //Here d[1] = 1, d[2] = 4 & d[3] = 6 for j ← 1 to NC[0, j] ← ∞ j=1C[0, 1] ←∞ j=2C[0, 2] ← ∞ j=12C[0, 12] ← ∞ Change-Making Problem (Dynamic Programming Aman Ullah Algorithms and Data Structures-2 j i 0123456789101112 0∞∞∞∞∞∞∞∞∞∞∞∞ 10 20 30

Coins(N) Given d= {1, 4, 6} //Here d[1] = 1, d[2] = 4 & d[3] = 6 for i ← 1 to k for j ← 1 to N if (j ≥ d[i]) C[i,j] ← min(C[i-1,j], 1 + C[i, j-d[i]]) else C[i,j] ← C[i-1,j] i=1,j=1 Since j≥d[1]=T => min(C[0,1],1+C[1,0])=min(∞,1+0)=1=C[1,1] i=1,j=2 Since j≥d[1]=T => min(C[0,2],1+C[1,1])=min(∞,1+1)=2=C[1,2] i=1,j=3 Since j≥d[1]=T => min(C[0,3],1+C[1,2])=min(∞,1+2)=3=C[1,3] i=1,j=12 Since j≥d[1]=T => min(C[0,12],1+C[1,11]) =min(∞,1+11)=12=C[1,12] Change-Making Problem (Dynamic Programming Aman Ullah Algorithms and Data Structures-2

Coins(N) Given d= {1, 4, 6} //Here d[1] = 1, d[2] = 4 & d[3] = 6 Change-Making Problem (Dynamic Programming Aman Ullah Algorithms and Data Structures-2 j i 0123456789101112 0∞∞∞∞∞∞∞∞∞∞∞∞ 10123456789101112 20 30

Coins(N) Given d= {1, 4, 6} //Here d[1] = 1, d[2] = 4 & d[3] = 6 for i ← 1 to k for j ← 1 to N if (j ≥ d[i]) C[i,j] ← min(C[i-1,j], 1 + C[i, j-d[i]]) else C[i,j] ← C[i-1,j] i=2,j=1 Since j≥d[2]=F => C[2,1]=C[1,1]=1 i=2,j=2 Since j≥d[2]=F => C[2,2]=C[1,2]=2 i=2,j=4 Since j≥d[2]=T => min(C[1,4],1+C[2,0])=min(4,1+0)=1=C[2,4] i=2,j=5 Since j≥d[2]=T => min(C[1,5],1+C[2,1])=min(5,1+1)=2=C[2,5] i=2,j=8 Since j≥d[2]=T => min(C[1,8],1+C[2,4])=min(8,1+1)=2=C[1,3] i=2,j=12 Since j≥d[2]=T => min(C[1,12],1+C[2,8]) =min(12,1+2)=3=C[1,12] Change-Making Problem (Dynamic Programming Aman Ullah Algorithms and Data Structures-2

Coins(N) Given d= {1, 4, 6} //Here d[1] = 1, d[2] = 4 & d[3] = 6 Change-Making Problem (Dynamic Programming Aman Ullah Algorithms and Data Structures-2 j i 0123456789101112 0∞∞∞∞∞∞∞∞∞∞∞∞ 10123456789101112 20123123423453 30

Coins(N) Given d= {1, 4, 6} //Here d[1] = 1, d[2] = 4 & d[3] = 6 for i ← 1 to k for j ← 1 to N if (j ≥ d[i]) C[i,j] ← min(C[i-1,j], 1 + C[i, j-d[i]]) else C[i,j] ← C[i-1,j] i=3,j=1 Since j≥d[3]=F => C[3,1]=C[2,1]=1 i=3,j=4 Since j≥d[3]=F => C[3,4]=C[2,4]=1 i=3,j=6 Since j≥d[3]=T => min(C[2,6],1+C[3,0])=min(3,1+0)=1=C[3,6] i=3,j=7 Since j≥d[3]=T => min(C[2,7],1+C[3,1])=min(4,1+1)=2=C[3,7] i=3,j=8 Since j≥d[3]=T => min(C[2,8],1+C[3,2])=min(2,1+2)=2=C[3,8] i=3,j=12 Since j≥d[3]=T => min(C[2,12],1+C[3,6]) =min(3,1+1)=2=C[3,12] Change-Making Problem (Dynamic Programming Aman Ullah Algorithms and Data Structures-2

Coins(N) Given d= {1, 4, 6} //Here d[1] = 1, d[2] = 4 & d[3] = 6 Change-Making Problem (Dynamic Programming Aman Ullah Algorithms and Data Structures-2 d[i] ji ji 0123456789101112 0∞∞∞∞∞∞∞∞∞∞∞∞ 110123456789101112 420123123423453 630123121224232

Greedy approach suggests constructing a solution through a sequence of steps, each expanding a partially constructed solution obtained so far, until a complete solution to the problem is reached. On each step-and this is the central point of this technique-the choice made must be –feasible, i.e., it has to satisfy the problem's constraints –locally optimal, i.e., it has to be the best local choice among all feasible choices available on that step –irrevocable, i.e., once made, it cannot be changed on subsequent steps of the algorithm These requirements explain the technique's name: on each step, it suggests a "greedy" grab of the best alternative available in the hope that a sequence of locally optimal choices will yield a (globally) optimal solution to the entire problem. Greedy Approach Aman Ullah Algorithms and Data Structures-2

Greedy Algorithm (N) (where N is the amount) Sort coins as d 1 > d 2 > d 3 >......... > d k S = Φ Change = 0 i = k While change ≠ N or i ≠ 0 do if Change + d[i] ≤ N then Change = Change + d[i] S = S U { d[i] } else i = i - 1 Change-Making Problem Aman Ullah Algorithms and Data Structures-2