1. Priority Queues, Heaps, and Heapsort CSE 2320 – Algorithms and Data Structures Vassilis Athitsos University of Texas at Arlington 1 Last modified: 2/22/2016

2. Overview Priority queue – list/array (sorted/unsorted) – Sorting with a priority queue Heap – Definition, properties, operations (insert, delete, create) – Array implementation – Batch initialization for a heap – Heapsort – Finding top k – Index items – the heap has the index of the element 2

3. Priority Queues Support (efficiently): – Insertion of a new element. – Deletion of the max element. – Initialization (organizing an initial set of data). Useful for online processing – So far we have seen sorting methods that work in batch mode: They are given all the items at once They sort the items. Done! – Batch methods are not efficient in cases like this. Applications: – Scheduling: flights take-off and landing, programs executed (CPU), database queries – Waitlists: patients in a hospital, student admission 3

4. Arrays and Lists as Priority Queues Arrays and lists either sorted or unsorted can be used as priority queues, but they require O(N) for either insert or delete max. 4 Data structureInsertDelete maxCreate from batch of N Array unsorted List unsorted Array sortedO(N) List sortedO(N) (find correct position)

5. Priority Queues and Sorting Sorting with a heap: – Given items to sort: – Create a priority queue that contains those items. – Initialize result to empty list. – While the priority queue is not empty: Remove max element from queue and add it to beginning of result. Heapsort – Θ(N lg N) – Must build the heap in O(N). 5

6. Heaps Initialization: – Given N data, heapify them (details in a few slides). – Time: Θ(N). Good! Insertion of a new item: – Insert the item at the right place, to maintain the heap property. (details in a few slides). – Time: O(lg N). Good! Deletion of the largest element (max-heap): – Delete the first item. – Rearrange other items, to maintain the heap property. (details in a few slides). – Time: O(lg N). Good! – Min-heap is similar. 6

7. Heap Intuition – Use a tree (‘trees are fast’, especially if they are ‘balanced’). – Keep the largest element in the root. (b.c. we delete the largest and trees are accessed from the root) – Keep the tree balanced after insert and delete. Specific heap properties: – Every node, N, is larger than or equal to any of his children (their keys). => root has the largest key – Complete tree: All levels are full except for possibly the last one If the last level is not full, all nodes are leftmost (no ‘holes’). This tree can be represented by an array, A. – Root stored at index 1, – The children of node at index i are at indexes: i (left child) and i+1 (right child) – The parent of node at index i is at index floor(i/2) 7

8. Array-Based Heap Consider this array: We can draw the array as a tree. – The children of A[N] are A[2N] and A[2N+1]. – This example shows that the tree and array representations are equivalent. 8 position123456789101112 valueXTOGSMNAERAI X T N O G S I M A A E R 1 23 7 6 5 4 109 8 11 12

9. Heap – Shape Property A binary tree representing a heap has to be complete: – All levels are full, except possibly for the last level. – At the last level: Nodes are placed on the left. Empty positions are placed on the right. – There is “no hole” 9 X T N O G S I M A A E R 1 23 7 6 5 4 109 8 11 12 Good Bad

10. Examples Invalid heaps – Node – children property violated – Complete tree property violated (‘tree with holes’) Valid heaps – Same key in node and one or both children – ‘Extreme’ heaps (all nodes in the left child are smaller than any node in the right child or viceversa) – Min-heaps Where can these elements be found? – Largest element? – 2-nd largest? – 3-rd largest? 10

11. Increasing a Key Also called “increasing the priority” of an item. Such an operation can lead to violation of the heap property. Easy to fix: – Exchange items as needed, between node and parent, starting at the node that changed key. 11 X T N O G S I M A A E R 1 23 7 6 5 4 109 8 11 12

12. Increasing a Key Also called “increasing the priority” of an item. Such an operation can lead to violation of the heap property. Easy to fix: – Exchange items as needed, between node and parent, starting at the node that changed key. Example: – An E changes to a V. 12 X T N O G S I M A A R 1 23 7 6 5 4 109 8 11 12 EVEV

13. Increasing a Key Also called “increasing the priority” of an item. Such an operation can lead to violation of the heap property. Easy to fix: – Exchange items as needed, between node and parent, starting at the node that changed key. Example: – An E changes to a V. – Exchange V and G. Done? 13 X T N O V S I M A A G R 1 23 7 6 5 4 109 8 11 12

14. Increasing a Key Also called “increasing the priority” of an item. Such an operation can lead to violation of the heap property. Easy to fix: – Exchange items as needed, between node and parent, starting at the node that changed key. Example: – An E changes to a V. – Exchange V and G. – Exchange V and T. Done? 14 X V N O T S I M A A G R 1 23 7 6 5 4 109 8 11 12

15. Increasing a Key Also called “increasing the priority” of an item. Such an operation can lead to violation of the heap property. Easy to fix: – Exchange items as needed, between node and parent, starting at the node that changed key. Example: – An E changes to a V. – Exchange V and G. – Exchange V and T. Done. 15 X V N O T S I M A A G R 1 23 7 6 5 4 109 8 11 12

16. Increasing a Key Implementation: fixUp(Item a[], int k){ while ((k > 1) && (less(a[k/2], a[k]))) { exch(a[k], a[k/2]); k = k/2; } 16 X V N O T S I M A A G R 1 23 7 6 5 4 109 8 11 12

17. Decreasing a Key Also called “decreasing the priority” of an item. Such an operation can lead to violation of the heap property. Easy to fix: – Exchange items as needed, between node and largest child, starting at the node that changed key. 17 X T N O G S I M A A E R 1 23 7 6 5 4 109 8 11 12

18. Decreasing a Key Also called “decreasing the priority” of an item. Can lead to violation of the heap property. Fix: – Exchange items as needed, between node and largest child, starting at the node that changed key. Example: – An X changes to a B. 18 XBXB T N O G S I M A A E R 1 23 7 6 5 4 109 8 11 12

19. Decreasing a Key B will move down until in a good position. Exchange B and T. 19 T B N O G S I M A A E R

20. Decreasing a Key B will move down until in a good position. Exchange B and T. Exchange B and S. 20 T S N O G B I M A A E R

21. Decreasing a Key B will move down until in a good position. Exchange B and T. Exchange B and S. Exchange B and R. 21 T S N O G R I M A A E B

22. Decreasing a Key Implementation: fixDown(Item a[], int k, int N) { int j; while (2*k <= N) { j = 2*k; // left child index // Update j to point to the larger child: if ((j < N) && less ((a[j], a[j+1]))) j++; // right child is larger. if (!less(a[k], a[j])) break; exch(a[k], a[j]); k = j; // prep to fix j next iter } 22 T S N O G R I M A A E B 1 23 7 6 5 4 109 8 11 12

23. Inserting a New Record This is a heap with 12 items. How will a heap with 13 items look? – Where can the new node be? (do not worry about the data in the nodes now) 23 T S N O G R I M A A E B 1 23 7 6 5 4 10 9 8 11 12 position123456789101112 valueTSOGRMNAEBAI

24. Inserting a New Record Let’s insert V 24 T S N O G R I M A A E B 1 23 7 6 5 4 10 9 8 11 12 N 13 position12345678910111213 valueTSOGRMNAEBAI

25. Inserting a New Record Let’s insert V Put V in the last position and fix up. 25 T S N O G R I M A A E B 1 23 7 6 5 4 10 9 8 11 12 N 13 position12345678910111213 valueTSOGRMNAEBAIV V

26. Inserting a New Record Fixed heap 26 V S N T G R I O A A E B 1 23 7 6 5 4 10 9 8 11 12 N 13 position12345678910111213 valueVSTGRONAEBAIM M

27. Inserting a New Record Insertion: Put in the last position and fix up. Discuss the runtime for insertion. – Let N be the number of nodes in the heap. 27 T S N O G R I M A A E B 1 23 7 6 5 4 10 9 8 11 12 N 13 position12345678910111213 valueTSOGRMNAEBAI V CaseDiscussionNotationExample Best Worst General

28. Inserting a New Record Insertion: Put in the last position and fix up. Discuss the runtime for insertion. – Let N be the number of nodes in the heap. 28 T S N O G R I M A A E B 1 23 7 6 5 4 10 9 8 11 12 N 13 position12345678910111213 valueTSOGRMNAEBAI V CaseDiscussionNotationExample Best1V was M WorstHeight of heap Shown here GeneralO(lgN)

29. Remove the Maximum This is a heap with 12 items. How will a heap with 11 items look like? – What node will disappear? (Again think about the ‘circles’, not the data in them) Where is the record with the highest key? 29 T S N O G R I M A A E B 1 23 7 6 5 4 10 9 8 11 12 position123456789101112 valueTSOGRMNAEBAI len

30. Remove the Maximum Removing the maximum – Save the record on top of the heap (e.g. as ret) – Copy the last record on the first position (on top) – Decrement heap size by 1 – Fix-down the top (invalid heap now) – Return the original top record, ret. ‘No holes’ – Because a valid item is placed in root. 30 I S N O G R M A A E B 1 23 7 6 5 4 10 9 8 11 position1234567891011 valueISOGRMNAEBA => T len

31. Remove the Maximum Fixed heap: Programming trick: – Swap the top with the last – … – Return from index len+1 31 S R N O G I M A A E B 1 23 7 6 5 4 10 9 8 11 position1234567891011 valueSROGIMNAEBAT => T len

32. Remove the Maximum Removing the maximum – Save the record on top of the heap (e.g. as ret) – Copy the last record on the first position (on top) – Decrement heap size by 1 – Fix-down the top – Return the original top record, ret. Discuss the runtime for remove max. 32 S R N O G I M A A E B 1 23 7 6 5 4 10 9 8 11 position1234567891011 valueSROGIMNAEBAT => T len CaseDiscussionNotationExample Best1 WorstHeight of heap Last node was A, not I General1<=…<=lgNO(lgN)

33. Insertions and Deletions - Summary Insertion: – Insert the item to the end of the heap. – Call fix up to restore the heap property. – Time = O(lg N) Deletion: – Will always delete the maximum element. This element is always at the top of the heap (the first element of the heap). – Deletion of the maximum element: Exchange the first and last elements of the heap. Delete the last element (which is the maximum element). Call fixDown to restore the heap property. Time = O(lg N) 33

34. Batch Initialization Batch initialization of a heap – The process of converting an unsorted array of data into a heap. – We will see 2 methods: top-down and bottom-up. 34 Batch Initialization Method TimeExtra space (in addition to the space of the input array) Top-down (insert in new array one by one) O(N lg N) Bottom-up (fix the given array)

35. Bottom-Up Batch Initialization 35 // pseudocode: void bottom_up_init(Item * a, int N) for i = N/2,..., 1 fixDown(array, i, N) See animation: https://www.cs.usfca.edu/~galles/visualization/HeapSort.htmlhttps://www.cs.usfca.edu/~galles/visualization/HeapSort.html o Note that they do not highlight the node being processed, but directly the children of it as they are compared to find the larger one of them. Another example is at the end of the slides.

36. Running Time How can we analyze the running time? To simplify, suppose that N = 2 n – 1 (=> heap height is lgN = n-1) Counter i starts at value 2 n-1 - 1. – That give the last node on level n-2. – At that point, we call fixDown on a heap of height 1. – For all the nodes at this level, we call fixDown on a heap of height 1 (nodes at this level are at indexes i s.t. 2 n-1 -1 ≤ i ≤ 2 n-2 ). … When i is 1 (=2 0 ) we call fixDown on a heap of height n-1. 36 // pseudocode: void bottom_up_init(Item * a, int N) for i = N/2,..., 1 fixDown(array, i, N)

37. Running Time: Θ(Ν) 37 Counter from: Counter to: LevelNodes per level Height of heaps rooted at these nodes Time per node (fixDown) Time for fixing all nodes at this level 2 n-2 2 n-1 – 1n-22 n-2 1O(1)O(2 n-2 * 1) 2 n-3 2 n-2 – 1n-32 n-3 2O(2)O(2 n-3 * 2) 2 n-4 2 n-3 – 1n-42 n-4 3O(3)O(2 n-4 * 3) … 2 0 = 12 1 -1 = 102 0 = 1n – 1O(n-1)O(2 0 * (n-1)) To simplify, assume: N = 2 n - 1. The analysis is a bit complicated. Total time: sum over the rightmost column: Θ(Ν) (linear!) // pseudocode: void bottom_up_init(Item * a, int N) for i = N/2,..., 1 fixDown(array, i, N)

38. Top-Down Batch Initialization 38

39. Heapsort Use a heap to sort an array: Build heap using bottom-up Repeat: – Swap top (largest element) and last element – Decrease heap size – Call fixDown the top node In the pseudocode below: Where is the sorted data? How does the heap get shorter?) 39 void heapsort(Item a[], int N) bottom_up_init(a, N) // build heap in Θ(N) for i = N, …, 2 exch(a[1], a[i]) // put max at the end fixDown(a, 1, i-1) // fixDown in heap of smaller size

40. Finding the Top k Largest Elements 40

41. Finding the Top k Largest Elements Assume N elements Using a max-heap – Build max-heap of size N from all elements, then – remove k times – Time: Θ(N + k*lgN) (build heap: Θ(N), k delete ops: Θ(k*lgN) ) Using a min-heap – Build a min-heap, H, of size k (from the first k elements). – (N-k) times perform both: insert and then delete in H. – After that, all N elements went through this min-heap and k are left so they must be the largest ones. – advantage: less space (constant space: k) – Time: Θ(k + (N – k)*lgk) (build heap + (N-k) insert & delete) 41

42. Index heap So far: – We assumed that the actual data is stored in the heap. – We can increase/decrease priority of any specific node and restore the heap. In a real application we need to be able to do more – Find a particular record in a heap John Doe got better and leaves. Find the record for John in the heap. (This operation will be needed when we use a heap later on.) – You cannot put the actual data in the heap You do not have access to the data (e.g. for protection) To avoid replication of the data. For example you also need to frequently search in that data so you also need to organize it for efficient search – Case 1: by a different field (e.g. ID number) – Case 2: by priority 42

43. Index Heap Example 43 1.Show a heap with this data. IndexHA (H->A) AH (A->H) NamePriorityOther data 152Aidan10 214Alice7 345Cam10 423Joe13 531Kate20 656Mary4 727Sam6

44. Index Heap Example IndexHA (H->A) AH (A->H) NamePriorityOther data 152Aidan10 214Alice7 345Cam10 423Joe13 531Kate20 656Mary4 727Sam6 44 HA – Heap to Array AH – Array to Heap (20) 5 1 (10) 1 2 (13) 4 3 (7) 2 7 (4) 6 (10) 3 5 (7) 2 4 Property: HA(AH(j) = j e.g. HA(AH(2) = 2 AH(HA(j) = j e.g. HA(AH(5) = 5