1. Algorithms and data structures Protected by http://creativecommons.org/licenses/by-nc-sa/3.0/hr/ 13.9.2015

2. Creative Commons n You are free to: share — copy and redistribute the material in any medium or format share — copy and redistribute the material in any medium or format adapt — remix, transform, and build upon the material adapt — remix, transform, and build upon the material n Under the following terms: Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. NonCommercial — You may not use the material for commercial purposes. NonCommercial — You may not use the material for commercial purposes. ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original. ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original. No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits. Text copied from http://creativecommons.org/licenses/by-nc-sa/3.0/ 13.9.2015Algorithms and data structures, FER Notices: You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation. No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material. 2 / 36

3. 13.9.2015 Algorithms and complexity Algorithms Complexity of algorithms

4. Algorithms and data structures, FER13.9.2015 Abu Ja'far Muhammad ibn Musa Al-Khwarizmi n Abu Ja'far Muhammad ibn Musa Al-Khwarizmi (the sources differ regarding his full name and transcription!) n أبو جعفر محمد بن موسى الخوارزمي Mohammad, father of Jafar, son of Musa from Khwarezm Mohammad, father of Jafar, son of Musa from Khwarezm n Born in Khwarezm, what is today Khiva, Uzbekistan, about year 780. Khiva, Uzbekistan Khiva, Uzbekistan n Died in Baghdad about year 850. n One of the 10 most appreciated mathematicians of all times. 4 / 36

5. Algorithms and data structures, FER13.9.2015 Achievements of al Khwarizmi n Encourages using of Hindu/Arabic numbers – introduces zero n About year 825 he wrote the book “Kitab al-jabr w'al-muqabala” in Baghdad الكتاب المختصر في حساب الجبر والمقابلة الكتاب المختصر في حساب الجبر والمقابلة The Compendious Book on Calculation by Completion and Balancing The Compendious Book on Calculation by Completion and Balancing jabr - moving to the other side of an equation jabr - moving to the other side of an equation x - 2 = 12  x = 12 + 2 muqabala - equal subtraction from both sides of an equation muqabala - equal subtraction from both sides of an equation – x + y = y + 7  x = 7 al-jabr  algebra - bone-setting al-jabr  algebra - bone-setting 5 / 36

6. Algorithms and data structures, FER13.9.2015 From al Khwarizmi to algorithms... n He believed that any mathematical problem can be split into steps, i.e. to a sequence of rules. n In the Latin translation of his book (12 th century), in front of every rule there is: Dixit Algorizmi - said Al Khwarizmi Dixit Algorizmi - said Al Khwarizmi The algorithm is as follows... The algorithm is as follows... n The very first understanding of the word “algorithm” refers only to calculation rules related to numbers; later it also refers to rules for solving other tasks in mathematics n In the 20 th century, with emerging of computers, the term is extended to computing, and later also to other fields n Rules to achieve a required goal 6 / 36

7. Algorithms and data structures, FER13.9.2015 What is an algorithm n A precisely described way to solve a problem n Uniquely defines what has to be done n Initial objects must be defined, belonging to a class of objects on which operations can be performed As outcome appear final objects or results As outcome appear final objects or results Final number of steps; every step is described by an instruction Final number of steps; every step is described by an instruction Execution is an algorithmic process Execution is an algorithmic process n Applicable, if the result can be obtained in a finite time n Examples for unallowed instructions: Calculate 5/0 Calculate 5/0 Increase x for 6 or 7 Increase x for 6 or 7 7 / 36

8. Algorithms and data structures, FER13.9.2015Algorithm n An algorithm must be effective: The result should be achievable by using pencil and paper, in a finite amount time The result should be achievable by using pencil and paper, in a finite amount time n Examples: Addition of integers is effective Addition of integers is effective Addition of real numbers is not because a number with infinite amount of digits may appear Addition of real numbers is not because a number with infinite amount of digits may appear n With programming skills and with understanding the nature of the problem to be solved, the student can write an effective algorithm The goal of this course is to learn how to design and program an efficient algorithm. The goal of this course is to learn how to design and program an efficient algorithm. n Effective and efficient are not synonyms effective = the one that brings effect, results effective = the one that brings effect, results efficient = effective & successful (regarding the consumption of resources - time, processor, disk, memory) efficient = effective & successful (regarding the consumption of resources - time, processor, disk, memory) – Multiplication can be performed as repetitive addition - effective but not efficient! – Solving of a large set of linear equations using the Kramer’s rule - effective but not efficient! n Wikipedia: A simple way of distinguishing between efficiency and effectiveness is the saying, " Efficiency is doing things right, while Effectiveness is doing the right things. " 8 / 36

9. Algorithms and data structures, FER13.9.2015Procedure n Procedure has all properties of an algorithm, except it does not necessarily terminate after a finite number of steps In language C that may be a void function In language C that may be a void function n Examples for procedure: Operating system Operating system Text editor Text editor n Execution time must be “reasonable” n Example: An algorithm that would play chess choosing at each move all the possible consequences of it, would require billions of years even on a quickest conceivable computer An algorithm that would play chess choosing at each move all the possible consequences of it, would require billions of years even on a quickest conceivable computer Algorithm Procedure 9 / 36

10. Algorithms and data structures, FER13.9.2015 Analysing algorithms n Program - description of an algorithm in a programming language that uniquely defines what the computer should do n Programming – to learn the syntax of a procedural language and to acquire basic intuitive skills for algorithmisation of a problem being verbally described n Algorithms + data structures = programs (Wirth) How do devise algorithms? How do devise algorithms? How to structure data? How to structure data? How to express the algorithms? How to express the algorithms? How to verify the algorithm correctness? How to verify the algorithm correctness? How to analyse algorithms? How to analyse algorithms? How to check (test) a program? How to check (test) a program? n The algorithm design procedures are not unique and require creativity. Otherwise, algorithm generators would already exist. It means that (for the time being?) the content of this course cannot be completely algorithmised. We shall use the programming language C. For a compact description of complex algorithms pseudo-code may be used. 10 / 36

11. Algorithms and data structures, FER13.9.2015 Effective but not efficient? Important! Due to a new computer program, which has slowed down our work I beg the patients for patience! Thanks, Your Doctor 11 / 36

12. 13.9.2015 Analysis of complexity "a priori" and "a posteriori“ analyses O-notation Asymptotic running time Examples

13. Algorithms and data structures, FER13.9.2015Basics n Purpose Intellectual amusement? Intellectual amusement? Prediction of execution time Prediction of execution time Search for more efficient algorithms Search for more efficient algorithms n Hypotheses: Sequential single processor computer Sequential single processor computer Fixed time (or limited by an upper bound) for retrieval of contents of a memory location Fixed time (or limited by an upper bound) for retrieval of contents of a memory location Time for processing of an operation (arithmetic, logical, assignment, function call) is upper bounded by a constant Time for processing of an operation (arithmetic, logical, assignment, function call) is upper bounded by a constant 13 / 36

14. Algorithms and data structures, FER13.9.2015 “a priori” and “a posteriori” analyses n Selection of data sets for exhaustive algorithm testing: best case scenario best case scenario worst case scenario worst case scenario average (typical) behaviour average (typical) behaviour n a priori Duration of algorithm execution ( worst case scenario ) expressed as a function of some relevant arguments ( e.g. amount of data ) Duration of algorithm execution ( worst case scenario ) expressed as a function of some relevant arguments ( e.g. amount of data ) n a posteriori Statistics achieved by measurements on computer Statistics achieved by measurements on computer 14 / 36

15. Algorithms and data structures, FER13.9.2015 “a priori” analysis n Estimation of execution time, independent of computer type, programming language, compiler... n examples: a) x += y;1 b) for(i = 1; i <= n; i++) {n x += y; x += y;} c) for(i = 1; i <= n; i++) {n 2 for(j = 1; j <= n; j++) { for(j = 1; j <= n; j++) { x += y; x += y; }} 15 / 36

16. Algorithms and data structures, FER13.9.2015 Algortihm complexity n A tool is needed for comparison of algorithms efficiency Application of mathematics Application of mathematics n Execution time is observed when the count of input data becomes sufficiently large n Example: execution of inversion of a n x n matrix, measured in seconds is in proportion to n 3 If the inversion of a 100 x 100 elements requires 1 minute, If the inversion of a 100 x 100 elements requires 1 minute, then the inversion of a 200 x 200 matrix will take 8 minutes (2 3 =8). then the inversion of a 200 x 200 matrix will take 8 minutes (2 3 =8). How to express the execution time as a function of n? How to express the execution time as a function of n? – E.g. execution time grows with the matrix size as n 3 16 / 36

17. Algorithms and data structures, FER13.9.2015 O - notation f(n) = O(g(n)) if there exist two positive constants c and n 0 such that is valid f(n)   c  g(n)  for all n  n 0 f(n) = O(g(n)) if there exist two positive constants c and n 0 such that is valid f(n)   c  g(n)  for all n  n 0 We look for the minimum g(n) to satisfy the above requirement We look for the minimum g(n) to satisfy the above requirement In other words, the execution time is estimated as an order of magnitude of f(n), determined by the quantity of data n, multiplied by some constant c In other words, the execution time is estimated as an order of magnitude of f(n), determined by the quantity of data n, multiplied by some constant c n examples: n 3 +5n 2 +77n = O(n 3 ) n 3 +5n 2 +77n = O(n 3 ) How much work is required to move 1 chair from the room A to the room B? How much work is required to move 1 chair from the room A to the room B? How much work is required to move n chairs from A to B? How much work is required to move n chairs from A to B? How much work is required to move n chairs from A to B, whereby when each new chair is brought to B, all the present chairs there have to be shifted, supposing that for shifting of a single chair, the same effort is required like when bringing it from A to B? How much work is required to move n chairs from A to B, whereby when each new chair is brought to B, all the present chairs there have to be shifted, supposing that for shifting of a single chair, the same effort is required like when bringing it from A to B? 1 + 2 + 3 + 4 +...+ n = n(n+1)/2 = n 2 /2 + n/2 = O (n 2 ) 17 / 36

18. Algorithms and data structures, FER13.9.2015 O - notation By “a priori” analysis, the execution time O(g(n)) of an algorithm is obtained By “a priori” analysis, the execution time O(g(n)) of an algorithm is obtained If the number of operations to be performed in an algorithm depends upon an input argument n in the form of a polynomial of m th degree, then the execution time for that algorithm equals O(n m ). If the number of operations to be performed in an algorithm depends upon an input argument n in the form of a polynomial of m th degree, then the execution time for that algorithm equals O(n m ). Proof: Proof: If the execution time follows a polynomial dependency: A(n) = a m n m +... + a 1 n + a 0 then it is valid:  A(n)    a m  n m +... +  a 1  n +  a 0   A(n)   (  a m  +  a m-1  / n +... +  a 1  /n m-1 +  a 0  / n m ) n m  A(n)   (  a m  +... +  a 1  +  a 0  ) n m, for every n  1 with: c =  a m  +... +  a 1  +  a 0  and n 0 = 1 the statement is proved. with: c =  a m  +... +  a 1  +  a 0  and n 0 = 1 the statement is proved. 18 / 36

19. Algorithms and data structures, FER13.9.2015 O - notation The value for c can be any constant grater then  a m  if n is big enough: The value for c can be any constant grater then  a m  if n is big enough: If an algorithm consists of k segments with respective execution times: c 1 n m1, c 2 n m2,...,c k n mk Then the total time is c 1 n m1 + c 2 n m2 +... + c k n mk This implies that for this algorithm the execution time is equal to O(n m ), where This implies that for this algorithm the execution time is equal to O(n m ), where m = max { m i }, i =1,...,k 19 / 36

20. Algorithms and data structures, FER13.9.2015 O - notation n For any n big enough, it holds: O(1) < O(log n) < O(n) < O(nlog n) < O(n 2 ) < O(n 3 ) <...< O(2 n ) < O(n!) O(1) means that the execution time is upper bounded by a constant O(1) means that the execution time is upper bounded by a constant Other values, until the penultimate, represent polynomial execution times Other values, until the penultimate, represent polynomial execution times – Each subsequent execution time is larger for an order of magnitude Penultimate expression represents exponential execution time Penultimate expression represents exponential execution time – No polynomial can be its upper bound because for n big enough, this function exceeds any polynomial Algorithms requiring exponential time can be insolvable in a reasonable time, regardless to the processing speed of a sequential computer Algorithms requiring exponential time can be insolvable in a reasonable time, regardless to the processing speed of a sequential computer 20 / 36

21. Algorithms and data structures, FER13.9.2015  - notation f(n) =  (g(n)) if there exist positive constants c and n 0 such that is valid  f(n)   c  g(n)  for all n > n 0 f(n) =  (g(n)) if there exist positive constants c and n 0 such that is valid  f(n)   c  g(n)  for all n > n 0 We look for the maximum g(n) to satisfy the above requirement We look for the maximum g(n) to satisfy the above requirement Lower bound for an algorithm execution time Lower bound for an algorithm execution time In no case the execution can be shorter than  In no case the execution can be shorter than  – E.g. multiplication of two n x n matrices always requires time  (n 3 ), O (n 3 ) – Addition of n numbers always requires the time  (n) = O (n) – Retrieval of a number among n unsorted numbers  (1)  O (n) 21 / 36

22. Algorithms and data structures, FER13.9.2015  - notation f(n) =  (g(n)) if there exist positive constants c 1, c 2 and n 0 such that is valid c 1  g(n)    f(n)   c 2  g(n)  for all n > n 0 f(n) =  (g(n)) if there exist positive constants c 1, c 2 and n 0 such that is valid c 1  g(n)    f(n)   c 2  g(n)  for all n > n 0 f and g grow equally fast for large n f and g grow equally fast for large n The ratio of f and g is between c 1 and c 2 The ratio of f and g is between c 1 and c 2 In other words, the execution times are equal for the best and the worst case In other words, the execution times are equal for the best and the worst case n Example: Sort n written examinations lexicographically according to students’ names. The process starts by finding the first one, than in the remaining set it looks for the first one, and so on. Sort n written examinations lexicographically according to students’ names. The process starts by finding the first one, than in the remaining set it looks for the first one, and so on. – The process takes the same time regardless of possible ordering of the elements: O(n 2 ) =  (n 2 ) =  (n 2 ) 22 / 36

23. Algorithms and data structures, FER13.9.2015 Asymptotic execution time f(n) ~ g(n) if f(n) ~ g(n) if It is pronounced: " f(n) is asymptotically equal to the function g(n) ” It is pronounced: " f(n) is asymptotically equal to the function g(n) ” A more precise estimation of the execution time than by O -notation A more precise estimation of the execution time than by O -notation The leading member’s order of magnitude is known but also its constant multiplier The leading member’s order of magnitude is known but also its constant multiplier n If, for example: f(n) = a k n k +... + a 0 then: f(n) = O(n k ) and f(n) ~ a k n k n Examples: 3*2 x +7 logx+x ~ 3*2 x 3*2 x +7 logx+x ~ 3*2 x For the example from the previous slide (sorting of written examinations) For the example from the previous slide (sorting of written examinations) – Due to linear decrement of the remaining set, where the first member is searched, asymptotic duration is: ~ n 2 /2 23 / 36

24. Algorithms and data structures, FER13.9.2015 n  i = n(n+1)/2 = O (n 2 ) i=1 Asymptotic execution time n While counting repetitions in some algorithms, the summations such as follows often appear:  n 2 /2 for (i = 1; i <= n; i++) for (j = 0; j < i; i++) {...} for (j = 0; j < i; i++) {...} 24 / 36

25. Algorithms and data structures, FER13.9.2015 Different complexities in logarithmic scale 25 / 36

26. Algorithms and data structures, FER13.9.2015 O -notation - examples IspisiTrazi.c (PrintSearch.c)  IspisiTrazi.c (PrintSearch.c) print print The loop is always repeated n times   (n), O(n),  (n) The loop is always repeated n times   (n), O(n),  (n) n search Execution time is O(n) Execution time is O(n) Lower bound is  (1) Lower bound is  (1) – In the best case, looked for array member is found in the first step – In the worst case, all the array members have to be examined 26 / 36

27. Algorithms and data structures, FER13.9.2015 “a posteriori” analysis n Real time necessary to execute an algorithm on a specific computer #include // where it is declared struct timeb { time_t time; // elapsed seconds since midnight, // 01.01.1970, UTC unsigned short millitm; // milliseconds short timezone; // difference to UTC, in minutes short dstflag; // <>0 if daylight saving time is active }; void ftime(struct timeb *timeptr); 27 / 36

28. Algorithms and data structures, FER13.9.2015 “a posteriori” analysis n For time measurement, the program should include: n n Universal Time Co-ordinated (UTC) The old term was Greenwich Mean Time (GMT) struct timeb time1, time2; long durationms; ftime (&time1);... ftime (&time2); durationms = 1000 * (time2.time - time1.time) + time2.millitm - time1.millitm; 28 / 36

29. Algorithms and data structures, FER13.9.2015Exercises n For the algorithm determine: a) a priori execution time b) asymptotic estimate for average execution time c) asymptotic estimate for the best case execution time d) asymptotic estimate for the worst case execution time e) for what value of the variable b happens the worst case? for (i = 0; i < n; i++) { if (a[i] == b) { if (a[i] == b) { printf ("i = %d\n", i); break;}} 29 / 36

30. Algorithms and data structures, FER13.9.2015Exercises n Write the function to calculate the sum of digits for a given natural number N. What is the function complexity? int sumDigits (int N) { int sum = 0; while (N > 0) { sum += N % 10; N /= 10; } return sum; } 30 / 36

31. Algorithms and data structures, FER13.9.2015Exercises Actual durationO~ 5n 2 +7n+4 n2n2n2n2 5n 2 3n 2 +700n+2 n2n2n2n2 3n 2 n 2 +7logn+40 n2n2n2n2 n2n2n2n2 0,1*2 n +100n 2 2n2n2n2n 0,1*2 n (2n+1) 2 n2n2n2n2 4n 2 6nlogn + 10n nlogn6nlogn 8logn + 1000 logn8logn 3n! + 2 n + n 10 n!3n! 3*5 n + 6*2 n + n 100 5n5n5n5n 3*5 n 31 / 36

32. Algorithms and data structures, FER13.9.2015 Examples of dependency of execution time on the amount of data and complexity nT(n) = nT(n) = n lg(n)T(n) = n 2 T(n) = n 3 T(n) = 2 n 5 0.005  s0.01  s0.03  s0.13  s0.03  s 10 0.01  s0.03  s0.1  s1  s 20 0.02  s0.09  s0.4  s8  s 1 ms 50 0.05  s0.28  s2.5  s125  s 13 days 100 0.1  s0.66  s10  s 1 ms4 x 10 13 years 32 / 36

33. Algorithms and data structures, FER13.9.2015 What can be solved in a given time Time complexity The size of the largest problem case that can be solved in 1 hour using... currently available computer 100 times faster computer 1000 times faster computer nN1100N11000N1 n2n2 N210 N231.6 N2 n3n3 N34.64 N310 N3 2n2n N4N4+6.64N4+9.97 3n3n N5N5+4.19N5+6.29 33 / 36

34. Algorithms and data structures, FER13.9.2015 Relative contribution of components in the expression: Relative contribution of components in the expression: n 4 + n 3 log n + 2 (n-1) O ( 2 n ) nn4n4 n 3 log n2 (n-1) 1101 1010 000100 0 512 1783 5216 04665536 18104 9767 321131072 506 250 000212 372562 949 953 421 312 100100 000 0002 000 0006,3 x 10 29 34 / 36

35. Algorithms and data structures, FER13.9.2015 Influence of the value of the constant Influence of the value of the constant K = 1 000 000 nK nK n 2 K n 4 K 2 n 110 6 2 x 10 6 1010 7 10 810 1,0 x 10 8 10010 810 10 14 1,2 x 10 36 1 00010 9 10 12 10 18 ~ 10 306 35 / 36

36. Algorithms and data structures, FER13.9.2015 What does it mean “n big enough”, i.e. n > n 0 Source: Cormen, Leiserson & Rivest: Introduction to algorithms, 2/e,MIT Press, 2001 36 / 36