Big-O and Friends

Formal definition of Big-O A function f(n) is O(g(n)) if there exist positive numbers c and N such that: f(n) = N Example: Let f(n) = 2n 2 + 3n + 1 n = f(n) = Let g(n) = n 2 and let c = 3 cg(n) = So f(n) = N, where c = 3, g(n) = n 2, and N = 4 Hence, f(n) = 2n 2 + 3n + 1 is O(n 2 )

Informal explanation A function f(n) is O(g(n)) if there exist positive numbers c and N such that: f(n) = N The part about “for all n >= N ” means we can ignore small values of n The c lets us adjust for functions that vary only by a constant N ok to ignore f(n) g(n) f(n)=g(n)+k 2g(n)

Big-O is an upper bound Big-O is an upper bound, not a lower bound Here is the same example as before: n = f(n) = Let g(n) = n 3 and let c = 1 cg(n) = So f(n) = N, where c = 1, g(n) = n 3, and N = 4 Hence, f(n) = 2n 2 + 3n + 1 is O(n 3 )

One more time Again, A function f(n) is O(g(n)) if there exist positive numbers c and N such that: f(n) N n = f(n) = Let g(n) = 2 n and let c = 1 cg(n) = So f(n) = N, where c = 1, g(n) = 2 n, and N = 7 That is, 2n 2 + 3n + 1 = 7 Hence, f(n) = 2n 2 + 3n + 1 is O(2 n )

Big-O is a sloppy bound Formally, Big-O is just an upper bound For example, all the sorting algorithms we have studied can be said to be O(n 2 ), or O(n 3 ), or O(n 3 log n), or O(2 n ), etc. However, these sorting algorithms are not O(1), or O(log n), or O(n) If you are asked on a test a question such as “Is insertion sort an O(n 3 ) algorithm?” the correct answer is “Yes” Informally, however, we use Big-O to mean “the least upper bound we can find”

Big-O is formally defined A function f(n) is O(g(n)) if there exist positive numbers c and N such that: f(n) = N Note that this does not put any restrictions on the form of the function g(n) Hence, it is reasonable and legal to say that, for example, an algorithm is O(2n 2 + 3n + 1) We usually want to simplify the function g(n) Since Big-O notation is formally defined, we can prove certain properties of Big-O that allow us to simplify the g(n) function

Big-O notation is transitive If f(n) is O(g(n)), and g(n) is O(h(n)), then f(n) is O(h(n)) –By definition, there exist c 1 and N 1 such that f(n) = N 1 –By definition, there exist c 2 and N 2 such that g(n) = N 2 –Hence, f(n) <= c 1 g(n) <= c 1 c 2 h(n) for N = max(N 1, N 2 ) –So if we take c = c 1 c 2 and N = max(N 1, N 2 ) then f(n) = N So what? –This allows us to simplify O(O(g(n))) to O(g(n))

More theorems about Big-O 1.If f(n) is O(h(n)) and g(n) is also O(h(n)), then f(n) + g(n) is O(h(n)) 2.The function an k is O(n k ) 3.The function n k is O(n k+j ) for any positive j From the above, we conclude that the Big-O of any polynomial is its highest power Example: 3n 4 + 5n 3 + 8n –By 2: O(3n 4 ) is O(n 4 ), O(5n 3 ) is O(n 3 ), and O(8n) is O(n) –By 3: O(n 4 ) is O(n 4 ), O(n 3 ) is O(n 4 ), and O(n) is O(n 4 ) –By 1: O(n 4 ) + O(n 4 ) + O(n 4 ) is O(n 4 )

“Is” is not “equals” We have been making statements such as, “ O(n) is O(n 4 ) ” It would be wrong to say O(n) = O(n 4 ) We are using “X is Y” to mean “Any X is also a Y” –We might say “Fido is a dog” –This is an asymmetric relation: It is not true that any dog is Fido When we say “ O(n) is O(n 4 ) ”, we mean that anything which is O(n) is also O(n 4 ), but not necessarily the reverse

Still more theorems about Big-O If f(n) = cg(n), then f(n) is O(g(n)) O(log A n) is O(log B n) for any positive numbers A ≠ 1 and B ≠ 1 –Changing the base of a logarithm just changes the result by a constant multiplier – log A X = (log A B)log B X O(log A n) is O(log 2 n) for any positive A ≠ 1 –Hence, we can always use logarithms base 2

Big-Ω Big-O is an upper bound Big-Ω is a lower bound A function f(n) is O(g(n)) if there exist positive numbers c and N such that: f(n) = N A function f(n) is Ω (g(n)) if there exist positive numbers c and N such that: f(n) >= cg(n) for all n >= N If a function is O(n 2 ), it is at least as fast as n 2 If a function is Ω (n 2 ), it is at least as slow as n 2

Big-  A function f(n) is  (g(n)) if there exist positive numbers c 1, c 2 and N such that: c 1 g(n) = N Big-  sets both a lower and an upper bound A function f(n) is  (g(n)) if it is both O(g(n)) and Ω(g(n)) This does not make  (g(n)) unique; f(n) = 2n 2 + 3n + 1 is  (n 2 ) but it is also  (2n 2 ),  (3n 2 ),  (86n 2 ), etc. Why not always use Big-  instead of Big-O? –Not every algorithm has a Big-  value –Quicksort, for example, can be O(n) or O(n 2 )

What you need to know Memorize and understand the definitions of Big- O, Big-Ω, and Big- , particularly Big-O Expect to see these on tests More importantly, you need to be able to examine an algorithm and determine its running time –Usually this is not difficult –Sometimes it can range from quite difficult to nearly impossible –We will come back to this in a later lecture

Simple analysis For series of loops, just add: –for (int i = 0; i < n; i++) {...some O(1) stuff...} for (int j = 0; j < m; j++) {...some more O(1) stuff...} –This is O(n) + O(m) time –If n is proportional to m, this simplifies to O(n) time For nested loops, just multiply: –for (int i = 0; i < n; i++) {...some O(1) stuff... for (int j = 0; j < m; j++) {...some more O(1) stuff } } –This is O(nm) time –If m is proportional to n, this is the same as O(n 2 ) time

More complex analysis Recursion can add some complications –static int depth(BinaryTree tree) { if (tree == null) return 0; int leftDepth = depth(tree.getLeftChild()); int rightDepth = depth(tree.getRightChild()); return Math.max(leftDepth, rightDepth) + 1; } –Formal analysis is a bit tricky, but... –This algorithm visits every node in the binary tree exactly once –Therefore, the algorithm is O(n), where n is the number of nodes in the binary tree

The End