2. Problem
A subsequence is a sequence derived from another sequence by deleting some elements without changing the order of the remaining elements.
Using code or pseudocode, write an algorithm that determines the longest common subsequence of two strings.

3. Introduction to Algorithms
Dynamic programming
An introduction
Simon Ellis
27th March, 2014
Introduction to Algorithms

4. Fibonacci numbers
Iterative solution

5. Fibonacci numbers (an aside)
Closed form solution exists
Binet’s formula
To calculate any Fibonacci number Fn

6. Pascal’s Triangle Iterative solution Begin with 1 on row 0
Each new row is offset to the left by the width of one number
Construct the elements of rows as follows:
For each new value k, sum the value above and left with the value above and right to find k
If there is no number, substitute zero in its place

7. Pascal’s Triangle

8. Pascal’s Triangle (an aside)
Closed form solution exists
To calculate values in row n
Use symmetry to derive right-hand side of row (or calculate)

9. Pascal’s Triangle (another aside)1 1 2 3 5 8 13 21 34

10. Calculating prime numbers
Iterative solution
Let P be the set of prime numbers; at start, P = ∅
For each value k = 2…n
For each value j = 2…k – 1
If k/j is not an integer for all j, P = P ∪ { k }
Output P
Naïve, slow solution

11. Dynamic programming May refer to two things
A mathematical optimisation method
An approach to computer programming and algorithm design
A method for solving complex problems using recursion
Problems may be solved by solving smaller parts
Keep addressing smaller parts until each subproblem becomes tractable
Problem must possess both of the following traits
Optimal substructure
Overlapping subproblems

12. Optimal substructure
A problem has optimal substructure if an optimal solution can be constructed efficiently from optimal solutions of its subproblems
May be solved using a greedy algorithm
Iff it can be proved by induction that this is optimal for all steps
Dynamic programming may be used if subproblems exist
Otherwise a brute-force search is required

13. Optimal substructure Optimal substructure Not optimal substructure
Finding the shortest path between two cities by car
e.g. if the shortest route from Seattle to Los Angeles passes through Portland and Sacramento, the shortest route from Portland to Los Angeles passes through Sacramento
Not optimal substructure
Buying the cheapest ticket from Sydney to New York
Even if a ticket has stops in Dubai and London we cannot conclude that the cheapest ticket will stop in London as the price at which an airline sells multi-flight tickets is not the sum of the prices for which it would sell individual flights on the same trip

14. Overlapping subproblems
A problem has overlapping subproblems if either of the following is true
The problem can be broken down into subproblems which are reused several times
A recursive algorithm solves the same subproblem over and over instead of generating new subproblems
Examples
Fibonacci numbers
Pascal’s Triangle …

15. Examples of dynamic programming
Fibonacci numbers
Pascal’s Triangle
Prime number generation
Dijkstra’s algorithm/A* algorithm
Towers of Hanoi game
Matrix chain multiplication
Beat tracking in music information retrieval software
Neural networks
Word-wrapping in word processors (especially LATEX)
Transposition and refutation tables in computer chess games

16. Recursion
Recursion is a powerful tool for solving dynamic programming problems
Not always an optimal solution on its own
Naïve
Wasteful
e.g. to calculate F100, a lot of work shall be wasted
Two standard approaches to improving performance
“Top-down”
“Bottom-up”

17. “Top-down” approach In software design In dynamic programming
Developing a program by starting with a large concept and adding increasing layers of specialisation
In dynamic programming
A solution which derives directly from the recursive solution
Performance is improved by memoisation
Solutions to subproblems are stored in a data structure
If a subproblem is solved, we read from the table, else we calculate and add it

18. “Bottom-up” approach In software design In dynamic programming
Developing a program by starting with many small objects and functions and building on them to create more functionality
In dynamic programming
Solving the subproblems first and aggregating them
Performance is improved using stored data
Memoisation may be used

19. Memoisation Optimisation technique
Reduces need to repeat function calls, which may be expensive
“Memoised” function stores results of previous calls with a specific set of inputs
Function must be ‘referentially transparent’
i.e. calling the function must have exactly the same outcome as returning the value that would be produced by calling the function

20. Memoisation Not the same as using a lookup table
Lookup tables are precalculated before use
Memoised tables are calculated transparently as required
Memoisation optimises for time over space
Time/space trade-off in all algorithms
‘Computational complexity’
Complexity in time (how much time is required)
Complexity in space (how much memory is required)
Cannot reduce one without increasing the other

21. Computational complexity
Roughly, the quantity of resources required to solve a problem computationally
Resources are whatever is appropriate to the situation
Time
Memory
Logic gates on a circuit board
Number of processors in a supercomputer
e.g. “Big-O” notation is a measure of complexity in time

22. Computational complexity
Cannot reduce all complexity values simultaneously
All software design requires a trade-off
Primarily between time (runtime) and space (memory requirements)
Memoisation optimises for time over space
Calculations are performed more quickly…
… but memory is required to store precalculated data

23. Longest common subsequence
Assume two strings, S and T
Solution?
Generate all possible subsequences of both sequences, the set Q
Return longest subsequence in Q
int lcs(char *S, char *T, int m, int n) {
   if (m == 0 || n == 0)
     return 0;
   if (S[m-1] == T[n-1])
     return 1 + lcs(S, T, m-1, n-1);
   else
     return max(lcs(S, T, m, n-1), lcs(S, T, m-1, n)); }

24. Longest common subsequence
Time: O(2n)
Why?
Is this an efficient algorithm for this problem?
int lcs(char *S, char *T, int m, int n) {
   if (m == 0 || n == 0)
     return 0;
   if (S[m-1] == T[n-1])
     return 1 + lcs(S, T, m-1, n-1);
   else
     return max(lcs(S, T, m, n-1), lcs(S, T, m-1, n)); }

25. Longest common subsequence
Can we use dynamic programming to improve performance?
Does the problem meet the requirements?
Does it have optimal substructure?
Does it have overlapping subproblems?

26. Optimal substructure of LCS
Assume
Input sequences S[0..m-1] and T [0..n-1] with lengths m and n
L(S[0..m-1], T [0..n-1]) is the length of the LCS of S and T
L(S[0..m-1], T [0..n-1]) is defined recursively as follows
If last characters of both sequences match
L(S [0..m–1], T [0..n–1])=1 + L(S [0..m–2], T [0..n–2])
If last characters of both sequences do not match
L(S [0..m–1], T [0..n–1]) = MAX( L(S [0..m–2], T [0..n–1]), L(S [0..m–1], T [0..n–2]) )

27. Optimal substructure of LCS
Examples
Last characters match
Consider the input strings “AGGTAB” and “GXTXAYB”
Length of LCS can be written as
L(“AGGTAB”, “GXTXAYB”) = 1 + L(“AGGTA”, “GXTXAY”)
Last characters do not match
Consider the input strings “ABCDGH” and “AEDFHR”
Last characters do not match for the strings.
L(“ABCDGH”, “AEDFHR”) = MAX ( L(“ABCDG”, “AEDFHR”), L(“ABCDGH”, “AEDFH”) )

28. Overlapping subproblems in LCS
Consider the original code
Many of the same problems being solved repeatedly
Subproblems overlap
Performance can be improved by memoisation
lcs("AXYT", "AYZX”)
/ \
lcs("AXY", "AYZX") lcs("AXYT", "AYZ")
/ \ / \
lcs("AX", "AYZX") lcs("AXY", "AYZ") lcs("AXY", "AYZ") lcs("AXYT", "AY")

29. Example of LCS of two strings∅ A G C T

30. Example of LCS of two strings∅ A G C T   
(G)

31. Example of LCS of two strings∅ A G C T   
(G)
(A)
(GA)

32. Example of LCS of two strings∅ A G C T   
(G)
(A)
(GA)
(AC)
(GC)

33. Example of LCS of two strings
New runtime O(n ⋅ m)
Much less than previously: O(2n)
But increased space requirement to store working data

34. ANY QUESTIONS?

35. References
ence