1. 1 1 © 2003 Thomson  /South-Western Slide Slides Prepared by JOHN S. LOUCKS St. Edward’s University

2. 2 2 © 2003 Thomson  /South-Western Slide Chapter 18 Dynamic Programming n Dynamic Programming Overview n Dynamic Programming Notation n Backwards Recursion n 3 Applications of Dynamic Programming n A Production and Inventory Control Problem

3. 3 3 © 2003 Thomson  /South-Western Slide Dynamic Programming n Dynamic programming (DP) is an approach to problem solving which permits decomposing of the original problem into a series of several smaller subproblems. n To successfully apply DP, the original problem must be viewed as a multistage decision problem. n Defining the stages of a DP problem is sometimes obvious, but at other times this requires subtle reasoning.

4. 4 4 © 2003 Thomson  /South-Western Slide Dynamic Programming n The power of DP is that one need solve only a small portion of all subproblems, due to Bellman's principle of optimality. n Bellman’s principle states that regardless of what decisions were made at previous stages, if the decision to be made at stage n is to be part of an overall optimal solution, then the decision made at stage n must be optimal for all remaining stages.

5. 5 5 © 2003 Thomson  /South-Western Slide Dynamic Programming Notation n At each stage, n, of the dynamic program, there is: a state variable, x n a state variable, x n an optimal decision variable, d n an optimal decision variable, d n n For each value of x n and d n at stage n, there is: a return function value, r n ( x n, d n ) a return function value, r n ( x n, d n ) n The output of the process at stage n is: the state variable for stage n- 1, x n- 1 the state variable for stage n- 1, x n- 1 x n- 1 is calculated by a stage transformation function, t n ( x n, d n ) x n- 1 is calculated by a stage transformation function, t n ( x n, d n ) n The optimal value function, f n ( x n ), is the cumulative return starting at stage n in state x n and proceeding to stage 1 under an optimal strategy.

6. 6 6 © 2003 Thomson  /South-Western Slide n Generally, a dynamic programming problem is solved by starting at the final stage and working backwards to the initial stage, a process called backwards recursion. n The following recursion relation can be used to operationalize the principle of optimality: f n ( x n ) = MAX { r n ( x n, d n ) + f n -1 ( t n ( x n, d n ))} f n ( x n ) = MAX { r n ( x n, d n ) + f n -1 ( t n ( x n, d n ))} d n d n n A problem is solved beginning at stage 0 with the boundary condition f 0 ( x 0 ) = 0, and working backwards to the last stage, N. Backwards Recursion

7. 7 7 © 2003 Thomson  /South-Western Slide Three Applications of Dynamic Programming Problems n Shortest Route Problem In solving a shortest route problem using dynamic programming, one should consider the network as a series of stages with a unique subset of nodes corresponding to each stage. The state variables correspond to the different nodes at each stage.

8. 8 8 © 2003 Thomson  /South-Western Slide Three Applications of Dynamic Programming Problems n Knapsack or Cargo Loading Problem The knapsack problem seeks to determine the optimal number of each of N items (which must not be fractional) to select in order to maximize profit subject to an overall capacity constraint. In solving a knapsack problem using dynamic programming, the stages correspond to the different items being placed into a knapsack. The state variables correspond to the capacity available at the stage.

9. 9 9 © 2003 Thomson  /South-Western Slide Three Applications of Dynamic Programming Problems n Production and Inventory Control Problems In production and inventory control problems, the stages correspond to time periods and the state variables generally will refer to the amounts of inventory on hand at the beginning of each stage.

10. 10 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation n Production and Inventory Control Problem Dicom Corporation wishes to determine a production schedule for its new Model 44/12 robotic welder. Because of differences in parts availability and spare production capacity, the cost of producing the machines will vary from month to month. The holding cost for each unsold welder still in inventory at the end of the month is $500,000. Corporate policy dictates that the maximum number of welders allowed in inventory at the end of any month is 8. Additional data appears on the next slide.

11. 11 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation Cost of Maximum Cost of Maximum Production Production Sales Production Production Sales Per Machine Level Demand Per Machine Level Demand Month (in $100,000's) for Month (in Units) Month (in $100,000's) for Month (in Units) August 32 4 2 August 32 4 2 September 18 3 1 September 18 3 1 October 26 4 5 October 26 4 5 November 45 5 3 November 45 5 3 Determine an optimal 4-month production schedule for the Model 44/12 welder.

12. 12 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation n 4-Stage Dynamic Programming Problem Working backwards, let stage 1 correspond to November, stage 2 to October, etc. The following data (costs in $100,000's) can be inferred: Product. Holding Product. Holding Product. Storage Cost Cost Product. Storage Cost Cost Month Dem. Capacity Capacity Per Unit Per unit Month Dem. Capacity Capacity Per Unit Per unit ( n ) D n P n W n C n H n ( n ) D n P n W n C n H n 1 3 5 8 45 5 1 3 5 8 45 5 2 5 4 8 26 5 2 5 4 8 26 5 3 1 3 8 18 5 3 1 3 8 18 5 4 2 4 8 32 5 4 2 4 8 32 5

13. 13 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation n State Variable Defined x n = number of computers in inventory at the beginning of month n. x n = number of computers in inventory at the beginning of month n. x 4 = 0 (Since the computer is new, there will be no inventory at the start of August.) x 4 = 0 (Since the computer is new, there will be no inventory at the start of August.) n Decision Variable Defined d n = production quantity for month n. d n = production quantity for month n.

14. 14 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation n Stage Transformation Function Then the stage transformations for months 0 through 4 can be defined by: Then the stage transformations for months 0 through 4 can be defined by: (Previous month's inventory) (Previous month's inventory) + (Production this month) + (Production this month) - (Demand this month) - (Demand this month) x n- 1 = x n + d n - D n x n- 1 = x n + d n - D n

15. 15 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation n Return Function r n ( x n, d n ) = (Production cost) + (Holding cost) for month n Production cost = (Production cost per unit) x (Number of units produced in month n ) Holding cost = (Holding cost per unit) x (Ending inventory for month n ) x (Ending inventory for month n ) Hence, r n ( x n, d n ) = C n d n + H n ( x n + d n - D n )

16. 16 © 2003 Thomson  /South-Western Slide n Return Function (continued) r n ( x n, d n ) = C n d n + H n ( x n + d n - D n ) r n ( x n, d n ) = C n d n + H n ( x n + d n - D n ) This gives: r 1 ( x 1, d 1 ) = 50 d 1 + 5 x 1 - 15 This gives: r 1 ( x 1, d 1 ) = 50 d 1 + 5 x 1 - 15 r 2 ( x 2, d 2 ) = 31 d 2 + 5 x 2 - 25 r 2 ( x 2, d 2 ) = 31 d 2 + 5 x 2 - 25 r 3 ( x 3, d 3 ) = 23 d 3 + 5 x 3 - 5 r 3 ( x 3, d 3 ) = 23 d 3 + 5 x 3 - 5 r 4 ( x 4, d 4 ) = 37 d 4 + 5 x 4 – 10 r 4 ( x 4, d 4 ) = 37 d 4 + 5 x 4 – 10 Example: Dicom Corporation

17. 17 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation n Restrictions on x n and d n Restriction (1) Restriction (1) Because backordering is not allowed, we must be able to meet the sales demand in any month. x n + d n > D n x n + d n > D n

18. 18 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation n Restrictions on x n and d n (continued) Restriction (2) Restriction (2) Because there is a maximum storage of W n at each stage n, the total inventory at the end of any month cannot exceed W n. x n + d n - D n < W n x n + d n - D n < W n or or x n + d n < W n + D n x n + d n < W n + D n

19. 19 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation n Restrictions on x n and d n (continued) Restriction (3) Restriction (3) The amount produced in any given month cannot exceed the production capacity for that month. d n < P n d n < P n

20. 20 © 2003 Thomson  /South-Western Slide n Optimal Value Function f n ( x n ) = optimal return (minimal cost) for stages 1 through n given one starts stage n with x n welders in inventory f n ( x n ) = Min { r n ( x n, d n ) + f n -1 ( x n -1 )} f n ( x n ) = Min { r n ( x n, d n ) + f n -1 ( x n -1 )} d n d n where d n is constrained by restrictions (1), (2), and (3). where d n is constrained by restrictions (1), (2), and (3). Example: Dicom Corporation

21. 21 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation n Stage 1 (November) Because f 0 ( x 0 ) = 0 is a boundary condition: Because f 0 ( x 0 ) = 0 is a boundary condition: f 1 ( x 1 ) = Min r 1 ( x 1, d 1 ) f 1 ( x 1 ) = Min r 1 ( x 1, d 1 ) d 1 d 1 Using the expression for r 1 ( x 1, d 1 ) and restrictions (1), (2), (3): Using the expression for r 1 ( x 1, d 1 ) and restrictions (1), (2), (3): f 1 ( x 1 ) = Min 5 x 1 + 50 d 1 - 15 f 1 ( x 1 ) = Min 5 x 1 + 50 d 1 - 15 s.t. x 1 + d 1 > 3 (1) s.t. x 1 + d 1 > 3 (1) x 1 + d 1 < 11 (2) x 1 + d 1 < 11 (2) d 1 < 5 (3) d 1 < 5 (3) and, d 1 > 0 and, d 1 > 0

22. 22 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation n Tabulated Values for 50 d 1 +5 x 1 –15 d 1 d 1 x 1 0 1 2 3 4 5 d 1 * f 1 ( x 1 ) x 1 0 1 2 3 4 5 d 1 * f 1 ( x 1 ) 0 135 185 235 3 135 0 135 185 235 3 135 1 90 140 190 240 2 90 1 90 140 190 240 2 90 2 45 95 145 195 245 1 45 2 45 95 145 195 245 1 45 3 0 50 100 150 200 250 0 0 3 0 50 100 150 200 250 0 0 Note: Having x 1 > 3 would result in having welders in inventory at the end of November which is undesirable. Note: Having x 1 > 3 would result in having welders in inventory at the end of November which is undesirable.

23. 23 © 2003 Thomson  /South-Western Slide n Stage 2 (October) f 2 ( x 2 ) = Min 5 x 2 + 31 d 2 -25 + f 1 ( x 1 ) f 2 ( x 2 ) = Min 5 x 2 + 31 d 2 -25 + f 1 ( x 1 ) d 2 d 2 Given the restrictions, the subproblem is: Given the restrictions, the subproblem is: f 2 ( x 2 ) = Min 5 x 2 + 31 d 2 -25 + f 1 ( x 2 + d 2 - 5) f 2 ( x 2 ) = Min 5 x 2 + 31 d 2 -25 + f 1 ( x 2 + d 2 - 5) s.t. x 2 + d 2 > 5 (1) s.t. x 2 + d 2 > 5 (1) x 2 + d 2 < 13 (2) x 2 + d 2 < 13 (2) d 2 < 4 (3) d 2 < 4 (3) and d 2 > 0 and d 2 > 0 Example: Dicom Corporation

24. 24 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation n Tabulated Values for 5 x 2 + 31 d 2 -25 + f 1 ( x 1 ) d 2 d 2 x 2 0 1 2 3 4 d 2 * f 2 ( x 2 ) x 2 + d 2 *-5 = x 1 x 2 0 1 2 3 4 d 2 * f 2 ( x 2 ) x 2 + d 2 *-5 = x 1 1 239 4 239 0 1 239 4 239 0 2 213 199 4 199 1 2 213 199 4 199 1 3 187 173 159 4 159 2 3 187 173 159 4 159 2 4 161 147 133 119 4 119 3 4 161 147 133 119 4 119 3 5 135 121 107 93 3 93 3 5 135 121 107 93 3 93 3 6 95 81 67 2 67 3 6 95 81 67 2 67 3 7 55 41 1 41 3 7 55 41 1 41 3 8 15 0 15 3 8 15 0 15 3 Note: x 2 = 0 is infeasible.

25. 25 © 2003 Thomson  /South-Western Slide n Stage 3 (September) f 3 ( x 3 ) = Min 5 x 3 + 23 d 3 - 5 + f 2 ( x 2 ) d 3 f 3 ( x 3 ) = Min 5 x 3 + 23 d 3 - 5 + f 2 ( x 2 ) d 3 Given the restrictions, the subproblem is: Given the restrictions, the subproblem is: f 3 ( x 3 ) = Min 5 x 3 + 23 d 3 - 5 + f 2 ( x 3 + d 3 - 1) f 3 ( x 3 ) = Min 5 x 3 + 23 d 3 - 5 + f 2 ( x 3 + d 3 - 1) s.t. x 3 + d 3 > 1 (1) s.t. x 3 + d 3 > 1 (1) x 3 + d 3 < 9 (2) x 3 + d 3 < 9 (2) d 3 < 3 (3) d 3 < 3 (3) and d 3 > 0 and d 3 > 0 Example: Dicom Corporation

26. 26 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation n Tabulated Values for 5 x 3 + 23 d 3 - 5 + f 2 ( x 3 + d 3 -1) x 3 0 1 2 3 d 3 * f 3 ( x 3 ) x 3 + d 3 *-1 = x 2 x 3 0 1 2 3 d 3 * f 3 ( x 3 ) x 3 + d 3 *-1 = x 2 0 280 263 3 263 2 0 280 263 3 263 2 1 263 245 228 3 228 3 1 263 245 228 3 228 3 2 244 227 210 193 3 193 4 2 244 227 210 193 3 193 4 3 209 192 175 172 3 172 5 3 209 192 175 172 3 172 5 4 174 157 154 151 3 151 6 4 174 157 154 151 3 151 6 5 139 136 133 130 3 130 7 5 139 136 133 130 3 130 7 6 118 115 112 109 3 109 8 6 118 115 112 109 3 109 8 7 97 94 91 1 91 8 7 97 94 91 1 91 8 8 76 73 1 73 8 8 76 73 1 73 8 d3d3d3d3

27. 27 © 2003 Thomson  /South-Western Slide n Stage 4 (August) f 4 ( x 4 ) = Min 5 x 4 + 37 d 4 - 10 + f 3 ( x 3 ) f 4 ( x 4 ) = Min 5 x 4 + 37 d 4 - 10 + f 3 ( x 3 ) d 4 d 4 Given the restrictions, the subproblem is: Given the restrictions, the subproblem is: f 4 ( x 4 ) = Min 5 x 4 + 37 d 4 - 10 + f 3 ( x 4 + d 4 - 2) f 4 ( x 4 ) = Min 5 x 4 + 37 d 4 - 10 + f 3 ( x 4 + d 4 - 2) s.t. x 4 + d 4 > 2 (1) s.t. x 4 + d 4 > 2 (1) x 4 + d 4 < 10 (2) x 4 + d 4 < 10 (2) d 4 < 4 (3) d 4 < 4 (3) and d 4 > 0 and d 4 > 0 Example: Dicom Corporation

28. 28 © 2003 Thomson  /South-Western Slide n Tabulated Values for 5 x 4 + 37 d 4 - 10 + f 3 ( x 4 + d 4 - 2) d 4 d 4 x 4 0 1 2 3 4 d 4 * f 4 ( x 4 ) x 4 + d 4 *-2 = x 3 x 4 0 1 2 3 4 d 4 * f 4 ( x 4 ) x 4 + d 4 *-2 = x 3 0 327 329 331 2 327 0 0 327 329 331 2 327 0 Note: Because August starts with x 4 = 0 inventory on hand, compute table only for x 4 = 0. Example: Dicom Corporation

29. 29 © 2003 Thomson  /South-Western Slide Example: Dicom Corporation n Solution Summary Working backwards through the tables, one can determine the optimal solution that gives the minimum cost of $327 x 100,000 = $32,700,000. Inventory On- Inventory On- Hand Beginning Hand Beginning Stage Month Produce ( d n *) Next Month x n -1 Stage Month Produce ( d n *) Next Month x n -1 4 August 2 0 4 August 2 0 3 September 3 2 3 September 3 2 2 October 4 1 2 October 4 1 1 November 2 0 1 November 2 0

30. 30 © 2003 Thomson  /South-Western Slide End of Chapter 18