1. Introduction and Preliminaries D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 Dynamic Programming and Applications

2. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 2 Objectives To discuss the Sequential Optimization an Multistage Decision Process To represent a Multistage Decision Process To describe the concept of sub-optimization and principle of optimality To formulate recursive equations for a multistage decision process In a backward manner In a forward manner To explain the difference between discrete and continuous dynamic programming To discuss about multiple state variables To discuss the curse of dimensionality in dynamic programming

3. Introduction D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 3  Complex problems are sometimes solved quickly if approached in a sequential manner  Dynamic Programming : Sequential or multistage decision making process  Basic approach behind dynamic programming: Solution is found out in multi stages  Works in a “divide and conquer” manner  Recursive equations are used to solve a problem in sequence  These equations are fundamental to the dynamic programming

4. Sequential Optimization D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 4  Problem is divided into smaller sub-problems  Optimize these sub-problems without losing the integrity of the original problem  Decisions are made sequentially or at a number of stages - Multistage decision problems  A N variable problem is represented by N single variable problems  These problems are solved successively to get the optimal value of the original problem

5. Sequential Optimization… D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 5  Serial multistage decision process: Single variable problems are connected in series so that the output of one stage will be the input to the succeeding stage  For example, consider a water allocation problem to N users  Objective function is to maximize the total net benefit from all users  Problem can be solved by considering each user separately and optimizing the individual net benefits, subject to constraints and then adding up the benefits from all users to get the total optimal benefit

6. Representation of Multistage Decision Process D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 6  Consider a single stage decision process  Here  S 1 is the input state variable  S 2 is the output state variable  X 1 is the decision variables and  NB 1 is the net benefit  The transformation function for the input and output is S 2 = g(X 1, S 1 )  Net benefits are expressed as a function of decision variables and input variable NB 1 = h(X 1, S 1 ) Stage 1 Input S 1 Net Benefit, NB 1 Decision variable, X 1 Output S 2

7. Representation of Multistage Decision Process… D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 7  Now, consider a serial multistage decision process consisting of T stages  For the t th stage the stage transformation and the benefit functions are S t+1 = g(X t, S t ) NB t = h(X t, S t ) Stage 1 Stage tStage T StSt NB t XtXt S t+1 S1S1 S2S2 NB 1 NB T X1X1 XTXT STST S T+1

8. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 8  Objective of this multistage problem is to find the optimum values of all decision variables X 1, X 2,…, X T such that the individual net benefits of each stage that is expressed by some objective function, f(NB t ) and the total net benefit which is expressed by f(NB 1, NB 2,…, NB T ) should be maximized  Dynamic programming can be applied to this multistage problem if the objective function is separable and monotonic  An objective function is separable, if it can be decomposed and expressed as a sum or product of individual net benefits of each stage or Representation of Multistage Decision Process…

9. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 9  An objective function is monotonic if for all values of a and b for which the value of the benefit function is, then, is satisfied Representation of Multistage Decision Process…

10. Types of Multistage Decision Process D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 10  A serial multistage problem can be divided into three categories  Initial value problem  Value of the initial state variable, S 1 is given  Final value problem  Value of the final state variable, S T is given  Final value problem can be transformed into an initial value problem by reversing the direction of the state variable, S t  Boundary value problem.  Values of both the initial and final state variables, S 1 and S T are given

11. Concept of Sub-Optimization D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 11 Consider the objective function consisting of T decision variables x 1, x 2, …, x T satisfying the equations, S t+1 = g(X t, S t ) NB t = h(X t, S t ) for t = 1,2,…,T To solve this problem through dynamic programming, the concepts of sub- optimization and principle of optimality are used

12. Concept of Sub-Optimization … D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 12  Consider the design of a water tank in which the cost of construction is to be minimized  Capacity of the tank to be designed is K  Main components of a water tank include (i) tank (ii) columns to support the tank and (iii) the foundation  Optimization can be done by breaking the system into individual parts and optimizing each part separately  While breaking and sub-optimizing, a logical procedure should be used; otherwise this approach can lead to a poor solution

13. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 13  For example, sub-optimization of columns without considering the other two components may cause the use of heavy concrete columns with less reinforcement, since the cost of steel is high  But while considering the sub-optimization of foundation component, the cost becomes higher as the foundation should be strong enough to carry these heavy columns  Thus, the sub-optimization of columns before considering the sub- optimization of foundation will adversely affect the overall design Concept of Sub-Optimization …

14. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 14  In most of the serial systems sub-optimization can start from the last component (or first component) since it does not influence the other components  Thus, for the above problem, foundation can be suboptimized independently  Then, the last two components (columns and foundation) are considered as a single component and sub-optimization is done without affecting other components  This process can be repeated for any number of end components Concept of Sub-Optimization …

15. Water Resources Planning and Management: M4L1 15 Process of sub-optimization TankColumnsFoundation Original System TankColumnsFoundation Suboptimize design of Foundation component TankColumnsFoundation Optimize complete system Suboptimize design of Foundation & Columns together TankColumnsFoundation

16. Principle of Optimality D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 16  Bellman (1957) stated the principle of optimality which explains the process of multi stage optimization as:  “An optimal policy (or a set of decisions) has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision.”

17. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 17 Recursive Equations  Recursive equations are used to structure a multistage decision problem as a sequential process  Each recursive equation represents a stage at which a decision is required  A series of equations are successively solved, each equation depending on the output values of the previous equations  A multistage problem is solved by breaking into a number of single stage problems through recursion  Approached can be done in a backward manner or in a forward manner

18. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 18 Backward Recursion  A problem is solved by writing equations first for the final stage and then proceeding backwards to the first stage  Consider a serial multistage problem  Let the objective function for this problem is Stage 1 Stage tStage T StSt NB t X t S1S1 S2S2 NB 1 NB T X1X1 XTXT STST S T+1 S t+1

19. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 19 Backward Recursion…  The relation between the stage variables and decision variables are S t+1 = g(X t, S t ), t = 1,2,…, T.  Consider the final stage as the first sub-problem. The input variable to this stage is S T.  Principle of optimality: X T should be selected such that is optimum for the input S T  The objective function for this stage is  Next, group the last two stages together as the second sub-problem. The objective function is

20. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 20  By using the stage transformation equation, can be rewritten as  Thus, a multivariate problem is divided into two single variable problems as shown  In general, the i+1 th sub-problem can be expressed as  Converting this to a single variable problem Backward Recursion…

21. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 21  denotes the optimal value of the objective function for the last i stages  Principle of optimality for backward recursion:  No matter in what state of stage one may be, in order for a policy to be optimal, one must proceed from that state and stage in an optimal manner sing the stage transformation equation Backward Recursion…

22. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 22 Forward Recursion  The problem is solved by starting from the stage 1 and proceeding towards the last stage  Consider a serial multistage problem  Let the objective function for this problem is Stage 1 Stage tStage T StSt NB t X t S1S1 S2S2 NB 1 NB T X1X1 XTXT STST S T+1 S t+1

23. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 23  The relation between the stage variables and decision variables are where S t is the input available to the stages 1 to t  Consider the stage 1 as the first sub-problem. The input variable to this stage is S 1  Principle of optimality: X 1 should be selected such that is optimum for the input S 1  The objective function for this stage is Forward Recursion…

24. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 24  Group the first and second stages together as the second sub-problem. The objective function is  By using the stage transformation equation, can be rewritten as  In general, the i th sub-problem can be expressed as Forward Recursion…

25. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 25  Converting this to a single variable problem  denotes the optimal value of the objective function for the last i stages  Principle of optimality for forward recursion:  No matter in what state of stage one may be, in order for a policy to be optimal, one had to get to that state and stage in an optimal manner  Refer lecture notes for computational procedure for backward regression. Forward Recursion…

26. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 26 Discrete versus Continuous Dynamic Programming Discrete dynamic programming problems: Number of stages is finite When the number of stages tends to infinity then it is called a continuous dynamic programming problem - Infinite-stage problem Continuous dynamic problems are used to solve continuous decision problem Classical method of solving continuous decision problems is by the calculus of variations

27. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 27 Discrete versus Continuous Dynamic Programming… Analytical solutions, using calculus of variations is applicable only for very simple problems Infinite-stage dynamic programming approach provides a very efficient numerical approximation procedure for solving continuous decision problems For discrete dynamic programming model, the objective function value is the sum of individual stage outputs For a continuous dynamic programming model, summation is replaced by integrals of the returns from individual stages Such models are useful when infinite number of decisions have to be made in finite time interval

28. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 28 Multiple State Problems Problems in which there are more than one state variable For example, consider a water allocation problem to n irrigated crops Let S i be the units of water available to the remaining n-i crops Considering only the allocation of water, the problem can be solved as a single state problem, with S i as the state variable Now, assume that L units of land are available for all these n crops Allocation of land also to be done to each crop after considering the units of water required for each unit of irrigated land containing each crop

29. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 29 Multiple State Problems… Let R i be the amount of land available for n-i crops An additional state variable R i is included while sub-optimizing different stages Thus, in this problem two allocations need to be made: water and land. A single stage problem consisting of two state variables, S 1 & R 1 is shown below Stage 1 Input S 1 & R 1 Net Benefit, NB 1 Decision variable, X 1 Output S 2 & R 2

30. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 30 In general, for a multistage decision problem of T stages, containing two state variables S t and R t, the objective function can be written as where the transformation equations are given as S t+1 = g(X t, S t ) for t =1,2,…, T & R t+1 = g’(X t, R t ) for t =1,2,…, T Multiple State Problems…

31. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 31 Curse of Dimensionality Limitation of dynamic programming: Dimensionality restriction Number of calculations needed increases rapidly as the number of variables and stages increase Increases the computational effort Increase in the number of stage variables causes an increase in the number of combinations of discrete states to be examined at each stage For a problem consisting of 100 state variables and each variable having 100 discrete values, the sub-optimization table will contain 100 100 entries

32. D Nagesh Kumar, IISc Water Resources Planning and Management: M4L1 32 Curse of Dimensionality… The computation of one table may take 100 96 seconds (about 100 92 years) even on a high speed computer Like this 100 tables have to be prepared for which computation is almost impossible This phenomenon as termed by Bellman, is known as “curse of dimensionality” or “Problem of dimensionality” of multiple state variable dynamic programming problems

33. D. Nagesh Kumar, IISc Water Resources Systems Planning and Management: M4L1 Thank You