1. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 31 Alternative System Description If all w k are given initially as Then, a system equation can be obtained as  System transition law  system equation

2. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 32 Alternative System Description System equation: better for models with continuous state and control spaces (e.g. S k =  ). Transition law: better for models with discrete state and control spaces (e.g. S k = {0,1,2, …, k}).

3. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 33 Alternative System Description Example: Integer, Finite Capacity Inventory Control Problem x k, u k : non negative integers. ( x k + u k )  2 : Maximum capacity is 2 units. S k = {0, 1, 2} C k = {0, 1, 2} Excess demand max{0, (w k -x k -u k )} is lost. D k ={0, 1, …}

4. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 34 Alternative System Description System (stock) equation to keep  2, put constraints on u k

5. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 35 Alternative System Description Assume Prob{w k =0}=0.1 Prob{w k =1}=0.7 Prob{w k =2}=0.2 Let’s find the (time independent) state transition law

6. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 36 Alternative System Description u k = 0 p 00 = 1, p 01 = p 02 = 0 p 10 = Prob{w k  1} = 0.7 + 0.2 = 0.9 p 11 = Prob{w k = 0} = 0.1; p 12 = 0 p 20 = Prob{w k = 2} = 0.2 p 21 = Prob{w k = 1} = 0.7 p 22 = Prob{w k = 0} = 0.1

7. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 37 Alternative System Description stock = 2 stock = 1 stock = 0 stock = 2 stock = 1 stock = 0 0.1 0.7 0.1 0.2 0.9 1 Transition Probability Diagram ( u k =0 ) k k+1

8. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 38 Alternative System Description stock = 2 stock = 1 stock = 0 stock = 2 stock = 1 stock = 0 0.7 0.2 0.9 Transition Probability Diagram ( u k =1 ) k k+1 0.1 Could put arbitrary numbers, then constraint u k * * : p 11 = Prob{w k = 0} = 0.7

9. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 39 Alternative System Description stock = 2 stock = 1 stock = 0 stock = 2 stock = 1 stock = 0 0.2 Transition Probability Diagram ( u k =2 ) k k+1 0.7 0.1

10. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 40 Alternative System Description Example: Finite Capacity Queue with variable service rate. arrival departure server 0 k-1k k+1 N nn +1.... 21 w k-1 w k+1 = arrivals Service can start/end only at the beginning (end) of the period. Customers finding system full depart and don’t come back. x k : queue length, 0 ≤ x k ≤ n (finite capacity) u k =

11. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 41 w k : # customer that arrive during period k if queue is full, customers leave and don’t return; {w k } i.i.d. p m : probability of m arrivals. q f > q s : probability of service complete in one period q f = 0.8, q s = 0.3 P( x k+1 = j │ x k = i, u k = F ) := p ij Alternative System Description

12. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 42 Alternative System Description u k = F i = 0 p 0j = p j, 0 ≤ j ≤ n - 1 i > 0 j < i -1 p ij = 0 j = i -1 P i i - 1 = Prob {service completed, no arrivals} = q f p 0 at most 1 customer serve in one period

13. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 43 u k = F i – 1 < j < n -1 p ij = Prob {service completed, j – i + 1 arrivals } + Prob {service not completed, j – i arrivals } = q f p j – i + 1 + ( 1 - q f ) p j - i j = n -1 j = n Alternative System Description

14. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 44 The Dynamic Programming Algorithm Example: (Deterministic) Shortest Path 0 1 2 4 3 6 7 8 5 4 1 4 3 6 2 4 7 1 1 5 1 11 15 0 1234 stages Start End

15. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 45 The Dynamic Programming Algorithm (i)Forward, myopic minimization: start at node 0, and proceed forward, minimization the length (cost) for present stage:  0  1 1  3 1  6 6  8 1 1 7 6 15  Not optimal, since  0  2 2  5 5  7 7  8 4 1 4 2 11

16. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 46 The Dynamic Programming Algorithm What we need to do is to minimize costs for current stage and the path these after. (x k, u k )  x k+1 Shortest path from x k+1 to end (ii) Backward minimization: stage 3 6  8 (6) 7  8 (2)

17. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 47 stage 2 6  8  6 (13) The Dynamic Programming Algorithm 4  7  8 (5) 5  7  8 (6) stage 1 6  8  7  8 (10) 2  4  7  8 (6) stage 0 : 2  4  7  8 (10)  J*(0) = 10  * = { 2, 4, 7, 8 }

18. ECES 741: Stochastic Decision & Control Processes – Chapter 1: The DP Algorithm 48 The Dynamic Programming Algorithm Subproblem: Suppose we start now at node 2, what is the shortest path? A/  * = { 4, 7, 8 } :truncated optimal solution for original problem