1. © 2004 Warren B. Powell Slide 1 Outline A car distribution problem

2. © 2004 Warren B. Powell Slide 2 Norfolk Southern

3. When a boxcar becomes empty, we have three options: Customers Regional depots General depots

4. Option 1: Send directly to customers

5. Option 2: Send to distribution areas

6. Option 1: Send directly to customers Option 2: Send to distribution areas Option 3: Send to general depots

7. © 2004 Warren B. Powell Slide 7 Forecasts of Car Demands ForecastActual

8. © 2004 Warren B. Powell Slide 8 Outline ADP for resource allocation problems with low dimensional attribute vectors: »Two-stage problems »Multistage problems

9. © 2004 Warren B. Powell Slide 9 Outline ADP for resource allocation problems with low dimensional attribute vectors: »Two-stage problems »Multistage problems

10. © 2004 Warren B. Powell Slide 10 Two-stage problems This period Future

11. © 2004 Warren B. Powell Slide 11 Two-stage problems Our basic strategy: Separable approximation 0 1 2 3 4 5

12. © 2004 Warren B. Powell Slide 12 Two-stage problems Two-stage resource allocation under uncertainty

13. © 2004 Warren B. Powell Slide 13 Two-stage problems

14. © 2004 Warren B. Powell Slide 14 Two-stage problems

15. © 2004 Warren B. Powell Slide 15 Two-stage problems

16. © 2004 Warren B. Powell Slide 16 Two-stage problems We estimate the functions by sampling from our distributions. Marginal value:

17. © 2004 Warren B. Powell Slide 17 Two-stage problems The time t subproblem: t (i-1,t+3) (i,t+1) (i+1,t+5)

18. © 2004 Warren B. Powell Slide 18 Two-stage problems Left and right gradients are found by solving flow augmenting path problems. t i (i-1,t+3) Gradients: The right derivative (the value of one more unit of that resource) is a flow augmenting path from that node to the supersink.

19. © 2004 Warren B. Powell Slide 19 Two-stage problems Left and right derivatives are used to build up a nonlinear approximation of the subproblem.

20. © 2004 Warren B. Powell Slide 20 Two-stage problems Left and right derivatives are used to build up a nonlinear approximation of the subproblem. Right derivativeLeft derivative

21. © 2004 Warren B. Powell Slide 21 Two-stage problems Each iteration adds new segments, as well as refining old ones.

22. © 2004 Warren B. Powell Slide 22 Two-stage problems Number of resources Approximate value function

23. © 2004 Warren B. Powell Slide 23 Two-stage problems It is important to maintain concavity:

24. © 2004 Warren B. Powell Slide 24 Value function A concave function… Slopes … has monotonically decreasing slopes. But updating the function with a stochastic gradient may violate this property. Two-stage problems

25. © 2004 Warren B. Powell Slide 25 A leveling algorithm

26. © 2004 Warren B. Powell Slide 26 A leveling algorithm

27. © 2004 Warren B. Powell Slide 27 A leveling algorithm Violates concavity (monotonicity of slopes)

28. © 2004 Warren B. Powell Slide 28 A leveling algorithm

29. © 2004 Warren B. Powell Slide 29 A projection algorithm (SPAR)

30. © 2004 Warren B. Powell Slide 30 A projection algorithm (SPAR)

31. © 2004 Warren B. Powell Slide 31 A projection algorithm (SPAR)

32. © 2004 Warren B. Powell Slide 32 A projection algorithm (SPAR)

33. © 2004 Warren B. Powell Slide 33 Two-stage problems A uniform sampling strategy: »Second stage sampled based on exogenous distribution.

34. © 2004 Warren B. Powell Slide 34 Two-stage problems Theorem (Topaloglu and Powell, 2003): Theorem (Powell, Ruszczynski and Topaloglu, 200?):

35. © 2004 Warren B. Powell Slide 35 Two-stage problems Projection algorithm Leveling algorithm

36. © 2004 Warren B. Powell Slide 36 Two-stage problems An optimizing sampling strategy:

37. © 2004 Warren B. Powell Slide 37 Two-stage problems Theorem (Powell, Ruszczynski and Topaloglu, 200?):

38. © 2004 Warren B. Powell Slide 38 Two-stage problems Separability implies:

39. © 2004 Warren B. Powell Slide 39 Two-stage problems We only sample the points in the second stage which were optimal given the second stage approximation. A stochastic allocation problem:

40. © 2004 Warren B. Powell Slide 40 Two-stage problems

41. © 2004 Warren B. Powell Slide 41 Two-stage problems Real problems are nonseparable.

42. © 2004 Warren B. Powell Slide 42 Two-stage problems Notes: »The use of separable approximations for two-stage problems can produce algorithms that produce solutions that converge almost surely to the optimal solution if the second stage problem is continuously differentiable (Cheung and Powell, Operations Research, 2000). »The use of separable approximations for two-stage problems will not generally produce optimal solutions when the second stage is nondifferentiable and nonseparable.

43. © 2004 Warren B. Powell Slide 43 Two-stage problems We are solving problems with the structure:

44. © 2004 Warren B. Powell Slide 44 Two-stage problems Duals reflect flow augmenting cycles through the network:

45. © 2004 Warren B. Powell Slide 45 Two-stage problems Duals reflect flow augmenting cycles through the network: 1 5 Captures impact on second stage

46. © 2004 Warren B. Powell Slide 46 Two-stage problems Duals reflect flow augmenting cycles through the network: 1 5

47. © 2004 Warren B. Powell Slide 47 Two-stage problems We solve a sample realization of the second stage: Capacitated arcs Uncapacitated arcs 1 5

48. © 2004 Warren B. Powell Slide 48 Two-stage problems (Sample) flow augmenting path from 5 to 1: 1 5

49. © 2004 Warren B. Powell Slide 49 Two-stage problems (Sample) flow augmenting path from 5 to 1: 1 5 Together they cancel, but one at a time may be infeasible.

50. © 2004 Warren B. Powell Slide 50 Two-stage problems (Sample) flow augmenting path from 5 to 1: 1 5

51. © 2004 Warren B. Powell Slide 51 Two-stage problems Questions: »How good is our approximation? »What is the speed of convergence? »How do we compare against deterministic approximations?

52. © 2004 Warren B. Powell Slide 52 Two-stage problems Nested Benders decomposition:

53. © 2004 Warren B. Powell Slide 53 Two-stage problems Point forecast Profits Iterations

54. © 2004 Warren B. Powell Slide 54 Two-stage problems Variations on Bender’s decomposition Point forecast Profits Iterations

55. © 2004 Warren B. Powell Slide 55 Two-stage problems Variations on Bender’s decomposition SPAR algorithm Point forecast Profits Iterations

56. © 2004 Warren B. Powell Slide 56 Two-stage problems Variations on Bender’s decomposition SPAR algorithm Deterministic approximation Iterations

57. © 2004 Warren B. Powell Slide 57 Two-stage problems Variations on Bender’s decomposition SPAR algorithm Deterministic approximation Iterations

58. © 2004 Warren B. Powell Slide 58 Two-stage problems

59. © 2004 Warren B. Powell Slide 59 Two-stage problems

60. © 2004 Warren B. Powell Slide 60 Two-stage problems

61. © 2004 Warren B. Powell Slide 61 Outline ADP for resource allocation problems with: »Low dimensional attribute vectors Two stage problems Multistage problems

62. © 2004 Warren B. Powell Slide 62 A pure (deterministic) network: Multistage problems

63. © 2004 Warren B. Powell Slide 63 A dynamic network: Multistage problems t

64. © 2004 Warren B. Powell Slide 64 Multistage problems Stepping through time:

65. © 2004 Warren B. Powell Slide 65 Multistage problems Iterative learning:

66. © 2004 Warren B. Powell Slide 66 Multistage problems Estimating values: »Single-pass algorithm: »Two-pass algorithm: Previous iterations This iteration

67. © 2004 Warren B. Powell Slide 67 Resource State-Type Time Multistage problems

68. © 2004 Warren B. Powell Slide 68 Time Resource State-Type Multistage problems

69. © 2004 Warren B. Powell Slide 69 Time Resource State-Type Multistage problems

70. © 2004 Warren B. Powell Slide 70 Resource State-Type Time Multistage problems

71. © 2004 Warren B. Powell Slide 71 Time Resource State-Type Multistage problems

72. © 2004 Warren B. Powell Slide 72 Time Resource State-Type Multistage problems

73. © 2004 Warren B. Powell Slide 73 Multistage problems The single-pass algorithm: »Easy to implement, but… »Suffers from slow backward communication of information: Information captured by value functions captures future time periods in previous iterations.

74. © 2004 Warren B. Powell Slide 74 Multistage problems Basic idea: »Perform forward pass as we did before, but do note update the value function. »Then step backward through the results, updating values given what happened in this iteration in the next time period.

75. © 2004 Warren B. Powell Slide 75 Backward pass

76. © 2004 Warren B. Powell Slide 76 Time Resource State-Type Backward pass

77. © 2004 Warren B. Powell Slide 77 Time Resource State-Type Backward pass

78. © 2004 Warren B. Powell Slide 78 Time Resource State-Type Backward pass

79. © 2004 Warren B. Powell Slide 79 Time Resource State-Type Backward pass

80. © 2004 Warren B. Powell Slide 80 Multistage problems Adaptive dynamic programming applied to a pure network.

81. © 2004 Warren B. Powell Slide 81 Chicag o Atlanta V0V0 V1V1 V2V2 V0V0 V1V1 V2V2 tt+1 2 Chicag o Atlanta V0V0 V1V1 V2V2 V0V0 V1V1 V2V2 2. ATL - CHI D 1 A multicommodity flow problem

82. © 2004 Warren B. Powell Slide 82 Chicag o Atlanta V0V0 V1V1 V2V2 V0V0 V1V1 V2V2 tt+1 Chicag o Atlanta V0V0 V1V1 V2V2 V0V0 V1V1 V2V2 2. ATL - CHI D 1 A multicommodity flow problem

83. © 2004 Warren B. Powell Slide 83 Chicag o Atlanta V0V0 V1V1 V2V2 V0V0 V1V1 V2V2 tt+1 2 Chicag o Atlanta V0V0 V1V1 V2V2 V0V0 V1V1 V2V2 2. ATL - CHI D 1 A multicommodity flow problem

84. © 2004 Warren B. Powell Slide 84 A multicommodity flow problem The mathematical optimum Using nonlinear approximations

85. © 2004 Warren B. Powell Slide 85 A multicommodity flow problem Planning horizon Percent of posterior bound Posterior bound Deterministic, rolling horizon

86. © 2004 Warren B. Powell Slide 86 A multicommodity flow problem Planning horizon Percent of posterior bound Posterior bound Results using value functions Deterministic, rolling horizon

87. © 2004 Warren B. Powell Slide 87 Multistage experiments Nested Benders, with 50, 100 and 250 iterationsA rolling horizon procedureAdaptive DP approximation

88. © 2004 Warren B. Powell Slide 88 Norfolk Southern

89. When a boxcar becomes empty, we have three options: Customers Regional depots General depots

90. Option 1: Send directly to customers

91. Option 2: Send to distribution areas

92. Option 1: Send directly to customers Option 2: Send to distribution areas Option 3: Send to general depots

93. © 2004 Warren B. Powell Slide 93 A car distribution problem For railroads, customers call in orders the week before: Requirement becomes knownRequirement becomes actionable Time

94. © 2004 Warren B. Powell Slide 94 Lagged information processes Ongoing research: we are currently testing an algorithmic strategy that explicitly captures this information process: Order is madeTransit time becomes knownCustomer accepts/rejects carOrder is released for shipping Time Destination of order becomes known

95. © 2004 Warren B. Powell Slide 95 A car distribution problem Repositioning movements based on forecasts Assignments to booked orders. Using value function approximations, we may reposition cars before orders become known:

96. © 2004 Warren B. Powell Slide 96 A car distribution problem Repositioning movements based on forecasts Assignments to booked orders. Using value function approximations, we may reposition cars before orders become known:

97. © 2004 Warren B. Powell Slide 97 A car distribution problem Profits Iterations

98. © 2004 Warren B. Powell Slide 98 A car distribution problem Profits Total revenue Empty repositioning costs Late service penalties Iterations

99. © 2004 Warren B. Powell Slide 99 A car distribution problem Empty miles as a percent of total miles History “Optimized” without adaptive learning

100. © 2004 Warren B. Powell Slide 100 A car distribution problem Empty miles as a percent of total miles History “Optimized” without adaptive learning “Optimized” with adaptive learning