© 2003 Warren B. Powell Slide 1 Approximate Dynamic Programming for High Dimensional Resource Allocation NSF Electric Power workshop November 3, 2003 Warren Powell CASTLE Laboratory Princeton University © 2003 Warren B. Powell, Princeton University

© 2003 Warren B. Powell Slide 2 Schneider National

© 2003 Warren B. Powell Slide 3 Schneider National

© 2003 Warren B. Powell Slide 4

© 2003 Warren B. Powell Slide 5

© 2003 Warren B. Powell Slide 6

© 2003 Warren B. Powell Slide 7 Air Mobility Command Air Mobility Command Fuel Cargo Handling Ramp Space Maintenance Cargo Holding

© 2003 Warren B. Powell Slide 8 The optimization challenge

Special equipment

© 2003 Warren B. Powell Slide 11 State variables Modeling the military airlift problem: »State variables: »Control variables:

© 2003 Warren B. Powell Slide 12 State variables We can formulate the problem of determining what to do with our aircraft as a dynamic program:

© 2003 Warren B. Powell Slide 13 State variables If we only have N=1 aircraft:

© 2003 Warren B. Powell Slide 14 State variables What if we have N>1 aircraft?

© 2003 Warren B. Powell Slide 15 State variables Number of resources Number of attributes Number of zeroes in size of state space

© 2003 Warren B. Powell Slide 16 Outline An algorithmic strategy for high-dimensional asset allocation problems

© 2003 Warren B. Powell Slide 17 Approximate dynamic programming Systems evolve through a cycle of exogenous and endogenous information Time

© 2003 Warren B. Powell Slide 18 Approximate dynamic programming Systems evolve through a cycle of exogenous and endogenous information Time

© 2003 Warren B. Powell Slide 19 Approximate dynamic programming Using this state variable, we obtain the optimality equations: Problem: Curse of dimensionality Three curses State space Outcome space Action space (feasible region)

© 2003 Warren B. Powell Slide 20 Approximate dynamic programming The computational challenge: How do we find ? How do we compute the expectation? How do we find the optimal solution?

© 2003 Warren B. Powell Slide 21 Approximate dynamic programming Approximation methodology: Can’t compute this!!!Don’t know what this is!

© 2003 Warren B. Powell Slide 22 Adaptive dynamic programming Alternative: Change the definition of the state variable: Time

© 2003 Warren B. Powell Slide 23 Adaptive dynamic programming Now our optimality equation looks like: We drop the expectation and solve the conditional problem: Finally, we substitute in our approximation:

© 2003 Warren B. Powell Slide 24 Adaptive dynamic programming Approximating the value function: »We choose approximations of the form:

© 2003 Warren B. Powell Slide 25 Approximate dynamic programming This period Future

© 2003 Warren B. Powell Slide 26 Approximate dynamic programming Our basic strategy: Separable approximation

© 2003 Warren B. Powell Slide 27 Research questions in electric power Special equipment

© 2003 Warren B. Powell Slide 28 Research questions in electric power Two-stage resource allocation under uncertainty

© 2003 Warren B. Powell Slide 29 Approximate dynamic programming

© 2003 Warren B. Powell Slide 30 Approximate dynamic programming

© 2003 Warren B. Powell Slide 31 Approximate dynamic programming

© 2003 Warren B. Powell Slide 32 Approximate dynamic programming We estimate the functions by sampling from our distributions. Marginal value:

© 2003 Warren B. Powell Slide 33 A dynamic network: Approximate dynamic programming t

© 2003 Warren B. Powell Slide 34 Approximate dynamic programming Stepping through time:

© 2003 Warren B. Powell Slide 35 Approximate dynamic programming Iterative learning:

© 2003 Warren B. Powell Slide 36 Nonlinear approximations Number of resources Approximate value function

© 2003 Warren B. Powell Slide 37 Competing algorithmic strategies Competing optimal algorithms: »Discrete dynamic programming Cannot handle even small problems Numerical comparisons are meaningless »Stochastic programming Bender’s decomposition is optimal for this problem class

© 2003 Warren B. Powell Slide 38 Benders decomposition Variations on Bender’s decomposition SPAR algorithm Deterministic approximation Iterations

© 2003 Warren B. Powell Slide 39 Conclusions: »Using sequences of separable, nonlinear approximations conquers the explosive growth with the number of resources. »We are now solving problems with thousands of resources. »But what about the attribute space? Complex equipment and people are typically described by vectors of attributes. We require multidimensional attributes to capture complex assets such as equipment and people. The size of the attribute space grows exponentially in the number of dimensions.

© 2003 Warren B. Powell Slide 40 Benders decomposition Variations on Benders decompositionSPAR Percent over optimal Attribute space = 10 Attribute space = 25 Attribute space = 50 Attribute space = 100

© 2003 Warren B. Powell Slide 41 Benders decomposition Variations on Benders decompositionSPAR Percent over optimal Increasing problem size makes solution much worse With SPAR, the solution gets better.

© 2003 Warren B. Powell Slide 42 Multidimensional attribute spaces decision d

© 2003 Warren B. Powell Slide 43 Multidimensional attribute spaces $450

© 2003 Warren B. Powell Slide 44 NE region PA TX NY Multidimensional attribute spaces

© 2003 Warren B. Powell Slide 45 Hierarchical Aggregation We can use a family of aggregation functions:

We can use different levels of aggregation to capture the value of an asset:

© 2003 Warren B. Powell Slide 47 Hierarchical aggregation Alternative: »Use multiple levels of aggregation at the same time Estimate at gth level of aggregation Weight on gth level of aggregation

© 2003 Warren B. Powell Slide 48 x f(x) Hierarchical aggregation

© 2003 Warren B. Powell Slide 49 x f(x) High structure Moderate structure Zero structure Hierarchical aggregation

© 2003 Warren B. Powell Slide 50 Bayesian weights Weight on disaggregate level Optimal weights Hierarchical aggregation

© 2003 Warren B. Powell Slide 51

© 2003 Warren B. Powell Slide 52 Hierarchical aggregation Aggregate Disaggregate Weighted Combination

© 2003 Warren B. Powell Slide 53 Hierarchical aggregation Iterations Weights Aggregation level 6767 Weight on most disaggregate level Weight on most aggregate levels Optimal weights change as the algorithm progresses:

© 2003 Warren B. Powell Slide 54 Conclusions »Hierarchical aggregation offers a powerful mechanism for handling high dimensional, arbitrary attribute spaces »Combined with the use of separable approximations for handling large numbers of assets, we have a powerful approach for large-scale resource allocation problems.

© 2003 Warren B. Powell Slide 55 Research questions Algorithmic questions: »Stepsizes and rate of convergence We need to improve our understanding of adaptive stepsizes.

© 2003 Warren B. Powell Slide 56 Research questions Algorithmic questions: »Stability: we would like to measure the responsiveness to small changes. The instability limits analyses to big changes. Research: Big simulations will also be unstable. We need to calculate derivatives of simulations. Theory needs to be extended to new problem classes.

© 2003 Warren B. Powell Slide 57 Research questions in electric power Application to electric power: »Fuel optimization (continuous assets): What fuel to purchase when we can switch between fuels Design of fuel contracts Determining prices of forward contracts How much and where to store fuel. »Asset management problems (discrete assets): Unit commitment problems –Control of hydro units Positioning of assets for emergency response –Special equipment –People with specialized training

© 2003 Warren B. Powell Slide 58 Research questions in electric power Special equipment