1. In practice, we run into three common issues faced by concurrent optimization algorithms. We alter our model-shaping to mitigate these by reasoning about the types of interactions we have: – Slow convergence  Prioritization – Oscillation  Probabilistic shaping – Local optima  Optimistic policy initialization Distributed POMDPs with Coordination Locales (DPCLs) This work uses the DPCL problem model [2]. DPCLs are similar to Dec-POMDPs in representing problems as sets of states, actions and observations with joint transition, reward, and observation functions. However, DPCLs differ in that they factor state space into global and per-agent local components, and interactions among agents are limited to coordination locales. Evaluation in a Heterogeneous Rescue Robot Domain Consider the problem of a team of robots planning to search a disaster area. Some robots can assist victims, while others can clear otherwise intraversable debris. Robot observations and movements are subject to uncertainty. We evaluate D-TREMOR’s performance on a number of these planning problems, in teams of up to 100 agents. Acknowledgements This research has been funded in part by the AFOSR MURI grant FA9550-08-1-0356. This material is based upon work supported under a National Science Foundation Graduate Research Fellowship. Tractable Planning in Large Teams Emerging team applications require the cooperation of 100s of members (humans, robots, agents). Team members must complete complex, collaborative tasks in dynamic and uncertain environments. How can we effectively and tractably plan in these domains? Scaling up from TREMOR [2] to D-TREMOR Conclusions and Future Work We introduce D-TREMOR, an approach to scale distributed planning under uncertainty into the hundreds of agents using information exchange and model-shaping. Results suggest competitive performance while improving scalability and reducing computational cost. We are working to further improve performance through better modeling of interaction dynamics and intelligent information dissemination between agents. References [1] M. Kearns, Y. Mansour, and A. Y. Ng. A sparse sampling algorithm for near-optimal planning in large Markov decision processes. Machine Learning. 2002. [2] P. Varakantham, J. Kwak, M. Taylor, J. Marecki, P. Scerri, and M. Tambe. Exploiting Coordination Locales in Distributed POMDPs via Social Model Shaping. Proc. of ICAPS, 2009. [3] P. Varakantham, R.T. Maheswaran, T. Gupta, and M. Tambe. Towards Efficient Computation of Error Bounded Solutions in POMDPs: Expected Value Approximation and Dynamic Disjunctive Beliefs. Proc. of IJCAI, 2007. Role AllocationPolicy SolutionInteraction DetectionCoordination TREMOR Branch & Bound MDP Independent EVA [3] solvers Joint policy evaluation Reward shaping of local models D-TREMOR Decentralized Auction Sampling & message passing Reward shaping of local models with convergence heuristics Rescue Agent Cleaner Agent Narrow Corridor Victim Unsafe Cell Clearable Debris Example Map: Rescue Domain Objective function: Get rescue agents to as many victims as possible within a fixed time horizon while minimizing collisions and unsafe cells. Agents can collide in narrow corridors (only one agent can fit at a time) and with clearable debris (blocks rescue agents, but can be cleared by cleaner agents). D-TREMOR policies – Max-joint-value – Last iteration Comparison policies – Independent – Optimistic – Do-nothing – Random Scaling dataset: – 10 to 100 agents – Random maps Density dataset – 100 agents – Concentric ring maps 3 problems/condition 20 planning iterations 7 time step horizon 1 CPU per agent D-TREMOR: Distributed Team REshaping of Models for Rapid-execution We extend the TREMOR [2] algorithm for solving DPCLs to produce D-TREMOR, a fully- distributed solver that scales to problems with hundreds of agents. It approximates DPCLs as a set of single-agent POMDPs which are solved in parallel, then iteratively reshaped using messages that describe CL interactions between agent policies. D-TREMOR rescues many more victims. D-TREMOR resolves many, but not all collisions. Task Allocation Local Planning Interaction Exchange Model Shaping Finding the probability of a CL [1] : Evaluate local policy Compute frequency of associated s i, a i Entered corridor in 95 of 100 runs: Pr CL = 0.95 Finding the value of a CL [1] : Sample local policy value with/without interactions – Test interactions independently Compute change in value if interaction occurred No collision Collision Val CL = -7 Send CL messages to teammates: Sparsity  Relatively small # of messages Shape local model rewards/transitions based on remote interactions Probability of interaction Interaction model functions Independent model functions Re-solve shaped local models to get new policies Result: new locally-optimal policies  new interactions Distributed Interaction Detection using Sampling and Message ExchangeImproved model shaping of local agent models with convergence heuristics Increases in time are related to # of CLs, not # of agents. Results of Scaling DatasetResults of Density Dataset Naïve Policies (optimistic, do- nothing, random) D-TREMOR Policies Number of agents and map size are varied as density of debris, corridors, and unsafe cells is held constant. Concentric rings of narrow corridors are added from outside in on a map where victims are at the center. High risk of collision penalties makes do-nothing seem competitive. Ignoring interactions = poor performance Independent & Optimistic ? Agents not interacting, use independent functions: Agents are interacting, use joint CL functions: CL = Nature of time constraint (e.g. affects only same-time, affects any future-time) Coordination Locales define regions of state-action space where joint transition/reward functions are needed Relevant region of joint state-action space : Joint Transition : Joint Reward : Joint Observation : Set of States: : Set of Actions: : Set of Observations: : Initial Belief State Time constraint