Beyond Human Factors: An Approach to Human/Automation Teams Haomiao Huang Jerry Ding Wei Zhang Claire J. Tomlin Hybrid Systems Lab Action Webs Meeting 11/17/2010 1

2 [nasa.gov, businessweek.com, tgdaily.com, techeasy.co.za, deere.com, aurore-sciences.org] Advances in complex multi-agent systems require smart integration of human elements.

[foxnews.com][wikipedia] 3 [media.weirdworm.com] [knowyourmeme.com] [adriandayton.com] This requires new approaches to analyze humans as part of the system! Let’s think about humans as part of the solution, not the problem.

Two related problems 4 2) Control - generating useful directives and controls for human agents 1) Modeling- Properly representing humans as components of the overall system

Outline  Motivation  Scenario for Research on Human/Automation Teams  Adversarial Game Problem  Reachability Based Approach  Results  Conclusions & Future Work 5

Choosing a Research Scenario Games are representative of hard, real-world problems, yet provide relatively benign “sandbox” environments for development Robocup Chess What is a good game to capture the aspects of human-automation teams that we want to explore? Starcraft 6

Time tested and fun Capture-the-Flag Capture-the-flag embodies the basic research challenges we are trying to address Limited Information Multiple Agents Competing Objectives Human players Adversarial 7

Automation-Assisted Human Capture-the-Flag Using mobile phones, computers, and UAVs, we have turned capture-the- flag into a testbed for advanced automation concepts involving human team members Game software on Android phones STARMAC Quadrotor UAVs Server-side Management Software 8

Time tested and fun Narrowing the problem Limited Information Multiple Agents Competing Objectives Human players Adversarial 9

Outline  Motivation  Scenario for Research on Human/Automation Teams  Adversarial Game Problem  Problem statement  Related Work  Solution Insights  Reachability Based Approach  Results  Conclusions & Future Work 10

Our Problem Capture Region Defender Attacker Flag Flag Region Return Region Game Domain Characterize and solve a 1-sided capture-the-flag game with a single attacker and defender 11

Related Work on Adversarial Games  Multi-agent games on discrete state spaces Greedy search Hespanha, Kim, and Sastry 1999 Approximate DP/Reinforcement Learning Lagoudakis and Parr 2002 Discrete Play Matching Browning, Bruce, and Veloso 2005  Pursuit-evasion games with continuous states Receding-Horizon Control Mcgrew, How, Bush, Williams and Roy 2008 Sprinkle, Eklund, Kim, and Sastry 2004 Optimal Trajectory Planning Earl and D’Andrea 2001 Chasparis and Shamma 2005 Analytical game theory approaches Basar 1989, Lewin 1994, Stipanovic, Melikyan, Hovakimyan 2010 Hamilton-Jacobi Reachability Mitchell, Bayen, and Tomlin, 2005 Ding, Sprinkle, and Tomlin Assumed, learned, or randomized opponent model

Reachability Approach, derived from pursuit-evasion games: CTF game can be posed as a reachability problem. Assume system dynamics Where is the input for Player I and is the input for Player II Define as the reach-avoid set where a player can arrive in a goal region in at most time while avoiding region, no matter what the other player does 13

Capture-the-Flag as Reachability Victory conditions for each player can be encoded as reach-avoid sets in the joint state-space Defender Attacker Joint Capture Set Joint Return Set 14 Flag Return Set (For Attacker) Game Domain

1-D Game 15

Geometric insights Geometric analysis allows some insight into the 2-D capture-the-flag problem 16

Geometric insights Geometric analysis allows some insight into the 2-D capture-the-flag problem 17

Utility of Reachability Analysis 18 Reachability analysis gives complete characterization of game, and are a natural display tool for guiding human decision-making and allowing least-restrictive control Teo and Tomlin, 2003 Geometric analysis is not terribly general, though…

Outline  Motivation  Scenario for Research on Human/Automation Teams  Adversarial Game Problem  Reachability Based Approach  Hamilton-Jacobi Reachability  Computation  Results  Conclusions & Future Work 19

Hamilton-Jacobi Reachability Reachability in continuous state-spaces can analyzed as a terminal cost- only optimization problem, solved backward in time Reachability Cost Function Classic Optimal Control Cost Function Tomlin

Level-Set Representation Sets can be represented using sub-level sets of signed distance functions as terminal cost functions Set operations using point-wise minimum and maximums can be used to create arbitrary sets Tomlin 2009, Mitchell

Solution Based on HJBI Equation The cost-to-go function is the unique viscosity solution to the Hamilton- Jacobi-Bellman-Isaacs equation Classic Optimal Control Cost Function Hamilton-Jacobi-Bellman-Isaacs Equation Optimal Hamiltonian 22

Reachability Via Modified HJBI Equation The backward reachable set is the zero sub-level set of the viscosity solution to a modified HJBI equation Modified HJBI Equation Optimal Hamiltonian Reachability Cost Function Mitchell, Bayen, Tomlin

Numerical Solution to the Modified HJBI Equation 24 The viscosity solution to the modified HJBI Equation can be computed on a grid using the Level Set Toolbox from UBC

Reach-Avoid & Control Inputs 25 Reach-avoid sets can be computed by masking the reach set at each integration time step with the avoid set Control inputs can be extracted using the co-states, which can be calculated by numerical differentiation of the value function

Outline  Motivation  Scenario for Research on Human/Automation Teams  Adversarial Game Problem  Reachability Based Approach  Results  HJBI Reachability applied to capture-the-flag  Simulation results  Experimental setup  Conclusions & Future Work 26

Problem Formulation for 1v1 Capture-the-Flag HJBI reachability analysis allows us to fully characterize the game 27 Dynamics Optimal Hamiltonian Optimal Inputs

Flag Return & Flag Capture 28 Winning regions for each portion of the game can be calculated directly from reach-avoid conditions

Sequenced Capture and Return 29 Winning regions for the full sequence (flag capture and subsequent return) can be computed by using the intersection of the flag return set and flag zone as the initial condition for flag capture

Simulation Results 30 Simulation results demonstrate the use of the reachability solutions

Field Experiments in Progress 31 Reachability-based control and input directives are being implemented on Droid Incredible phones Game software on Android phones Server-side Management Software Player Positions and State Reachable sets & optimal control inputs

Outline  Motivation  Scenario for Research on Human/Automation Teams  Adversarial Game Problem  Reachability Based Approach  Results  Conclusions & Future Work 32

Conclusions  Capture-the-flag is great platform for developing human- automation systems research.  A differential game formulation using HJBI reachability solves perfect information, 1v1 CTF 33

Future Work We have the “correct” answer to the adversarial problem… now what? Limited Information Multiple Agents Competing Objectives Human players Adversarial 34

Thank you! Questions? 35