1. Beyond Human Factors: An Approach to Human/Automation Teams Haomiao Huang Jerry Ding Wei Zhang Claire J. Tomlin Hybrid Systems Lab Action Webs Meeting 11/17/2010 1

2. 2 [nasa.gov, businessweek.com, tgdaily.com, techeasy.co.za, deere.com, aurore-sciences.org] Advances in complex multi-agent systems require smart integration of human elements.

3. [foxnews.com][wikipedia] 3 [media.weirdworm.com] [knowyourmeme.com] [adriandayton.com] This requires new approaches to analyze humans as part of the system! Let’s think about humans as part of the solution, not the problem.

4. Two related problems 4 2) Control - generating useful directives and controls for human agents 1) Modeling- Properly representing humans as components of the overall system

5. Outline  Motivation  Scenario for Research on Human/Automation Teams  Adversarial Game Problem  Reachability Based Approach  Results  Conclusions & Future Work 5

6. Choosing a Research Scenario Games are representative of hard, real-world problems, yet provide relatively benign “sandbox” environments for development Robocup Chess What is a good game to capture the aspects of human-automation teams that we want to explore? Starcraft 6

7. Time tested and fun Capture-the-Flag Capture-the-flag embodies the basic research challenges we are trying to address http://www.goforyourlife.vic.gov.au/hav/articles.nsf/pages/Capture_the_Flag Limited Information Multiple Agents Competing Objectives Human players Adversarial 7

8. Automation-Assisted Human Capture-the-Flag Using mobile phones, computers, and UAVs, we have turned capture-the- flag into a testbed for advanced automation concepts involving human team members Game software on Android phones STARMAC Quadrotor UAVs Server-side Management Software 8

9. Time tested and fun Narrowing the problem http://www.goforyourlife.vic.gov.au/hav/articles.nsf/pages/Capture_the_Flag Limited Information Multiple Agents Competing Objectives Human players Adversarial 9

10. Outline  Motivation  Scenario for Research on Human/Automation Teams  Adversarial Game Problem  Problem statement  Related Work  Solution Insights  Reachability Based Approach  Results  Conclusions & Future Work 10

11. Our Problem Capture Region Defender Attacker Flag Flag Region Return Region Game Domain Characterize and solve a 1-sided capture-the-flag game with a single attacker and defender 11

12. Related Work on Adversarial Games  Multi-agent games on discrete state spaces Greedy search Hespanha, Kim, and Sastry 1999 Approximate DP/Reinforcement Learning Lagoudakis and Parr 2002 Discrete Play Matching Browning, Bruce, and Veloso 2005  Pursuit-evasion games with continuous states Receding-Horizon Control Mcgrew, How, Bush, Williams and Roy 2008 Sprinkle, Eklund, Kim, and Sastry 2004 Optimal Trajectory Planning Earl and D’Andrea 2001 Chasparis and Shamma 2005 Analytical game theory approaches Basar 1989, Lewin 1994, Stipanovic, Melikyan, Hovakimyan 2010 Hamilton-Jacobi Reachability Mitchell, Bayen, and Tomlin, 2005 Ding, Sprinkle, and Tomlin 2008 12 Assumed, learned, or randomized opponent model

13. Reachability Approach, derived from pursuit-evasion games: CTF game can be posed as a reachability problem. Assume system dynamics Where is the input for Player I and is the input for Player II Define as the reach-avoid set where a player can arrive in a goal region in at most time while avoiding region, no matter what the other player does 13

14. Capture-the-Flag as Reachability Victory conditions for each player can be encoded as reach-avoid sets in the joint state-space Defender Attacker Joint Capture Set Joint Return Set 14 Flag Return Set (For Attacker) Game Domain

15. 1-D Game 15

16. Geometric insights Geometric analysis allows some insight into the 2-D capture-the-flag problem 16

17. Geometric insights Geometric analysis allows some insight into the 2-D capture-the-flag problem 17

18. Utility of Reachability Analysis 18 Reachability analysis gives complete characterization of game, and are a natural display tool for guiding human decision-making and allowing least-restrictive control Teo and Tomlin, 2003 Geometric analysis is not terribly general, though…

19. Outline  Motivation  Scenario for Research on Human/Automation Teams  Adversarial Game Problem  Reachability Based Approach  Hamilton-Jacobi Reachability  Computation  Results  Conclusions & Future Work 19

20. Hamilton-Jacobi Reachability Reachability in continuous state-spaces can analyzed as a terminal cost- only optimization problem, solved backward in time Reachability Cost Function Classic Optimal Control Cost Function Tomlin 2009 20

21. Level-Set Representation Sets can be represented using sub-level sets of signed distance functions as terminal cost functions Set operations using point-wise minimum and maximums can be used to create arbitrary sets Tomlin 2009, Mitchell 2003 21

22. Solution Based on HJBI Equation The cost-to-go function is the unique viscosity solution to the Hamilton- Jacobi-Bellman-Isaacs equation Classic Optimal Control Cost Function Hamilton-Jacobi-Bellman-Isaacs Equation Optimal Hamiltonian 22

23. Reachability Via Modified HJBI Equation The backward reachable set is the zero sub-level set of the viscosity solution to a modified HJBI equation Modified HJBI Equation Optimal Hamiltonian Reachability Cost Function Mitchell, Bayen, Tomlin 2005 23

24. Numerical Solution to the Modified HJBI Equation 24 The viscosity solution to the modified HJBI Equation can be computed on a grid using the Level Set Toolbox from UBC http://www.cs.ubc.ca/~mitchell/ToolboxLS/index.html

25. Reach-Avoid & Control Inputs 25 Reach-avoid sets can be computed by masking the reach set at each integration time step with the avoid set Control inputs can be extracted using the co-states, which can be calculated by numerical differentiation of the value function

26. Outline  Motivation  Scenario for Research on Human/Automation Teams  Adversarial Game Problem  Reachability Based Approach  Results  HJBI Reachability applied to capture-the-flag  Simulation results  Experimental setup  Conclusions & Future Work 26

27. Problem Formulation for 1v1 Capture-the-Flag HJBI reachability analysis allows us to fully characterize the game 27 Dynamics Optimal Hamiltonian Optimal Inputs

28. Flag Return & Flag Capture 28 Winning regions for each portion of the game can be calculated directly from reach-avoid conditions

29. Sequenced Capture and Return 29 Winning regions for the full sequence (flag capture and subsequent return) can be computed by using the intersection of the flag return set and flag zone as the initial condition for flag capture

30. Simulation Results 30 Simulation results demonstrate the use of the reachability solutions

31. Field Experiments in Progress 31 Reachability-based control and input directives are being implemented on Droid Incredible phones Game software on Android phones Server-side Management Software Player Positions and State Reachable sets & optimal control inputs

32. Outline  Motivation  Scenario for Research on Human/Automation Teams  Adversarial Game Problem  Reachability Based Approach  Results  Conclusions & Future Work 32

33. Conclusions  Capture-the-flag is great platform for developing human- automation systems research.  A differential game formulation using HJBI reachability solves perfect information, 1v1 CTF 33

34. Future Work We have the “correct” answer to the adversarial problem… now what? http://www.goforyourlife.vic.gov.au/hav/articles.nsf/pages/Capture_the_Flag Limited Information Multiple Agents Competing Objectives Human players Adversarial 34

35. Thank you! Questions? 35