2. Pursuit Evasion Games (PEGs) for Multiple UUVs
UUVs have the potential to provide an effective defense against submersible threats to military and civilian assets
The strategies and protocols for their operation are at least as big a challenge as the design and construction of the vehicles themselves
This project address the strategies and coordination protocols necessary to enable this technology
Defense against enemy subs is the number one FNC that is called for in every briefing since 2000.

3. Bear UUV Being Built by Lt Tulio Celano III, USN
8 ft X 10 in., 400 lbs. displacement, upto 100 ft. depth, Top speed 12 knots, effective cruising speed 5 knots, endurance at 5 knots is 45 hours, Modular design

4. BEAR 1-UUV, Nov 2004 Showing Modules with Mast Up Celano Machine Shop
Ballast Pump and Ballast Module

5. Prior PEG Experience PEG experience in:
Unmanned aerial and ground vehicle (UAVs and UGVs)
Two and three dimensions.
Symmetrical and asymmetrical games
Proven tool: Nonlinear Model Predictive Trajectory Control (NMPTC)
Explicitly addresses nonlinear systems with constraints on operation and performance
A cost minimization problem in the presence of state and input constraints
Control resulting in the minimum cost is determined over a model predicted horizon
Previously demonstrated in rotary-wing and fixed-wing UAVs

6. NMPTC: Cost Function Definition
Cost function is defined by:
x is the state vector
u is the control vector
y tilde is the trajectory error
d is the pursuer/evader position difference
The first three terms in L are attractive functions, i.e. values closer to zero are lower cost
The last term is a repulsive function, i.e. values further from zero are lower cost
The ½ and ¼ powers are used to reduce the gradient of the cost as a function of d and a
The function is tuned with Q, S, R, G, T and also the ½ and ¼ terms

7. NMPTC: Cost function illustration
All matrices are square
All functions are easily differentiated

8. NMPTC: Cost function minimization
A gradient decent method is used to minimize the cost function
Initialization with previous result reduces the number of iterations required
Usually 3-4 iterations are required
The number of iterations in limited to prevent overruns in real-time
In rapidly changing situation this can result in suboptimal solution
Sudden changes may take several time steps
However, this is alright since the situation is changing and unpredictable

9. Previous UC Berkeley PEG Experiments
Berkeley Aerobot Project (BEAR)
Goal: to build a coordinated, intelligent network with multiple heterogeneous agents
11 Rotorcraft-based unmanned aerial vehicles (UAVs)
5 Unmanned ground vehicles (UGVs)
Shipdeck simulator (landing platform)
Stochastic Pursuit-Evasion Games (PEG)
Self-localization
Target detection
Map building
Pursuit policy
Trajectory generation
Control / Action

10. Previously: PEGs with 4 UGVs and 1 UAV
Sub-problems for Pursuit Evasion Games
Sensing
Navigation sensors -> Self-localization
Detection of objects of interest
Framework for communication and data flow
Map building of environments and evaders
How to incorporate sensed data into agents’ belief states
probability distribution over the state space of the world (I.e. possible configuration of locations of agents and obstacles)
How to update belief states
Strategy planning
Computation of pursuit policy
mapping from the belief state to the action space
Control / Action

11. PEG Experiment with UAV/UGVs
PEG with four UGVs
Global-Max pursuit policy
Simulated camera view
(radius 7.5m with 50degree conic view)
Pursuer=0.3m/s Evader=0.5m/s MAX

12. Evaluation of Policies for different visibility
Capture time of greedy and glo-max
for the different region of visibility
of pursuers
3 Pursuers with
trapezoidal or omni-directional view
Randomly moving evader
Global max policy performs better than greedy, since the greedy policy selects movements based only on local considerations.
Both policies perform better with the trapezoidal view, since the camera rotates fast enough to compensate the narrow field of view.
This picture compares the performance of two policies with different visibility region.
In all cases, three pursuers with either conic or omnidirectional view, greedy or global max strategy try to capture a randomly moving single evader.

13. Evader’s Speed vs. Intelligence
Capture time for different speeds
and levels of intelligence of the evader
3 Pursuers with trapezoidal view
& global maximum policy
Max speed of pursuers: 0.3 m/s
Having a more intelligent evader increases the capture time
Harder to capture an intelligent evader at a higher speed
The capture time of a fast random evader is shorter than that of a slower random evader, when the speed of evader is only slightly higher than that of pursuers.
This picture shows the capture time while changing the speed and intelligence of evaders.
In all cases, three pursuers with conic view and global max policy try to capture a single evader.

14. SEC Capstone Demo: Fixed-Wing PEGs
Capstone Demonstrations were proposed to highlight and test the technologies developed in the SEC program
One was a fixed-wing UAV flight test
6 participant technology developers (TDs)
Honeywell, Northrop Grumman, U Minnesota, MIT, Stanford, and UCB/U Colorado/CalTech
System Integrator was Boeing
OCP would be software framework
Autonomous T-33 trainer as UAV surrogate
Piloted F-15 as wingman/opponent
13 month schedule May 03 – June 04
UCB Contribution: Fixed-Wing PEGs

15. Demo: UCB PEG Scenario
20 – 60 min. games confirm NMPTC feasibility at real-time
Evader goal: get to final waypoint or avoid evader
Pursuer goal: ‘target’ evader
Pursuer and evader restricted to same performance limits
Scenarios:
UAV as evader
UAV can become pursuer
OCP Experiment Controller Snapshot
T-33: Evader (yellow) F-15: Pursuer (blue)
(1) (Very) brief overview of the experiment, decoupled from the rest of the slides, so that Shankar can take it out of the darpa guys are already keen on the SEC demo
    a) Aircraft type(s) in use
    b) approx. time constraints
    c) Description of game rules (including AOT, etc.)
The F15 altitude range is 18,000’ to 20,000’
The UAV altitude range is 10,000’ to 12,000’
Target cone definition (θ=10˚,d=3 nm)
Left: F15 not behind UAV, middle: F15 not pointed at UAV, right: F15 behind AND pointed at UAV

16. Flight Test 1 (UAV as evader)

17. Flight Test 2 (UAV as evader/pursuer)

18. UUV PEGs: Multiplayer Games
In littoral waters the pursuit evasion game consists of an enemy submarine attempting to cross a line of UUVs which are protecting an asset
The enemy submarine has a speed advantage over the blue force UUVs
UUVs play a role in between a sensor web and a group of pursuers
Research aimed at determining new approaches to teaming for multi-player games. Current literature focuses exclusively on either Nash or Stackleberg solutions

19. Multiplayer PEG Challenges
The research challenge includes extending the strategies to:
Large multi-player teams
Asymmetric platform characteristics
Limited communications
High level of uncertainly

20. UUV PEG: Approach The UUV PEG involves two distinct phases:
Detection phase
Maximize chances of detection constrained by:
Area to cover
Number of UUVs available
Possible evader strategies
Capabilities of UUVs and evader (sensors and noise signatures)
Response phase
Maximize chances of catching the pursuer constrained by:
Capabilities of UUVs and evader (speed, manueverability and communications)
These two phases also depend on each other as both must succeed
How to share resources to maximize overall chance of success
How to overlap the strategies: detectors are responders as well

21. Multiplayer PEGs: Proposed Solution
A close analogy is football or other team games:
Multi player
Initial (global) strategies well defined
Limited (local) coordination after the snap
What can we learn?
How can we apply this?
How far does the analogy go?

22. Multiplayer PEGs Preseason (Off-line precomputed strategy)
Play book:
Evaluate strategies and configurations that will maximize chance of success based on best estimate of other team’s tactics
Practice and preseason games:
Test playbook and find problems
Game time (On-line adaptive strategy)
Choose play based on best knowledge and experience
Line up (in best detection configuration, not necessarily static)
Execute the play
Active and reactive actions (respond to detected evader)
Local communication
Adapt to evolving behavior
Learn from experience, repeat as necessary (Learning by Doing)

23. Pre-game: Strategies for Detection
Maximize the chance of detecting the evader
Tradeoffs
Movement of the pursuer:
Moving quickly covers more area, but
This makes it easier for evader to see the pursuer and avoid the pursuer
Sensors:
Using passive sonar reduces the range of detection
Using active sonar reveals the pursuers location
Number of pursuers in detector role:
Increases chance of detection

24. Basic principle: Defense in depth
May not be the optimal with limited resources,
for instance if there are not enough UUVs to ensure detection of the evader by the front line.

25. Options: Zone Defense
Would this leave seams for an evader to exploit if they have superior sensors, for instance?
Is communication necessary to make such a zone defense effective?
Is there an alternative?

26. Options: Channeling the evader
A coordinated, heterogenous detection strategy.
For instance, some pursuers could use a very active strategy that exposed them to intentional detection by the evader, with the intension of “channeling” the evader towards other more passive pursuers.

27. Pursuer Strategies: Capture
Speed disadvantage means that simply optimizing the detection probability is not sufficient
Reachability of UUVs must be known
Communication and coordination will be necessary to overcome speed disadvantage

28. Defining the Problem: Basic UUV PEG Scenarios
Single evader infiltration
Objectives
Red: pass through game area undetected
Blue: detect red team only
Scenario 2
Single evader attack
Red: get within weapons range of some objective
Blue: prevent red attack on objective
Capabilities
Red: limited number of torpedoes available to attack target or blue team UUVs
Red: suicide attacks only, must get within effective range
Scenario 3
Multiple evader attack
Objectives & Capabilities
Same as Scenario 2

29. UUV Characteristics in General
Performance
Speed and acceleration
Maximum rate of turn
Maximum rate of ascent/descent (not symmetric in general)
Maximum depth
Sensors
Effective range in passive detection mode
Effective range in active detection mode
Deployable sonar buoys
Communications
Effective communication range (variable in general)

30. UUV Characteristics, cont.
Method of attack, including:
Self detonation, effective range and perhaps effectiveness as a function of range
Missile (i.e. torpedo) capabilities
Counter measures, including:
Sonar buoys
Noise canisters
Noise signature as a function of:
Speed
Acceleration
Rate of turn
Rate of ascent/descent
Sensor mode
Communication mode
And others

31. The First Problem Definition
Based on “Scenario 2”:
Single evader attack, many defenders
Objectives
Red: get within weapons range of some objective
Blue: prevent red attack on objective
Capabilities
Red team:
Limited number of torpedoes available to attack target or blue team UUVs
3 times speed advantage over the Blue (pursuer) team
Blue team:
Suicide attacks only, must get within effective range
3 times manuever (turn rate ) advantage over the Red team
Limited communication range
Passive and active sensors available

32. The First Problem Definition, cont
Characteristics
Speed
3X advantage to Red Team
Maneuver advantage
3X to Blue Team
Detection a function only of
speed,
communication use
Distance
Sonar
Active & passive available
Communications available for each team:
range a function of power
detectability also a function of power

33. Detection: Strategy Comparison

34. Detection: Monte Carlo results
Line abreast
Staggered
Goals:
Statistical model as a function of configuration, spacing, etc.
Test strategies
(Recall: detection function)

35. Capture: Strategies based on reachability (Mitchell’s Level Set Toolbox)
From Airplane example
Still even, turn rate reduced to 1/3
Pursuer speed reduced to 1/3
Pursuer speed reduced to 1/3, turn rate increased by 3

36. Combined the Strategies: Advances in Game Theory
These PEGs fit Game Theory descriptions as:
mixed strategy
simultaneous move
multiplayer
coordinated games
games with incomplete information
Specific tactics can be evaluated to find the equilibria and optimal strategies in certain situations, e.g.:
the best search patterns within a single zone for one UUV, or for two adjacent zones/UUVs
Statistical methods or Monte Carlo methods could be used to determine the changes of success for each player

37. Communication Between UUVs
The issue of the short range of underwater communications with multiple players is very similar to the problem of ad hoc wireless networks of motes devices
This experience is directly adaptation to the multiple, coordinated UUV scenarios and used to disseminate information within the team regarding:
Detection of an evader
Likely detection by the other team
Coordination instructions
Strategic commands
etc.
This is integral into the simulation environment

38. Future Research
We are building a group of UUVs at the USNA in Annapolis. These are also being shared with NUWC, Newport
We will develop a theory of multi-player pursuit evasion games with off-line strategies (using robust optimization, the level set toolbox), on-line modifications of the strategy using Model Predictive Control, and an outer-loop of learning
Expect to have simulation results by June 05, experiments for single UUV by June 05, multiple UUV experiments by June 06

39. UCAR Transition: NMPC in a Complex 3-D Environment
Potential
function
NMPC

40. UCAR-Obstacle Sensing
Map-based approach
- Obstacle map is measured and stored in the computer
- Upon request, the nearest obstacle coordinate is passed to the MPC unit
- Sensing is always perfect, thus reducing any risks due to sensing failure or any unpredicted control behavior

41. Obstacle Sensing using Laser Scanner
Control Computer
Tilt Mount
Position Command
- Encoder
Servo
Micro controller
Encoder reading
- PIII 700MHz PC104 module
Measured range data
Ground Station
Measured range data
361meas/scan
Minimum range data
Real-time 3D Visualization
Vehicle state
Light weight
2D Laser Scanner
Flight Computer
Real-time optimization
Reference trajectory
GPS+INS
PIII 700MHz
PC104 module
Navigation data
Minimum range data
MPC Engine

42. November 2004 Flight Tests

43. Multi-Player Games: The Play of the Big Game 1982