1. Modelling CGFs for tactical air-to-air combat training
Motivation-based behaviour and Machine Learning in a common architecture
Jan Joris Roessingh, Ph.D., Maj. Roel Rijken, M.Sc.
National Aerospace Laboratory (NLR), Royal Netherlands Air Force

2. Contents Part 1(Intelligent CGFs) Smart Bandits Requirements
State-of-the-art in CGFs
Architecture
Cognitive models
Part 2 (Machine Learning)
Experiment RL (example)
Pros & Cons ML
Towards hybrid models (example)
Conclusions
Exploratory Team under IST panel

3. Project Goals “Smart Bandits”
Development of intelligent Computer Generated Forces
for tactical mission training of fighter pilots
in the opponent role
‘humanlike’ behaviour
capable of
tactical reasoning
intelligent decision making
team work ..
Constrained by
Situation Awareness
Memory capacity
Intelligent CGFs should be suitable for use in simulations at MoD

4. Application of intelligent CGFs in Embedded Training?

5. Current Scope Tactical mission training
Tactics are techniques for using aircraft and weapons in a combined fashion with the purpose to gain advantage over / defeat the enemy
Air-to-Air
Beyond Visual Range (>10 NM)
Offensive and Defensive Counter Air (picture)
1v1, 2v2, 4v4 engagements

6. Requirements Operational CGFs in the opponent role
should be autonomous
should exhibit credible behaviour
should contribute to training value of simulation
Functional
weapon system functions
human functions
more specific functions per mission phase
planning & briefing
targeting
executing the game plan
self-defence

7. Research facility: NLR’s “Fighter 4-Ship”
One station of the Fighter 4-Ship (Four networked F-16 simulations)

8. F-16 executes OCA mission (Offensive Counter Air) Su-27 executes DCA mission (Defensive Counter Air)
FLOT

9. State-Of-the-Art in CGFs
Scenario-management packages
behaviour of CGFs is “scripted”
pre-defined CGFs lack appropriate weapon and human models
limited possibilities for the use of AI
Agent Qualities
Non-responsive behaviour
Stimulus-Response (S-R) behaviour
Delayed Response (DR) behaviour
motivation-based behaviour
combines S-R and DR behaviour + ‘motivational states’
TAC-AIR SOAR
cognitive architecture deals with observations, decisions and coordination
Order of magnitude: tactical decision rules
Machine Learning techniques

10. Agent Development Approach
Cognitive (BDI) Models
Machine Learning Techniques
Tactical Scenarios
(scripted)
Single Agent - 1v1
Situation Awareness
Evolutionary
techniques
Theory
of Mind
Reinforcement Learning
Multi Agent - 2v2
Decision Making
Neural Networks
Multi Agent - 4v4
Smart Adversary Behaviour

11. Architecture
Simulator – CGF package
Agent-models are functionally separated from simulation environment
Human-like behaviour can be linked to CGFs
Different agent models can run on different machines

12. Cognitive Models
(Team) Situation Awareness
Naturalistic Decision Making
Theory of Mind

13. Example : Situation Awareness
Definition Mica Endsley (1988)
three levels of SA:
the perception of the environment,
the comprehension and integration of information, and
the projection of information into future events.
Translation to “BDI” framework
Perceive: Observations/ Simple beliefs
Understand: Complex beliefs
Anticipate: Future beliefs
Human constraints
belief formation constrained by workload

14. Cognitive model for situation awareness: overview
from Hoogendoorn, van Lambalgen & Treur, 2011

15. Example belief network for SA model
from Hoogendoorn, van Lambalgen & Treur, 2011

16. Reinforcement Learning Experiment

17. Pros and Cons Machine Learning
Save development time (less knowledge elicitation required)
Adaptation to environment and opponent
Complex behaviour in complex domains
New tactics and evaluation of human tactics
Cons
Learning speed
Effectiveness (unpredictable behaviour)
Computation time and memory requirements
Adaptation to game randomness
Increase development time (tweaking)

18. Hybrid models (Dynamic Scripting, Spronck et al., 2005)
Reinforcement learning
Scripts
‘Adaptive game AI with dynamic scripting’, Spronck et al., 2005

19. Conclusions
Cognitive modelling one of the fundamental techniques for motivation-based behaviour CGFs
Machine Learning is powerful tool to:
enhance and complement cognitive models
reduce knowledge elicitation efforts
Smart Bandits: combination of models, utilizing advantages of different approaches

20. Let us know whether you are interested to participate!
Technical Activity Proposal (TAP) Machine Learning Techniques for Battlefield Agents
Exploratory Team under the IST panel
Some topics to be covered:
Current applications of ML
Potential applications in Defence (all Forces)
Potential barriers for application
Most appropriate ML techniques
Systems engineering aspects of ML
3 meetings in 2012, 1st in Amsterdam, early 2012
Leading to a Research Task Group
TAP available!
Let us know whether you are interested to participate!