1. Synthetic Teammate Project March 2009
Jerry Ball
Air Force Research Laboratory

2. Synthetic Teammate Project
Project Goal: Develop a Synthetic Teammate capable of functioning as the Air Vehicle Operator (AVO) or pilot in a 3-person simulation of a Unmanned Air Vehicle (UAV) performing reconnaissance missions
Cognitively Plausible
Using ACT-R
Functional
Large-scale
Empirically Validated
Not valid if it’s not functional!
few research teams
attempting to do these
at once!

3. Synthetic Teammate Project
Guiding principle: Don’t use any computational techniques which are obviously cognitively implausible
Key Assumption: Adhering to well-established cognitive constraints may actually facilitate development by pushing development in directions that are more likely to be successful
Short-term costs associated with adherence to cognitive constraints may ultimately yield long-term benefits
Don’t know what you’re giving up when you adopt cognitively implausible techniques

4. Synthetic Teammate Project
Collaborative project between the Air Force Research Laboratory (AFRL) and Cognitive Engineering Research Institute (CERI)
Applied research funds from AFRL/RHA
Basic research funds from AFOSR
Basic research funds from ONR
Using the Cognitive Engineering Research on Team Tasks (CERTT) Synthetic Task Environment (STE)
Developed with funds from AFOSR

5. CERTT Synthetic Task Environment
Team Goal: Fly UAV
Reconnaissance
Missions
PLO (takes pics)
DEMPC (plans route)
AVO (flies UAV)

6. UAV Reconnaissance Missions
AVO, DEMPC and PLO collaborate to complete a 40 minute reconnaissance mission
AVO must fly UAV past a sequence of waypoints which are determined by the DEMPC and communicated to the AVO as a flight plan
Waypoints may have altitude and airspeed restrictions and have an effective radius for fly by
Route based restrictions, waypoint type and effective radius must be communicated from DEMPC to AVO
Photo restrictions must be communicated from PLO to AVO
PLO must take pictures of target waypoints within the effective radius, but does not take pictures of entry and exit waypoints

7. Importance of Communication
Communication is critical to the success of reconnaissance missions
PLO and DEMPC must communicate restrictions to AVO
DEMPC must communicate flight plan to AVO
When the unexpected happens—e.g. unplanned waypoint added to mission, photo missed—teammates must develop workarounds and communicate adjustments

8. AVO Workstation Instruments Warnings Text Chat
DEMPC to AVO: LVN is our first waypoint
AVO to INTEL: Copy
INTEL to all: OK team, mission 1, good luck.
Text
Chat
Are there any restrictions for LVN?

9. Synthetic Teammate Integration
AVO
Synthetic AVO
Teammate

10. Synthetic Teammate Integration Standalone Mode
Using an agent development framework to provide “light-weight” implementations of the DEMPC and PLO for development purposes
Low-cognitive fidelity, scripted agents
Eliminate need to have humans acting as DEMPC and PLO during development

11. System Overview Text Chat Text Chat Input Output Motor Visual Actions
Dialog
Manager
Text Chat
Input
Language
Comprehension
Language
Generation
Text Chat
Output
Situation Model
Motor
Actions
Visual
Input
Task Behavior Model

12. System Overview Text Chat Text Chat Input Output Motor Visual Actions
Dialog
Manager
Text Chat
Input
Language
Comprehension
Language
Generation
Text Chat
Output
Situation Model
Motor
Actions
Visual
Input
Task Behavior Model

13. Language Comprehension
Theory of Language Processing (Ball 2007…1991)
Activation, selection and integration of constructions corresponding to the linguistic input
Nearly deterministic, serial processing mechanism (integration) operating over a parallel, probabilistic (constraint-based) substrate (activation & selection)
Theory of Linguistic Representation (Ball 2007)
Focus on encoding of referential and relational meaning
Implemented in a Computational Cognitive Model
Using the ACT-R Cognitive Architecture
Adheres to well-established Cognitive Constraints

14. Cognitive Constraints
Incremental processing – word by word
Interactive processing – lexical, syntactic, semantic, pragmatic and task environment information used simultaneously to guide processing
Highly context sensitive – but limited to preceding context (no access to subsequent context)
Word recognition and part-of-speech determination integrated with higher-level syntactic, semantic and discourse processing (single pass)
Robust processing
Must handle ungrammatical input, incorrectly spelled words and non-sentential input
Minimize number of “hard constraints” (e.g. whole word matching) which can lead to failure when they aren’t satisfied

15. Cognitive Constraints  Processing Mechanisms
Serial, nearly deterministic (controlled) processing operating over a parallel, probabilistic (automatic) substrate
Parallel, probabilistic substrate interactively integrates all contextual information leading to selection of the best choice given the available local context at each incremental choice point
Soft constraints or biases
Once a choice is made the processor proceeds serially and deterministically forward in real-time
When a locally preferred choice turns out to be dispreferred in wider context, context sensitive context accommodation mechanism kicks in

16. Language Processing in the Model
The following example is from the Language Processing Model
“no airspeed or altitude restrictions”

17. “no”  object specifier  object referring expression
= nominal construction

18. “airspeed”  object head
no airspeed
integration
Tree structures created from output of model
automatically with a tool for dynamic visualization
of ACT-R declarative memory (Heiberg, Harris & Ball 2007)

19. “airspeed or altitude”  object head
no airspeed or altitude
override
Accommodation
of conjunction via
function overriding

20. “airspeed or altitude”  modifier “restrictions”  object head
no airspeed or altitude restrictions
shift
Appearance of parallel processing!
airspeed or altitude = head vs.
airspeed or altitude = mod
Accommodation
of new head via
function shift

21. Computational Constraints
Processor needs to operate in near real-time to be functional
Large-scale systems that can’t handle non-determinism efficiently (e.g. Context-Free Grammars) typically collapse under their own weight
Deterministic processing is computationally efficient
Probabilistic and Parallel processing—often combined with a limited “spot light”—are alternative mechanisms for dealing with non-determinism
Parallel processing can be computationally explosive on serial hardware
Forced to use some “hard constraints”—e.g. first letter match—in word recognition subcomponent

22. Computational Constraints
No limited domain assumption to simplify model
CERTT text chat shows broad range of grammatical constructions and thousands of lexical items
Relational database integrated with ACT-R to support scaling up model to a full mental lexicon
Plan to integrate sizeable subset ( > 15,000 lexical items) of most common words in WordNet lexicon ( > 100,000 lexical items)
Can’t ignore lexical ambiguity!
Study underway to compare performance of model when Declarative Memory (DM) is stored in an external DB vs. internal Lisp process
Internal Lisp process is faster for small DM, but can only handle 30% of WordNet before running out of memory!

23. Start with a Domain General Language Processing System
Contains 2000 most common words in English and 2500 words in total
Handles a broad range of construction types
Declarative, Imperative, Yes-No Question, Wh-Question
Intransitive, Transitive & Ditransitive Verbs, Verbs with Clausal Complements, Predicate Nominals, Predicate Adjectives and Predicate Prepositions
Specifier, Head, Complement, Pre- and Post-Head Modifier
Conjunctions of numerous functional categories
Relative Clauses, Wh-Clauses, Infinitive, -ing, -en & Bare Verb Clauses
Long-distance dependencies
Passive constructions

24. Start with a Domain General Language Processing System
Representations are in the spirit of the “Simpler Syntax” of Culicover & Jackendoff (2005) except that there are no purely syntactic representations
Referring Expression
Functional
Categories
Predicates
Semantic Features
Trace bound
to subject
He is eager to please.

25. Extend to Handle Scripted Comm
AVO: DEMPC, please let me know the first waypoint!
DEMPC: The first waypoint is LVN. It’s an entry point. There are no airspeed or altitude restrictions. The effective radius is 2.5 miles.
AVO: PLO, I’m heading towards LVN.
DEMPC: We’re within the effective radius so go to the second waypoint.
AVO: Are there any altitude or airspeed restrictions for the second waypoint?
DEMPC: The second waypoint is H-AREA. It’s a target. The airspeed restriction is between 50 and 200 knots. There is no altitude restriction. The effective radius is 5 miles.
PLO: AVO, please keep the altitude over 3000 feet for the photo!
PLO: I have a good photo of H-AREA.

26. Scripted Comm Full sentences Correct spelling Explicit discourse acts
Still lots of variability
Declarative sentences
Imperative sentences
Questions
Conjunctions

27. Extend to Handle Text Chat for a 40 Minute Mission – without editing!
PLO to AVO: avo-don't ever proceed from a target if i haven't taken the picture
AVO to PLO: ok -- keep me in the loop!
INTEL to all: ok team, mission 2
PLO to AVO: effective radiu
PLO to AVO: avo i need to be below 3000
AVO to PLO: copy, will 2000 do?
DEMPC to AVO: LVN is our 1st entry point with a radius of 2.5
AVO to PLO: speed?
AVO to DEMPC, PLO: 1 mile out/ 30 seconds
PLO to AVO: i don't have a speed for lvn so go faster
AVO to DEMPC, PLO: speed 340
PLO to AVO: avo i'll need to be above 3000 for h area
AVO to PLO: above 3000 copy -- can we proceed to h-area yet?

28. Extend to Handle Text Chat for a 40 Minute Mission – without editing!
PLO to AVO: lets get out of effective zone
DEMPC to AVO: Speed=50-200, Altitude=
AVO to DEMPC, PLO: wait -- my flight plan changed -- are we going to Z1?
PLO to AVO: can yougo faster yet or is it stll 200
DEMPC to AVO: no speed or alt. restrictions
PLO to AVO: avo i need to be above 3000 for s ste- go there when you think it would be most effective
PLO to AVO: avo 3000
DEMPC to AVO: YES to S-StE=Target
PLO to AVO: `avo get back within 5 miles of s ste
PLO to AVO: aavo dont slow down

29. Handle Communication with Unscripted Human DEMPC and PLO
Language varies significantly from team to team
Can’t predict vocabulary requirements in advance
Teams adapt particular ways of communicating which can’t be predicted in advance
Text becomes more cryptic as mission continues
Discourse acts are often implicit

30. Word Recognition Subcomponent
Word recognition subcomponent largely compatible with the E-Z Reader model of reading (cf. Reichle, Warren & McConnell 2009) with extensions to support higher-level language processing
Perceptual window used for low-level processing of linguistic input
Model can “see” space delimited “word” in focus of attention
Model can “see” up to first 3 letters of word in right periphery following space
Retrieved word is verified against actual input
Consistent with Activation-Verification model of Word Recognition (Paap et al. 1982)

31. Word Recognition
Word recognition is an interaction between low-level perceptual and higher-level cognitive processing
Perceptually identified letters, trigrams and space delimited “words” spread activation to words (and multi-word units) in DM
Most-highly activated word or multi-word unit consistent with retrieval template is retrieved
Need not be a space delimited “word”

32. Generating Linguistic Representations
Incremental, interactive generation of linguistic representations which encode referential and relational meaning
Referring Expressions
Relations
He is eager to please.

33. Mapping into the Situation Model
Referring expressions in the linguistic representation get mapped to objects and situations in the situation model
Indefinite object referring expression typically introduces a new object into the situation model
Definite object referring expression typically identifies and existing object either in the situation model or salient in the context
Situation referring expressions typically introduce a new relation into the situation

34. System Overview Text Chat Text Chat Input Output Motor Actions Visual
Dialog
Manager
Text Chat
Input
Language
Comprehension
Language
Generation
Text Chat
Output
Motor
Actions
Situation Model
Visual
Input
Task Behavior Model

35. Centrality of Situation Model
Task Behavior
World Knowledge
Situation
Model
Language Output
Language Input
Language
Knowledge
Task Input
Domain
Knowledge

36. Situation Model Situation Model (Zwann & Radvansky, 1998)
Spatial-Imaginal (and Temporal) representation of the objects and situations described by linguistic expressions and encoded directly from the environment
Non-propositional (at least in part)
Non-textual
No available computational implementations
Provides grounding for linguistic representations
Integrates task environment and linguistic information

37. Abstract Concepts vs. Perceptually Grounded Language
The Prevailing View
An Emerging View
Real World
Mental Box
Real World
Mental Box
“pilot”
perception
Language
of Thought
“pilot”
“pilot”
grounding
PILOT
Explicit
(Perceptual)
Implicit
(Abstract)
perception

38. Abstract Concepts vs. Perceptually Grounded Language
The Prevailing View
An Emerging View
Real World
Mental Box
Real World
Mental Box
“pilot”
perception
Language
of Thought
“pilot”
“pilot”
grounding
PILOT
Explicit
(Perceptual)
Implicit
(Abstract)
perception

39. Situation Model Propositional Content
Planning to use Hobbs’ theory of “ontological promiscuity” and his well-developed logical notation (translated into ACT-R chunks) to represent propositional content
The logical notation should be as close to English as possible
The logical notation should be syntactically simple to support inferencing

40. Situation Model Spatial Content
Planning to use Scott Douglass’ spatial module extension to ACT-R which implements a matrix-like representation of spatial information
Discourse Content
Working on identification and representation of Discourse Acts which are often only implied in linguistic input
“I need to be above 3000 feet for the photo”
This is a request to increase the altitude of the UAV (human is not actually in UAV)

41. Empirical Validation
Experiment conducted with human subjects in conditions using 1) spoken language and 2) text chat to provide data for model development
AVO station moved into separate room so DEMPC and PLO don’t see AVO
Text chat condition showed team performance effect similar to spoken language condition
Goal is to conduct an experiment with Synthetic AVO Teammate interacting with human DEMPC and PLO

42. Questions?