Presentation on theme: "Synthetic Teammate Project March 2009"— Presentation transcript:
1 Synthetic Teammate Project March 2009 Jerry BallAir Force Research Laboratory
2 Synthetic Teammate Project Project Goal: Develop a Synthetic Teammate capable of functioning as the Air Vehicle Operator (AVO) or pilot in a 3-person simulation of a Unmanned Air Vehicle (UAV) performing reconnaissance missionsCognitively PlausibleUsing ACT-RFunctionalLarge-scaleEmpirically ValidatedNot valid if it’s not functional!few research teamsattempting to do theseat once!
3 Synthetic Teammate Project Guiding principle: Don’t use any computational techniques which are obviously cognitively implausibleKey Assumption: Adhering to well-established cognitive constraints may actually facilitate development by pushing development in directions that are more likely to be successfulShort-term costs associated with adherence to cognitive constraints may ultimately yield long-term benefitsDon’t know what you’re giving up when you adopt cognitively implausible techniques
4 Synthetic Teammate Project Collaborative project between the Air Force Research Laboratory (AFRL) and Cognitive Engineering Research Institute (CERI)Applied research funds from AFRL/RHABasic research funds from AFOSRBasic research funds from ONRUsing the Cognitive Engineering Research on Team Tasks (CERTT) Synthetic Task Environment (STE)Developed with funds from AFOSR
6 UAV Reconnaissance Missions AVO, DEMPC and PLO collaborate to complete a 40 minute reconnaissance missionAVO must fly UAV past a sequence of waypoints which are determined by the DEMPC and communicated to the AVO as a flight planWaypoints may have altitude and airspeed restrictions and have an effective radius for fly byRoute based restrictions, waypoint type and effective radius must be communicated from DEMPC to AVOPhoto restrictions must be communicated from PLO to AVOPLO must take pictures of target waypoints within the effective radius, but does not take pictures of entry and exit waypoints
7 Importance of Communication Communication is critical to the success of reconnaissance missionsPLO and DEMPC must communicate restrictions to AVODEMPC must communicate flight plan to AVOWhen the unexpected happens—e.g. unplanned waypoint added to mission, photo missed—teammates must develop workarounds and communicate adjustments
8 AVO Workstation Instruments Warnings Text Chat DEMPC to AVO: LVN is our first waypointAVO to INTEL: CopyINTEL to all: OK team, mission 1, good luck.TextChatAre there any restrictions for LVN?
10 Synthetic Teammate Integration Standalone Mode Using an agent development framework to provide “light-weight” implementations of the DEMPC and PLO for development purposesLow-cognitive fidelity, scripted agentsEliminate need to have humans acting as DEMPC and PLO during development
11 System Overview Text Chat Text Chat Input Output Motor Visual Actions DialogManagerText ChatInputLanguageComprehensionLanguageGenerationText ChatOutputSituation ModelMotorActionsVisualInputTask Behavior Model
12 System Overview Text Chat Text Chat Input Output Motor Visual Actions DialogManagerText ChatInputLanguageComprehensionLanguageGenerationText ChatOutputSituation ModelMotorActionsVisualInputTask Behavior Model
13 Language Comprehension Theory of Language Processing (Ball 2007…1991)Activation, selection and integration of constructions corresponding to the linguistic inputNearly deterministic, serial processing mechanism (integration) operating over a parallel, probabilistic (constraint-based) substrate (activation & selection)Theory of Linguistic Representation (Ball 2007)Focus on encoding of referential and relational meaningImplemented in a Computational Cognitive ModelUsing the ACT-R Cognitive ArchitectureAdheres to well-established Cognitive Constraints
14 Cognitive Constraints Incremental processing – word by wordInteractive processing – lexical, syntactic, semantic, pragmatic and task environment information used simultaneously to guide processingHighly context sensitive – but limited to preceding context (no access to subsequent context)Word recognition and part-of-speech determination integrated with higher-level syntactic, semantic and discourse processing (single pass)Robust processingMust handle ungrammatical input, incorrectly spelled words and non-sentential inputMinimize number of “hard constraints” (e.g. whole word matching) which can lead to failure when they aren’t satisfied
15 Cognitive Constraints Processing Mechanisms Serial, nearly deterministic (controlled) processing operating over a parallel, probabilistic (automatic) substrateParallel, probabilistic substrate interactively integrates all contextual information leading to selection of the best choice given the available local context at each incremental choice pointSoft constraints or biasesOnce a choice is made the processor proceeds serially and deterministically forward in real-timeWhen a locally preferred choice turns out to be dispreferred in wider context, context sensitive context accommodation mechanism kicks in
16 Language Processing in the Model The following example is from the Language Processing Model“no airspeed or altitude restrictions”
18 “airspeed” object head no airspeedintegrationTree structures created from output of modelautomatically with a tool for dynamic visualizationof ACT-R declarative memory (Heiberg, Harris & Ball 2007)
19 “airspeed or altitude” object head no airspeed or altitudeoverrideAccommodationof conjunction viafunction overriding
20 “airspeed or altitude” modifier “restrictions” object head no airspeed or altitude restrictionsshiftAppearance of parallel processing!airspeed or altitude = head vs.airspeed or altitude = modAccommodationof new head viafunction shift
21 Computational Constraints Processor needs to operate in near real-time to be functionalLarge-scale systems that can’t handle non-determinism efficiently (e.g. Context-Free Grammars) typically collapse under their own weightDeterministic processing is computationally efficientProbabilistic and Parallel processing—often combined with a limited “spot light”—are alternative mechanisms for dealing with non-determinismParallel processing can be computationally explosive on serial hardwareForced to use some “hard constraints”—e.g. first letter match—in word recognition subcomponent
22 Computational Constraints No limited domain assumption to simplify modelCERTT text chat shows broad range of grammatical constructions and thousands of lexical itemsRelational database integrated with ACT-R to support scaling up model to a full mental lexiconPlan to integrate sizeable subset ( > 15,000 lexical items) of most common words in WordNet lexicon ( > 100,000 lexical items)Can’t ignore lexical ambiguity!Study underway to compare performance of model when Declarative Memory (DM) is stored in an external DB vs. internal Lisp processInternal Lisp process is faster for small DM, but can only handle 30% of WordNet before running out of memory!
23 Start with a Domain General Language Processing System Contains 2000 most common words in English and 2500 words in totalHandles a broad range of construction typesDeclarative, Imperative, Yes-No Question, Wh-QuestionIntransitive, Transitive & Ditransitive Verbs, Verbs with Clausal Complements, Predicate Nominals, Predicate Adjectives and Predicate PrepositionsSpecifier, Head, Complement, Pre- and Post-Head ModifierConjunctions of numerous functional categoriesRelative Clauses, Wh-Clauses, Infinitive, -ing, -en & Bare Verb ClausesLong-distance dependenciesPassive constructions
24 Start with a Domain General Language Processing System Representations are in the spirit of the “Simpler Syntax” of Culicover & Jackendoff (2005) except that there are no purely syntactic representationsReferring ExpressionFunctionalCategoriesPredicatesSemantic FeaturesTrace boundto subjectHe is eager to please.
25 Extend to Handle Scripted Comm AVO: DEMPC, please let me know the first waypoint!DEMPC: The first waypoint is LVN. It’s an entry point. There are no airspeed or altitude restrictions. The effective radius is 2.5 miles.AVO: PLO, I’m heading towards LVN.DEMPC: We’re within the effective radius so go to the second waypoint.AVO: Are there any altitude or airspeed restrictions for the second waypoint?DEMPC: The second waypoint is H-AREA. It’s a target. The airspeed restriction is between 50 and 200 knots. There is no altitude restriction. The effective radius is 5 miles.PLO: AVO, please keep the altitude over 3000 feet for the photo!PLO: I have a good photo of H-AREA.
26 Scripted Comm Full sentences Correct spelling Explicit discourse acts Still lots of variabilityDeclarative sentencesImperative sentencesQuestionsConjunctions
27 Extend to Handle Text Chat for a 40 Minute Mission – without editing! PLO to AVO: avo-don't ever proceed from a target if i haven't taken the pictureAVO to PLO: ok -- keep me in the loop!INTEL to all: ok team, mission 2PLO to AVO: effective radiuPLO to AVO: avo i need to be below 3000AVO to PLO: copy, will 2000 do?DEMPC to AVO: LVN is our 1st entry point with a radius of 2.5AVO to PLO: speed?AVO to DEMPC, PLO: 1 mile out/ 30 secondsPLO to AVO: i don't have a speed for lvn so go fasterAVO to DEMPC, PLO: speed 340PLO to AVO: avo i'll need to be above 3000 for h areaAVO to PLO: above 3000 copy -- can we proceed to h-area yet?
28 Extend to Handle Text Chat for a 40 Minute Mission – without editing! PLO to AVO: lets get out of effective zoneDEMPC to AVO: Speed=50-200, Altitude=AVO to DEMPC, PLO: wait -- my flight plan changed -- are we going to Z1?PLO to AVO: can yougo faster yet or is it stll 200DEMPC to AVO: no speed or alt. restrictionsPLO to AVO: avo i need to be above 3000 for s ste- go there when you think it would be most effectivePLO to AVO: avo 3000DEMPC to AVO: YES to S-StE=TargetPLO to AVO: `avo get back within 5 miles of s stePLO to AVO: aavo dont slow down
29 Handle Communication with Unscripted Human DEMPC and PLO Language varies significantly from team to teamCan’t predict vocabulary requirements in advanceTeams adapt particular ways of communicating which can’t be predicted in advanceText becomes more cryptic as mission continuesDiscourse acts are often implicit
30 Word Recognition Subcomponent Word recognition subcomponent largely compatible with the E-Z Reader model of reading (cf. Reichle, Warren & McConnell 2009) with extensions to support higher-level language processingPerceptual window used for low-level processing of linguistic inputModel can “see” space delimited “word” in focus of attentionModel can “see” up to first 3 letters of word in right periphery following spaceRetrieved word is verified against actual inputConsistent with Activation-Verification model of Word Recognition (Paap et al. 1982)
31 Word RecognitionWord recognition is an interaction between low-level perceptual and higher-level cognitive processingPerceptually identified letters, trigrams and space delimited “words” spread activation to words (and multi-word units) in DMMost-highly activated word or multi-word unit consistent with retrieval template is retrievedNeed not be a space delimited “word”
32 Generating Linguistic Representations Incremental, interactive generation of linguistic representations which encode referential and relational meaningReferring ExpressionsRelationsHe is eager to please.
33 Mapping into the Situation Model Referring expressions in the linguistic representation get mapped to objects and situations in the situation modelIndefinite object referring expression typically introduces a new object into the situation modelDefinite object referring expression typically identifies and existing object either in the situation model or salient in the contextSituation referring expressions typically introduce a new relation into the situation
34 System Overview Text Chat Text Chat Input Output Motor Actions Visual DialogManagerText ChatInputLanguageComprehensionLanguageGenerationText ChatOutputMotorActionsSituation ModelVisualInputTask Behavior Model
35 Centrality of Situation Model Task BehaviorWorld KnowledgeSituationModelLanguage OutputLanguage InputLanguageKnowledgeTask InputDomainKnowledge
36 Situation Model Situation Model (Zwann & Radvansky, 1998) Spatial-Imaginal (and Temporal) representation of the objects and situations described by linguistic expressions and encoded directly from the environmentNon-propositional (at least in part)Non-textualNo available computational implementationsProvides grounding for linguistic representationsIntegrates task environment and linguistic information
37 Abstract Concepts vs. Perceptually Grounded Language The Prevailing ViewAn Emerging ViewReal WorldMental BoxReal WorldMental Box“pilot”perceptionLanguageof Thought“pilot”“pilot”groundingPILOTExplicit(Perceptual)Implicit(Abstract)perception
38 Abstract Concepts vs. Perceptually Grounded Language The Prevailing ViewAn Emerging ViewReal WorldMental BoxReal WorldMental Box“pilot”perceptionLanguageof Thought“pilot”“pilot”groundingPILOTExplicit(Perceptual)Implicit(Abstract)perception
39 Situation Model Propositional Content Planning to use Hobbs’ theory of “ontological promiscuity” and his well-developed logical notation (translated into ACT-R chunks) to represent propositional contentThe logical notation should be as close to English as possibleThe logical notation should be syntactically simple to support inferencing
40 Situation Model Spatial Content Planning to use Scott Douglass’ spatial module extension to ACT-R which implements a matrix-like representation of spatial informationDiscourse ContentWorking on identification and representation of Discourse Acts which are often only implied in linguistic input“I need to be above 3000 feet for the photo”This is a request to increase the altitude of the UAV (human is not actually in UAV)
41 Empirical ValidationExperiment conducted with human subjects in conditions using 1) spoken language and 2) text chat to provide data for model developmentAVO station moved into separate room so DEMPC and PLO don’t see AVOText chat condition showed team performance effect similar to spoken language conditionGoal is to conduct an experiment with Synthetic AVO Teammate interacting with human DEMPC and PLO