1. Machine Learning Approaches to Cognitive Parameter Acquisition Terran Lane University of New Mexico terran@cs.unm.edu Chris Forsythe, Patrick Xavier Sandia National Labs {jcforsy,pgxavie}@sandia.gov

2. Sandias Cognitive Modeling Framework Computational models of human decision- makers Models attention, perceptual cues, situational awareness, decision making Based on oscillatory models of activation Spreading activation networks and feedback loops between functional elements Applications -- data analysis, security, tutoring… Bottleneck: models hand-built/tuned Expensive and slow!

3. The Big Picture World Cue 0 0 Cue 1 1 Cue N N Situation 0 Situation 1 Situation M Actions/ Decisions 01 10 N1 NM

4. Automated Model Acquisition High predictive accuracy 87% correct prediction of operators interpretation of scenario (incl. relevance) 91% correct in recognizing situation only Insights into operator decision-making process Models are task & user specific Only 26% overlap between users Large effort in building and tuning models Project goal: (semi-)automate acquisition of parameters, network topologies, etc. Prediction accuracy secondary concern

5. Roles for Machine Learning Parameter acquisition Interconnection weights Activation levels Oscillator frequencies Network topologies Inter-cue spreading activation network Cue situation relations Feedbacks Cues and situation identification

6. Parameter Acquisition World Cue 0 0 Cue 1 1 Cue N N Situation 0 Situation 1 Situation M Actions/ Decisions 01 10 N1 NM

7. Parameter Acquisition: Issues Superficially supervised learning Observe features/cues and operator actions; induce params (find s.t. f :C A) Similar to ANN backprop, EM, etc. Many effective, well understood techniques Problem: not just high-likelihood params Actually want params used by human operator Much harder – observable stimuli dont directly reflect operators internal state Cognitive plausibility constraint

8. Parameter Acquisition: Approaches Additional instrumentation Measure characteristics of operator Biometrics – eye tracking, MEG, etc. Expensive, not widespread Maybe not informative to params anyway Utility elicitation techniques Software queries user about why decisions were made / state of attention Picks questions to maximally improve model Emulates expert knowledge engineer

9. Network Topology Induction World Cue 0 0 Cue 1 1 Cue N N Situation 0 Situation 1 Situation M Actions/ Decisions 01 10 N1 NM

10. Topology Induction: Issues Find structure of interconnections between variables (I.e., cues, situations) Much harder than parameter acquisition Formally, maximum likelihood/MAP search through all possible networks

11. Topology Induction: Issues Find structure of interconnections between variables (I.e., cues, situations) Much harder than parameter acquisition Formally, maximum likelihood/MAP search through all possible networks L=137

12. Topology Induction: Issues Find structure of interconnections between variables (I.e., cues, situations) Much harder than parameter acquisition Formally, maximum likelihood/MAP search through all possible networks L=137L=238

13. Topology Induction: Issues Find structure of interconnections between variables (I.e., cues, situations) Much harder than parameter acquisition Formally, maximum likelihood/MAP search through all possible networks L=137L=238L=493

14. Topology Induction: Issues Find structure of interconnections between variables (I.e., cues, situations) Much harder than parameter acquisition Formally, maximum likelihood/MAP search through all possible networks L=137L=238L=493L=318

15. Topology Induction: Issues Find structure of interconnections between variables (I.e., cues, situations) Much harder than parameter acquisition Formally, maximum likelihood/MAP search through all possible networks L=137L=238L=493L=318

16. Topology Induction: Approaches Principles of structure search well understood Gradient ascent, annealing, genetic search, constrained search, etc. Difficult in practice Computationally intractable Resulting models very sensitive to data Spurious likelihood spikes low confidence models Compounded by cognitive plausibility constraint Can get leverage from cognitive plausibility, though

17. Cue and Situation Identification World Cue 0 0 Cue 1 1 Cue N N Situation 0 Situation 1 Situation M Actions/ Decisions 01 10 N1 NM

18. Cue and Situation Identification: Issues Discern cues and whole environmental situations employed by user Related to constructive feature induction, nonlinear projection identification, relational learning, etc. Search across all possible nodes/relations N=2N=3

19. Cue and Situations: Approaches Cutting-edge ML problem Direct elicitation is probably most promising approach Formulating search space/uncertainty reduction not straightforward Even user interface is difficult (naming synthetic nodes/relations)

20. Conclusions Decrease time/effort/cost to construct and tune cognitive model Constrained to correspond to humans internal model Both bane and boon to automated model construction Insights into operators mental state/decision- making process Requires/drives novel ML algorithms Future work: all of it…

21. Questions?