Presentation is loading. Please wait.

Presentation is loading. Please wait.

A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State.

Similar presentations

Presentation on theme: "A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State."— Presentation transcript:

1 A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State University 5 May 2010

2 Background Education: Cognitive Psychology/Human Factors George Mason University, B.A. New Mexico State University, M.A., Ph.D. Positions Rice University New Mexico State University Arizona State University & Cognitive Engineering Research Institute Applied Experience: U.S Air Force, Navy, Army, NASA, NTSB, VA Section Editor, Human Factors USAF Scientific Advisory Board National Research Council Committee on Human Systems Integration Relevant Research Team Cognition Military, Cyber, and Medical Applications Communication Analysis Metrics for Coordination and Collaboration Sponsors Air Force Office of Scientific Research Air Force Research Laboratory Office of Naval Research Army Research Office Leonard Wood Institute Veterans Administration – MWM VERC Cookes Background

3 Overview MURI and ASU Team Team Cognition and Team Situation Awareness Other Team Testbeds CyberCog – New Testbed

4 MURI: Computer-aided Human Centric Cyber Situation Awareness DoD Multidisciplinary University Research Initiative (MURI) program project, funded through Army Research Office Two fundamental limitations of Cyber Situation Awareness (C-SA) Gap: human cognition C-SA tools – Situation data exceeds cognitive throughput of human analysts Blind spots in views of cyber situation for existing C-SA tools (including auditing, vulnerability scanners, attack graph tools, intrusion detection systems, damage assessment tools, and forensics tools) Cyber-SA Vision Build data human decision links through innovations – knowledge fusion – cognitive automation – artificial intelligence – visual analytics Awareness-driven cyber defense vs. malware behavior dependent defense Automatic blind spot identification and monitoring techniques

5 MURI Partners Professor Peng Liu, Penn State University, Overall PI Professor Nancy Cooke, Arizona State University Professor Coty González, Carnegie Mellon University Professor Dave Hall, Penn State University Professor Sushil Jajodia, George Mason University Professor Mike McNeese, Penn State University Professor Peng Ning, NC State University Professor VS Subrahmanian, Univ. of Maryland Professor John Yen, Penn State University Professor Michael Young, NC State University

6 ASU MURI Team Nancy J. Cooke Professor, Cognitive Science & Engineering College of Technology and Innovation Prashanth Rajivan Graduate Student Masters in Computing Studies College of Technology Innovation Shankaranarayanan Venkatanarayanan Graduate Student Masters in Computing Studies College of Technology and Innovation

7 Teams and Cognitive Tasks Team is unit of analysis = Heterogeneous and interdependent group of individuals (human or synthetic) who plan, decide, perceive, design, solve problems, and act as an integrated system. Cognitive activity at the team level= Team Cognition Improved team cognition Improved team/system effectiveness Heterogeneous = differing backgrounds, differing perspectives on situation (surgery, basketball)

8 Unmanned Aerial Vehicles USS Vincennes shoots down Iranian airbus (1988) Challenger/Columbia accidents tied to poor organizational decision making (1986/2003) Response to 9/11 reveals communication breakdowns (2001) Katrina response lacked coordination (2005) Sago Mine disaster report cites poor command-and-control (2006) VA Tech communications substandard (2007) Friendly fire incidents Various health care mishaps attributed to poor teamwork Some Instances of Failures of Team Cognition

9 Miracle on the Hudson And some successes… Response to Fargo flooding

10 Interactive Team Cognition in a Nutshell Team interactions often in the form of explicit communications are the foundation of team cognition ASSUMPTIONS 1)Team cognition is an activity; not a property or product 2)Team cognition is inextricably tied to context 3)Team cognition is best measured and studied when the team is the unit of analysis

11 US 2004 Olympic Basketball Team " We still have a couple of days, but I don't know where we are," replied USA head coach Larry Brown to a question Wednesday on where his team was in its preparations. "We have good moments and bad, but I've got a pretty good understanding of who needs to play. Now the job is to get an understanding of how we have to play." A team of experts does NOT make an expert team Collaborative skill is not additive

12 US 1980 Olympic Ice Hockey Team Herb Brooks and 20 young no-names won the 1980 Olympic Gold Medal in Ice Hockey An expert team made up of no-names…

13 In our UAV STE three operators must coordinate over headsets in order to maneuver their UAV to take pictures of ground targets Our UAV Testbed UAV-STE: Uninhabited Air Vehicle (ground control station) Synthetic Task Environment for research on team cognition (DURIP 1997; USAF funded)

14 Air Vehicle Operator controls UAV airspeed, heading, and altitude and monitors air vehicle systems Payload Operator controls camera settings, takes photos, and monitors camera systems DEMPC navigator, mission planner, plans route from target to target under constraints Interdependence requires interaction, communication, & coordination Three team members with inter- dependent tasks

15 Our MacroCog (Macro-Cognition Testbed) MacroCog Testbed Navy-funded lab for strategic planning and decision- making in the context of noncombatant evacuation operations

16 MacroCog Roles in Current Experiment Information Warfare Specialist Personnel Specialist: Military Equipment Specialist: Land/Sea Vehicles Personnel Specialist: Humanitarian Equipment Specialist: Air Vehicles Experimenter 1 Experimenter 2

17 Example of Empirical Results on Team Cognition As teams acquire experience, performance improves, interactions improve, but not individual or collective knowledge Individuals are trained to criterion prior to M1 Asymptotic team performance after 4 40-min missions (robust finding) Knowledge changes tend to occur in early learning (M1) and stabilize Process improves and communication becomes more standard over time 40-min missions Spring Break

18 Team Situation Awareness A teams coordinated perception and action in response to a change in the environment How can we exercise team SA in a testbed? How can we measure it? How can we intervene to improve it? Contrary to view that all team members need to be on the same page

19 What is Meant by Coordinated Perception and Action?

20 Measure of Team Situation Awareness Change is introduced (communication breakdown, enemy in area, storm) that will impact mission 2-3 team members are presented cues regarding change Team members need to perceive cues in a coordinated way (i.e., connect the dots) to identify the change Team members coordinate to take action relevant to the change (e.g., change altitude, communicate indirectly) Measure in terms of outcome and process – who on team was involved?

21 CyberCog Simulator Web based Simulator application for measuring individual interaction and team collaboration (e.g., team situation awareness) in a Cyber security analysis situation

22 CyberCogSimulator – System Overview

23 CyberCogSimulator – Components Cyber Security Analyst (User) – Assigned a specific role such as Denial of Service (Dos) specialist, Malware specialist and Phishing specialist – Understands the scenario given, use events and attack symptoms, collaborates with other participants to identify a potential attack or a combination of attacks – The team reaches a common consensus on the type of attack and its corresponding events

24 CyberCogSimulator – Components Master controller and Evaluator – Queries attack scenarios, events and symptoms from the database – Distributes the events and symptoms to the participants – Logs the interaction between participants at real time – Evaluates and scores the participants findings with the expected results

25 CyberCogSimulator – Components Database server – MySQL database server stores :- Attack Scenarios Events corresponding to attack scenarios including some false positives & noise events Attack Symptoms for each specialization (E.g., Dos, Malware, Phishing) identified The expected results, interaction (between participants ) logs and attack conclusion arrived at by each team for each session

26 User and Team Views Legends Functions Data

27 CyberCog Simulator- Interaction

28 CyberCogSimulator- Architecture Microsoft IIS Database Intra/Internet Malware Specialist Phishing Specialist Dos Specialist Client Tier Controller & View Tier Controller & View Tier POCOs Web Services Model Tier

29 Conclusion There are current gaps and limitations in Cyber Situation Awareness Cyber situation awareness by teams involves the coordinated perception and action in the face of a change in the cyber situation CyberCog will allow the MURI team and others to better understand team-based cyber SA and to test algorithms and tools developed for improving it

30 Team Cognition Research Program Testbeds: 1) UAS C2 2) Navy Strategic Planning Empirical Studies in Testbed UAS Field Data Theory Development ACT-R Model of Synthetic Teammate Dynamical Systems Modeling Measures

Download ppt "A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State."

Similar presentations

Ads by Google