ELICIT Experimental Laboratory for Investigating Collaboration, Information-sharing, and Trust

2 What is ELICIT? ELICIT = Experimental Laboratory for Investigating Collaboration, Information-sharing, and Trust U.S. DoD (OASD/NII) Command and Control Research Program (CCRP) sponsored the design and development of the ELICIT platform for experimentation focused on information, cognitive, and social domain phenomena Purpose of ELICIT-related Experimentation and Analysis is to investigate the cognitive and social impacts of C2 approach and organizational structure (e.g. information sharing, trust, shared awareness, and task performance) Initial applications focus on a comparison of traditional hierarchical and edge C2 approaches

3 ELICIT Scenario The goal of each set of participants is to build situational awareness and identify the who, what, when, and where of a pending attack –Participants can share factoids directly with each other or post factoids to websites –Participants build awareness by gathering and analyzing factoids and interacting with one another Participants receive factoids about a future attack –Factoids fall into four task categories: who, what, when, and where –Factoids are periodically distributed to the participants –No one is given sufficient information to solve their assigned problem without receiving information from others The receiving, sharing, and posting of factoids and the nature of the interactions between and among participants can be constrained

4 Experiment Details Sixty-eight (68) total factoids: –17 when; 17 where; 17 who; 17 what –Factoids may be key, supportive, or nonessential Factoids are distributed in three waves –34 factoids at start –17 factoids at 5 minutes –17 factoids at 10 minutes C2 approach for this series of experiments were designated prior to the start of the run as –Hierarchy or Edge 17 participants make up an organization Output: –Transaction log –Scratch paper collected –Survey conducted at the conclusion of the run Hierarchy Edge

5 Website Access The commander (gray) can access/post to all four websites All other participants can only access/post their group’s website Each participant can access/post to all four websites HierarchyEdge Access to all websites Where When Who What Team Member Website

6 Illustrative Factoids and Identifies Who Factoid: The Lion is known to work only with the Azur, Brown, or Violet groups (classification – key fact) What Factoid: Bloggers are discussing the role of financial institutions in oppressing the Coral, Violet and Chartreuse groups (classification – supportive fact) When Factoid: The Brown group needs time to regroup (classification – nonessential fact) Where Factoid: The Azur, Brown, Coral, and Violet groups have the capacity to operate in Tau, Epsilon, Chi, Psi and Omega-lands (classification – supportive fact) Sample Identify: "who=Red Group; what=TV station; where=B-land; when=Nov 8, 10:00 pm“ Sample Correct Solution: The Green group plans to attack a TV station in A-land on 25 Nov at 11AM

7 Variables of Interest Info Sharing & Collaborative Behaviors Shared Information Quality of Information Shared Awareness Quality of Awareness Shared Understanding Quality of Understanding Task Performance Task Difficulty Measures of Merit Network Characteristics & Performance Individual & Team Characteristics Culture Allocation of Decision Rights Quality of Information Sources Patterns of Interaction Distribution of Information C2 Maturity Level Partially Controllable Controllable Legend

8 Dependent Variables MOE = Quality of Awareness and Shared Awareness –Correctness (Authorized Correct IDs) –Timeliness (Person-Minutes with Correct IDs) –Accuracy rate (Correct IDs/Total IDs) Efficiency, Given Effectiveness –Productivity (Correct IDs/Total Actions; Correct IDs/Person- Minutes Available) –Speed (Time of Earliest Correct ID) Agility –Effectiveness over problem difficulty

9 C2 Approach Independent Variables Hierarchy v. Edge –We expect Hierarchy to map to De-conflicted and Edge to map to a more mature level –Each run will be mapped to a point in the C2 Approach Space based on observed behaviors Rules of Interaction –Website access –Sharing permissions Initial Distribution of Factoids –Invariant in existing runs

10 C2 Approach Intervening Variables Patterns of Interaction –Characteristic path length –Clustering coefficient –Connectedness Distribution of Information –The average number of unique facts to which each participant has access as a function of time

11 Measures of C2 Effectiveness (MOCE) Quality of Information Position –Percentage of relevant facts for the assigned task that a participant can access as a function of time –Percentage of key facts for the assigned task that a participant can access as a function of time Extent of Shared Information –The average number of participants that have access to each fact as a function of time –The average number of participants that have access to each key fact as a function of time

12 Intervening Behavioral Variables Activity over time (sharing, website posts, website pulls, ID attempts) –Sharing Peer-to-peer sharing Posting –Information Seeking Pulling –Identification Attempts

13 Other Independent Variables IDs allowed ELICIT experience of the player Factoid set (problem difficulty) Translated factoids v. original Native Language (English v. Other) Communications media –Postcards –Chat Time available Degree of Education (Graduate, Undergraduate) Seniority (Rank) Subcultures (Military, Civilian, Special Forces, Civil Servants)

14 ELICIT Data Set Includes data from 37 ELICIT experimentation trials Venues –Boston University (2 runs) –Naval Postgraduate School (16 runs) –Portugal (6 runs) –US Military Academy (3 runs) –Singapore (10 runs) Organization Types –18 Edge –19 Hierarchy Past excursions include –Chat capability –Varying the subjects (Intel, Military, PhD) Future use planned for Germany, UK, SAS-065, NDU, AWC

15 Extracting Data from ELICIT ELICIT generates output in a number of forms –Participant survey responses and “scratch paper” –Transaction logs recording all information exchange and identify actions While rich in data, transaction logs must be processed in order to be formulated for analysis –Transactions of each type must be extracted and tabulated –Identify attempts must be graded for correctness Scripts (Python, C++) and macros (Visual Basic) were employed to facilitate data structuring for analysis –Software is available for use by others Transactions can be parsed and organized into time intervals to enable observation of how behaviors change over time, and of the timing of identifies –Whole-trial statistics also taken to characterize trials, but difficult to compare due to differences in trial duration