1. Scenario Wrapup “Shangri-la” – QA Support in 7-10 Year Dr. John D. Prange AQUAINT Program Manager jprange@nsa.gov 301-688-7092 http://www.ic-arda.org

2. 2 Task Two Approaches –Top Down –Bottom Up Both Approaches worked simultaneously by Analysts Goal is to see if the two approaches can be connected somewhere in the middle; Can we “Marshal the Evidence to Build a Case” that will directly addresses the assigned task. How many operatives remain in America & what forms of terror do they have in mind?

3. 3 Top Down Top Level Questions: –What are the possible forms of terror? Candidate: Man Portables –Who are the possible perpetrators/operatives? Suspected Al-Qaeda operatives Members of God’s Aryan Militia –What are the possible modes of operations? Weapons theft from military Surplus weapons purchases Suspicious transactions Transportation of suspicious materials / movement of suspects Task: For each possibility – decompose into sufficiently fine sub-elements; indicators, plan outlines, time lines, and then search for data that supports these elements. How does the data that the analyst has in hand or can find fit these finer-grained models?

4. 4 An Asymmetric Threat Scenario * How safe is the Muscat harbor for refueling US Navy ships? ? What recent terrorist activity/Incidents in Yemen? Against US Military world-wide? What are the potential threats? ? How secure is the Aden Harbor? Are our security plans adequate? ? What is the Current Political/Social/ Cultural climate in Yemen? In the region? ? What is the current state of US relations with Yemen Other countries in region? * Reference: SRI International AQUAINT Briefing

5. 5 ? What recent terrorist activity/Incidents in Yemen? Against US Military world-wide? What are the potential threats? ? How secure is the Aden Harbor? Are our security plans adequate? ? What is the Current Political/Social/ Cultural climate in Yemen? In the region? ? What is the current state of US relations with Yemen Other countries in region? An Asymmetric Threat Scenario * How safe is the Muscat harbor for refueling US Navy ships? POSSIBLE SOURCES Current Intel Reports; Latest Terrorist Assessments; Historical Incident Reports Terrorist Activities How secure is the Aden Harbor? Are our security plans adequate? POSSIBLE SOURCES Imagery of Harbor; Annotated Maps; Military Attaché Video of Port Facilities; Naval Orders/ SOPS/Plans Harbor Facilities/ Security Plans POSSIBLE SOURCES Local Print, Radio, Broadcast News Reports Other Analysts Current Yemen Environment ……………………….………………….. Current US /Yemen Relations POSSIBLE SOURCES Other Analysts; State Department/ Country Team Cables; Recent Contact with Embassy/Consulate * Reference: SRI International AQUAINT Briefing

6. 6 Bottom Up Try to form a more coherent picture of the available data / evidence that has been obtained. Use “Shangri-la” QA system to: –Create detailed timelines of the available data –Create multiple Link Diagrams to display different types of relationships between events, people, places, materials –Create overlays of these timelines and link diagrams Use “Shangri-la” QA system to –Develop detailed profiles, background information on each event, people involved, organizations, materials, weapons, vehicles etc –Cast a “Wide Net”; Follow Leads –Create a broad foundation of known / suspected facts that appear to be related to the current investigation –Identify gaps, missing data  What don’t I know? And if I knew it what good would it do me?

7. 7 A Different Paradigm may be useful when handling QA Scenarios: Current Analytic Paradigm: QA Scenarios: A Different Paradigm? – Sequentially “Filter Down” to the final result Processing & Analysis Data Results – Works when QA’s are independent, isolated activities – Cast a “wider net” while searching for “golden nuggets” (Answers) Answers Space of Data Objects and Sources How Wide to Cast the “Net”? What Info to Retain? In what form? For how long? – Automatically Extract, Represent, and Preserve “closely related” background information within context of the QA Scenario Background Discarded

8. 8 Marshal the Evidence to Build a Case Shangra-La QA System assists Analyst in finding closure between the two approaches Top Down Approach Bottom Up Approach Case

9. 9 Improved Reasoning & Learning Associates Follow-up Leads Follow-up Leads Cross Fertilization Advanced Reasoning: Use Multi-level Plans Create and evaluate chains of reasoning Reason across hetero- geneous data sources Infer answers from data extracted from multiple sources when the answer is not explicitly stated Utilize Link Analysis & Evidence Discovery Plus other strategies Advanced Learning: Automatically learn new or modify existing reasoning strategies New Senior Advisor “Views: Past & Present”. ….… ….…...……. ….….. Summarized Results Collected Views TV & Radio Broadcasts, Newspapers & Other Archives “Bio” ………..…. ……..……. ………..…. ……..……. ………..…. ……..……. …………... Raw “Bio” Information Education Past Positions Family Travels Other Activities Summarized Results

10. 10 Problems Partially Solved by the Shangri-la QA System Being able to adequately explain the limits of its capabilities so that Analyst can ask reasonable questions. Developing / Implementing a Detailed, Complex Plan to Solve the QA Task at Hand Decomposing Complex Questions into a series / sequence of Simpler Questions whose Answers can be found Selecting the appropriate sources to search Knowing when No Answer is Available; Being able to then give a partial, incomplete answer Giving understandable explanations of the “Plan”, the “Reasoning Used” and the “Answers Found”

11. 11 Final Comments Shangra La QA System: –Not really a system but an integrated suite of tools –Allows access to tools/capabilities that fall outside the scope of Advanced QA –Learns and adapts as the task unfolds –Capabilities in the small exist today but they need to be done far more automatically with little or no direct analysts involvement (e.g. the metadata hotlink tags done automatically in the large) –Knows its limitations and can adequately share those with analysts –Can provide explanation and maintains understandable audit trails –Guts of the system are transparent to the analyst –HCI issues/concerns have been significantly addressed

12. 12 Final Comments IC Analysts: –Understands the capabilities and limitations of the Shangra-la QA system; How the system interfaces with the Analyst’s methods –Knows how to ask questions that the system is capable of responding to –Uses “Evidence Marshalling Strategies” and knows how to use the Shangra-la System to: Formulate hypotheses and then decompose them into subcomponents that can be researched and investigated Develop appropriate arguments and build a “Defendable Case” Seek both confirming and disconfirming evidence Construct scenarios out of the available evidence

13. 13 Final Comments It’s a team effort –IC Analyst + Shangra-la QA System Use a Football Analogy Shangra-la QA System –“Franchise” Player BUT … IC Analyst is the “Quarterback/Coach” –Knows the scouting report –Develops the game plan –Calls the plays –Make the tough decisions