Scenario Wrapup “Shangri-la” – QA Support in 7-10 Year Dr. John D. Prange AQUAINT Program Manager 301-688-7092

Slides:



Advertisements
Similar presentations
The Design and Problem Solving Process. What is the point? A proven method and process Can be utilized for any problem Helps you to produce quality work.
Advertisements

Project-Based vs. Text-Based
Analyzing Student Work
KEEN Conference Created by Kansans 35 Individuals, most of them educators From all 10 Board of Education districts From all levels of education.
Chapter 1 Business Driven Technology
When the bartender asked, "How's it going, Norm
Intro. Website Purposes  Provide templates and resources for developing early childhood interagency agreements and collaborative procedures among multiple.
Introduction to: Assessing Pupil Progress (APP)
Marzano Art and Science Teaching Framework Learning Map
How do I progress in History? What does the National Curriculum mean for me?
Beth Rous University of Kentucky Working With Multiple Agencies to Plan And Implement Effective Transitions For Head Start Children Beth Rous University.
Helpful Hints to Conduct and Write a Literature Review October 2006.
1 14. Project closure n An information system project must be administratively closed once its product is successfully delivered to the customer. n A failed.
EVIDENCE BASED WRITING
Workshop Summary ISPS Drills & Exercises Workshop Port Moresby 2006.
Integrative Model Designed to help students develop a deep understanding of organized bodies of knowledge while simultaneously developing critical thinking.
By: Becky Guzie Chapter 5: Developing Adaptations to Promote Participation in Inclusive Environment.
Literature Review and Parts of Proposal
Intelligence is being able to add value to information by subjecting it to the elements of the intelligence process. Intelligence has become the driving.
CriteriaExemplary (4 - 5) Good (2 – 3) Needs Improvement (0 – 1) Identifying Problem and Main Objective Initial QuestionsQuestions are probing and help.
IAEA International Atomic Energy Agency Reviewing Management System and the Interface with Nuclear Security (IRRS Modules 4 and 12) BASIC IRRS TRAINING.
AGENDA  A teacher’s perspective  Barb Schmidt Stevens High School  Acacia Trevillyan South Park Elementary  Review steps to create a quality CFA 
Investors in People Champions. Interpretation of the Standard Purpose To give a brief overview with the necessary background information on the Investors.
1 Customized Employment Strategic Service Delivery Component Disability Employment Initiative.
Big Idea 1: The Practice of Science Description A: Scientific inquiry is a multifaceted activity; the processes of science include the formulation of scientifically.
RAKESS PYP Mathematics Parents’ Evening February 1 st, 2010.
New Teachers’ Induction January 20, 2011 Office of Curriculum and Instruction.
AGSM 440 April 5 AnnouncementsAnnouncements HW#12 team peer review due today HW#12 team peer review due today Final draft of the Economic Analysis due.
National Commission for Academic Accreditation & Assessment Developmental Reviews at King Saud University and King Faisal University.
Measuring the Quality of Decisionmaking and Planning Framed in the Context of IBC Experimentation February 9, 2007 Evidence Based Research, Inc.
“The great end of education is to discipline rather than furnish the mind; to train it to the use of its own powers, rather than fill it with the accumulation.
Enterprise Risk Management Chapter One Prepared by: Raval, Fichadia Raval Fichadia John Wiley & Sons, Inc
The Steps Involved Select a Company who has internationalised or is intending to Age of Internationalisation event Check existing papers about this coy.
COOP Continuity Operations A Grass Roots Approach COL Ray Horoho OASA M&RA (TRM)
Literacy I can recall main info, know where to look for it, make inferences linked to evidence, show awareness of characters’ intentions, adapt speech.
From Question-Answering to Information-Seeking Dialogs Jerry R. Hobbs Artificial Intelligence Center SRI International Menlo Park, California (with Douglas.
AQUAINT Scenario Breakout -- Group 2, Team 1 12 June 2002.
Systems Analysis and Design in a Changing World, Fourth Edition
AQUAINT Scenario Breakout -- Group 2, Team 3 12 June 2002.
Splitting Complex Temporal Questions for Question Answering systems ACL 2004.
© 2010 Health Information Management: Concepts, Principles, and Practice Chapter 5: Data and Information Management.
Breakout # 1 – Data Collecting and Making It Available Data definition “ Any information that [environmental] researchers need to accomplish their tasks”
Writing Proposals Nayda G. Santiago Capstone CpE Jan 26, 2009.
Keys to the Comparison Essay. What is the Comparison essay? THE BASICS  An essay discussing the similarities and differences between two given regions.
Module 12: Responding to Security Incidents. Overview Introduction to Auditing and Incident Response Designing an Audit Policy Designing an Incident Response.
AQUAINT Scenario Breakout -- Group 2, Team 6 12 June 2002.
Copyright © 2014 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Performance Task Overview Introduction This training module answers the following questions: –What is a performance task? –What is a Classroom Activity?
How to structure good history writing Always put an introduction which explains what you are going to talk about. Always put a conclusion which summarises.
Discuss how researchers analyze data obtained in observational research.
Greenbush. An informed citizen possesses the knowledge needed to understand contemporary political, economic, and social issues. A thoughtful citizen.
ITEC 1010 Information and Organizations Chapter V Expert Systems.
BSBPMG501A Manage Project Integrative Processes Manage Project Integrative Processes Project Integration Processes – Part 2 Diploma of Project Management.
Marketing Essentials Mark Davis Senior Examiner Exam guidance June 2014.
Strategic Communications Training Crisis Communications X State MDA 1.
Analysing different levels of sporting performance.
Understanding Standards: Advanced Higher Event
Understanding Standards: Advanced Higher Statistics
Integration of Temperature Control into the Supply Chain
Writing a sound proposal
Project Overview – Phase 1
The crime scene sketch:
CREATE - ASSESSMENT LEVEL GRADE CREATE: Your final Game design.
Performance Task Overview
Learning Objectives: Understand the interdependence of DM and KM
Cyber Threat Intelligence Sharing Standards-based Repository
Increasing Rigor to Develop Critical Thinking Skills
Setting Writing Goals in Science The Living Environment
CBMS4303: Management Information System
Managerial Decision Making and Evaluating Research
Presentation transcript:

Scenario Wrapup “Shangri-la” – QA Support in 7-10 Year Dr. John D. Prange AQUAINT Program Manager

2 Task Two Approaches –Top Down –Bottom Up Both Approaches worked simultaneously by Analysts Goal is to see if the two approaches can be connected somewhere in the middle; Can we “Marshal the Evidence to Build a Case” that will directly addresses the assigned task. How many operatives remain in America & what forms of terror do they have in mind?

3 Top Down Top Level Questions: –What are the possible forms of terror? Candidate: Man Portables –Who are the possible perpetrators/operatives? Suspected Al-Qaeda operatives Members of God’s Aryan Militia –What are the possible modes of operations? Weapons theft from military Surplus weapons purchases Suspicious transactions Transportation of suspicious materials / movement of suspects Task: For each possibility – decompose into sufficiently fine sub-elements; indicators, plan outlines, time lines, and then search for data that supports these elements. How does the data that the analyst has in hand or can find fit these finer-grained models?

4 An Asymmetric Threat Scenario * How safe is the Muscat harbor for refueling US Navy ships? ? What recent terrorist activity/Incidents in Yemen? Against US Military world-wide? What are the potential threats? ? How secure is the Aden Harbor? Are our security plans adequate? ? What is the Current Political/Social/ Cultural climate in Yemen? In the region? ? What is the current state of US relations with Yemen Other countries in region? * Reference: SRI International AQUAINT Briefing

5 ? What recent terrorist activity/Incidents in Yemen? Against US Military world-wide? What are the potential threats? ? How secure is the Aden Harbor? Are our security plans adequate? ? What is the Current Political/Social/ Cultural climate in Yemen? In the region? ? What is the current state of US relations with Yemen Other countries in region? An Asymmetric Threat Scenario * How safe is the Muscat harbor for refueling US Navy ships? POSSIBLE SOURCES Current Intel Reports; Latest Terrorist Assessments; Historical Incident Reports Terrorist Activities How secure is the Aden Harbor? Are our security plans adequate? POSSIBLE SOURCES Imagery of Harbor; Annotated Maps; Military Attaché Video of Port Facilities; Naval Orders/ SOPS/Plans Harbor Facilities/ Security Plans POSSIBLE SOURCES Local Print, Radio, Broadcast News Reports Other Analysts Current Yemen Environment ……………………….………………….. Current US /Yemen Relations POSSIBLE SOURCES Other Analysts; State Department/ Country Team Cables; Recent Contact with Embassy/Consulate * Reference: SRI International AQUAINT Briefing

6 Bottom Up Try to form a more coherent picture of the available data / evidence that has been obtained. Use “Shangri-la” QA system to: –Create detailed timelines of the available data –Create multiple Link Diagrams to display different types of relationships between events, people, places, materials –Create overlays of these timelines and link diagrams Use “Shangri-la” QA system to –Develop detailed profiles, background information on each event, people involved, organizations, materials, weapons, vehicles etc –Cast a “Wide Net”; Follow Leads –Create a broad foundation of known / suspected facts that appear to be related to the current investigation –Identify gaps, missing data  What don’t I know? And if I knew it what good would it do me?

7 A Different Paradigm may be useful when handling QA Scenarios: Current Analytic Paradigm: QA Scenarios: A Different Paradigm? – Sequentially “Filter Down” to the final result Processing & Analysis Data Results – Works when QA’s are independent, isolated activities – Cast a “wider net” while searching for “golden nuggets” (Answers) Answers Space of Data Objects and Sources How Wide to Cast the “Net”? What Info to Retain? In what form? For how long? – Automatically Extract, Represent, and Preserve “closely related” background information within context of the QA Scenario Background Discarded

8 Marshal the Evidence to Build a Case Shangra-La QA System assists Analyst in finding closure between the two approaches Top Down Approach Bottom Up Approach Case

9 Improved Reasoning & Learning Associates Follow-up Leads Follow-up Leads Cross Fertilization Advanced Reasoning: Use Multi-level Plans Create and evaluate chains of reasoning Reason across hetero- geneous data sources Infer answers from data extracted from multiple sources when the answer is not explicitly stated Utilize Link Analysis & Evidence Discovery Plus other strategies Advanced Learning: Automatically learn new or modify existing reasoning strategies New Senior Advisor “Views: Past & Present”. ….… ….…...……. ….….. Summarized Results Collected Views TV & Radio Broadcasts, Newspapers & Other Archives “Bio” ………..…. ……..……. ………..…. ……..……. ………..…. ……..……. …………... Raw “Bio” Information Education Past Positions Family Travels Other Activities Summarized Results

10 Problems Partially Solved by the Shangri-la QA System Being able to adequately explain the limits of its capabilities so that Analyst can ask reasonable questions. Developing / Implementing a Detailed, Complex Plan to Solve the QA Task at Hand Decomposing Complex Questions into a series / sequence of Simpler Questions whose Answers can be found Selecting the appropriate sources to search Knowing when No Answer is Available; Being able to then give a partial, incomplete answer Giving understandable explanations of the “Plan”, the “Reasoning Used” and the “Answers Found”

11 Final Comments Shangra La QA System: –Not really a system but an integrated suite of tools –Allows access to tools/capabilities that fall outside the scope of Advanced QA –Learns and adapts as the task unfolds –Capabilities in the small exist today but they need to be done far more automatically with little or no direct analysts involvement (e.g. the metadata hotlink tags done automatically in the large) –Knows its limitations and can adequately share those with analysts –Can provide explanation and maintains understandable audit trails –Guts of the system are transparent to the analyst –HCI issues/concerns have been significantly addressed

12 Final Comments IC Analysts: –Understands the capabilities and limitations of the Shangra-la QA system; How the system interfaces with the Analyst’s methods –Knows how to ask questions that the system is capable of responding to –Uses “Evidence Marshalling Strategies” and knows how to use the Shangra-la System to: Formulate hypotheses and then decompose them into subcomponents that can be researched and investigated Develop appropriate arguments and build a “Defendable Case” Seek both confirming and disconfirming evidence Construct scenarios out of the available evidence

13 Final Comments It’s a team effort –IC Analyst + Shangra-la QA System Use a Football Analogy Shangra-la QA System –“Franchise” Player BUT … IC Analyst is the “Quarterback/Coach” –Knows the scouting report –Develops the game plan –Calls the plays –Make the tough decisions