1. Exercises in Defending Cyberspace: The Capstone of Education, Training, and Awareness Craig E. Kaucher LTC, U.S. Army Professor of Information Operations and Assurance Information Resources Management College National Defense University kaucherc@ndu.edu “My opinions: not necessarily the USG, DOD, or NDU!”

2. AgendaAgenda Why Exercises ?Why Exercises ? Developing ExercisesDeveloping Exercises –Sponsor –Objectives –Scope and Format –Participants –Scenario –Controllers –Models –Testing and Validation Case StudiesCase Studies Why Exercises ?Why Exercises ? Developing ExercisesDeveloping Exercises –Sponsor –Objectives –Scope and Format –Participants –Scenario –Controllers –Models –Testing and Validation Case StudiesCase Studies

3. Why Exercises ? “To test civilian agencies’ security preparedness and contingency planning, DHS will use exercises to evaluate the impact of cyberattacks on governmentwide processes. Weaknesses discovered will be included in agency corrective action plans and submitted to the OMB. DHS also will explore such exercises as a way to test the coordination of public and private incident management, response and recovery capabilities.” (A/R 1-3)“To test civilian agencies’ security preparedness and contingency planning, DHS will use exercises to evaluate the impact of cyberattacks on governmentwide processes. Weaknesses discovered will be included in agency corrective action plans and submitted to the OMB. DHS also will explore such exercises as a way to test the coordination of public and private incident management, response and recovery capabilities.” (A/R 1-3) “Corporations are encouraged to regularly review and exercise IT continuity plans and to consider diversity in IT service providers as a way of mitigating risk.” (A/R 1-4)“Corporations are encouraged to regularly review and exercise IT continuity plans and to consider diversity in IT service providers as a way of mitigating risk.” (A/R 1-4) Appendix, Actions and Recommendations (A/R) Summary, The National Strategy to Secure Cyberspace, February 2003.

4. Why else… (Obligatory Dead Guy Quote) “To rely on rustics and not prepare is the greatest of crimes; to be prepared beforehand for any contingency is the greatest of virtues.”“To rely on rustics and not prepare is the greatest of crimes; to be prepared beforehand for any contingency is the greatest of virtues.” Sun Tzu, on the need to wargame strategies, from “Sun Tzu and the Art of Business: Six Strategic Principles”, Mark McNeilly, Oxford University Press, 1996.

5. Developing Exercises The Role of the Sponsor What does the sponsor want to learn or demonstrate? What does the sponsor want to learn or demonstrate? What does the sponsor want the participants to learn or demonstrate? What does the sponsor want the participants to learn or demonstrate? How can the exercise best assure that the sponsor’s goals are met? How can the exercise best assure that the sponsor’s goals are met? What information must be What information must be provided by the sponsor? provided by the sponsor? What information will be gathered for the sponsor? What information will be gathered for the sponsor? What does the sponsor want to learn or demonstrate? What does the sponsor want to learn or demonstrate? What does the sponsor want the participants to learn or demonstrate? What does the sponsor want the participants to learn or demonstrate? How can the exercise best assure that the sponsor’s goals are met? How can the exercise best assure that the sponsor’s goals are met? What information must be What information must be provided by the sponsor? provided by the sponsor? What information will be gathered for the sponsor? What information will be gathered for the sponsor?

6. Developing Exercises Specifying Objectives Educational and Training Objectives Educational and Training Objectives –Teach or train new tasks and procedures –Reinforce previous training and education –Evaluate training and education Research and Procedural Objectives Research and Procedural Objectives –Develop new strategies, plans, procedures –Test execution of strategies, plans, procedures –Identify issues and gaps in current strategies, plans, procedures –Build consensus for strategies, plans, procedures Educational and Training Objectives Educational and Training Objectives –Teach or train new tasks and procedures –Reinforce previous training and education –Evaluate training and education Research and Procedural Objectives Research and Procedural Objectives –Develop new strategies, plans, procedures –Test execution of strategies, plans, procedures –Identify issues and gaps in current strategies, plans, procedures –Build consensus for strategies, plans, procedures

7. Full Scale or “Live” Exercise Command Post Exercise Tabletop Full Scale or “Live” Exercise Command Post Exercise Tabletop Developing Exercises Scope of Activities Exercises Education Training Awareness Exercises Education Training Awareness

8. Developing Exercises Tabletop Exercises “One step that any organization can take is to reach out to other public and private entities in its region to conduct joint tabletop exercises.” ( Andrews, 2003) “One step that any organization can take is to reach out to other public and private entities in its region to conduct joint tabletop exercises.” ( Andrews, 2003) Normally very low cost Anyplace, anytime Small number of participants Could be for any type of objective Could be the first phase of a larger exercise Normally very low cost Anyplace, anytime Small number of participants Could be for any type of objective Could be the first phase of a larger exercise

9. Many organizations, not many peopleMany organizations, not many people Frequently examinesFrequently examines existing or new procedures existing or new procedures Also could be part ofAlso could be part of an exercise “buildup” an exercise “buildup” More costs, more disruption to regular activitiesMore costs, more disruption to regular activities Developing Exercises Command Post Exercises

10. Highest costHighest cost Most people involvedMost people involved Inter-agency, inter- governmental, inter-sectorInter-agency, inter- governmental, inter-sector Occasional (but required)Occasional (but required) Impressions and perceptions countImpressions and perceptions count Developing Exercises Full Scale Exercises “That’s why the most comprehensive cyberpreparedness exercises bring together people from different, interdependent sectors and government agencies and include practicing how information will be shared.” Dr. Craig Koerner, Naval War College

11. Developing Exercises Identifying Participants Organization(s) Organization(s) Individuals Individuals Who is essential? Who is essential? Controllers Controllers Organization(s) Organization(s) Individuals Individuals Who is essential? Who is essential? Controllers Controllers

12. Developing Exercises Developing the Scenario The ScenarioThe Scenario –A situation into which participants are placed that requires them to make decisions Scenario-related informationScenario-related information –Who and what will decisions affect? –What operational information is required? is required? –How will the scenario be changed or updated? changed or updated? The ScenarioThe Scenario –A situation into which participants are placed that requires them to make decisions Scenario-related informationScenario-related information –Who and what will decisions affect? –What operational information is required? is required? –How will the scenario be changed or updated? changed or updated? “The scenario can have a significant, if not overwhelming effect on the decisions players are able to make.” (Perla, 1990)

13. Developing Exercises The Role of Controllers Monitor participant actions Monitor participant actions Assess interactions Assess interactions Inform participants about outcomes Inform participants about outcomes Monitor participant actions Monitor participant actions Assess interactions Assess interactions Inform participants about outcomes Inform participants about outcomes

14. Developing Exercises Using Models Models can have several purposes Models can have several purposes –Provide inputs to the exercise –Keep the exercise moving –Replicate realistic organizations, events or functions Examples Examples –Physical or logical environment –Functional activities (logistics, intelligence) –Sensors –Command and control –Weapons Models can have several purposes Models can have several purposes –Provide inputs to the exercise –Keep the exercise moving –Replicate realistic organizations, events or functions Examples Examples –Physical or logical environment –Functional activities (logistics, intelligence) –Sensors –Command and control –Weapons

15. Developing Exercises Testing and Validation Model, data, and scenario validation Model, data, and scenario validation Play testing Play testing Pre-play Pre-play Final Rules Final Rules Model, data, and scenario validation Model, data, and scenario validation Play testing Play testing Pre-play Pre-play Final Rules Final Rules

16. Exercise Case Study: Eligible Receiver “The eye-opener exercise”“The eye-opener exercise” Live cyberattacks involvedLive cyberattacks involved DOD focused and directedDOD focused and directed No notice to “participants”No notice to “participants” Key lesson learned: DOD networks are highly vulnerableKey lesson learned: DOD networks are highly vulnerable Led to the formation of Joint Task Force Computer Network DefenseLed to the formation of Joint Task Force Computer Network Defense “The eye-opener exercise”“The eye-opener exercise” Live cyberattacks involvedLive cyberattacks involved DOD focused and directedDOD focused and directed No notice to “participants”No notice to “participants” Key lesson learned: DOD networks are highly vulnerableKey lesson learned: DOD networks are highly vulnerable Led to the formation of Joint Task Force Computer Network DefenseLed to the formation of Joint Task Force Computer Network Defense

17. Focused on regional CIP in preparation for 2002 Winter Olympics, co-sponsored by Utah Dept. of Public Safety, US DOE Office of CIP, Utah Olympic Public Safety CommandFocused on regional CIP in preparation for 2002 Winter Olympics, co-sponsored by Utah Dept. of Public Safety, US DOE Office of CIP, Utah Olympic Public Safety Command Tabletop exerciseTabletop exercise Used to surface issues, develop and implement an action plan for “disaster resistant Olympics”Used to surface issues, develop and implement an action plan for “disaster resistant Olympics” Key lessons learned in understanding interdependencies, communication, coordination, and resource allocationKey lessons learned in understanding interdependencies, communication, coordination, and resource allocation Focused on regional CIP in preparation for 2002 Winter Olympics, co-sponsored by Utah Dept. of Public Safety, US DOE Office of CIP, Utah Olympic Public Safety CommandFocused on regional CIP in preparation for 2002 Winter Olympics, co-sponsored by Utah Dept. of Public Safety, US DOE Office of CIP, Utah Olympic Public Safety Command Tabletop exerciseTabletop exercise Used to surface issues, develop and implement an action plan for “disaster resistant Olympics”Used to surface issues, develop and implement an action plan for “disaster resistant Olympics” Key lessons learned in understanding interdependencies, communication, coordination, and resource allocationKey lessons learned in understanding interdependencies, communication, coordination, and resource allocation Exercise Case Study: Black Ice

18. US Air Force exercise focused on internal networks and operatorsUS Air Force exercise focused on internal networks and operators Used to evaluate detection, response, recovery proceduresUsed to evaluate detection, response, recovery procedures Live and simulated (range) playLive and simulated (range) play Validated operational procedures, and gathered best practicesValidated operational procedures, and gathered best practices US Air Force exercise focused on internal networks and operatorsUS Air Force exercise focused on internal networks and operators Used to evaluate detection, response, recovery proceduresUsed to evaluate detection, response, recovery procedures Live and simulated (range) playLive and simulated (range) play Validated operational procedures, and gathered best practicesValidated operational procedures, and gathered best practices Exercise Case Study: Black Demon

19. Pacific Northwest critical infrastructure owners with federal, state, local governments (US and Canada)Pacific Northwest critical infrastructure owners with federal, state, local governments (US and Canada) Tabletop exerciseTabletop exercise Physical attacks (in the scenario) led to IT failuresPhysical attacks (in the scenario) led to IT failures Key lessons learned:Key lessons learned: –Number/degree of interdependencies unknown –Regional and US/Canada coordination lacking –Unanticipated loss of communications –No mechanism for cross-border analysis and reporting –Roles, missions, role of law enforcement not understood Pacific Northwest critical infrastructure owners with federal, state, local governments (US and Canada)Pacific Northwest critical infrastructure owners with federal, state, local governments (US and Canada) Tabletop exerciseTabletop exercise Physical attacks (in the scenario) led to IT failuresPhysical attacks (in the scenario) led to IT failures Key lessons learned:Key lessons learned: –Number/degree of interdependencies unknown –Regional and US/Canada coordination lacking –Unanticipated loss of communications –No mechanism for cross-border analysis and reporting –Roles, missions, role of law enforcement not understood Exercise Case Study: Blue Cascades

20. Local/regional exercise involving federal, state, local govt., industry, academia, militaryLocal/regional exercise involving federal, state, local govt., industry, academia, military “Congressionally” directed“Congressionally” directed Three PhasesThree Phases –Tabletop –Lessons learned implementation –Live exercise Key lessons learned: Start small and build, broadest participation is best, many information gaps existKey lessons learned: Start small and build, broadest participation is best, many information gaps exist Local/regional exercise involving federal, state, local govt., industry, academia, militaryLocal/regional exercise involving federal, state, local govt., industry, academia, military “Congressionally” directed“Congressionally” directed Three PhasesThree Phases –Tabletop –Lessons learned implementation –Live exercise Key lessons learned: Start small and build, broadest participation is best, many information gaps existKey lessons learned: Start small and build, broadest participation is best, many information gaps exist Exercise Case Study: Dark Screen

21. Department of Homeland Security and Dartmouth University sponsored/run exerciseDepartment of Homeland Security and Dartmouth University sponsored/run exercise Simulated attacks (physical and cyber)Simulated attacks (physical and cyber) Focus on banking and financial sector, with other sector involvementFocus on banking and financial sector, with other sector involvement Government performance “certainly a B+, better than my personal expectations” – Amit YoranGovernment performance “certainly a B+, better than my personal expectations” – Amit Yoran Key Lessons Learned: inter-sector coordination and information sharing need improvementKey Lessons Learned: inter-sector coordination and information sharing need improvement Department of Homeland Security and Dartmouth University sponsored/run exerciseDepartment of Homeland Security and Dartmouth University sponsored/run exercise Simulated attacks (physical and cyber)Simulated attacks (physical and cyber) Focus on banking and financial sector, with other sector involvementFocus on banking and financial sector, with other sector involvement Government performance “certainly a B+, better than my personal expectations” – Amit YoranGovernment performance “certainly a B+, better than my personal expectations” – Amit Yoran Key Lessons Learned: inter-sector coordination and information sharing need improvementKey Lessons Learned: inter-sector coordination and information sharing need improvement Exercise Case Study: Livewire

22. Other Views of Exercises How do exercises affect industry ? How do exercises affect industry ? –Participation Scope (number of participants) Scope (number of participants) Business Impact Business Impact RepetitionRepetition –Cost Who pays ? Who pays ? Overhead & overtime Overhead & overtime –Interrelated sectors How do exercises affect industry ? How do exercises affect industry ? –Participation Scope (number of participants) Scope (number of participants) Business Impact Business Impact RepetitionRepetition –Cost Who pays ? Who pays ? Overhead & overtime Overhead & overtime –Interrelated sectors

23. Closing Thoughts Education, training, and awareness are valuable countermeasures, but exercises are where “the rubber meets the road”Education, training, and awareness are valuable countermeasures, but exercises are where “the rubber meets the road” “If you’ve never been under mass fire and suddenly you are, the odds are that your brain will shut down and you’ll do everything wrong.” - Stephen Northcutt, SANS Institute.“If you’ve never been under mass fire and suddenly you are, the odds are that your brain will shut down and you’ll do everything wrong.” - Stephen Northcutt, SANS Institute. Education, training, and awareness are valuable countermeasures, but exercises are where “the rubber meets the road”Education, training, and awareness are valuable countermeasures, but exercises are where “the rubber meets the road” “If you’ve never been under mass fire and suddenly you are, the odds are that your brain will shut down and you’ll do everything wrong.” - Stephen Northcutt, SANS Institute.“If you’ve never been under mass fire and suddenly you are, the odds are that your brain will shut down and you’ll do everything wrong.” - Stephen Northcutt, SANS Institute. Graphic courtesy of US Naval Postgraduate School, Winners of the 2002 DOD Cyber Defense Exercise Downloaded from: www.nps.navy.mil/PAO/Internal/ Cyber_Defense.htm

24. ReferencesReferences “The Art of Wargaming”, Peter P. Perla, Naval Institute Press, Annapolis, MD, 1990. “The Art of Wargaming”, Peter P. Perla, Naval Institute Press, Annapolis, MD, 1990. “How can information exchange be enhanced”, Richard Andrews, Security Management, vol. 47/6, pg. 162. Arlington, VA, 2003. “How can information exchange be enhanced”, Richard Andrews, Security Management, vol. 47/6, pg. 162. Arlington, VA, 2003. “More than a game”, Deborah Padcliff, Computerworld, vol. 36/37, September 2002. “More than a game”, Deborah Padcliff, Computerworld, vol. 36/37, September 2002. “Blue Cascades” Final Report, Pacific Northwest Economic Region, 18 July 2002. “Blue Cascades” Final Report, Pacific Northwest Economic Region, 18 July 2002. “Infrastructure Interdependencies Tabletop Exercise: Summary of Key Issues and Actions to Date”, Paula Scalingi, DOE Office of CIP, May 2001 “Infrastructure Interdependencies Tabletop Exercise: Summary of Key Issues and Actions to Date”, Paula Scalingi, DOE Office of CIP, May 2001 “Black demon tests tactics, improves network defense”, Dom Cardonita, HQ AIA/PA, Lackland AFB, Texas, Summer 2002. “Black demon tests tactics, improves network defense”, Dom Cardonita, HQ AIA/PA, Lackland AFB, Texas, Summer 2002. “Dark Screen: A Cyber Security Exercise for San Antonio/Bexar County. Final Report”, Gregory B. White, University of Texas – San Antonio, 26 September 2003. “Dark Screen: A Cyber Security Exercise for San Antonio/Bexar County. Final Report”, Gregory B. White, University of Texas – San Antonio, 26 September 2003. “Simulated terrorist Attack Exposes Problems”, Ted Bridis, Associated Press, downloaded from http://www.informationweek.com, 25 November 2003. “Simulated terrorist Attack Exposes Problems”, Ted Bridis, Associated Press, downloaded from http://www.informationweek.com, 25 November 2003. http://www.informationweek.com “Cyberexercises”, Seth Cowand, University of Texas-San Antonio, unpublished manuscript, December 2003. “Cyberexercises”, Seth Cowand, University of Texas-San Antonio, unpublished manuscript, December 2003. “Current Issues in US Homeland Security and Critical Infrastructure Protection”, Cristin L. Flynn, MCI, Inc., briefing at National Defense University, November 6, 2003 “Current Issues in US Homeland Security and Critical Infrastructure Protection”, Cristin L. Flynn, MCI, Inc., briefing at National Defense University, November 6, 2003