Presentation on theme: "INCOSE International Symposium 2008"— Presentation transcript:
1INCOSE International Symposium 2008 Factors Related to the Implementation of ISSE: A Quality Perspective Bryan S. Cline, CISSP-ISSEP, CISM, CISA, CPP, CAP Doctoral Candidate, Information Assurance University of Fairfax Vienna, VA (USA) Director, IS Risk Management The Children’s Hospital of Philadelphia Philadelphia, PA (USA)
2ISSE in the Acquisition of Secure IT IntroductionBackgroundResearch ProposalResearch Review and SynthesisPreliminary ConclusionsPreliminary Recommendations
3Introduction Technical Risk Impacts the Corporate “Bottom Line” Individual Companies Estimate Annual Losses of up to $5M (Deloitte, 2006)Loss of Mobile DevicesIdentity Theft Considered “Number Two Hot-Button” and “Crime of the 21st Century”Viruses/WormsPhishing/PharmingSpyware/MalwareIncreasing Cost of “Compliance”Sarbanes-Oxley (SOX)Federal Information Security Act (FISMA)Health Insurance Portability and Accountability Act (HIPAA)Payment Card Industry (PCI)
4IntroductionTechnical Risk Generally Attributed to System Design, Development, and/or ConfigurationSystem Design & DevelopmentIncorrect SystemInadequate Understanding of User NeedsSystem Specification Doesn’t Reflect User NeedsSystem IncorrectSystem Not Designed to SpecificationsSystem Not Built to the DesignSystem ConfigurationUnnecessary ServicesIncorrect SettingsInadequate Patching
5IntroductionEnsures Requirements are Addressed Early in (and throughout) the System Development Life Cycle (SDLC)INCOSE SE Hdbk ver. 2a
6IntroductionReturn on Investment (ROI) Related to Stage “Defect” is Identified and Subsequently CorrectedINCOSE SE Hdbk ver. 3.1
7Background Federal Law National Policy DoD Policy, Regulations Service Policy, Regulations
8Background DoD Rules and Regulations CJCSI D, IA & Computer Network Defense (CND)Policy & guidance for IA and CNDIA architectureC&AMAC levelsDefense-in-Depth“Incorporate … IA … into [IS] during all phases of [the] system design life cycle including analysis, design, development, test and operation …”
9Background DoD Rules and Regulations (Cont) CJCSM , Defense-in-Depth: IA and CNDGuidance & procedures for implementing Defense-in-Depth strategy and standardsDefense-in-Depth elements:
10Background DoD Rules and Regulations (Cont) CJCSM D, Interoperability and Supportability of IT and National Security SystemsNR-KPP mandatoryIA compliance one of 4 NR-KPP requirementsSpecifies use of Information Systems Security Engineering (ISSE)IA component of the GIGThroughout system lifecycle
11Background DoD Rules and Regulations (Cont) DoDD , Data Sharing in a Net-Centric DoD“… data assets shall have associated [IA] and security metadata, and an authoritative source for the data …”DoDD , IA in the Defense Acquisition SystemDescribes required & recommended levels of IA activities related to acquisitionDescribes the IA strategyPrescribes an IA strategy submission and review processCaptures the acquisition-related IA policies of and
13Background DoD Rules and Regulations DoDI , Information AssuranceEstablishes IA policy and assigns responsibilitiesRequires a defense-in-depth approach to IADoDI , IA ImplementationPrescribes DoD IA Requirements via Minimum Baseline IA ControlsAdditional Controls Specified in Related Documentation
14Background DoD Implementation Controls-based C&A Since 2003 (DoDI )Security Technical Implementation Guides (STIGs) predate DoDIPMs Required to Perform Information Systems Security Engineering (ISSE) (DoDD , DoDI , DoDI )Ensure Security is Incorporated into Overall DesignSupport Successful Certification of System Security Design and Configuration
15BackgroundContinuing Perception Security is Not Adequately Addressed in System Design/ConfigurationGeneral increase in number and types of exploitable vulnerabilities (Deloitte, 2006)Excessive Technical Risk Often Identified During Security/Certification Testing (Cline, 2007)Need to Incorporate Security in Development Practices (Mouratidis, 2007; Davis 2004)—Limited Practice, Limited Research (Siponen, 2007; Woon, 2007)Problems/Issues with Specification/Incorporation of Security Requirements (Mayer, 2005; Mead, 2005; Wilander, 2005)
16Background Impact Program Managers: Scope, Cost, & Schedule Certifiers: Cost & ScheduleDAAs: Cost, Schedule, Acceptance of Excessive Residual RiskUsers: Scope, Cost, Schedule, & Acceptance of Excessive Residual Risk
17Research Proposal Problem Statement Opportunity for Research Research Objective
18Problem StatementBarriers to the successful implementation of ISSE in DoD acquisition organizations result in IS/IT with excessive residual security risk—as determined by the DAA based on the certifiers recommendation—that potentially impacts:Project Scope (Requirements / Capabilities)Project and/or System CostProject ScheduleTechnical Security Risk
19Opportunity for Research NSA ResponsibilityProvides:Technical guidance for protecting information and infrastructuresPrescribes a Defense-in-Depth approachDefines a process for system development …Information Assurance Technical Framework (IATF)
20Opportunity for Research Two Principal Disciplines in System Development* (Cline, 2007)Program ManagementSystems EngineeringIntersection forms Technical Managementaka Engineering Managementaka Systems Engineering Management*in addition to specialty engineeringSystemsEngineeringTechnicalManagementProgramManagement
21Opportunity for Research Two Principal Disciplines in System Security Risk Management (Cline, 2007)Information AssuranceIT/IS ComplianceIntersection forms Certification & AccreditationCertification&AccreditationInformationAssuranceIT/ISCompliance
22Opportunity for Research Information Systems Security Engineering (ISSE) … Systems Development meets Security Risk MgmtFour Domains in the “ISSE Rose” (Cline, 2007)Security EngineeringTechnical ManagementC&ARules & RegulationsTechnicalManagementSystemsEngineeringProgramManagementInformationSystemsSecurityEngineeringSecurityEngineeringRules &RegulationsInformationAssuranceIT/ISComplianceCertification &Accreditation
23Opportunity for Research Information Systems Security Engineering (ISSE)System development process described by the IATFTightly integrated into the systems engineering model1. DiscoverInformationProtectionNeeds2. DefineSystemSecurityRequirements3. DesignArchitecture4. DevelopDetailedDesignUser Involvement6. AssessInformation ProtectionEffectiveness5. Implement
24Opportunity for Research SE ActivitiesISSE ActivitiesDiscover NeedsThe systems engineer helps the customer understand and document the information management needs that support the business or mission. Statements about information needs may be captured in an information management model (IMM).Discover Information Protection NeedsThe ISSE helps the customer understand the information protection needs that support the mission or business. Statements about information protection needs may be captured in an Information Protection Policy (IPP).Mission/Business FunctionInformation Management Functions
25Opportunity for Research SE ActivitiesISSE ActivitiesDefine System RequirementsThe systems engineer allocates identified needs to systems. A system context is developed to identify the system environment and to show the allocation of system functions to that environment. A preliminary system Concept of Operations (CONOPS) is written to describe operational aspects of the candidate system (or systems). Baseline requirements are established.Define System Security RequirementsThe ISSE allocates information protection needs to systems. A system security context, a preliminary system security CONOPS, and baseline security requirements are developed.
26Opportunity for Research SE ActivitiesISSE ActivitiesDesign System ArchitectureThe systems engineer performs functional analysis and allocation by analyzing candidate architectures, allocating requirements, and selecting mechanisms. The systems engineer identifies components or elements, allocates functions to those elements, and describes the relationships between the elements.Design System Security ArchitectureThe ISSE works with the systems engineer in the areas of functional analysis and allocation by analyzing candidate architectures, allocating security services, and selecting security mechanisms. The ISSE identifies components or elements, allocates security functions to those elements, and describes the relationships between the elements.
27Opportunity for Research SE ActivitiesISSE ActivitiesDevelop Detailed DesignThe systems engineer analyzes design constraints, analyzes trade-offs, does detailed system design, and considers life-cycle support. The systems engineer traces all of the system requirements to the elements until all are addressed. The final detailed design results in component and interface specifications that provide sufficient information for acquisition when the system is implemented.Develop Detailed Security DesignThe ISSE analyzes design constraints, analyzes trade-offs, does detailed system and security design, and considers life-cycle support. The ISSE traces all of the system security requirements to the elements until all are addressed. The final detailed security design results in component and interface specifications that provide sufficient information for acquisition when the system is implemented.
28Opportunity for Research SE ActivitiesISSE ActivitiesImplement SystemThe systems engineer moves the system from specifications to the tangible. The main activities are acquisition, integration, configuration, testing, documentation, and training. Components are tested and evaluated to ensure that they meet the specifications. After successful testing, the individual components—hardware, software, and firmware—are integrated, properly configured, and tested as a system.Implement System SecurityThe ISSE participates in a multidisciplinary examination of all system issues and provides inputs to C&A process activities, such as verification that the system as implemented protects against the threats identified in the original threat assessment; tracking of information protection assurance mechanisms related to system implementation and testing practices; and providing inputs to system life-cycle support plans, operational procedures, and maintenance training materials.
29Opportunity for Research SE ActivitiesISSE ActivitiesAssess EffectivenessThe results of each activity are evaluated to ensure that the system will meet the users’ needs by performing the required functions to the required quality standard in the intended environment. The systems engineer examines how well the system meets the needs of the mission.Assess Information Protection EffectivenessThe ISSE focuses on the effectiveness of the information protection—whether the system can provide the confidentiality, integrity, availability, authentication and non-repudiation for the information it is processing that is required for mission success.1. DiscoverInformationProtectionNeeds2. DefineSystemSecurityRequirements3. DesignArchitecture4. DevelopDetailedDesignUser Involvement6. AssessInformation ProtectionEffectiveness5. Implement
30Opportunity for Research ISSE ActivityAssess Information Protection Effectiveness TaskDiscover Information Protection NeedsPresent an overview of the process.Summarize the information modelDescribe threats to the mission or business through information attacksEstablish security services to counter those threats and identify their relative importance to the customer.Obtain customer agreement on the conclusions of this activity as a basis for determining system security effectiveness.Define System Security RequirementsEnsure that the selected solution set meets the mission or business security needs.Coordinate the system boundaries.Present security context, security CONOPS, and system security requirements to the customer and gain their concurrence.Ensure that the projected security risks are acceptable to the customer.Design System Security ArchitectureBegin the formal risk analysis process to ensure that the selected security mechanisms provide the required security services and to explain to the customer how the security architecture meets the security requirements.Develop Detailed Security DesignReview how well the selected security services and mechanisms counter the threats by performing an interdependency analysis to compare desired to effective security service strengths.Once completed, the risk assessment results, particularly any mitigation needs and residual risk, will be documented and shared with the customer to obtain their concurrence.Implement System SecurityThe risk analysis will be conducted/updated.Strategies will be developed for the mitigation of identified risksIdentify possible mission impacts and advise the customer and the customer’s Certifiers and Accreditors.
32Opportunity for Research Little in the Formal Literature on ISSE ImplementationGeneral Failure to Implement ISSEDavis (2004), Mouratidis (2007)Literature Focused on Methods, Processes, ToolsApplication of Quality Construct to ISSESecurity is a Non-Functional or Quality RequirementSSE-CMM is ISSE Maturity (Quality) ModelFurther Indicates Applicability of Quality ConceptsIndicates ISSE both Technical & Mgmt InnovationAllows Synthesis of Process & Factor Models / Frameworks (Adoption Literature)
33Opportunity for Research Construct Proposed by Sebastianelli & Tamimi (2004)Barriers Based on Factor Analysis:Human Resources Development & Mgmt, Planning, Leadership, Resources, Customer FocusDifficult to Interpret from Survey ItemsConstruct modified by Cline (2007)Factors Based on a Synthesis of the Literature (Hill, 2006)Training, Planning, Leadership, Resources, CultureFactors Formally Defined
34Research ObjectiveThe research seeks to determine if the implementation of ISSE in support of acquired DoD IS/IT encounters the same impediments or barriers to implementation as traditional quality programs.
35Research Review and Synthesis Approach to the ReviewEmpirical Research in Information Security and AssuranceEmpirical Research in Relevant Disciplines
36Approach to the Review Purpose of Methodical Review Understand Research Topic and Justify the ResearchIdentify Research Problem and Gaps in the LiteratureProvides Suitable MethodologiesRelevance to Research Problem is Key Discriminator“Each literature piece should constantly be evaluated on how applicable it is to the proposed study. Thus, only the applicable literature articles that are relevant to build the theoretical foundations for the validity of the theories, constructs, and measures should be noted.” (Levy & Ellis, 2006, p. 188)ISSE is “an engineering process … that captures and refines information protection requirements [SRE] and ensures their integration into IT acquisition processes [implementation] through purposeful security design and configuration [development]”
37Non-Functional Requirements Approach to the ReviewPrimary Bodies of KnowledgeQualityManagementHypothesized relationshipDemonstrated relationshipSREImplementationDevelopment(Design &Configuration)Capability MaturityFactorsNon-Functional RequirementsISSEAdoption Theory
38ISSE / Quality Implementation Approach to the ReviewBased on IS/ISSE Research Models (Lowry et al., 2002; Bjorck, 2001)RigorousRelevantEstablishedEmergentISSEQualityManagementDevelopment(Design & Configuration)SREISSE / Quality ImplementationAdoption Theory
39Approach to the ReviewInfoSec Research Model (Siponen & Oinas-Kukkonen, 2007)Levels of AbstractionOrganizational, Conceptual, TechnicalInformation Security Area / Topic / ContributionAccess to ISSecure CommunicationSecurity ManagementDevelopment
40Approach to the ReviewApplicable Categories (Italicized Most Relevant to Research Problem)Access to IS / Conceptual (e.g., modeling of access control policy)Secure Communication / Conceptual (e.g., modeling in distributed systems)Security Management / Organizational (e.g., organizational ISSE implementation policies and guidelines)Development / Organizational (e.g., ISSE methods, processes)
41Empirical Research in Information Security and Assurance ISSE MethodologyISSE Implementation
42ISSE MethodologyOrganizational-level Literature on Development of Secure IS/ITLandwehr (1981)Assurance-based Security Design (circa )Development Using Formal Models (TCSEC, ITSEC, CC)Baskerville (1993)Checklist-based Security Design (circa )Design Based on Available (Known) ControlsUse of Risk in Cost-Benefit Analysis (Controls Selection)Mechanistic Engineering Security Design (circa )Design Based on RequirementsContinued Use of Risk in Cost-Benefit AnalysisFormative Logical Security Design (circa )Design Based on Abstract Models (Functional/Organizational)Limited or No Use of Risk in Cost-Benefit Analysis)
43ISSE Methodology Weiss et al. (1996) Discussed Integration of Engineering and Risk ManagementStraub & Welke (1998)Discussed Systems Risk in Organizational Risk ManagementSecurity Design a Component of Systems Risk AssessmentChivers (2004)Traditional Methods Ineffective for Distributed SystemsIntegrated Risk/Engineering Security Design Methods (circa )Failure Mode Analysis, Fault Tree AnalysisIncreased Complexity Degrades Abstraction
44ISSE Methodology Chivers (2004) (Continued) Need to Integrate Security Analysis in the Systems Engineering ProcessMost Relevant Work in Requirements Engineering (circa )Functional Requirements: Use Cases, GoalsNon-Functional Requirements: Patterns, Abuse Cases & Misuse Cases, Anti-GoalsNeed for Integration of Functional and Non-Functional RequirementsObject-Oriented Patterns; Goal-Based RefinementMost Promising Area is Goal-Based Refinement
45ISSE ImplementationOrganizational-level Literature on Security Management (ISSE)Marmor-Squires & Rougeau (1988) (cited in Ashby et al.,1989)Early Paper Addressing Security Engineering in Systems EngineeringDid Not Address ISSE Implementation IssuesFroscher et al. (1990)Specific to C&A But Addressed Secure IS/IT DevelopmentRecommended Certification Activity Early in DevelopmentRecommended Specific Interactions: CA, DAA, PM, DeveloperWeiss et al. (1996)Discussed Integration of Engineering & Security Risk Mgmt
46ISSE Implementation Peter & Schleipfer (2004) Recommended Addressing Security Engineering Early in DevelopmentDid Not Address ISSE Implementation IssuesDavis (2004)Advocated Security Engineering Throughout Development Lifecycle
47ISSE Implementation Mouratidis (2007) Security Engineering and IS Engineering Communities Work SeparatelyLittle Integration of Security and IS Engineering Principles/PracticeResearch in Security Engineering Primarily Technical IssuesLittle Consideration of Social Aspects / Social ContextRecommended New Discipline: Secure Information Systems Engineering (Note: No Significant Literature on DoD ISSE)Did Not Address ISSE Implementation Issues
48Empirical Research in Relevant Disciplines Quality Management ModelsQuality Implementation
50Quality Implementation Cline (2007)Sebastianelli & Tamimi (2003)Nwabueze (2001) 1Deming (1987) 2Amar & Zain (2002) 3Juran (1993) 4Williams et al. (2004) 5Leonard & McAdam (2001) 6Chin & Pun (2002) 7Kwak & Anbari (2006) 8CultureLack of customer focusNo focus on customer satisfaction 5No understanding of cultural change 1Culture or relationships between departments 3Resistance to change 6No long-term management (mgmt) commitment due to rewards based on short-term successes 5Inappropriate organizational culture 7, 8Incompatibility of attitudes between management and workers 1Attitudes towards quality 3Employee ‘buy-in’ 4Insufficient cooperation 7
51Quality Implementation Cline (2007)Sebastianelli & Tamimi (2003)Nwabueze (2001) 1Deming (1987) 2Amar & Zain (2002) 3Juran (1993) 4Williams et al. (2004) 5Leonard & McAdam (2001) 6Chin & Pun (2002) 7Kwak & Anbari (2006) 8LeadershipInadequate human resources (HR) development and managementHR issues 3No focus on employee satisfaction, innovation, or quality systems 5Failure of management to recognize employee importance 7Lack of leadership for qualityLack of management commitment 1Ineffective mgmt 1, 2Lack of management commitment 3Unstable management due to high turnover 3Lack of long-term management commitment due to rewards based on short-term successes 5Lack of management leadership 7Lack of realism for what TQM can do 7PlanningLack of quality planningPoor planning 1Lack of goals 1Implementation without regard to context 1Use of improper framework 1Divide between strategic & operational goals/planning 6Lack of strategic planning 8
52Quality Implementation Cline (2007)Sebastianelli & Tamimi (2003)Nwabueze (2001) 1Deming (1987) 2Amar & Zain (2002) 3Juran (1993) 4Williams et al. (2004) 5Leonard & McAdam (2001) 6Chin & Pun (2002) 7Kwak & Anbari (2006) 8ResourcesInadequate resourcesLack of resources 1Insufficient raw materials 3Lack of machines & equipment 3TrainingLack of training 1Inadequate training 3Inadequate training 7Training in leadership & project mgmt 8
53Preliminary Conclusions Limited Research on Implementation in the ISSE Literature (Mouratidis, 2007; Siponen, 2007)Applicability of quality concepts to ISSE (Cline, 2007)Non-Functional Requirements (Chung, 1995)Capability Maturity (SSE-CMM) (ISSEA, 2007)Factors Related to Quality Implementation (Hill, 2006)Factors Related to the Implementation of ISSE by DoD Acquisition Agencies are Similar to the Implementation of Traditional Quality Management Models (Cline, 2007)CultureLeadershipPlanningResourcesTraining
54Preliminary Recommendations In Addition to Mandating Use of ISSE in System Acquisition, the DoD Should Provide a Level-of-Effort Similar to the Implementation of Traditional Quality ProgramsRecommendations Specific to Each Barrier:Culture—Stress the Importance of ISSE through Policy; Provide Penalties for Non-Compliance and Enforce the PenaltiesLeadership—Ensure Leadership/Management Understand their Compliance Requirements and the Penalties for Non- Compliance
55Preliminary Recommendations Recommendations Specific to Each Barrier (cont):Planning—Formally Incorporate ISSE in Systems Engineering Plans, Processes, Procedures, Standards and GuidanceFully Integrate ISSE in All Engineering & Program ReviewsImplement the SSE-CMM as part of CMMI in Acq AgenciesResources—Mandate ISSEs for Acquisition ProgramsProgram/Budget Needed Tools (e.g., Mgmt and Assessment)TrainingRequire ISSE Awareness Training for Systems Engineers and PMsRequire Properly Trained and Certified ISSEs
56Preliminary Recommendations Senior Management Responsible for Fielding Secure IS/IT (e.g., DAA, CA, PM, & User Rep) Should:Receive Briefings on the Role of ISSEApply Lessons Learned From Prior Quality Model Implementations to ISSEBe Tracked to DetermineRate & Level of AdoptionSubsequent Impact on Residual RiskTechnical Vulnerabilities Associated with System Development and Configuration
57Recommended ReadingDS Herrmann (2002). A practical guide to security engineering and information assurance. Boca Raton, FL: Auerbach Publications.57
58Questions … that haven’t already been asked? Bryan S. Cline, CISSP-ISSEP, CISM, CISA, CPP, CAPUniversity of FairfaxThe Children’s Hospital of Philadelphia
60Justification for Research Answering the research questions will allow DoD acquisition agencies to:Address the most critical issues facing program and project managersProvide targeted application of scarce resources (dollars and personnel)Improve overall system securityReduce risk to project scope, cost and schedule
61Theoretical Foundation Extensive Literature on ISSE MethodologyLittle or No Relevant Literature on ISSE ImplementationNo Construct for Barriers to ISSE ImplementationApplicability of Quality Concepts to ISSENonfunctional Requirements (Engineering)Capability Maturity Models (Management)Quality Construct – Barriers to ImplementationSebastianelli and Tamimi (2003)Exploratory Quantitative ResearchCorrelational Survey StudyBarriers Based on Interpretation of Optimal Factor AnalysisCline (2007)Barriers Based on Synthesis of Relevant Literature
62Research Methodology Theoretical Framework Research Design Approach Context of StudyFeasibility Analysis and Design Selection
63Theoretical Framework Research HypothesesOperational Definition of VariablesRival HypothesesPlausibility of Rival Hypotheses
64Research QuestionsQuestion 1: Is there a relationship between the successful implementation of ISSE and the DoD IS/IT acquisition organization’s culture?Question 2: Is there a relationship between the successful implementation of ISSE and the DoD IS/IT acquisition organization’s leadership?Question 3: Is there a relationship between the successful implementation of ISSE and the acquisition organization’s planning?Question 4: Is there a relationship between the successful implementation of ISSE and the DoD IS/IT acquisition organization’s resources?Question 5: Is there a relationship between the successful implementation of ISSE and the DoD IS/IT acquisition organization’s training?
65Research HypothesesH1H10 (Null): Implementation of ISSE is independent of the DoD IS/IT acquisition organization’s culture.H11 (Alternative): Implementation of ISSE is dependent on the DoD IS/IT acquisition organization’s culture.H2H20 (Null): Implementation of ISSE is independent of the DoD IS/IT acquisition organization’s leadership.H21 (Alternative): Implementation of ISSE is dependent on the DoD IS/IT acquisition organization’s leadership.H3H30 (Null): Implementation of ISSE is independent of the DoD IS/IT acquisition organization’s planning.H31 (Alternative): Implementation of ISSE is dependent on the DoD IS/IT acquisition organization’s planning.H4H40 (Null): Implementation of ISSE is independent of the DoD IS/IT acquisition organization’s resources.H41 (Alternative): Implementation of ISSE is dependent on the DoD IS/IT acquisition organization’s resources.H5H50 (Null): Implementation of ISSE is independent of the DoD IS/IT acquisition organization’s training.H51 (Alternative): Implementation of ISSE is dependent on the DoD IS/IT acquisition organization’s training.
66Operational Definition of Variables Independent Variable: ISSE implementation is defined as the implementation of ISSE within a DoD IS/IT acquisition organization, where ISSE is defined as “an engineering process … that captures and refines information protection requirements and ensures their integration into IT acquisition processes through purposeful security design and configuration” (DoD, 2006, p. 19).Dependent Variable 1: Culture is defined as “a set of shared assumptions, values, and behaviors that characterize the functioning of an organization” (Schwalbe, 2006, p. Glossary 8).Dependent Variable 2: Leadership is defined as the actions of an individual, usually in a formal position of authority in an organization (management), “who focuses on long-term goals and big-picture objectives, while inspiring people to reach those goals” (Schwalbe, 2006, p. Glossary 6).
67Operational Definition of Variables Dependent Variable 3: Planning is defined as the activities and processes used to devise and maintain a workable scheme to ensure organizational needs are met (adapted from Schwalbe, 2006, p. Glossary 8).Dependent Variable 4: Resources are defined as “skilled human resources (specific disciplines either individually or in crews or teams), equipment, services, supplies, commodities, materiel, budgets, or funds” (Program Management Institute, 2004, p. 372) but not the management of personnel as defined by leadership.Dependent Variable 5: Training is defined as “the level of learning required to adequately perform the responsibilities designated … and accomplish the mission” (DAU, 2005, p. B 170).
68Rival HypothesesUse of integrated commercial off-the-shelf (COTS) or non- developmental items (NDI)Burbank & Kasch (2004); Chung & Cooper (2002); Steves (1997)Use of a rapid acquisition process
69Plausibility of Rival Hypotheses Both type of development (NDI vice developmental) and type of acquisition (rapid vice traditional) will be accounted for and controlled as mediating variables.
70Research Design Approach Post-Positivistic with Hermeneutic (Constructivist) ElementsAppliedExplanatory (Confirmatory?)QuantitativeCorrelational (Predictive?)Survey
71Context of Study Setting Population Limitations Sample Design and Selection
72Setting Product Group Twelve (PG-12), Marine Corps Systems Command Multiple government and contractor facilities and workspaces
73Population Theoretical Study Population All military, government civilian and civilian contractor acquisition and engineering personnel supporting DoD IS/IT acquisition programs on behalf of a DoD acquisition agencyTarget Study PopulationRestricted to Marine Corps Systems Command
74Limitations Facility Access Duty Days, Duty Hours Formal Approval and/or Escort RequiredPersonnel AccessNot-to-Interfere BasisInformationRestrictions on Use of Classified and SBU
75Sample Design and Selection Random Selection InfeasibleResearch Setting RestrictedSmall Target Study PopulationNon-Random AlternativeGrounded Theory of Generalized Causal Inference (Shadish et al., 2002)Purposive Sample of a Typical Instance (Theoretical Study Population)Marine Corps Systems CommandProduct Group Twelve
76Sample Design and Selection Principles of Generalization (Framework for Validity Claims)Surface SimilarityCategorization of Phenomena Based on Similar Characteristics (Homogeneity)Ruling Out IrrelevanciesIgnoring Characteristics of Phenomena Not Relevant to Categorization (Heterogeneity)Making DiscriminationsDiscarding Phenomena That Do Not Fit CategorizationInterpolation and ExtrapolationGeneralizing Between or Beyond Sampled ValuesCausal ExplanationAttributing Causation Based on Theorized Structural Similarity
77Feasibility Analysis and Design Selection AssumptionsLimitationsDelimitationsDesign ApproachesResources
78AssumptionsAdditional technical risk discovered during IV&V is due to a failure of one or more aspects of the ISSE implementationTheoretical framework provided by identified barriers to quality implementations is applicable to ISSETarget study sample is assumed extensible to the target and theoretical study populationsPerceptions of surveyed personnel are a valid indicator of the variables measured
79Limitations Threats to Validity External Validity of the Purposive SampleAddressed Though Five Principles of GeneralizationConstruct Validity of Barriers to ImplementationAddressed Through Analytical Comparison with Original Construct (Factor Definitions / Item Loadings)Internal Validity due to Construct ModificationsAddressed Through Grounded Interpretation of Factors (Factor Definitions / Item Loadings)Statistical Conclusion ValidityAddressed Through Use of Generally Accepted Statistical Procedures and Statistical Software PackageThreats to Predictive AnalysisNon-Experimental Design; Small Sample SizeAddressed Through Full Disclosure of the Issues
80Delimitations Research Restricted to: DoD IS/IT DoD Acquisition Organizations
81Design Approach Operational Constraints Limited Invasiveness / InterferenceSuggests Non-Experimental DesignSite ConstraintsSmall PopulationLimited Sample SizeSuggests Non-Random SelectionSuggests Purposive SampleFoundational ConstraintsPartial Replication of Prior StudySuggests Exploratory or Confirmatory ResearchRequires Correlational DesignRequires Survey Study (Approach to Data Collection)
82Resources Limited Requirements General Computing and Printing ResourcesStatistical Software PackageWeb Hosting (Survey Instrument)Direct (e.g., Books, Literature Sources)Indirect (e.g., Communications, Travel)All Costs Born by the Author
83Research Design Specification RDS Completed and ApprovedStudy has IRB Approval; Permission to Collect Data ReceivedNext Steps …Survey Instrument Completed / Hosted on SurveyMonkeyData Collection to Start as Early as First Week in June 2008
85Clinger-Cohen Act Clinger-Cohen Act (CCA) IA-related requirements: IT RegistrationDoD IT RegistryDoN IT Registration DatabaseUSMC IT Registration Database ??IA StrategyStand-alone documentWritten at program inceptionMore comprehensive for higher levels of assuranceSupplemental/supporting documents:ISP (Policies, Standards, & Architecture)SSAA (Certification & Accreditation)
86National Policy, Regulations National Security Directive 42 (NSD-42)Established the National Security Telecommunications and Information Security Committee (NSTISSC)Chaired by the DoDProvides security guidance for National Security Systems (NSS)Evaluates the status of security for NSSNote: NSTISSC is now the Committee on National Security Systems, CNSS)
87FISMA Federal Information Security Management Act (FISMA) Federal Agencies must:Inventory systems (IT Database)Identify & provide appropriate security protections (IA Controls / C&A)Institute an INFOSEC programNIST provides security standards & guidelinesDoD provides own standards & guidanceOMB oversees FISMA (and CCA) complianceAuthorizes annual program reviewsNon-compliance can result in severe delays or elimination of funding!
88FISMA FISMA-compliance IA Controls C&A INFOSEC Program Reporting DODIC&ADoDI & DoD MINFOSEC ProgramDoDD & DoDIReportingHandled by MCSC IA Division
89OMB Circular A-130Office of Management and Budget (OMB) Circular A-130, Appendix IIIProvides security guidance and defines responsibilitiesDefines the use of “adequate security”“Commensurate with risk and magnitude of harm”Implements risk management vice risk avoidance
90NSTISSP 11National Security Telecommunications and Information Systems Security Policy (NSTISSP) No. 11“National Policy Governing the Acquisition of IA and IA-Enabled IT”Replaced “Orange Book”Requires use of National Information Assurance Program (NIAP)-certified productsBased on the Common Criteria (CC)Products evaluated against:Protection Profile (PP)Evaluation Assurance Level (EAL)
91NSTISSI 1000National Security Telecommunications and Information Systems Security Instruction (NSTISSI) No. 1000“National IA C&A Program (NIACAP)”Standardizes C&A for Federal AgenciesImplements C&A using FISMA guidelinesBasis for revision of current DoD C&A process
92Key Performance Parameters KPPs establish requirements (including security) specific to a particular systemObjective: Preferred capabilityThreshold: Minimum capabilityFailure to meet a threshold may result in a reevaluation, re-assessment or termination of the program, or a modification of the production incrementsExample: Time to restore full functionality into a degraded or corrupted systemObjective: 5 minutesThreshold: 15 minutes
95Relevant TermsAccreditation – Formal declaration by a DAA that an IS is approved to operate at an acceptable level of risk, based on the implementation of an approved set of technical, managerial, and procedural safeguards.Acquisition – The conceptualization, initiation, design, development, test, contracting, production, deployment, logistics support, modification, and disposal of systems to satisfy DoD needs intended for use in, or in support of, military missions.Authority to Operate (ATO) – The authorization granted by a DAA for a DoD IS to process, store, or transmit information.Certification – A comprehensive validation of actual IA capabilities and services of a DoD IS to establish compliance with assigned IA Controls.
96Relevant TermsCertifier – Individual responsible for making a technical judgment of the system’s compliance with stated requirements.Certification Determination – A certifier’s validation of the systems compliance with IA controls, identifying and assessing the risks with operating the system, and the cost to correct or mitigate the IA security weakness.Designated Accrediting Authority (DAA) – Official with the authority to formally assume responsibility for operating a system at an acceptable level of risk.DoD Information System (IS) – Set of information resources organized for the collection, storage, processing, maintenance, use, sharing , dissemination, disposition, display, or transmission of information.
97Relevant TermsInformation Assurance Control – An objective IA condition of integrity, availability, or confidentiality achieved through the application of specific safeguards or through the regulation of specific activities.Independent Verification and Validation (IV&V) – An independent review … [of a system] performed by an organization that is technically, managerially, and financially independent of the development organization.Information Systems Security Engineering – An engineering process that captures and refines information protection requirements and ensures their integration into [IS] acquisition processes through purposeful security design and configuration.Residual Risk – Risk due to a partial or unsatisfactory implementation of assigned IA controls.
98Relevant TermsRisk – Possibility that a particular threat will adversely impact an IS by exploiting a particular vulnerability.Technical Controls – Security controls (i.e., safeguards or countermeasures) for an IS that are primarily implemented and executed by the IS through mechanisms contained in the hardware, software, or firmware components supporting the system.Test & Evaluation – Process by which a system or components are exercised and results analyzed to provide performance-related information.Threat – Any circumstance or event with the potential to adversely impact an IS through unauthorized access, destruction, disclosure, modification of data, and/or denial of service.
99Relevant TermsValidation – Activity applied throughout the system lifecycle, to confirm or establish by testing, evaluation, examination, investigation, or competent evidence that a DoD information system’s assigned IA Controls are implemented correctly and are effective in their application.Vulnerability – Weakness in an IS, system security procedures, internal controls, or implementation that could be exploited.