Presentation on theme: "AF Systems Engineering Assessment Model (AF SEAM) Presented To:"— Presentation transcript:
1 AF Systems Engineering Assessment Model (AF SEAM) Presented To: International Council on Systems Engineering Finger Lakes ChapterGeorge Richard FreemanTechnical Director, US Air Force Center for Systems Engineering ext 341917 Sep 09
2 Agenda Background Development process AF SEAM construct AFSEAM process Results reportingSpiral 2 considerations / Lessons Learned
3 Why AF SEAM Challenges: Goals: AF programs late, over cost, & do not provide the performance expectedSECAF directed action to revitalize SE across the AFNo standard tool/method for assessing SE processesGoals:Promote consistent understanding of SEEnsure core SE processes are in place and being practicedFacilitate sharing “Best Practices”Provide “Brain Drain” insuranceImprove AF leadership visibility into SE process maturityImproved program performance & reduced technical risk
4 Defining the Methodology Assessment ContinuumLowHighAF SEAMCollaborative & inclusiveLeanest possible best practices “Must Dos”Clearly stated expectationsProgram team & assessor teamTrainingSelf-assessment of program with optional validationHands OffPromulgate PolicyDirectivesInstructionsChecklistsGuidanceExpect ComplianceHands OnComprehensive Continuous Process ImprovementHighly detailed process booksTrainingIndependent AssessmentDeep divesAssessment methods that balance time & depth to need
5 AFSEAM Development Team Members CenterMembersAACMr. Ian TalbotAEDCMr. Neil Peery, Maj Mark JenksASCMr. Gary BaileyAF CSEMr. G. Richard Freeman & Mr. Randy BullardHQ AFMCMrs. Caroline BuckeyESCMr. Bob Swarz & Mr. Bruce AllgoodOC-ALCMr. Cal Underwood & Mr. Bill RaphaelOO-ALCMr. Jim Belford & Ms. Mahnaz MaungSMCMs. Linda TaylorWR-ALCMr. Jim Jeter & Mr. Ronnie RogersPhenomenal Team Support !
6 Background Original task: AFMC EC Action Item Objective: “Develop standard AF assessment model”Tools were in 4 Centers12 On-Site Team EngagementsRepresentatives from EN Home Offices4 Product Centers, 3 ALCs, AEDC, HQ AFMC/EN, CSEMet 9 times at 5 different locations in one yearConducted 3 baseline assessments at 3 Centers12 Briefings to Senior LeadersAFMC Engineering Council Meetings (4)ALC EN MeetingSAF/AQR (2)AF Tech Leaders Round TableOSD (AT&L) & Boeing SE Advisory GroupNational Research Council (National Academies)Final to AFMC/EN – 5 Aug 08, & Final to SAFF/AQR – 11 Aug 08
7 Collaborative build – Included greater SE community Development ProcessEnvironmental Scan Up FrontExternal BenchmarkingExisting Best PracticesCollaborative Reviews/InputsSoftware Engineering Institute (CMMI)NDIAAF HSIOLHA Development TeamTD 1-12INCOSEIndustry PartnersCollaborative build – Included greater SE community
8 AF SEAM PedigreeSeveral AF product Centers selected and tailored some version of the Software Engineering Institute (SEI) Capability Maturity Model Integration (CMMI®) to baseline process institutionalizationSEI CMMI® is the Defense Industry-wide accepted method for process appraisal and improvementThe SEI CMMI® incorporates principles and practices from recognized industry and US Government system engineering and related standards such as:AFI Life Cycle Systems Engineering, Defense Acquisition Guidebook, Chapter 4MIL-STD 499B System EngineeringANSI/EIA Processes for Engineering a SystemIEEE/EIA 731 Systems Engineering Capability ModelISO/IEEE Systems Engineering-System Life Cycle ProcessesINCOSE System Engineering StandardIEEE Application and Management of the Systems Engineering Process
9 Formula For Change C fn[ N, V, M]>C of C C = Change N = Need Leaders Create the need to changeV = VisionLeaders provide a Vision of a better futureM = MeansLeaders manage the charge processC of C = Cost of ChangeMust be less than the benefit anticipated
10 Environment Sensitivity Sustaining Change Over TimeUnfreeze > Changes > Refreeze1 to 24 months 12 to to 48Case for Change Early wins Irreversible momentumRef: Lewin's Freeze Phases,
11 Individually Tailored SEAM AF SEAM ConstructAF SEAMAFCenter SpecificCenterSize/ ComplexityPoint in Life-CycleIndividually Tailored SEAMProgram
12 AF SEAM - CMMI-ACQv1.2 Requirements Design V&V Decision Analysis AF SEAM ProcessesCMMI-ACQ Processes v1.2RequirementsDesignV&VDecision AnalysisConfiguration MgmtRisk MgmtProject PlanningSustainmentManufacturingTech Mgmt & CtrlGeneric PracticesREQM – Requirements Management (RM)MA – Measurements & AnalysisPMC – Project Monitoring & ControlPP – Project PlanningPPQA – Process and Product Quality AssuranceSSAD – Solicitation & Supplier Agreement DevCM – Configuration ManagementDAR – Decision Analysis and ResolutionAM – Agreement ManagementARD – Acq Requirements DevelopmentATM – Acq Technical ManagementVAL – Acq ValidationVER – Acq VerificationOPD – Organizational Process DefinitionOPF – Organizational Process FocusIPM – Integrated Project Management (IPPD)RSKM – Risk ManagementOT – Organizational TrainingOPP – Organizational Process PerformanceQPM – Quantitative Project ManagementOID – Organizational Innovation & DeploymentCAR – Causal Analysis & Resolution23Reviewed CMMI and incorporated/tailored where appropriate. Noting that all areas are not being addressed equally nor to the same level as CMMI.45CMMI Color Legend: Green = Covered, Yellow = Partially, Red = Not CoveredCMMI Maturity Levels:1 Initial, 2 Managed, 3 Defined, 4 Quantitatively Managed, 5 Optimizing
13 AF SEAM - CMMI-DEVv1.2 Requirements Design V&V Decision Analysis Process AreaMaturity LevelCausal Analysis and Resolution5Configuration Management2Decision Analysis and Resolution3Integrated Project Management +IPPDMeasurement and AnalysisOrganizational Innovation and DeploymentOrganizational Process Definition +IPPDOrganizational Process FocusOrganizational Process Performance4Organizational TrainingProduct IntegrationProject Monitoring and ControlProject PlanningProcess and Product Quality AssuranceQuantitative Project ManagementRequirements DevelopmentRequirements ManagementRisk ManagementSupplier Agreement ManagementTechnical SolutionValidationVerificationAF SEAM ProcessesRequirementsDesignV&VDecision AnalysisConfiguration MgmtRisk MgmtProject PlanningSustainmentManufacturingTech Mgmt & CtrlGeneric PracticesReviewed CMMI and incorporated/tailored where appropriate. Noting that all areas are not being addressed equally nor to the same level as CMMI.CMMI Color Legend: Green = Covered, Yellow = Partially, Red = Not CoveredCMMI Maturity Levels:1 Initial, 2 Managed, 3 Defined, 4 Quantitatively Managed, 5 Optimizing
14 AF SEAM Content Process Areas (PAs) Goals Practices Informative MaterialDescriptionTypical Work ProductsReference MaterialOther ConsiderationsBroadestLevel of SpecificityMost Detailed
15 Process Areas # Symbol Process 1 CM Configuration Management 2 DA Decision Analysis3DDesign4MManufacturing5PPProject Planning6RRequirements7RMRisk Management8SSustainment9TMCTechnical Management & Control10VVerification & ValidationReviewed CMMI and incorporated/tailored where appropriate. Noting that all areas are not being addressed equally nor to the same level as CMMI.
16 Generic Practices # Practice Description GP1 Description of process Plans for performing the processGP3Adequate resources for performing the processGP4Responsibility & authority for performing the processGP5Train the people performing the processGP6Monitor & control the processGP7Review activities, status, & results of the processReviewed CMMI and incorporated/tailored where appropriate. Noting that all areas are not being addressed equally nor to the same level as CMMI.
17 AF SEAM Umbrella AF SEAM Manufacturing Readiness Assessments Safety Technology Readiness AssessmentsHuman Systems IntegrationLogistics Health AssessmentsRisk AssessmentsAnd More…….
18 Where They Intersect AF SEAM Risk ID Enhances existing TRA process to:Address issues wrt integration and ‘ilities of new technologiesForecast cost/sched/risk of moving to next TRL levelDevelops TRA & MRA courses to train workforceReduce ambiguities on interpreting TRLsEnsure understanding of TRLs & MRLsPrepare teams to conduct TRAs & MRAsEnables Acq & S&T communities to communicate on desired levels of tech maturityAF SEAMDetermines existence of defined SE practicesTen “classical SE” process areasValidates documented processes are being followedA self-inspection or independent-inspection toolWhat to DoHow to Do It
19 AF SEAM Process Descriptions 8.3 Design (D)DG3: Assemble the design/development prototype(s) in accordance with the detailed design and integration strategy.DG3P2 Establish and maintain procedures and criteria for integration of the product-componentsDescription: Procedures for the integration of the product-components can include such things as details of the expected tests and other evaluations to be carried out at each stage. Criteria can indicate the readiness of a product-component for integration or its acceptability. Assure component compatibility with the integration environment. Provide sufficient data to support certifications.Typical Work Products:Product and product-component certificationsIntegration and test documentsReference Material: AFI , AFI
20 AF SEAM Process Descriptions DG3P3 Manage internal and external interface definitions, designs, and changes for products and product-componentsDescription: Periodically review all interface descriptions for coverage and completeness. Effective management of product-component interface requirements, specifications, and designs helps ensure that implemented interfaces will support integration.Typical Work Products:Assembly procedures and sequencesInterface drawings and documentsIntegration and test documentsTCTO (Time Compliance Technical Orders)Digital System ModelsReference Material: AFI , AFIOther Considerations: Many product integration problems arise from unknown or uncontrolled aspects of both internal and external interfaces. Participation in all relevant interface control working groups (ICWGs) helps ensure that implemented interfaces will support integration.
21 AF SEAM Process Descriptions 8.10 Verification and Validation (V)VG1 Prepare for verificationVG1P1 Establish and maintain the overall verification strategy and plan, including integrated testing approachDescription: Address, in the verification strategy, the technical approach to product verification and the specific approaches that will be used to verify that work products meet their specified requirements. Establish verification criteria and methods for each requirement. Proactively seek opportunities for combined testing with the operational test agencies and oversight offices and form an Integrated Test Team.Typical Work Products:Document modeling, simulation, ground and flight test activities planned for product verification in the Test and Evaluation Master Plan (TEMP) or similar document if a TEMP does not exist.Use Cases are used to define test requirements.Reference Material: AFI , AFI , AFIOther Considerations: Work products can be developed by either the acquirer or suppliers. They should be selected based on their contribution to meeting project objectives and requirements, and to addressing project risks.
27 Tool Suite Management Guide Assessment Tool (Spreadsheet) Training Orientation/OverviewSelf-AssessmentValidation Team“I highly encourage each program at your Center to conduct a self-assessment using this model.” Mr. Jaggers, SAF/AQR, 14 May 2008
28 Development Schedule DELIVERED ON TIME ! 2007 2008 TASKJ J A S O N D J F M A M J J AID Process Areas (10)Define Process Area Goals (34)Defined Practices aligned to goals (120)Formative Material Includes:- Detail Practice Description- Practice Assessment Criteria- Reference MaterialsDevelop Assessment & Scoring MethodologyDevelop & Implement Training PlanID Baseline Test CandidatesPerform Baseline Training & AssessmentsIncorporate Baseline Test Feedback & Revise AF SEAMCoordinate Final Version (Spiral 1)Ground Theater Air Control SystemsTSAT at SMCMission Planning SystemsDELIVERED ON TIME !
29 Implementation By Center 5 AUG 08 - FEEDBACKAAC"AAC began integrating AF SEAM in our established program assessment process in January 2008 and expects to complete this integration in FY09."AEDC"We will begin implementing AF SEAM in October."ASC"We are creating a plan to migrate from our current tool to SEAM, tailored with AFMC and ASC specific areas of interest."ESC"We have initiated tailoring efforts to implement AF SEAM by the end of the calendar year. We will be working closely with SMC, our acquisition partner, on the tailoring and implementation effort."OC-ALC"Strongly support, have plans in place, ready to go!"OO-ALC"We are implementing now."SMC"SMC plans to adopt AF SEAM and comply with related policies."WR-ALC"We'll begin implementation at Robins with pilot assessments in F-15 and Avionics."Awaiting issuance of AQR policy letter.Development process yielded 100% buy-in
30 Spiral 2 Considerations / Lessons Learned Capability EnhancementRe-look at process areas for improvementsFurther refine assessment methodologyStrengthen inclusion of softwareCapture and promulgate best practices/lessons learnedReview scoringExamine potential use for SE health assessmentMigrate to web-based platformCharterEstablish vision & missionEstablish governanceSupport team by providing resourcesappropriate levelFundingSpiral 2 & SustainmentLead POC/Steering GroupStaff supportCommunity of InterestConfiguration control