1. University of Southern California Center for Systems and Software Engineering Integrating Systems and Software Engineering: Technical Aspects Workshop 30 October 2007

2. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE2 IS&SE Workshop Objectives: Identify Biggest issues and opportunities? –Technical aspects –Management aspects –Complex systems aspects Inhibitors to progress and how to overcome them? Ability of six principles to improve IS&SE? Ability of Incremental Commitment Model to improve IS&SE? What else is needed? –Technology/management research –Education and training –Regulations, specifications and standards –Other?

3. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE3 Attendees Rick Selby, NGC/USC (Chair) Anthony Peterson, Raytheon Karen Owens, Aerospace Corporation George Huling, LA Spin JinBo Chen, NGC Wilson Rosa, Air Force Cost Analysis Agency Rich Turner, Stevens Institute of Technology John Miller, MITRE Elliot Axelband, USC/RAND Jared Fortune, Aerospace/USC Shawn Rahmani, Boeing Daniel Winton, Aerospace Sue Koolmanojwong, USC (Scribe) Winsor Brown, USC

4. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE4 IS&SE for Technical Aspects : Summary SE/SW integration –Systems engineering is doing an commendable job in incorporating software, hardware, and human factors as it has for years however improvements are needed –A lot of initiatives are pushed by software people, because they suffered from the consequences of inadequate SE (integration) –Significant benefits can be achieved (or are possible) by greater integration and synergy between systems and software engineering in many areas, such as communication, education, research, cross training, integrated life cycle models Personnel/Education/Communication –Too few “software engineers” have adequate engineering background, domain expertise, or physics understanding / exposure –At the engineering executive and chief engineering level, few come from software engineering background Research, initiatives, guidance, policy –Prototype technical policy –Need major DoD initiative to fund research centers for SE/SW

5. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE5 IS&SE for Technical Aspects: Issues Earned value is a rear-view mirror metric; does not adapt to added work over time, not reflect the rework Commitment and accountability: how to make sure/ verify the commitment and accountability (more than powerpoint) For project implementation, it is difficult to predict schedule; requirement volatility drives schedule unpredictability Lack of standardization within DoD for technical standards such as software requirement specifications, capability design documents Include mechanism for monitoring cost and technical performance throughout the life cycle of the systems One-sided systems and software engineering marriage attempt –Integration attempt pushed by software engineering and not adequately pulled by system engineering (e.g. CMM and CMMI, system and software consortium – SSCI, UML and SysML, software technology conference and systems and software technology conference, center for software engineering and center for systems and software engineering) –Need system engineering pushed/pulled integration initiatives for an acceptable marriage

6. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE6 IS&SE for Technical Aspects : Biggest Opportunities(1) Define integrated measures for systems and software engineering Define a common core set of metrics (common across acquisition organization and prime contractor) for software development that visible at system level (build on ASN RDA initiative) –Need Software-specific items within the system WBS to a level to support the metrics –Include software-specific criteria into checklist and entry/exit criteria at system engineering technical review (SETR) Augment technology readiness assessment (TRA) deskbook with a second set of software technology readiness levels (TRL) Leverage the attention and focus provided by OSD-AT&L on integration of systems and software engineering desire to provide center of excellence

7. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE7 IS&SE for Technical Aspects : Biggest Opportunities(2) Use true technical reviews to communicate between different disciplines, requiring the small set of the right people to be there, small set of people Ask systems engineers to develop test procedures (test group does not have the capacity to develop test procedure) Model-based engineering Prototype policy using a true IRT For systems engineering requirements and architecting processes, define and implement a quantitative set of quality attributes and checklist for coverage of software engineering

8. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE8 IS&SE for Technical Aspects : Biggest Opportunities(3) Develop tools for automated measurement of systems sizing metrics Use requirement document to construct end-to-end threads to cover every requirement such as not just users but sensors, in-bound radar data (JPL experience) Have systems engineers write test requirements, such as “What do we have to show to sell off this requirement” Combine earned value systems with agile development approach Systems engineering to co-own software engineering risks Explore automated generating code because we are unable to keep up with the systems/software size growth trend –Needs domain specific languages and domain specific models –Need to validate the results of the translation; compilers are not perfect

9. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE9 IS&SE for Technical Aspects: Inhibitors to progress and how to overcome them (1) InhibitorHow to Overcome Inhibitor “Never changing” earned value is required (reflects spent costs) An agile rebaselining process Lack of communication between system engineering and other types of engineering and software engineering due to lack of overlapping of educational background Define and deliver 90-day “crash courses” on software to non-software engineer and on physics/math/etc. to software people; small group of key people Culture: company1.Verifiable demonstrated prototype evaluated by a true IRT 2.Properly selected and trained program manager who appreciate software, hardware and people; must be managed from the system perspective Culture: government; a history of predicating policy on unvalidated pilot, e.g., acquisition reform, CMM, CMMI, and DoD BMP (business management plan) ; Prototype policy using a IRT Culture: congress; Not interested in open, e.g., cost plus or incremental contract (want fixed price) Demonstrated success and a selling campaign to explain same Culture: engineersDefine new lexicon of terminology; same words mean different things to SE and SW Communication; lack of prioritization, priorities commonly poorly communicated

10. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE10 IS&SE for Technical Aspects: Inhibitors to progress and how to overcome them (2) InhibitorHow to Overcome Inhibitor Large differences in technical background, scope, focus, and priority Do the old fashioned way mentoring TRA Deskbook software TRLs deal with domains and algorithms but do not address the development of software with respect to systems Develop system applicable TRL and TRA for software (and an assessment process) Separate, overly complex, poorly synchronized software engineering and systems engineering life cycles Replace traditional system life cycle, e.g., 1521 with an integrated systems and software life cycle Failure by OSD to include software-specific guidance in their system engineering plan (SEP) guidance Traditional waterfall acquisition model is not as flexible as would be optimal for opportunities such as ICM Prototyping different acquisition process

11. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE11 Process Model Principles (0) 1.Commitment and accountability 2.Success-critical stakeholder satisficing 3.Incremental growth of system definition and stakeholder commitment 4, 5.Concurrent, iterative system definition and development cycles Cycles can be viewed as sequential concurrently- performed phases or spiral growth of system definition 6.Risk-based activity levels and anchor point commitment milestones

12. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE12 IS&SE for Technical Aspects : Ability of six principles to improve IS&SE? Step 1 through 5 are what we do now Step 1 through 5 are things that we do now, but I question the level of rigor, such as original Fagan Inspections required a high level of rigor Risk-focused anchor point decisions are a great value-added concept of ICM. Unfortunately, most programs perform risk identification and management activities poorly Software has multiple, overlapping risk reduction possibilities but systems engineering risk mitigation plans have sometimes lacked integration; they do not reflect the overlap Risk-focused anchor point decisions duplicate what is already present and practice in current DoD risk management policies. What is lacking is a rigorous adherence to establish entry and exit criteria General concern that the six principles are already in place but lack rigor –ICM could introduce/bring about/reinforce a greater degree of rigor and formalism The rational that lead to the six principles is based on 19 years of misuse of using spiral model, as arrive by expert group responding to explaining the intended use of the spiral to the general accounting office (GAO)

13. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE13 Process Model Principles (1) 1.Commitment and accountability Should be supported, not prohibited, by RFP Should accommodate changes driven by risks 2.Success-critical stakeholder satisficing 3.Incremental growth of system definition and stakeholder commitment 4, 5.Concurrent, iterative system definition and development cycles Cycles can be viewed as sequential concurrently-performed phases or spiral growth of system definition 6.Risk-based activity levels and anchor point commitment milestones

14. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE14 Process Model Principles (2) 1.Commitment and accountability 2.Success-critical stakeholder satisficing Require access to success critical stakeholder and frequently they are in short supply Potential concern that emergent behavior by success critical stakeholder or non success critical stakeholder can be lost. Final system may not do what really needs to be done. Satisficing is a condition that may not be achievable; therefore Kill the program early Create conditions where satisficing is achievable Push forward regardless e.g. eliminate success critical stakeholder or criteria or knowingly proceed on the path to partial success Concept of satisficing requires Openness and honesty regarding win conditions Success critical stakeholder are milestone dependent Need to build in flexibility accommodate/ anticipate the changing mind of success critical stakeholders Keep stakeholders involved and continue to identify new ones 3.Incremental growth of system definition and stakeholder commitment 4, 5.Concurrent, iterative system definition and development cycles Cycles can be viewed as sequential concurrently-performed phases or spiral growth of system definition 6.Risk-based activity levels and anchor point commitment milestones

15. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE15 Process Model Principles(3) 1.Commitment and accountability 2.Success-critical stakeholder satisficing 3.Incremental growth of system definition and stakeholder commitment Improve method for quantifying the system/software size at the concept exploration phase Define representations to capture integrated systems / software artifacts such as specifications/architectures /implementations Imagine a future where there are executable artifacts delivered early in the lifecycle (other than code )such as executable specifications and executable architectures Executable specification captures nominal behavioral properties but not necessarily / off-nominal or realistic efficiency/fault-tolerance needs 4, 5.Concurrent, iterative system definition and development cycles Cycles can be viewed as sequential concurrently-performed phases or spiral growth of system definition 6.Risk-based activity levels and anchor point commitment milestones

16. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE16 Process Model Principles(4&5) 1.Commitment and accountability 2.Success-critical stakeholder satisficing 3.Incremental growth of system definition and stakeholder commitment 4, 5.Concurrent, iterative system definition and development cycles Cycles can be viewed as sequential concurrently- performed phases or spiral growth of system definition –Impossible to do concurrent, iterative definition/development unless you have got stakeholders involved concurrently and iteratively –Requires agile program management method 6. Risk-based activity levels and anchor point commitment milestones

17. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE17 Process Model Principles(6) 1.Commitment and accountability 2.Success-critical stakeholder satisficing 3.Incremental growth of system definition and stakeholder commitment 4, 5.Concurrent, iterative system definition and development cycles Cycles can be viewed as sequential concurrently-performed phases or spiral growth of system definition 6.Risk-based activity levels and anchor point commitment milestones Need to have greater emphasis on feasibility rationale Tom Schroeder’s presentation on difficulty of finding independent experts Milestone definition and feasibility rationale are independent; Feasibility rationale could be used to measure progress Define additional metaphors (engage, marriage, kids, and gambling)

18. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE18 IS&SE for Technical Aspects: Ability of ICM to improve IS&SE? (1) The rubber will hit the road when you need to define the specific deliverables capabilities in the first increment, how long it takes to get there? Many people may feel that there is nothing new here and “we are already doing this” –They often overlook on minimizing the risk-driven aspects –Seldom have explicit evidence of feasibility –Main point of ICM is that feasibility rationale became a first class citizen Define a detailed plan for prototyping/piloting, evaluating, and deploying (initial, medium, large scale) ICM including success criteria and a business case analysis general concern of premature focus on ICM as a point solution as opposed to the other way to implement the six principles Is ICM actually being proposed as a point solution or is this just a way to try something new on a small scale (need strategic positioning on ICM) To make ICM work, it requires the funding agency and the LSI / prime contractor to do much more than they are currently doing such as con ops development, testbed development, frequent evaluation of program status and making adjustments, gathering information to evaluate subcontractors/suppliers, and team building –LSI/prime contractor must engage the developer earlier in the process, so they do not end up with a poor con ops or design

19. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE19 IS&SE for Technical Aspects: Ability of ICM to improve IS&SE? (2) Could/should feasibility rationale be developed in the same way that the safety and assurance cases ? Can the guidance for those support the feasibility rationale development? Could the FR be seen as uber-case that organizes and gathers the information from both systems and software specific attribute cases as well as the business and stakeholder satisfaction cases?

20. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE20 IS&SE for Technical Aspects: What else is needed? (1) Technology/management research –Create a catalog of successful integrated systems and software architectural patterns, will help get systems engineering to think about what they need to do with software –Model-driven approach for integrated systems and software engineering –Simulation-based acquisition approaches, e.g., processes, methods, tools for integrated systems and software engineering –Launch a major DoD funded initiative for creating joint university-industry-government of research centers to address integrated systems and software engineering to catalyze research breakthroughs –Apply, and further research where necessary social science to overcome cultural barrier and facilitate team interaction, e.g., common training, symbolic language

21. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE21 IS&SE for Technical Aspects: What else is needed? (2) Education and training –Need hands-on manager as a part of the education / training process, e.g., six sigma training “too abstract and not applicable to the work” to fine tune the process –Need for personnel that have expertise in multiple domain such as software engineer that also know aerospace/autos/automobile/telecom –The regular engineer need exposure to software engineering –At what level is the expertise needed (MS level? ) Awareness would be an improvement

22. University of Southern California Center for Systems and Software Engineering October 2007©USC-CSSE22 IS&SE for Technical Aspects: What else is needed? (3) Regulations, specifications and standards –Real consequences for a non-compliance with policy or incentive structures (carrot vs 2x4) –Relieve those who are forced to follow policy that leads to failure from the consequences of the failure (JPL failure for 2 MARS missions that were failure as a consequence of faster better cheaper policy) –“None before their time”, e.g., before understood and their value demonstrated Other? –Prototype technical policy –Senate staffers at the sub committee level reduced the FCS program proposed fiscal year 08 budget by $ 635 M primarily from system engineering and management accounts. WA HA –Need root cause analysis and action plan to prevent for future case