1. University of Southern California Center for Software Engineering CSE USC Engineering Systems at USC: Center for Systems and Software Engineering Barry Boehm CESUN University Roundtable April 2, 2008 boehm@usc.eduboehm@usc.edu; http://csse.usc.edu

2. University of Southern California Center for Software Engineering CSE USC 04-02-2008©USC-CSE2 Outline CSE, SAE, and CSSE CSSE scope, objectives, and strategy Research and education examples Recent major developments –Integrating systems and software engineering –Value-based systems and software engineering

3. University of Southern California Center for Software Engineering CSE USC CSE, SAE, and CSSE Center for Software Engineering (CSE) –Large research program, Affiliates’ program –MSCS-SwE and PhD program Systems Architecting and Engineering (SAE) Program –Large MS program –Plans for PhD, Affiliates’ program Proposal to Affiliates –Set up SAE Affiliates’ program –Membership: $25K/year for either; $40K/year for both Affiliates’ response –We’re integrating SysE and SwE; you should do this too Result: Center for Systems and Software Engineering –Membership: $35K/year ($5K/year for small organizations) 04-02-2008©USC-CSE3

4. University of Southern California Center for Software Engineering CSE USC 10-08-2007©USC-CSE4 Research, Transition, and Education Strategies Stakeholder Win-Win –Principals, affiliates, project clients, practitioners, students Opportunistic collaboration –Focus on high-impact current, future problem areas –Principals’ criterion: shared interest in SysE/SwE Commercial or open-source versions of tools Research/education integration via real-client projects –Primarily USC campus, neighborhood e-services –Validate new methods and tools via project usage –Partial basis of 11 PhD dissertations –Rqts. negotiation, formalization (2), COTS integration (2), Value-based methods (2), Agile methods (1), Quality tradeoffs (1), Risk analysis (1), Cost estimation (1), Testbeds (1)

5. University of Southern California Center for Software Engineering CSE USC 10-08-2007©USC-CSE5 Why Software Projects Fail

6. University of Southern California Center for Software Engineering CSE USC 10-08-2007©USC-CSE6 “Software Engineering:” The disciplines which distinguish the coding of a computer program from the development of a software system product Requirements, Architecture Design, Code Implement, Maintain Computer ScienceCS Focus User Applications Economics People Accommodate new tools and techniques Web apps, Architecting tools, WikiWinWin, Spiral processes Integrate all these considerations –Via Incremental Commitment Model (ICM) –Necessarily emphasizes systems architecting, engineering Stages Issues - Involves 9 times as much effort [Brooks, 1975]

7. University of Southern California Center for Software Engineering CSE USC 2007-08 Software Engineering Projects Box Office Database SystemJay McAdams24th Street Theatre Online Peer Review SystemStephen BucherUSC, Engineering Writing Program Thai CDC Software ToolPheel WangThai Community Development Center USC COINCOMOEd Colbert, Vu NguyenUSC CSSE Crystal Stairs DashboardRick CapellaCrystal Stairs, Inc. Family & Homeless Well-Being ProjectJill RemelskiSt. Francis Center (SFC) BTI Appraisal ProjectsMegan Tunnell O'RourkeBTI Appraisal LAMAS Customer Service ApplicationCelia CastellanosLos Angeles Music and Art School Web-based service for TPC Foundation Pat Means Turning Point URBAN Magazine REEO Database Scott Stimpfel Resources for Educational and Employment Opportunities BID review SystemKeith CullingBackground Investigation Division, LA city THSA Website ProjectPoonchana ThitametakulUSC THSA EEO Section DatabaseArt IrigoyenLA city Conference Room Reservation SystemLinda LanierLA city Applicant-processing systems for CLAPDDee Dee CoultasLA city E-Mentoring programCarlene Davis Los Angeles Urban League Social Networking Tools for LibrariansJulie KwanSocial Networking Tools for Librarians EZSIM IIRay MadachyUSC-CSSE Los Angeles County Generation Web InitiativeFrank Cheng, Jeramy GrayLA County Development Of Hunter-gatherer DatabaseChris BoehmUSC 10-08-2007©USC-CSE7

8. University of Southern California Center for Software Engineering CSE USC 4/18/06©USC-CSE8 First Six Weeks of Project Course Domain Model WinWin Taxonomy Basic Concept of Operation Frequent Risks Stakeholders, Primary win conditions WinWin Negotiation Model IKIWISI Model, Prototypes, Properties Models Environment Models WinWin Agreements, Shared Vision Viable Architecture Options Updated Concept of Operation Life Cycle Plan elements Outstanding LCO risks Requirements Description LCO Rationale Life Cycle Objectives (LCO) Package Anchor Point Model determines identifies determines situatesexercise focus use of focus use of determines guides determination of validate inputs for provides initializeadoptidentify update achieveiterate to feasibility, consistency determines exit criteria for validates readiness of initializesinitializes

9. University of Southern California Center for Software Engineering CSE USC CSSE Principals (37) Core in CS, SAE, ISE (20) Engineering: Aerospace, Civil, Electrical, Mechanical (8) Schools of Business, Public Policy (4) Information Sciences Institute, Institute for Creative Technology, CREATE Center (5) 7 NAE members; 4 INCOSE Fellows; 9 IEEE Fellows 10-08-2007©USC-CSE9

10. University of Southern California Center for Software Engineering CSE USC Major Research Projects Integrating systems and software engineering (OSD-SSE) –2007: Inhibitors and enablers study –2008: IS&SE via Incremental Commitment Model guidebook Systems of systems engineering (Army FCS, OSD-SSE) –2001- : Future Combat Systems S&SE support –2007- : OSD-SSE Systems of Systems Guidebook support –2007: DACS system of systems cost estimation framework Next-generation space systems (DARPA, NASA) –2003- : Software reliability; processes; risk assessment Scaling up agile methods (Affiliates) –2003- : Architecture-based agile methods Value-based science of design (NSF) –2003- : Value-based theories of software, systems engineering 04-02-2008©USC-CSE10

11. University of Southern California Center for Software Engineering CSE USC 08/31/06©USC-CSE11 FCS WinWin Spiral Model 1b. Stakeholders Identify System Objectives, Constraints, & Priorities (OC&Ps); Alternative Solution Elements 1a. Identify Success-Critical Stakeholders 2a. Evaluate Alternatives with respect to OC&Ps 2b. Assess, Address Risks 3. Elaborate Product and Process Definition 4. Verify and Validate Product and Process Definitions Stakeholders’ Commitment 4 5 6 8 2 1 Stakeholders’ Review 7 3 L COL CA Build 2 Build 3 Progress Through Steps Bl 1 Driven By: Success- critical stakeholders’ win conditions Risk Management Spiral anchor point milestones Feasibility Rationale

12. University of Southern California Center for Software Engineering CSE USC 10-08-2007©USC-CSE12 How much Architecting is Enough? - A COCOMOII Analysis Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution Added Schedule Devoted to Rework (COCOMO II RESL factor) Total % Added Schedule 10000 KSLOC 100 KSLOC 10 KSLOC Sweet Spot Sweet Spot Drivers: Rapid Change: leftward High Assurance: rightward

13. University of Southern California Center for Software Engineering CSE USC 10-08-2007©USC-CSE13 Tradeoffs Among Cost, Schedule, and Reliability: Want $5.5M, 20 months, 10K hours MTBF -- Cost/Schedule/RELY: “pick any two” points (RELY, MTBF (hours)) Cost, schedule, quality: pick any two? For 100-KSLOC set of features Can “pick all three” with 77-KSLOC set of features

14. University of Southern California Center for Software Engineering CSE USC 08/31/06©USC-CSE14 COSYSMO Size Drivers Effort Multipliers Effort Calibration to 42 projects # Requirements # Interfaces # Scenarios # Algorithms + 3 adjustment factors - Application factors -8 factors - Team factors -6 factors COSYSMO Operational Concept

15. University of Southern California Center for Software Engineering CSE USC 04-02-2008©USC-CSE15 Outline CSE, SAE, and CSSE CSSE scope, objectives, and strategy Research and education examples Recent major developments –Integrating systems and software engineering –Value-based systems and software engineering

16. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE16 2007 OSD-SSE Study Scope and Context Statement of Work –Evaluate current DoD SysE, SwE processes and standards Primarily SEP Guide, DAG Ch. 4 (SysE), DoDI 5000.2 Identify deltas, issue areas, recommendations –Evaluate ICM for integration applicability –Evaluate relevant DoD guidance, education, and training Provide recommendations based on evaluations Context: current and future DoD S&SE challenges –Multi-owner, multi-mission systems of systems –Rapid pace of change –Always-on, never-fail systems –Need to turn within adversaries’ OODA loop Observe, orient, decide, act –Within asymmetric adversary constraints

17. University of Southern California Center for Software Engineering CSE USC An omniscient authority pre-establishes the requirements, preferred system concept, etc. (A4.1, 4.2, 4.4) Technical solutions are mapped and verified with respect to these (A4.1, 4.2, 4.4) They and technology do not change very often (A4.2, 4.5, 6.2) –Emphasis on rigor vs. adaptability The program has stable and controllable external interfaces (A4.5) Project organization is function-hierarchical and WBS-oriented (A3.1, 3.2) All requirements are equally important (A4.2) Reviews event/product-based vs. evidence-based (A5.2) The program’s critical path can be defined up front (A6.1) Can achieve optimum readiness at minimum life-cycle cost (A6.5) –No slack for contingencies 03/19/2008©USC-CSSE17 Current SysE Guidance: Outdated Assumptions Example: SysE Plan Preparation Guide Sometimes OK for hardware; generally not for software

18. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE18 System/Software Architecture Mismatches - Maier, 2006 System Hierarchy –Part-of relationships; no shared parts –Function-centric; single data dictionary –Interface dataflows –Static functional- physical allocation Software Hierarchy –Uses relationships; layered multi-access –Data-centric; class-object data relations –Interface protocols; concurrency challenges –Dynamic functional- physical migration

19. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE19 Fractionated, incompatible sensor data management “Touch Football” interface definition earned value –Full earned value taken for defining interface dataflow –No earned value left for defining interface dynamics Joining/leaving network, publish-subscribe, interrupt handling, security protocols, exception handling, mode transitions –Result: all green EVMS turns red in integration Examples of Architecture Mismatches … Sensor 1 SDMS1 Sensor 2 SDMS2 Sensor 3 SDMS3 Sensor n SDMSn ……

20. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE20 The Incremental Commitment Life Cycle Process: Overview Stage I: DefinitionStage II: Development and Operations Anchor Point Milestones Synchronize, stabilize concurrency via FEDs Risk patterns determine life cycle process

21. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE21 Anchor Point Feasibility Evidence Description Evidence provided by developer and validated by independent experts that: If the system is built to the specified architecture, it will –Satisfy the requirements: capability, interfaces, level of service, and evolution –Support the operational concept –Be buildable within the budgets and schedules in the plan –Generate a viable return on investment –Generate satisfactory outcomes for all of the success-critical stakeholders All major risks resolved or covered by risk management plans Serves as basis for stakeholders’ commitment to proceed Can be used to strengthen current schedule- or event-based reviews

22. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE22 Incremental Commitment in Gambling Total Commitment: Roulette –Put your chips on a number E.g., a value of a key performance parameter –Wait and see if you win or lose Incremental Commitment: Poker, Blackjack –Put some chips in –See your cards, some of others’ cards –Decide whether, how much to commit to proceed

23. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE23 Scalable remotely controlled operations

24. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE24 Total vs. Incremental Commitment – 4:1 RPV Total Commitment –Agent technology demo and PR: Can do 4:1 for $1B –Winning bidder: $800M; PDR in 120 days; 4:1 capability in 40 months –PDR: many outstanding risks, undefined interfaces –$800M, 40 months: “halfway” through integration and test –1:1 IOC after $3B, 80 months Incremental Commitment [number of competing teams] –$25M, 6 mo. to VCR [4]: may beat 1:2 with agent technology, but not 4:1 –$75M, 8 mo. to ACR [3]: agent technology may do 1:1; some risks –$225M, 10 mo. to DCR [2]: validated architecture, high-risk elements –$675M, 18 mo. to IOC [1]: viable 1:1 capability –1:1 IOC after $1B, 42 months

25. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE25 The Incremental Commitment Life Cycle Process: Overview Stage I: DefinitionStage II: Development and Operations Anchor Point Milestones Concurrently engr. OpCon, rqts, arch, plans, prototypes Concurrently engr. Incr.N (ops), N+1 (devel), N+2 (arch)

26. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE26 ICM HSI Levels of Activity for Complex Systems

27. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE27 Different Risk Patterns Yield Different Processes

28. University of Southern California Center for Software Engineering CSE USC 03/19/2008 Common Risk-Driven Special Cases of the ICM Special CaseExampleSize, Complexity Change Rate % /Month CriticalityNDI SupportOrg, Personnel Capability Key Stage I Activities : Incremental DefinitionKey Stage II Activities: Incremental Development, Operations Time per Build; per Increment 1. Use NDISmall AccountingCompleteAcquire NDIUse NDI 2. AgileE-servicesLow1 – 30Low-MedGood; in place Agile-ready Med-high Skip Valuation, Architecting phasesScrum plus agile methods of choice<= 1 day; 2-6 weeks 3. Architected Agile Business data processing Med1 – 10Med-HighGood; most in place Agile-ready Med-high Combine Valuation, Architecting phases. Complete NDI preparation Architecture-based Scrum of Scrums2-4 weeks; 2-6 months 4. Formal MethodsSecurity kernel; Safety-critical LSI chip Low0.3Extra HighNoneStrong formal methods experience Precise formal specificationFormally-based programming language; formal verification 1-5 days; 1-4 weeks 5. HW component with embedded SW Multi-sensor control device Low0.3 – 1Med-Very High Good; In place Experienced; med-high Concurrent HW/SW engineering. CDR- level ICM DCR IOC Development, LRIP, FRP. Concurrent Version N+1 engineering SW: 1-5 days; Market-driven 6. Indivisible IOCComplete vehicle platform Med – High 0.3 – 1High-Very High Some in placeExperienced; med-high Determine minimum-IOC likely, conservative cost. Add deferrable SW features as risk reserve Drop deferrable features to meet conservative cost. Strong award fee for features not dropped SW: 2-6 weeks; Platform: 6-18 months 7. NDI- IntensiveSupply Chain Management Med – High 0.3 – 3Med- Very High NDI-driven architecture NDI-experienced; Med-high Thorough NDI-suite life cycle cost- benefit analysis, selection, concurrent requirements/ architecture definition Pro-active NDI evolution influencing, NDI upgrade synchronization SW: 1-4 weeks; System: 6-18 months 9. Hybrid agile / plan-driven system C4ISRMed – Very High Mixed parts: 1 – 10 Mixed parts; Med-Very High Mixed parts Full ICM; encapsulated agile in high change, low-medium criticality parts (Often HMI, external interfaces) Full ICM,three-team incremental development, concurrent V&V, next- increment rebaselining 1-2 months; 9-18 months 9. Multi-owner system of systems Net-centric military operations Very HighMixed parts: 1 – 10 Very HighMany NDIs; some in place Related experience, med- high Full ICM; extensive multi-owner team building, negotiation Full ICM; large ongoing system/software engineering effort 2-4 months; 18- 24 months 10. Family of systems Medical Device Product Line Med – Very High 1 – 3Med – Very High Some in placeRelated experience, med – high Full ICM; Full stakeholder participation in product line scoping. Strong business case Full ICM. Extra resources for first system, version control, multi- stakeholder support 1-2 months; 9- 18 months C4ISR: Command, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance. CDR: Critical Design Review. DCR: Development Commitment Review. FRP: Full-Rate Production. HMI: Human-Machine Interface. HW: Hard ware. IOC: Initial Operational Capability. LRIP: Low-Rate Initial Production. NDI: Non-Development Item. SW: Software

29. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE29 Example ICM Commercial Application: Symbiq Medical Infusion Pump Winner of 2006 HFES Best New Design Award Described in NRC HSI Report, Chapter 5

30. University of Southern California Center for Software Engineering CSE USC 04-02-2008©USC-CSE30 Outline CSE, SAE, and CSSE CSSE scope, objectives, and strategy Research and education examples Recent major developments –Integrating systems and software engineering –Value-based systems and software engineering (VBSEE)

31. University of Southern California Center for Software Engineering CSE USC 06/25/07© USC-CSE31 VBSEE Motivation and Context INCOSE Definition of “Systems Engineering” –An interdisciplinary approach and means to enable the realization of successful systems –A systems engineering theory should provide criteria and guidance for realizing successful systems Addressing the Intellectual Content of Systems Engineering –How distinguished from other engineering disciplines –How related to other theories with intellectual content

32. University of Southern California Center for Software Engineering CSE USC 06/25/07© USC-CSE32 Intellectual Content of Systems Engineering: Distinguishing Features The intellectual content of most engineering disciplines is component-oriented and value-neutral –Ohm’s Law, Hooke’s Law, Newton’s Laws The intellectual content of systems engineering is focused on: –How people and components can be combined into a cost- effective system System definition, architecting, analysis, planning, evaluation, improvement Cost-effectiveness in terms of value to stakeholders The ability to address value considerations is an essential qualification for a theory of systems engineering

33. University of Southern California Center for Software Engineering CSE USC 06/25/07© USC-CSE33 Theory W: Enterprise Success Theorem – And informal proof Theorem: Your enterprise will succeed if and only if it makes winners of your success-critical stakeholders Proof of “if”: Everyone that counts is a winner. Nobody significant is left to complain. Proof of “only if”: Nobody wants to lose. Prospective losers will refuse to participate, or will counterattack. The usual result is lose-lose.

34. University of Southern California Center for Software Engineering CSE USC 06/25/07© USC-CSE34 Theory W: WinWin Achievement Theorem Making winners of your success-critical stakeholders requires: i.Identifying all of the success-critical stakeholders (SCSs). ii.Understanding how the SCSs want to win. iii.Having the SCSs negotiate a win-win set of product and process plans. iv.Controlling progress toward SCS win-win realization, including adaptation to change.

35. University of Southern California Center for Software Engineering CSE USC 06/25/07© USC-CSE35 VBSET Component Theories Theory W (Stakeholder win-win) –Enterprise Success Theorem, Win-Win Achievement Theorem Dependency Theory (Product, process, people interdependencies) –Systems architecture/performance theory, costing and scheduling theory; organization theory Utility Theory –Utility functions, bounded rationality, Maslow need hierarchy, multi-attribute utility theory Decision Theory –Statistical decision theory, game theory, negotiation theory, Theory of Justice Control Theory –Observability, predictability, controllability, stability theory

36. University of Southern California Center for Software Engineering CSE USC 08/2007©USC-CSSE36 Initial VBSE Theory: 4+1 Process – With a great deal of concurrency and backtracking Provides process for executing ICM Exploration phase

37. University of Southern California Center for Software Engineering CSE USC 06/25/07© USC-CSE37 Value-Based Testing: Empirical Data and ROI — LiGuo Huang, ISESE 2005 (a) (b)

38. University of Southern California Center for Software Engineering CSE USC 04/19/0738 By NumberP-value% Gr A higher By ImpactP-value% Gr A higher Average of Concerns 0.20234 Average Impact of Concerns 0.04965 Average of Problems 0.05651 Average Impact of Problems 0.01289 Average of Concerns per hour 0.02655 Average Cost Effectiveness of Concerns 0.004105 Average of Problems per hour 0.02361 Average Cost Effectiveness of Problems 0.007108 Group A: 15 IV&V personnel using VBR procedures and checklists Group B 13 IV&V personnel using previous value-neutral checklists – Significantly higher numbers of trivial typo and grammar faults Experiment: Reviews of Prioritized Ops Concept, Requirements, and Architecture Descriptions Value-Based Reading (VBR) Experiment — Keun Lee, ISESE 2005

39. University of Southern California Center for Software Engineering CSE USC 03/19/2008 Conclusions: IS&SE with ICM Current DoD (and other) SysE guidance seriously inhibits SwE –Largely sequential definition, design, development, integration, and test –Slows agility, ability to turn inside adversaries’ OODA loop –Functional “part-of” vs. layered “served by” product hierarchy –Static vs. dynamic interfaces: messages vs. protocols –One-size-fits-all process guidance inhibits balancing agility and assurance Incremental Commitment Model (ICM) enables better IS&SE –Integrates hardware, software, human factors aspects of SysE –Concurrent exploration of needs, opportunities, solutions Enabled by value-based systems engineering theory and process –Successfully addresses issues above in commercial practice –Only partially proven in DoD practice (examples: CrossTalk Top-5 projects) –Key practices applied to help major programs (example: Future Combat Systems) –Being adopted by major programs (example: Missile Defense Agency) –Effort underway to work with adopters to develop DoD ICM Guidebook, in coordination with other USD(AT&L)SSE initiatives ©USC-CSSE 39

40. University of Southern California Center for Software Engineering CSE USC 2008 SOW: DoD ICM Guidebook Develop a draft Guidebook for next-generation DoD IS&SE based on the ICM –In collaboration with DoD and industry participants Collaborate with selected DoD and industry parties in experimental tailoring of draft Guidebook elements Hold a series of Guidebook workshops –Mar 19-20 (USC), July 15-17 (DC area), Oct 29-30 (USC) Coordinate Guidebook with other IS&SE initiatives –Systems of systems engineering, systemic analysis, acquisition, education, assessment, research and technology 03/19/2008 ©USC-CSSE 40

41. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE41 References - I Beck, K., Extreme Programming Explained, Addison Wesley, 1999. Boehm, B., “Some Future Trends and Implications for Systems and Software Engineering Processes”, Systems Engineering 9(1), pp. 1-19, 2006. Boehm, B., Brown, W., Basili, V., and Turner, R., “Spiral Acquisition of Software-Intensive Systems of Systems, CrossTalk, Vol. 17, No. 5, pp. 4-9, 2004. Boehm, B. and Lane J., "21st Century Processes for Acquiring 21st Century Software-Intensive Systems of Systems." CrossTalk: Vol. 19, No. 5, pp.4-9, 2006. Boehm, B., and Lane, J., “Using the ICM to Integrate System Acquisition, Systems Engineering, and Software Engineering,” CrossTalk, October 2007, pp. 4-9. Boehm, B., and Lane, J., “A Process Decision Table for Integrated Systems and Software Engineering,” Proceedings, CSER 2008, April 2008. Boehm, B., “Future Challenges and Rewards for Software Engineers,” DoD Software Tech News, October 2007, pp. 6-12. Boehm, B. et al., Software Cost Estimation with COCOMO II, Prentice Hall, 2000. Boehm, B., Software Engineering Economics, Prentice Hall, 2000. Carlock, P. and Fenton, R., "System of Systems (SoS) Enterprise Systems for Information-Intensive Organizations," Systems Engineering, Vol. 4, No. 4, pp. 242-26, 2001. Carlock, P., and J. Lane, “System of Systems Enterprise Systems Engineering, the Enterprise Architecture Management Framework, and System of Systems Cost Estimation”, 21st International Forum on COCOMO and Systems/Software Cost Modeling, 2006. Checkland, P., Systems Thinking, Systems Practice, Wiley, 1980 (2 nd ed., 1999). Department of Defense (DoD), Defense Acquisition Guidebook, version 1.6, http://akss.dau.mil/dag/, 2006.http://akss.dau.mil/dag/ Department of Defense (DoD), Instruction 5000.2, Operation of the Defense Acquisition System, May 2003. Department of Defense (DoD), Systems Engineering Plan Preparation Guide, USD(AT&L), 2004. Electronic Industries Alliance (1999); EIA Standard 632: Processes for Engineering a System Galorath, D., and Evans, M., Software Sizing, Estimation, and Risk Management, Auerbach, 2006. Hall, E.T., Beyond Culture, Anchor Books/Doubleday, 1976.

42. University of Southern California Center for Software Engineering CSE USC 03/19/2008©USC-CSSE42 References -II Highsmith, J., Adaptive Software Development, Dorset House, 2000. International Standards Organization, Information Technology Software Life Cycle Processes, ISO/IEC 12207, 1995 ISO, Systems Engineering – System Life Cycle Processes, ISO/IEC 15288, 2002. Jensen, R. “An Improved Macrolevel Software Development Resource Estimation Model,” Proceedings, ISPA 5, April 1983, pp. 88-92. Krygiel, A., Behind the Wizard’s Curtain; CCRP Publication Series, July, 1999, p. 33 Lane, J. and Boehm, B., "System of Systems Cost Estimation: Analysis of Lead System Integrator Engineering Activities", Information Resources Management Journal, Vol. 20, No. 2, pp. 23-32, 2007. Lane, J. and Valerdi, R., “Synthesizing SoS Concepts for Use in Cost Estimation”, Proceedings of IEEE Systems, Man, and Cybernetics Conference, 2005. Lientz, B., and Swanson, E.B., Software Maintenance Management, Addison Wesley, 1980. Madachy, R., Boehm, B., Lane, J., "Assessing Hybrid Incremental Processes for SISOS Development", USC CSSE Technical Report USC-CSSE-2006-623, 2006. Maier, M., “Architecting Principles for Systems-of-Systems”; Systems Engineering, Vol. 1, No. 4 (pp 267- 284). Maier, M., “System and Software Architecture Reconciliation,” Systems Engineering 9 (2), 2006, pp. 146- 159. Northrop, L., et al., Ultra-Large-Scale Systems: The Software Challenge of the Future, Software Engineering Institute, 2006. Pew, R. W., and Mavor, A. S., Human-System Integration in the System Development Process: A New Look, National Academy Press, 2007. Putnam, L., “A General Empirical Solution to the Macro Software Sizing and Estimating Problem,” IEEE Trans SW Engr., July 1978, pp. 345-361. Rechtin, E. Systems Architecting, Prentice Hall, 1991. Schroeder, T., “Integrating Systems and Software Engineering: Observations in Practice,” OSD/USC Integrating Systems and Software Engineering Workshop, http://csse.usc.edu/events/2007/CIIForum/pages/program.html, October 2007. http://csse.usc.edu/events/2007/CIIForum/pages/program.html

43. University of Southern California Center for Software Engineering CSE USC 03/19/2008 ©USC-CSSE List of Acronyms B/LBaselined C4ISRCommand, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance CDConcept Development CDRCritical Design Review COTSCommercial Off-the-Shelf DCRDevelopment Commitment Review DIDevelopment Increment DoDDepartment of Defense ECRExploration Commitment Review EVMSEarned Value Management System FCRFoundations Commitment Review FDNFoundations, as in FDN Package FEDFeasibility Evidence Description FMEAFailure Modes and Effects Analysis FRPFull-Rate Production GAOGovernment Accountability Office GUIGraphical User Interface

44. University of Southern California Center for Software Engineering CSE USC 03/19/2008 ©USC-CSSE List of Acronyms (continued) HMIHuman-Machine Interface HSIHuman-System Interface HWHardware ICMIncremental Commitment Model IOCInitial Operational Capability IRRInception Readiness Review IS&SEIntegrating Systems and Software Engineering LCALife Cycle Architecture LCOLife Cycle Objectives LRIPLow-Rate Initial Production MBASEModel-Based Architecting and Software Engineering NDINon-Developmental Item NRCNational Research Council OCOperational Capability OCROperations Commitment Review OO&DObserve, Orient and Decide OODAObserve, Orient, Decide, Act O&MOperations and Maintenance

45. University of Southern California Center for Software Engineering CSE USC 03/19/2008 ©USC-CSSE List of Acronyms (continued) PDRPreliminary Design Review PMProgram Manager PRPublic Relations PRRProduct Release Review RUPRational Unified Process SoSSystem of Systems SoSESystem of Systems Engineering SSESystems and Software Engineering SWSoftware SwESoftware Engineering SysESystems Engineering Sys EngrSystems Engineer S&SESystems and Software Engineering USD (AT&L)Under Secretary of Defense for Acquisition, Technology, and Logistics VCRValidation Commitment Review V&VVerification and Validation WBSWork Breakdown Structure WMIWarfighter-Machine Interface