1. Page 1 October 31, 2000 An Introduction to Large-Scale Software Development Steve Varnau Core HP-UX Operation October 31, 2000

2. Large- Scale SW Devel. October 31, 2000 Page 2 Agenda Software Change Software Configuration Management (SCM) Change Control Large-Scale Software Development Example: HP-UX Operating System

3. Large- Scale SW Devel. October 31, 2000 Page 3 The Need for Managing Software Change Environment Changes Computer Hardware Competition Business Processes External Requirements (Y2k, Euro,…) New Standards (XML, MP3, e-Speak,…) All Software Demands Change Incremental Development Maintenance (Corrective, Perfective, Adaptive) Software Evolution

4. Large- Scale SW Devel. October 31, 2000 Page 4 Benefits of Source Control / SCM Manage Constant Software Change Improve Debugging Productivity Trace Defect Reports to Correct Versions of Source Code Understand Change History Reproducibility Build (Compile) Software Accurately, Repeatably Grow Project Size Avoid Over-writing and Losing Changes Concurrent Development (Improve TTM) Coordinate Work with Development Policies SCM

5. Large- Scale SW Devel. October 31, 2000 Page 5 Terminology - File Versioning Revision / Version - Single copy of a source file as it appeared at given point in its change history (checkpoint) File / Element - Complete change history; all versions Branch - Set of successive versions; parallel to other branches Delta - Change from one file version to the next SCM

6. Large- Scale SW Devel. October 31, 2000 Page 6 Terminology - Version Tree SCM

7. Large- Scale SW Devel. October 31, 2000 Page 7 Terminology - Multiple File Configuration Management Variant - File based on another, co-exists with original Derived Object - File automatically generated from source files Change Set - Set of versions across multiple files making up a single logical change (e.g., feature or fix) Baseline - Consistent set of versions across all files SCM

8. Large- Scale SW Devel. October 31, 2000 Page 8 Terminology - Baseline SCM

9. Large- Scale SW Devel. October 31, 2000 Page 9 Coordinating Work Problem: Multiple people working on same file at the same time can result in lost work. Solution 1: Serialize work with file locking. Solution 2: Work in isolation (branch) and merge. Branching and merging scales much better. Merging can be semi-automated using 3-way merge algorithms. SCM

10. Large- Scale SW Devel. October 31, 2000 Page 10 Merging - Version Tree SCM

11. Large- Scale SW Devel. October 31, 2000 Page 11 Formal definition (IEEE) SCM is the process of identifying and defining the items in the system, controlling the change of these items throughout their life cycle, recording and reporting the status of items and change requests and verifying the completeness and correctness of the items. Related Terms: Source Control, Version Control SCM

12. Large- Scale SW Devel. October 31, 2000 Page 12 Managing Change Change Control - Formal processes for tracking and approval of software modifications. Managed with SCM and defect tracking tools. Usual Policy Areas: Documentation - Change request / defect report filed. Approval - Change request reviewed by release management or technical board. Quality Requirements - Code reviewed, regression tests passed. Policies often change by release stage and in response to problems. Change Control

13. Large- Scale SW Devel. October 31, 2000 Page 13 Development Process Key Components of Development Process: Software Configuration Management Change Control Quality Control Change Control

14. Large- Scale SW Devel. October 31, 2000 Page 14 Principles of Large-Scale Software Development Reproducibility Policy Enforcement versus Policy Auditing Process, Process, Process A process must be defined to be improved Automation A process must be automated to be repeatable Continuous Functional Integration and Test Scaling

15. Large- Scale SW Devel. October 31, 2000 Page 15 Software Integration Integration: Combining multiple features or components developed in isolation. Large projects are subject to constant change, so isolation on a stable base is necessary. Continuous Functional Integration and Test (CFIT) Changes integrated throughout development. Changes not allowed to regress functionality or quality. Scaling

16. Large- Scale SW Devel. October 31, 2000 Page 16 HP-UX Development 220,000+ Source Files ~2.5 Million Versions 52,000+ Derived Files (Single Nightly Build) 900+ User Accounts More Than 1 Terabyte of Disk Space HP-UX

17. Large- Scale SW Devel. October 31, 2000 Page 17 HP-UX Concurrent Development Normally 2-3 Releases Under Development Several Releases Under Maintenance Heavy Use of Branching and Merging HP-UX

18. Large- Scale SW Devel. October 31, 2000 Page 18 HP-UX Multisite Development Four Primary Development Sites Three in USA (Three Time Zones) One in India One USA Site Hosts Three Logical Sites Partner Companies Several Secondary Sites HP-UX

19. Large- Scale SW Devel. October 31, 2000 Page 19 Tiered Software Integration Model 1. Developer Task - Edit, Compile, Test 2. Submit - Batch Changes, Acceptance Test 3. Baseline - Nightly Build, Regression Test 4. Integration - Monthly Packaging, Partner Test 5. Release - Semi-annual Delivery, Production Use HP-UX

20. Large- Scale SW Devel. October 31, 2000 Page 20 Integration Levels Build Submit Baseline Integration Cycle Release Modifications HP-UX

21. Large- Scale SW Devel. October 31, 2000 Page 21 Process Evolution Continuous Process Improvement Root Cause Analysis (RCA) Capability Maturity Model (CMM) 1. Initial 2. Repeatable 3. Defined 4. Managed 5. Optimizing