CBS Development: Guidelines Based on Lessons Learned Betsy Clark Software Metrics Inc. February 7, 2001 Sponsored by the Federal Aviation Administration’s Software Engineering Resource Center

2 COCOTS Background USC –model development and calibration SMI –rounding up projects to provide data asking, persuading, begging, cajoling Jointly –workshops for model definition –data collection

3 COTS Lessons Learned At the conclusion of the data collection interviews, we asked about lessons learned Sixteen (of twenty projects) responded –Twelve were FAA Air Traffic Management: 6 Air-Ground Communications: 2 Support systems (non-operational): 2 Administrative: 2 –Four were DoD Missile Launch Control: 2 Mission Planning: 1 Administrative: 1

4 Areas Addressed Problems with vendors (6) Need for flexibility in defining requirements (5) Importance of operational demonstrations (5) Assessment of specific product attributes (5) Life-cycle issues (5) COTS integrator experience (3) Training on COTS packages (2) Need for technology watch to keep up with vendors (2) Interface to legacy systems (1) Vendor management (1) Impacts of volatility during development (1)

5 Guidelines -1 Don’t rely on vendor claims. Verify with operational demonstrations. –Time spent on detailed operational demonstrations was consistently viewed as time well spent. –Know what you’re buying! “Our biggest problem was with the vendors, especially with believing their claims. Very few components worked as advertised.” “Operational demos are important. At that point, vendors are bending over backwards to sell their components so they’ll participate.”

6 Guidelines - 2 Bring the users into the operational demonstrations, not just the vendors. –The more diverse the user community and the greater the impact on their business processes, the more important this is.

7 Guidelines - 3 Understand that profits motivate vendors. Whether they are cooperative or not depends to a large degree on anticipated profits. –In our sample, vendor cooperation ranged from superb to indifferent. –Superb: Vendor wanted to enter the federal marketplace and foresaw a lot more business with success of first project. –Indifferent: Government market for product was not materializing; not worthwhile for vendor to “knock themselves out”.

8 Guideline - 4 Your leverage occurs before the contract is signed. Negotiate all prices up front. “Up-front time is critical. That’s when you have leverage with vendors. Once you buy their product, they are a lot less willing to help out.”

9 Guidelines - 5 Establish a technology watch to track vendors and products. “You have to constantly monitor the state of the COTS components. We had a company fold and we were taken by surprise. A technology watch would have prevented us from getting stuck.”

10 Guidelines - 6 Consider including a refresh cycle prior to fielding the system so that the products are not nearing end-of- life. –Even with a system development of 24 months, COTS components are likely to be obsolete by one or more versions

11 Guidelines - 7 Distinguish between essential requirements and those that can be negotiated. Use mature products for safety-critical applications. “If you can bend your requirements, COTS is cheaper. Otherwise you’re better off with custom developed.” “Never use an untried operating system.”

12 Guidelines - 8 Be forward looking in assessing attributes. –e.g., changes in hardware platforms

13 General Comments With COTS, you are buying into a different process –operational demos are critical –requirements must be flexible –costs shift to the right “People have to look at the entire life cycle realistically – not just the development cost but consider what it’s going to cost to maintain over a number of years.” “COTS is high risk because we are dependent on someone else. There needs to be a process to help people evaluate their risks.”

14 Contact Information Betsy Clark Software Metrics Inc High Ridge Rd. Haymarket, Virginia (703) (703) fax We are ALWAYS looking for data!