1. COTS Metrics and Modeling Working Group Chris Abts, Vic Basili, Jim Cannon, Brad Clark, Nancy Eickelmann, Peter Hantos, Keun Lee, Gary Thomas, Lori Vaughn

2. © 2001 USC-CSE 2 What should I measure?  In what activities did I spend my time, money? In what phases did I spend my money? What problems are we having? What advantages does the vender give us over our internal staff? What are the vender capabilities? How do I characterize the environment? How do I characterize the volatility of the system? –How do I characterize the volatility of each COTS product? –How do I aggregate that volatility?

3. © 2001 USC-CSE 3 SEI Activity Sets Engineering Activity Area  Evaluation –Construction –Configuration Management –Deployment and Sustainment –System Context –Architecture and Design –Marketplace Business Activity Area  Vendor Relationship –Intergovernmental Supplier Relationships –COTS Business Case –COTS Cost Estimation Ref: "An Activity Framework for COTS-Based Systems,” CMU/SEI-2000-TR-010, by Oberndorf, Brownsword, Sledge.

4. © 2001 USC-CSE 4 Possible Effort Model Outputs Amount of time we should spend on evaluation –Product (effort) –Vendor (rating) Confidence in our assessment Risk exposure associated with the decision

5. © 2001 USC-CSE 5 Product Evaluation Effort Model Inputs (In what activities did I spend my time, money?) How much effort does it take? –Number of potential products available –Technology maturity / volatility –Number of features to be evaluated –Number / complexity of interfaces –Number of other COTS products it has to interface to –Criticality to system –Organization maturity with COTS based-development –Vendor Rating How much confidence do we have in our answers

6. © 2001 USC-CSE 6 Vendor Relations Effort Model (What is the Vendor’s influence on my evaluation effort?) Inputs –What size of the vendor’s market share does your need represent? –Vendor’s market share –Vendor’s product maturity –Criticality of COTS product to system –Vendor responsiveness –Trade exposure –Vendor level of support (training, help desk, tech. support) How much confidence do we have in our answers

7. © 2001 USC-CSE 7 Risk Model for Exposure (How much resource should I invest?) Effort Spent in Evaluation Risk Exposure

8. © 2001 USC-CSE 8 Evaluation Effort Model Discussion Effort to evaluate = F (#potential products to evaluate, # interfaces, #features) But most important factor is organization maturity with COTS Meta Model (what have you got available)

9. © 2001 USC-CSE 9 Priority Chart for Models Difficulty Importance LM H M H L X Risk Exposure X Product EvaluationX Vendor Evaluation X Meta Model X SIZE

10. © 2001 USC-CSE 10 Conclusion There is a lot we don’t know about how to conduct a COTS based project so that it ends successfully Modeling activity is very important to project success –How much time should I spend doing an activity? –What happens when I spend more or less time doing an activity Building activity models is very hard –Hard to identify a primary relationship between effort and something else (e.g. size and effort in COCOMO II) –Lots of different things affect effort in different activities –Little information is collected in addition to effort (if effort is collected at all down to the activity level)