1. COTS Projects: learning from experience Dr Jenny Longster Prof Martin Shepperd BSERC, BT Submitted to TSE 25 th January 2007

2. Article Objectives Collaborative work with BT plc Perform a Systematic Review of Industrial Case Studies concerning COTS Projects Identify the difficulties and challenges that are associated with COTS Extract lessons learnt in terms of success and failure This is to our knowledge the first review of its kind for COTS integration projects

3. Systematic Reviews “A systematic review is a means of evaluating and interpreting all available research relevant to a particular research question” [1] aim to minimise bias identify all relevant material by adhering to a strict scientific methodology traceable and rigorous process

4. COTS Projects  What is COTS Integration and what are COTS? “A COTS product is a commercially available or open source piece of software (OSS) that other software projects can reuse and integrate into their own products”

5. Results 32 primary case studies were identified. Of these 21 (66%) were considered a success, 10 (31%) were unknown, and 1 (3%) was considered a failure and aborted Studies revealed that COTS projects either lead to, or are expected to lead to, high quality systems, effective maintenance, and that time/schedule and cost/budget issues are high on the agenda. Process issues, reuse, and having standard terms and interfaces across the organisation and system are also seen as a motivation for the adoption of COTS. Curiously benefits from an organised software architecture and training issues were not generally regarded as very significant.

6. Results Success factors were then analysed in decreasing order of mention:  Process Success Factors 50.0%  Reuse Success Factors 37.5%  Success Factors for Choosing Components 31.3%  Standards/Interfaces Success Factors 28.1%  Integration Success Factors 15.6%  Requirements Success Factors 15.6%  Vendor Success Factors 15.6%  Maintenance/Change Success Factors 15.6%  Technical Expertise Success Factors 15.6%  BPR Success Factors 9.3%  Quality Success Factors 9.3%  Resources Success Factors 9.3%  User Success Factors 6.2%  Organisational Success Factors 3.1%  Legacy Success Factors 3.1%

7. Results Success factors were then analysed in decreasing order of mention:  Issues with Choosing Components 28.1%  Requirements Issues 28.1%  Process Issues 25.0%  Standards/Interfaces 21.9%  Change/ Maintenance Challenges 18.8%  Problems when designing for reuse 12.5%  Integration Challenges 12.5%  Technical expertise issues 12.5%  Vendor Issues 12.5%  Quality Issues (Usability, Performance etc) 9.4%  Resourcing Issues 9.4%  User Issues 9.4%  Organisational Barriers 6.3%  Legacy Challenges 6.3%  BPR - Business Process 3.1%

8. Conclusions significant number of independent case studies of report successful outcomes, only one study reported a failure  This suggests that at the very least, integration technology can be considered as promising Clearly different studies have adopted different views as to what constitutes success there are various threats to validity of our work  rely on the reporting of the authors though we have restricted our analysis to peer reviewed research papers.  Authors may more readily report on success, although the success and failure factors are equally reported on More analysis of data possible  Issues with lack of data points  Define qualitative terms such as “strongly significant”, “weakly suggestive”