1. An industrial study in Norway, Germany and Italy Seminar on CBSE (component-based software engineering) Simula Research Lab., Oslo, 4 Feb. 2005 http://www.idi.ntnu.no/grupper/su/cbse-survey Method Issues in the CBSE survey An industrial study in Norway, Germany and Italy Seminar on CBSE (component-based software engineering) Simula Research Lab., Oslo, 4 Feb. 2005 http://www.idi.ntnu.no/grupper/su/cbse-survey Odd Petter N. Slyngstad Department of Computer and Information Science (IDI) Norwegian University of Science and Technology (NTNU) oslyngst@idi.ntnu.no

2. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 2 Overview Survey details Survey details The ICT-sector in Norway The ICT-sector in Norway Sample & sampling process Sample & sampling process Company contact process Company contact process Prestudy convenience sample Prestudy convenience sample Resulting representative sample selection Resulting representative sample selection SESE tool SESE tool Reflection Reflection Conclusions Conclusions

3. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 3 Survey details First survey with a representative sample performed since Frakes et.al. ”Sixteen Questions About Software Reuse” (1995) First survey with a representative sample performed since Frakes et.al. ”Sixteen Questions About Software Reuse” (1995) High cost in terms of effort High cost in terms of effort Questionnaire took ½ year to finalize Questionnaire took ½ year to finalize A pretest was performed in early 2004 to check the questionnaire (16 projects in 13 companies) A pretest was performed in early 2004 to check the questionnaire (16 projects in 13 companies) Finally, the questionnaire was translated into the native language of each country before starting the survey Finally, the questionnaire was translated into the native language of each country before starting the survey Project is the unit of study Project is the unit of study

4. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 4 The ICT-sector in Norway ICT- manufacturing industry (hardware) ICT- manufacturing industry (hardware) –30.xx - 33.xx series –Manufacture of computers (and other equipment), transmitters, receivers, wire, cable etc. –ex: Ericsson, Nera, etc. ICT- retail ICT- retail –51.xxx, 52.xxx series –Wholesale of software, hardware etc. –ex: Computerland, etc. ICT- telecommunications ICT- telecommunications –64.xx series – telecommunications –ex: Telenor, Netcom, etc. ICT- software ICT- software –72.xx series –Consultancy, data processing, database activities etc. –ex: EDB, Software Innovation, Opera Software, etc. Source: Norwegian Census Bureau Nace codes (OECD):

5. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 5 The ICT-sector in Norway (2) As of 2002 EmployeesTurnover (mill. NOK) Value Added (mill. NOK) 4ICT-software 38 952 44 633 44 633 22 863 3ICT-retail 22 133 66 231 66 231 11 143 2ICT-telecom 13 546 52 342 52 342 19 756 1ICT-hardware 12 043 24 174 24 174 6 797 6 797 Total 86 674 187 380 60 559 Content Industry 39 642 47 362 47 362 17 246 Source: Norwegian Census Bureau

6. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 6 The ICT-sector in Norway (3) ICT sector (overall and software dev.) ICT sector (overall and software dev.) –Second largest industry in Norway –median: 2 –mean: 11 Three groups with approximately 1/3 of the total employees in each: Three groups with approximately 1/3 of the total employees in each: Companies with 0-19 employees Companies with 0-19 employees Companies with 20-99 employees Companies with 20-99 employees Companies with over 100 employees Companies with over 100 employees Source: Norwegian Census Bureau

7. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 7 Sample Main problem: a representative sample of companies does not mean a representative sample of CBSE projects; Main problem: a representative sample of companies does not mean a representative sample of CBSE projects; –Some business areas may be more inclined than others to use CBSE and COTS/OSS –Important issues include security, quality certification –Initially, we only had access to company lists - no knowledge of individual projects in these companies

8. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 8 Sample (2) But a representative sample is better than a convenience sample used in previous studies: But a representative sample is better than a convenience sample used in previous studies: –Morisio, Torchiano, TSE 2002, April 2002: success/failure factors analysis of 30 EU projects –Torchiano, Morisio: (rejects) Common Wisdom on COTS; 7 structured interviews

9. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 9 Sampling process Process in Norway Process in Norway –Obtained lists of company names from the Norwegian Census Bureau/IKT-Norge/Infosector.net company –Three groups with approximately 1/3 of the total employees in each: Companies with 0-19 employees from 72.xx series Companies with 0-19 employees from 72.xx series Companies with 20-99 employees from 72.xx series Companies with 20-99 employees from 72.xx series 100 largest companies (by turnover) in the ICT sector + 3 largest in 5 sectors (finance etc.) 100 largest companies (by turnover) in the ICT sector + 3 largest in 5 sectors (finance etc.) –Lots of imprecise information, wrong classifications (no software dev. even in nace 72.xx) Process in the other countries Process in the other countries –Italy: Obtained 400 companies from yellow pages, checked if they had sw development Obtained 400 companies from yellow pages, checked if they had sw development –Germany: Data points from the German census bureau, took subset of IESE contacts. Have also included some major companies. Data points from the German census bureau, took subset of IESE contacts. Have also included some major companies.

10. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 10 Company contact process Made guidelines for the process as follows: Made guidelines for the process as follows: Call/contact company Call/contact company –get contact person information Call this contact person (”gatekeeper”) Call this contact person (”gatekeeper”) –explain context –send email with questionnaire if any CBSE dev. + ok/relevant (30% of initial sample) –Benefits for respondents include report, lottery and free seminar All respondent information entered into the SESE tool All respondent information entered into the SESE tool

11. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 11 Company contact process (2) Rephone Rephone –is it really relevant, willing to participate? (40% of relevant) –if yes, send login info for web questionnaire and explain respondent options: Electronic questionnaire (web - preferred) Electronic questionnaire (web - preferred) Phone interview Phone interview Word questionnaire (by email or printed version by fax) Word questionnaire (by email or printed version by fax) –then wait for response, rephone etc. to remind many times

12. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 12 Prestudy convenience sample initial selection 50% 75 % relevant for willing to participate i.e. = 37,5% of total

13. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 13 Resulting representative sample selection initial (three groups) selection 30% 40 % relevant for willing to participate i.e. = 12% of total - Of the companies which were willing to participate, 10% had several projects

14. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 14 SESE webtool at Simula Pros Pros –All researchers have access to all data simultaneously –Ease of use in filling in the questionnaire Issues Issues –Requires a small learning curve for researchers –Minor technical ”problems” experienced, reporting back to Simula Research Lab.

15. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 15 Reflection High effort cost, but yet low response rate (12% in Norway) High effort cost, but yet low response rate (12% in Norway) In Germany and Italy many questionnaires were filled out in phone interviews In Germany and Italy many questionnaires were filled out in phone interviews –this approach was discussed and rejected in March 2004, due to the presumed effort required Motivation & environment differences Motivation & environment differences –Italy lost summer student in October 2004 –IESE (Germany) had 3 months delay in PhD student upstart –Norway hired extra research assistant for 4 months (March – June 2004)

16. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 16 Conclusions Lessons learned: Perform survey by phone interview if less than 100 respondents Keep active contact with contact persons (gatekeepers)/respondents; agree on ”deadline” for next contact/completing questionnaire, preferably by phone Allow all researchers involved to share data by electronic means (online DB or similar)

17. Odd Petter N. Slyngstad, IDI, NTNU CBSE Seminar, Simula Research Lab., 4th of February, 2005 17 Questions & Comments? oslyngst@idi.ntnu.no