1. HiST Tacitly developed research methods in ICS Ivar Tormod Berg Ørstavik

2. HiST Outline Introduction: Methodology in ICS (ICS = CS + SE + IS) a. KWIC: Where and how is ICS methodology discussed in NTNU PhD theses? b. Text classification: Research method per chapter in PhD theses. c. Trend exemplification: ICS engineering as a research method in PhD theses. Conclusion

3. HiST Introduction ICS as applied science “A large part of CS research consists of proposing new designs: systems, algorithms, and models. Such designs must be judged by whether they increase our knowledge about what are useful and cost-effective problem solutions” Tichy (1995: 9). “Software engineering is concerned with techniques useful for the development of effective software programs” (Zelkowitz 1997: 735). “A justified theory that is not useful for the environment contributes as little to the IS literature as an artifact that solves a nonexistent problem” Hevner (2004: 81).

4. HiST Introduction Research methods’ status in ICS applied science From a strong applied research position, the internal workings of research processes are only indirectly relevant. Research method is a means to an end. “highly practical research in the field of information systems simply ignores the research method issue” (Baskerville 1998: 90). “CS has not developed a concise taxonomy of methods applicable for demonstrating the validity of a new technique” (Zelkowitz 1997: 1). “a lack of methodology for study and research in scientific software development” (Diaconescu 2002: 72). “SE researchers rarely write explicitly about their paradigms of research and their standards for judging quality of research” (Shaw 2002: 1).

5. HiST Introduction Does ICS lack methods? NO! Missing is not the methods themselves, but discussions of them. Does ICS lack a methodology altogether? No, in Software Engineering a vibrant discussion about the need for empirical experiments and observation as research methods in SE thrives. No, discussions of research methods in ICS pop up everywhere. Yes, in ICS and particularly CS the researcher’s own role and methods are often implicit, not discussed. Yes, a coherent methodological discussion in which all the three disciplines participate is lacking. Does poor methodology equal poor methods? No, the ICS disciplines and many of their methods has clearly demonstrated their value! No, an explicitly discussed method does not necessarily equals a better method. Yes, by discussing strengths and weaknesses of methods openly, it is much easier to improve on them.

6. HiST Material for a, b, c Material: 93 PhD theses IDI, NTNU (dr.ing.) 1980 - 2006

7. HiST a. KWIC -Not all “methods” are research methods (i.e. “system methods”, “development methods”, etc.). Each keyword is interpreted manually. Here, “research method” is “methods” that is to be performed by the researcher(s) in their capacity as researcher. -Synonyms such as ”research approach” =an overall perspective, not exact, hard facts. -> 1995 very little explicit discussion about research methods. 1995 -> several theses mainly within SE devote lengthy discussions to research method issues. still, more than half the theses present research methods implicitly.

8. HiST b. Text classification

9. HiST b. Text classification Text classification is highly problematic! –As in the two original studies, most categories are practically not present in this material. –Some categories are too broad and function as storage containers. –Research methods are not spread out 1:1 per article or PhD chapter. –Research methods do (and should) not follow a rigid template. They are tailored to the individual project’s needs and context. To classify a research method therefore implies a subjective interpretation of each category that both varies from text to text and researcher to researcher. The resulting graph and research method taxonomies hide these essential interpretations. –The existing taxonomies are biased. Some categories are highly nuanced, others wide open. In my material, far too many texts used methods not really part of the taxonomy.

10. HiST c. Trend exemplification “an ultrasound system”, (Lindseth 2002). “a database system”, (Thorbjørnsen 1995). Shared tacit assumptions Solutions and designs are implementable in hardware and software. Both projects engineer solutions and only devote time to problems that can be described as solutions. Any analytical process (process that takes things apart) is subsumed in this constructive outlook. The projects goals are applied knowledge and are justified by their usefulness. The main source of insight is gained not from observing systems but from participation in their construction. The researcher himself is the brains behind many of the ideas and solutions discussed. The researcher’s role should be understood as subjective participation, not objective observation. “the fishing boats with fishermen system”, (Digernes 1982). “The development process system”, (Opdahl 1992). Testing tacit assumptions Diversity of problems is overcome through unity in solutions. Internal coherence is achieved through holistic solutions. The research method is optimistic. Research questions are not formulated skeptically (“Can X be solved?”), but optimistically (“What is the best solution for X?”). Problems pursued are solvable and implementable: unsolvable problems is not part of ICS engineering’s scope. 4 PhDs using ICS Engineering as a research method

11. HiST c. ICS engineering As a research method, ICS engineering is strongly solution oriented. The aim of ICS engineering is to develop applicable, useful solutions. This applied goal is propped up by an optimistic outlook that filters out problems that cannot be solved. While working with solvable problems, the active process of designing a solution is central. The researchers participate in these constructive processes and therefore act subjectively when they later analyze their own and others’ rival or allied technical concepts. Internal coherence in the research project is primarily achieved through a holistic solution.

12. HiST Lessons learned ICS research methods are worth studying! Before methods can be counted, they must first be described. Studies of research methods are likely to be qualitative and subjective, even when they are presented as quantitative and objective. How do design and implementation languages influence research methods?