1. Hans AkkermansIS Research Methodology (06Sep2010)1 Notes on Scientific Method Hans Akkermans

2. Issues of Research Methodology If so, what are these rules actually in our science(s)? Hans AkkermansIS Research Methodology (06Sep2010)2 Do we do it this way? … or that way? … Or do we follow The Rules (of Scientific Method)

3. Hans AkkermansIS Research Methodology (06Sep2010)3 What is Science? (MyDef) Originally, term for systematic inquiry (broad church, inclusive formulation) [ what I personally still continue to prefer] A social – institutionalized – practice focused on the production of claims to knowledge through a process of inquiry, in a way that is: Relevant: produced knowledge is about something of interest/importance [Q1 what does that mean?] Systematic: process of inquiry is carried out in a well- organized, critical and rigorous fashion [Q2 i.e?] Transparent: claims are produced and argued for such that they are clear and open to critical scrutiny for/by others [Q3 who are they?]

4. Hans AkkermansIS Research Methodology (06Sep2010)4 Many different scientific research goals exist [as a matter of fact] Description Understanding Explanation (e.g. causal) Prediction Guidance for action Design (e.g. informatics, engineering, architecture, urban planning) Intervention (e.g. therapy, medicine, organizational change) Professional practice (e.g. law); decision making (e.g. management, policy), Combinations possible and quite common Note1: different disciplines strongly differ in view on what are proper goals of science and research Note2: also linked to different philosophical positions on science Reference vs. coherence vs. interpretivist vs. pragmatic theories of truth Realism vs. positivism vs. constructivism vs. postmodernism vs. pragmatism (cf. reflective practitioner) Hence: varying positions on what science is or ought to be

5. Hans AkkermansIS Research Methodology (06Sep2010)5 Typical academic misunderstandings By other disciplines about I/CS Its engineering (i.e. useful but not really science) Where is the theory? Results: Any universal laws? Does I/CS have any empirical nature/content? Isnt experiment absent? Big difficulty in understanding I/CS-style conceptualization Idem: what counts as a proper research claim or contribution Generally, no clue about the computational paradigm as scientific method Design as a proper research goal? How can that be?? By I/CS researchers about others How key notions such as theory, hypothesis, empirical research, experiment, validation/validity are seen in other disciplines Knowledge about quantitative and qualitative research methodology Although commonly used in I/CS Methodology often left quite (too) implicit [you wouldnt get away with this elsewhere!], OR: naïve import Ignorance about thinking and results from e.g. social and other sciences Generally, big cultural gap wrt. nature and esp. justification of research claims

6. Method as the toolbox of the researcher Formal methods F Logic(s) Math Formal specs Inductive-statistical (quantitative) methods S Qualitative methods Q Interview Field observation Case study Action research Computational methods C Modelling and simulation Demo programs and architectures Design methods and rules Protocols, algorithms Experimental methods E Hans AkkermansIS Research Methodology (06Sep2010)6

7. Hans AkkermansIS Research Methodology (06Sep2010)7 Experimental Method(s) The classic laboratory experiment [physics etc.] Randomized Controlled Trials (RCT) [medicine etc.] Field test [e.g. engineering, innovation] Quasi-experiment [psychology, cf. Donald Campbell] Thought experiment [e.g. Einstein, Bohr] One may view modelling & simulation as a further developed, formalized and computerized version of this Campbell (2003, discussing case study vs. experiment) è Core of the scientific method is strategy to deal with plausible rival hypotheses (and not experimentation per se) è Not (positivistic) corroboration of context-independent hypotheses, but evaluation of extended networks of implications of rival hypotheses [ramification extinction]

8. Research is a 3-place relation (not 2) Research = make [argued] claims to knowledge: ABOUT the world W (some part of reality) TO a community H (some part of humankind) Hans AkkermansIS Research Methodology (06Sep2010)8 W (world) H (audience, your peers) S (you as researcher) REPRESENTATION APPEAL EXPRESSION Cf. Buhlers linguistics/ communication diagram (1930s) Note1: Context W importance Note2: Context H importance

9. Hans AkkermansIS Research Methodology (06Sep2010)9 Argument and Informal Logic D + T R C - Core idea of scientific argument Data plus Theory produce Conclusions by means of Reasoning Toulmin (1958, 2000): Reasoning is field-dependent: each domain has its own knowledge and forms of reasoning (non-universal logic) Test questions for argument acceptability (Fisher, 2004) [Note: argument can be represented as directed and/or graph] AR (argument reconstruction): what is the claim, and what reasons are presented to justify the conclusion? Principle of Charity: construe argument favourably, by adding unstated, implicit but necessary, reasons or assumptions TQ1: Are the reasons (premisses) true? TQ2: Are there situations in which all premisses can be true and conclusion false? AQ (assertability question): What evidence/reasons would justify one to assert the conclusion?

10. Hans AkkermansIS Research Methodology (06Sep2010)10 Validity and the Structure of Argument (1) Descriptive validity (D) (2) Theoretical validity (T) (3) Interpretive validity [D + T] (I) (4) Validity of Reasoning [consistency, coherence] (R) (5) Internal validity [+ qualifiers …] (C INT ) (6) External validity (C EXT ) = Generalizability (Toulmin, 1958)

11. Freedom & the Principle of Methodological Accountability I/CS is methodologically relatively liberal (good) but also often relatively unreflective (not so good) Either: you follow standardized, accepted method in a field (e.g. quantitative, economics) Safe research route (mainstream, but danger: maybe boring) Or research freedom: construct your own method But then: you are accountable and have to argue for it Science historical note: ground-breaking research often invents its own (new, so contested) method E.g. Leibniz/Newton: differential calculus (etc. etc.) I/CS: computational paradigm (!) Hans AkkermansIS Research Methodology (06Sep2010)11

12. How/where I/CS can become (methodologically) better What is your research claim precisely? Context!: What difference does your contribution make (the delta) And how did you get there? Argument A B C …. Claim Transparent = scrutinizable for others Note: overall argument is informal (even in math/logic!) Typical mistakes: Claim too implicit NOT: the work you did (e.g. simulations; but: what do they tell us) Claim too strong or too weak (qualifications etc.) Overall argument too implicit, or components of argument weak / dubious = do not pass test questions of argument validity Hans AkkermansIS Research Methodology (06Sep2010)12