1. Issues in Study Design
Petri Nokelainen

2. Database of scientific knowledge
Original idea for the research
Literature review
Research questions / hypotheses
Design
Database of scientific knowledge
Discussion
RQ’s
Method
Intro/theory
Methodology
Publication of the study
Design
vs.
Methodology?
Data collection
Results
Primary / existing data
Measurements
Conclusions
Peer review
Writing scientific report
Data analyses

3. Design vs. Methodology
Design focuses on the procedures related to outcomes
Historical, comparative, interpretive, exploratory research
What evidence is needed to answer research question(s)
Methodology focuses on the research process (instrumentation and analyses)
Primary, secondary data
How to conduct analyses in robust and unbiased way

4. D = Design (ce = controlled experiment, co = correlational study)
N = Sample size
IO = Independent observations
ML = Measurement level (c = continuous, d = discrete, n = nominal)
MD = Multivariate distribution (n = normal, similar)
O = Outliers
C = Correlations
S = Statistical dependencies (l = linear, nl = non-linear)
(Nokelainen, 2008, p. 119.)

5. Experimental design ‘Pretest post-test randomized experiment’
Applied in many fields, but needs a random sample (‘probability sample’) and random assignment (participants are randomly selected for the experimental and control groups).
Research is conducted in a controlled environment (e.g., laboratory) with experiment and control groups (threat to external validity due to artificial environment).
Using experimental design, both reliability and validity are maximized via random sampling and control in the given experiment (de Vaus, 2004).

6. Experimental design Random sample Exp. Contr. Pre I - Post
Random assignment

7. Random assignment to groups
Experimental design
Random assignment to groups
Pretest
Intervention
Post-test
Experimental group
Measurement (X)
Treatment
Measurement (Y)
Control group
Measurement (X)
No treatment
Measurement (Y)

8. Quasi-experimental design
‘Non-equivalent groups design’
Resembles experimental design but lacks random assignment (sometimes also random sampling) and controlled research environment.
This type of design is sometimes the only way to do research in certain populations as it minimizes the threats to external validity (natural environments instead of artificial ones).
Random / convenience sample
Exp.
Contr.
Pre I -
Post

9. Correlational design ‘Descriptive study’ or ‘observational study’
Allows the use of non-probability sample (a.k.a ‘convenience sample’).
Most correlational designs are missing control, and thus loose some of their scientific power (Jackson, 2006).
Some research journals accept factorial analysis (main and interaction effects, e.g., MANOVA) based on correlational design.
Convenience sample
Exp.
Pre I
Post

10. RS RS CS RANDOM SAMPLING RANDOM SELECTION
pretest-posttest randomized experiment
Pre
Post
TEST I RS
Pre -
Post
CONTROL
Non-Equivalent Groups Design
Pre
Post
TEST I RS
Pre
Post
CONTROL -
Correlational design CS
Pre
Post
TEST I

11. Time and design
Observational studies can utilize cross-sectional or longitudinal designs (see Caskie & Willis, 2006).
Longitudinal design includes series of measurements over time.
Change over time, age effect.
Cross-sectional study involves usually one measurement and is thus considerably cheaper and faster to conduct (although producing less controllable and less powerful results).
If there are several measurements, individual participants answers are not connected over time (e.g., due to anonymity).
Causal conclusions are usually out of scope of this research type.

12. Longitudinal design
One sample that remains the same throughout the study.
Longitudinal study produces more convincing results as it allows the understanding of change in a construct over time and variability and predictors of such change over time.
However, it takes more time to carry out and suffers from participant drop-out (imputation of missing data, e.g., Molenberghs, Fitzmaurice, Kenward, Tsiatis, & Verbeke, 2014).

13. Cross-sectional design
Measurement is conducted once (or several times) and the sample varies throughout the study.

14. Case study design Applied in qualitative research.
The aim is to collect information from one or more cases and study, describe and explain them through how and why questions.
Cases are represented, for example, by individuals, their communication and experiences. (For thorough discussion, see Flyvbjerg, 2004.)

15. About designs
Controlled experiment designs, when conducted properly, rule out IO violations quite effectively (Martin, 2004), but correlational designs usually lack such control (e.g., to rule out employee’s co-operation when they respond to the survey questions).
On the other hand, some qualitative techniques, like focus group analysis (Macnaghten & Myers, 2004), are heavily based on non-independent observations as informants may (or are asked) talk to each other during the data collection.

16. What really matters Most important questions: Answered by design
Scientific impact
Societal impact
Answered by design
So, what drives us:
Design or method?

17. Education Science Team
Regulation of learning and active learning methods in the context of engineering education (REALMEE)
Research Design
Pre- and post tests
Event measures
Research Team
Intervention group
Course Planning
Control group
Pedagogical intervention
Education Science Team
TUT Course
Contents Expert Team

18. Regulation of learning and active learning methods in the context of engineering education (REALMEE)
Research Design

19. Lack of design shows up Dissertations Journal manuscripts
Funding applications
Even in published research!

20. Review
Total number of participants in the 18 reviewed articles was 3485, of which 681 participated in qualitative and 2804 in quantitative studies.
Only 11 articles contained both explanation and justification of selected methodological approach and robust description of data analysis.
Only eight articles had a section about critical examination of the method(s) and limitations of the study.
Two articles based on group level data did not discuss about rationale of choosing such approach and related validity issues (Chioncel et al., 2003).
(Pylväs et al., in press.)

21. What really matters Scientific impact
Existing research, review (Paré et al., 2015).
Research gap

22. What really matters Scientific impact
Trends in publication policies research design and methodology
qual vs. quan, generalizability vs. representativeness
Gobo (2004) defines a concept of generalizability for qualitative research by arguing that the concept of generalizability is based on the idea of social representativeness, which allows the generalizability to become a function of the invariance (regularities) of the phenomenon.

23. What really matters
Thus, “The ethnographer does not generalize one case or event … but its main structural aspects that can be noticed in other cases or events of the same kind or class.” (Gobo, 2004, p. 453.)

24. What really matters Scientific impact
Trends in publication policies research design and methodology
Data, investigator, theory and methodological triangulation (Denzin, 1978) are applied to compensate design limitations, reduce possible researcher bias, and increase the strength of conclusions.
Design research approach (Bannan-Ritland, 2003).

25. What really matters Scientific impact
Trends in publication policies research design and methodology
longitudinal studies (qual & quan), latent variable modeling (e.g., R, lavaan)
effect size (Barry et al., 2016), CI for effect sizes (Thompson, 1994, 1996)
critical examination of p-values and NHSTP

26. NHSTP
‘null hypothesis significance testing procedure’ and featured product, p-value.
Gigerenzer, Krauss and Vitouch (2004, p. 392) describe ‘the null ritual’ as follows:
1) Set up a statistical null hypothesis of “no mean difference” or “zero correlation.” Don’t specify the predictions of your research or of any alternative substantive hypotheses;
2) Use 5 per cent as a convention for rejecting the null. If significant, accept your research hypothesis;
3) Always perform this procedure.

27. NHSTP
A p-value is the probability of the observed data (or of more extreme data points), given that the null hypothesis H0 is true, P(D|H0) (id.).
The first common misunderstanding is that the p-value of, say t-test, would describe how probable it is to have the same result if the study is repeated many times (Thompson, 1994).
Gerd Gigerenzer and his colleagues (id., p. 393) call this replication fallacy as “P(D|H0) is confused with 1—P(D).”

28. NHSTP
The second misunderstanding, shared by both applied statistics teachers and the students, is that the p-value would prove or disprove H0. However, a significance test can only provide probabilities, not prove or disprove null hypothesis.
Gigerenzer (id., p. 393) calls this fallacy an illusion of certainty: “Despite wishful thinking, p(D|H0) is not the same as P(H0|D), and a significance test does not and cannot provide a probability for a hypothesis.”

29. What really matters Scientific impact
Trends in publication policies research design and methodology
paradigmatic vs. algorithmic modeling (Breiman, 2001)
Seeking or learning structures from data? Exploratory vs confirmatory approach …

30. Learning structures
The target population of the study consisted of ATCOs in Finland (N=300) of which 28, representing four different airports, were interviewed.
The research data also included interviewees’ aptitude test scoring, study records and employee assessments.
(Pylväs, Nokelainen, & Roisko, 2015.)

31. Learning structures
The research questions were examined by using theoretical concept analysis.
The qualitative data analysis was conducted with content analysis and Bayesian classification modeling.
What are the differences in characteristics between the air traffic controllers representing vocational expertise and vocational excellence?

32. Learning structures

33. Learning structures
"…the natural ambition of being good. Air traffic controllers have perhaps generally a strong professional pride." ”Interesting and rewarding work, that is the basis of wanting to stay in this work until retiring.”
"I read all the regulations and instructions carefully and precisely, and try to think …the majority wave aside of them. It reflects on work."

34. Learning structures

35. Learning structures

36. Learning structures

37. Conclusions
Data analysis should not be pointlessly formal, but instead “ ... it should make an interesting claim; it should tell a story that an informed audience will care about and it should do so by intelligent interpretation of appropriate evidence from empirical measurements or observations” (Abelson, 1995, p. 2).

38. Conclusions
Reviewers (mostly seasoned scientists) usually accept the intellectual challenge of an innovative methodological approach.
Means to reach an interesting academic end are usually supported … and that builds YOUR scientific credibility over time.

39. References
Abelson, R. P. (1995). Statistics as Principled Argument. Hillsdale, NJ: Lawrence Erlbaum Associates.
Anderson, J. (1995). Cognitive Psychology and Its Implications. Freeman: New York.
Bannan-Ritland, B. (2003). The Role of Design in Research: The Integrative Learning Design Framework. Educational Researcher, 32(1),
Barry, A. E., Szucs, L. E., Reyes, J. V., Ji, Q., Wilson, K. L., & Thompson, B. (2016). The Handling of Quantitative Results in Published Health Education and Behavior Research. Health Education & Behavior, 43(5), 518–527.
Brannen, J. (2004). Working qualitatively and quantitatively. In C. Seale, G. Gobo, J. Gubrium, & D. Silverman (Eds.), Qualitative Research Practice (pp ). London: Sage.
Breiman, L. (2001). Statistical Modeling: The Two Cultures. Statistical Science, 16(3), 199–231.
Chioncel, N. E., Van Der Veen, R.G.W., Wildemeersch, D., and Jarvis, P “The validity and reliability of focus groups as a research method in adult education.” International Journal of Lifelong Education 22(5):
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Second edition. Hillsdale, NJ: Lawrence Erlbaum Associates.
Fisher, R. (1935). The design of experiments. Edinburgh: Oliver & Boyd.

40. References
Flyvbjerg, B. (2004). Five misunderstandings about case-study research. In C. Seale, J. F. Gubrium, G. Gobo, & D. Silverman (Eds.), Qualitative Research Practice (pp ). London: Sage.
Gigerenzer, G. (2000). Adaptive thinking. New York: Oxford University Press.
Gigerenzer, G., Krauss, S., & Vitouch, O. (2004). The null ritual: What you always wanted to know about significance testing but were afraid to ask. In D. Kaplan (Ed.), The SAGE handbook of quantitative methodology for the social sciences (pp ). Thousand Oaks: Sage.
Gobo, G. (2004). Sampling, representativeness and generalizability. In C. Seale, J. F. Gubrium, G. Gobo, & D. Silverman (Eds.), Qualitative Research Practice (pp ). London: Sage.
Hair, J. F., Anderson, R. E., Tatham, R. L., & Black, W. C. (1998). Multivariate Data Analysis. Fifth edition. Englewood Cliffs, NJ: Prentice Hall.

41. References
Jackson, S. (2006). Research Methods and Statistics. A Critical Thinking Approach. Second edition. Belmont, CS: Thomson.
Lavine, M. L. (1999). What is Bayesian Statistics and Why Everything Else is Wrong. The Journal of Undergraduate Mathematics and Its Applications, 20,
Nokelainen, P. (2006). An Empirical Assessment of Pedagogical Usability Criteria for Digital Learning Material with Elementary School Students. Journal of Educational Technology & Society, 9(2),
Nokelainen, P. (2008). Modeling of Professional Growth and Learning: Bayesian Approach. Tampere: Tampere University Press.
Nokelainen, P., & Ruohotie, P. (2009). Non-linear Modeling of Growth Prerequisites in a Finnish Polytechnic Institution of Higher Education. Journal of Workplace Learning, 21(1),

42. References
Paré, G., Trudel, M. C., Jaana, M., & Kitsiou, S. (2015). Synthesizing information systems knowledge: a typology of literature reviews. Information and Management, 52(2),
Pylväs, L., Mikkonen, S., Rintala, H., Nokelainen, P., & Postareff, L. (in press). Guiding the workplace learning in vocational education and training: A literature review. To appear in Empirical Research in Vocational Education and Training.
Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54(4),
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25(2),
de Vaus, D. A. (2004). Research Design in Social Research. Third edition. London: Sage.