1. Educational Research Methods
Dr. K. A. Korb
University of Jos

2. Introduction
There is no point to analyzing data from a study that was not properly designed to answer the research question under investigation. In fact, there’s a real point in refusing to analyze such data less faulty results be responsible for implementing a program or policy contrary to what’s really needed.
Two of the valuable things a researcher can possess are:
Knowledge of the principle of good study design
The courage to refuse to cut corners
(Dallal, 1998)
Dr. K. A. Korb
University of Jos

3. Scientific Method 1: Ask Question 6: Share Findings 2: Design Study
5: Reach Conclusions
3: Collect Data
4: Analyze Results
Dr. K. A. Korb
University of Jos

4. 1. Ask a Question What is your general research question?
What is the independent variable in the research question?
What is the dependent variable in the research question?
Dr. K. A. Korb
University of Jos

5. 2. Design the Study
Research Design: Structure the research to show how the major parts of the research project work together to address the central research questions
Participants
What is the population of people I am interested in?
Instruments
How will I accurately measure my independent and dependent variables?
Procedure
How will I collect data in an appropriate manner?
Dr. K. A. Korb
University of Jos

6. Outline Types of Research Methodologies
Descriptive
Causal Comparative
Correlation
Experimental
Quasi-Experiment
Threats to valid Research Design
Dr. K. A. Korb
University of Jos

7. Descriptive
Purpose: Carefully describe a naturally occurring educational phenomenon through systematic observation
Key characteristic: Describes one (sometimes more) variable within a particular population
Descriptive research designs are sometimes also called Survey Research Designs.
Dr. K. A. Korb
University of Jos

8. Descriptive
Research hypotheses are impossible for descriptive designs. Only research questions should be posed.
There are no independent or dependent variables, simply the key psychological variables that the researchers is interested in.
Example questions:
What are teachers’ attitude towards a new moral educational program?
What would counselors recommend be included in an HIV/AIDS curriculum?
How many students engage in exam malpractice?
Dr. K. A. Korb
University of Jos

9. Descriptive
Select participants: Define the group that possess the variables you want to study
Data collection: Administer valid measures of the variables of interest
Data analysis: Compute descriptive statistics
Dr. K. A. Korb
University of Jos

10. Descriptive
Dr. K. A. Korb
University of Jos

11. Causal Comparative
Purpose: Determine a cause/effect relationship where the independent variable cannot be manipulated
Key characteristic: Compare two or more naturally-occurring groups on the dependent variable
Causal Comparative designs can also be called ex post facto designs
Dr. K. A. Korb
University of Jos

12. Causal Comparative
Null hypothesis: There is no significance difference between GROUPS on DEPENDENT VARIABLE.
Independent variable: Group category
Dependent variable: What the groups are hypothesized to differ on.
Examples:
There is no significant differences between boys and girls on interest in mathematics.
There is no significant differences between children from low and high socioeconomic status in the number of books read in a month.
There is no significance differences between Nigeria and British children in academic self efficacy.
Dr. K. A. Korb
University of Jos

13. Causal Comparative
Select participants: Obtain a critical number of participants in each group
Data collection: Administered valid measures of your variables to all groups of participants
Data analysis: Calculate the mean score for each group on the dependent variable.
However, any difference in the mean score between groups might be due to error.
Therefore, inferential statistics are necessary to determine if the difference are significant.
Dr. K. A. Korb
University of Jos

14. Inferential Statistics
The inferential statistics to be depends on how many groups are being compared
Two groups: t-test
Three or more groups: ANOVA
If the value is greater than.05, retain the null hypothesis.
There is no significant difference between boys and girls on interest in math.
If the p-value is less than .05, reject the null hypothesis.
There is a significant difference between boys and girls on interest in math.
Look at the mean score to determine which group has the stronger interest.
Dr. K. A. Korb
University of Jos

15. Causal Comparative
Dr. K. A. Korb
University of Jos

16. Correlational
Purpose: Quantity the extent to which two variables are associated
Key characteristic: The same group of participants are given measures of both key variables in order to calculate the correlation coefficient
Because of the directionality and third variable problems, there are no independent or dependent variables in correlational designs
Dr. K. A. Korb
University of Jos

17. Correlational
Null hypothesis: There is no significant relationship between VARIABLE 1 and VARIABLE 2.
Examples
There is no significant relationship between number of books a child reads at home and reading ability.
There is no significant relationship between WAEC scores and university GPA.
There is no significant relationship between the amount of time spent in lecture and frequency of exam malpractice.
Dr. K. A. Korb
University of Jos

18. Correlational Select participants: A homogeneous sample
Data collection: Administer valid measures of the variables
Data Analysis: Calculate the correlation coefficient and the significance test for correlation
Dr. K. A. Korb
University of Jos

19. Interpreting Correlations
Dr. K. A. Korb
University of Jos
Interpreting Correlations
Nature:
Negative
Positive -1 +1
Strength:
Nature
Positive: Two variables increase or decrease together
Negative: As one variable increases, the other decreases
Strength
Closer to -1 or +1 is stronger relationship
0 is no relationship
Dr. K. A. Korb
University of Jos

20. Correlation = .78
Dr. K. A. Korb
University of Jos

21. Correlation = -.86
Dr. K. A. Korb
University of Jos

22. Correlation: .04
Dr. K. A. Korb
University of Jos

23. Research Example
The effect of regular leisure reading on reading achievement in primary school.
Students in a primary school will be randomly assigned to either a treatment or a control group.
The treatment group will spend 10 minutes everyday reading a book with an adult.
In an attempt to keep the treatment and control groups as similar as possible, the control group will also spend 10 minutes with an adult everyday, but will instead do math problems.
Dr. K. A. Korb
University of Jos

24. Amount of time spent studying
Correlational
Correlation does not prove causation.
Amount of time spent studying
Interest
Academic
Achievement
Interest
Academic
Achievement
Interest
Academic
Achievement
Directionality Problem
Third Variable Problem
Dr. K. A. Korb
University of Jos

25. Experimental Purpose: Establish cause and effect
Key characteristic: Statement about how one variables affects (causes) another
Three essential components of an experiment
Treatment
Control
Random assignment to treatment and control groups
Dr. K. A. Korb
University of Jos

26. Experimental
Null hypothesis: There is no significant effect of TREATMENT on DEPENDENT VARIABLE.
Independent variable: Treatment
Dependent variable: What the treatment should change
Examples
There is no significant effect of a counseling intervention on homework completion.
Treatment: Counseling
There is no significance effect of reading a book a day on reading achievement.
Treatment: Reading a book a day
There is no significant effective of an anti-exam malpractice campaign on the frequency of exam malpractice.
Treatment: Anti-exam malpractice campaign
Dr. K. A. Korb
University of Jos

27. Experimental
Experimental Group
Goal: Keep the experiences of the treatment and control groups as identical as possible except for the treatment in order to conclude that the change in the dependent variable is the result of the treatment.
Read a book every day
Reading
Achievement
Reading Class
Control Group
Reading Class
Reading
Achievement
Dr. K. A. Korb
University of Jos

28. Experimental
Interest
Experimental Group
Random Assignment: Each subject has an equal chance of being assigned to either the treatment or control group
Minimize individual differences in extra variables that might influence the dependent variable
Read a book every day
Reading
Achievement
Reading Class
Control Group
Reading Class
Reading
Achievement
Dr. K. A. Korb
University of Jos

29. Random Assignment Draw names out of a hat Random number table S/No
Group 1
Treatment 2 3
Control 4 5 6 7 8 9 10
Dr. K. A. Korb
University of Jos

30. Experiment Pretest, Post-test Procedure
Random assignment of participants to experimental and control groups
Administered valid measure of DV as pretest to all groups
Administer treatment to experimental groups(s)
Administer identical (or conceptually identical) measure as posttest
Statistical Analysis
Descriptive statistics of pretest and post-test score for experiment and control groups separately
Significance tests to determine significance between differences (Analysis of Covariance: ANCOVA)
Dr. K. A. Korb
University of Jos

31. Experimental
Dr. K. A. Korb
University of Jos

32. Experimental Post-test Only Procedure
Random assignment of participants to experimental and control groups
Administered treatment to experimental group(s)
Administer valid measure of DV post-test
Statistical Analysis
Descriptive statistics of post-test score for experimental and control groups separately
Significance tests to determine significance between differences (Analysis of Variance – ANOVA or t-test)
The post-test only procedure is typically recommended because of the possibility that the pretest may have an effect on the experiment treatment
Dr. K. A. Korb
University of Jos

33. Experimental
Dr. K. A. Korb
University of Jos

34. Quasi-Experiment
Purpose: Conduct an experiment when random assignment is not possible
Key characteristic: Treatment and control groups, but no random assignment
The null hypotheses are exactly the same as the Experimental method
Dr. K. A. Korb
University of Jos

35. Quasi-Experiment Pretest-Posttest Procedure
Administer valid measure of DV as pretest
Administer treatment to experimental group(s)
Administer identical (or conceptually identical) measure as posttest
Statistical Analysis
Descriptive statistical of pretest and posttest score for experiment and control groups separately
Significance tests to determine significance between difference (ANCOVA)
Dr. K. A. Korb
University of Jos

36. Review: Research Designs
Descriptive: Carefully describe a naturally occurring educational phenomenon through systematic observation
Causal Comparative: Compare two or more groups on a dependent variable
Correlational: Quantify the extent to which two variables are related
Experiment: Establish cause and effect
Three requirement:
Treatment Groups(s)
Control Group
Random Assignment
Quasi-Experiment: Experiment where random assignment is not possible
Dr. K. A. Korb
University of Jos

37. Research Designs Conclusion
Testing the effectiveness of a counseling or teaching intervention MUST either use an experimental or quasi-experimental design
Simply administering surveys CANNOT establish the quality of a counseling or teaching intervention
Dr. K. A. Korb
University of Jos

38. Research Example
The effect of regular leisure reading on reading achievement in primary school
Students in a primary school will be randomly assigned to either a treatment or a control group.
The treatment group will spend 10 minutes everyday reading a book with an adult.
In an attempt to keep the treatment and control groups as similar as possible, the control group will also spend 10 minutes with an adult everyday, but will instead do math problems.
After one month of the treatment, both groups will be tested on a reading achievement test.
Dr. K. A. Korb
University of Jos

39. Research Example IV: Regular Leisure Reading DV: Reading Achievement
Experimental Group
Reading
Achievement
Read Book
Every Day
with Adult
Reading
Achievement
Control Group
Reading
Achievement
Do Maths
Every Day
with Adult
Reading
Achievement
Dr. K. A. Korb
University of Jos

40. Construct Validity
Construct Validity: Establishing valid operational measures for the concepts being studied
This relates to:
Independent Variable
Dependent Variable
Definitions of Variable:
Construct Definition: General explanation of the construct
Operational Definition: Statement of specifically how the construct will be measured or implemented in the study
Dr. K. A. Korb
University of Jos

41. Construct vs. Operational Definitions
WAEC Scores
Final Course Exam
Academic Achievement
NECO Scores
State Exams
Self-Report of Parents’
Yearly Income
Economic
Status
Public/Private School
Dr. K. A. Korb
University of Jos

42. Research Example Construct validity focuses on:
IV:
Regular
Reading
DV:
Reading
Achievement
Experimental Group
Reading
Achievement
Read a Book
Every Day
Reading
Achievement
Reading
Achievement
Control Group
Reading
Achievement
Dr. K. A. Korb
University of Jos

43. Construct Validity
Research studies begin with hypotheses about psychological construct.
There is no significant effect of regular leisure reading on reading achievement scores.
Independent variable: Regular leisure reading
Dependent variable: Reading achievement score
The psychological constructs are then operationalized into manipulatable, measurable terms.
Regular leisure reading: Reading a book with an adult everyday for 10 minutes
Reading achievement score: Classroom exam scores
Dr. K. A. Korb
University of Jos

44. Construct Validity
When finished analyzing the data, the result will be translated back into psychological constructs.
The conclusion should not be limited to Reading a book for 10 minutes everyday with an adult leads to higher score on a classroom reading exams.
The conclusion should be Regular leisure reading will lead to higher reading achievement scores in general.
However, in order to translate the results back to psychological construct, the study must have construct valid measures of the independent and dependent variables.
Dr. K. A. Korb
University of Jos

45. Construct Validity
When you develop your research project, you start with a theory.
For example, regular leisure reading will lead to increased reading achievement scores.
Then you operationalize your theory into your research study.
You translate “regular leisure reading” into how it will be implemented in your study – reading a book with an adult everyday for 10 minutes.
You translate “reading achievement scores” into how it will be measured in your study – classroom exam scores.
When you finish collecting data, you hope to be able to move back to theory.
You want to be able to say that not only did reading a book for 10 minutes everyday with an adult lead to higher classroom exam scores on reading achievement, but also that regular leisure reading will lead to higher reading achievement scores in general.
However, in order to do that, you must have construct valid measures of your independent and dependent variables.
Dr. K. A. Korb
University of Jos

46. Research Example Constructs: Operationalized: Constructs: IV: Regular
Reading
DV:
Reading
Achievement
Constructs:
Read book
every day with
an adult for
10 min.
Scores on
reading section
of classroom
exams
Operationalized:
IV:
Regular
Reading
DV:
Reading
Achievement
Constructs:
Dr. K. A. Korb
University of Jos

47. Construct Validity
Construct Validity of Research Designs: Allows generalization from the operationalized treatment and measures to the general psychological constructs
Threats to Construct Validity
Poor construct definitions in the paper
Flawed matching of operationalization to constructs
The measures do not align with the constructs they were designed to measure.
Dr. K. A. Korb
University of Jos

48. Construct Validity
Research studies begin with hypotheses about psychological construct.
There is no significant effect of regular leisure reading on reading achievement scores.
Independent variable: Regular leisure reading
Dependent variable: Reading achievement score
The psychological constructs are then operationalized into manipulatable, measurable terms.
Regular leisure reading: Reading a book with an adult everyday for 10 minutes
Reading achievement score: Classroom exam scores
Dr. K. A. Korb
University of Jos

49. Construct Validity
When finished analyzing the data, the result will be translated back into psychological constructs.
The conclusion should not be limited to Reading a book for 10 minutes everyday with an adult leads to higher score on a classroom reading exams.
The conclusion should be Regular leisure reading will lead to higher reading achievement scores in general.
However, in order to translate the results back to psychological construct, the study must have construct valid measures of the independent and dependent variables.
Dr. K. A. Korb
University of Jos

50. Construct Validity
When you develop your research project, you start with a theory.
For example, regular leisure reading will lead to increased reading achievement scores.
Then you operationalize your theory into your research study.
You translate “regular leisure reading” into how it will be implemented in your study – reading a book with an adult everyday for 10 minutes.
You translate “reading achievement scores” into how it will be measured in your study – classroom exam scores.
When you finish collecting data, you hope to be able to move back to theory.
You want to be able to say that not only did reading a book for 10 minutes everyday with an adult lead to higher classroom exam scores on reading achievement, but also that regular leisure reading will lead to higher reading achievement scores in general.
However, in order to do that, you must have construct valid measures of your independent and dependent variables.
Dr. K. A. Korb
University of Jos

51. Internal Validity
Internal validity: Extent to which variables other than the treatment provide plausible explanations to the experimental results
Only relevant for experimental designs
Primary consideration for interventions
Extraneous variable: Any variable other than treatment variable that, if not controlled, can affect the experimental outcome
Hold constant or eliminate all extraneous variables that might affect the posttest
Goal of Research Design: Create set of conditions so any observed changes in your dependent variable can be attributed to experimental treatment instead of extraneous variables
Do variables other than the treatment provide plausible explanations to the experimental results?
Dr. K. A. Korb
University of Jos

52. Research Example Maturation IV: Regular Reading DV: Reading
Achievement
Interest
Experimental Group
Reading
Achievement
Read a Book
Every Day
Reading
Achievement
Reading
Achievement
Control Group
Reading
Achievement
Reading
Ability
Adult
Attention
Dr. K. A. Korb
University of Jos

53. Threats to Internal Validity
History: If treatment lasts over time, other events may influence dependent variable
To prevent: Keep the control group equivalent in all aspects but treatment
Maturation: Physical or psychological changes may influence the dependent variable
To prevent: Use a control group
Selection Bias: Treatment and control groups are different on an important extraneous variable
To prevent: Random assignment OR Use a pre-test
Dr. K. A. Korb
University of Jos

54. Threats to Internal Validity
Testing: Giving a pretest may increase performance on the post-test
To prevent: Don’t use a pre-test OR Use different pre- and post-tests
Instrumentation: Nature of measuring instrument has changed between pre- and post-test
To prevent: Do not use a pre-test OR Use the same for pre- and post-test
Many of these solutions to prevent threats to internal validity contradict.
Consider the threats that are most relevant to a particular study and then develop a strategy for overcoming the most important threats.
Dr. K. A. Korb
University of Jos

55. External Validity
External Validity: Establishing the group of people to which the research findings can be generalized
Dr. K. A. Korb
University of Jos

56. Population Sample Research Example Experimental Group Control Group
Reading
Achievement
Read a Book
Every Day
Reading
Achievement
Sample
Control
Group
Reading
Achievement
Reading
Achievement
Dr. K. A. Korb
University of Jos

57. Types of External Validity
Population Validity: Generalizing the results from an experimental sample to a defined population
To establish: Give a thorough explanation of the constitution of the constitution of the sample and how the sample was selected.
Ecological Validity: Generalizing the results of a study from the artificial conditions created by researcher to real-life conditions
To establish:
Give an adequate description of treatment
Give an adequate description of the measure for the dependent variable
Threats to Ecological Validity:
Novelty: The treatment was different than typical treatment
Experimenter Effect: The particular person delivering the treatment affects results
Dr. K. A. Korb
University of Jos

58. Reliability
Reliability: Demonstrating that the operations of the study can be repeated with the same results
Conduct research as if someone were always looking over your shoulder
Report the procedure of your study in explicit detail
Dr. K. A. Korb
University of Jos

59. Other Issues in Valid Research Design
Experimenter Bias: Researcher’s expectations about the outcome of experiment influences participants’ response
To prevent: Use research assistants who are blind to the study (aka do not know the purpose of the study)
Treatment Fidelity: Extent to which treatment conditions are implemented according to the researcher’s specifications
To support:
Carefully train research assistants
Periodically check up on research assistants unexpectedly to determine if they are following guidelines
Dr. K. A. Korb
University of Jos

60. Other Issues in Valid Research Design
Strong Experimental Treatments: Developing a treatment that will make a robust effect on the dependent variable
To support:
Develop a thorough understand of the dependent variable and how it can be influenced.
Spend much thoughtful time developing your treatment.
Dr. K. A. Korb
University of Jos

61. Revision Describe the five types of educational research designs.
What is construct validity as related to research designs? How can it be supported?
What is internal validity? What are threats to internal validity?
What is external validity? What are types of external validity?
What are some other issues to consider when conducting educational research?
Dr. K. A. Korb
University of Jos