1. Designing and Developing Research Projects and Dissertation on IT in Education
Alvin Kwan
Division of Information & Technology Studies
Faculty of Education
University of Hong Kong
IT in Education Symposium 2004

2. IT in Education Symposium 2004
The aim
The aim of this seminar is to share with the audience some common mistakes I found from the dissertations or projects my students, who were mostly first-time educational researchers, so that those mistakes can hopefully be avoided when the audience conduct their own research in the future.
IT in Education Symposium 2004

3. IT in Education Symposium 2004
My assumptions
The audience have no experience of conducting academic research, in particular, in the educational sector.
Given the time constraint, I would talk more on how not to do educational research as I believe this will help the audience to pick up the key ideas of doing research rightly faster.
IT in Education Symposium 2004

4. IT in Education Symposium 2004
Topics of discussion
Types of educational research
Characteristics of experimental studies
Typical research tools
Some common mistakes in research design
Some common problems in writing a dissertation
IT in Education Symposium 2004

5. Types of educational research
Confirmatory research
Researcher has generated a theoretical model that requires to be tested (e.g., can educational games motivate students’ learning?)
Exploratory research
To study a phenomenon so as to identify the main variables of interest, their relationships and any potential causal linkages (e.g., what types of educational games motivate students’ learning most?)
IT in Education Symposium 2004

6. Another classification
Historical
Heavily rely on the use of source documents
Case studies
No intention to generalize inferences to wider populations
Longitudinal
Study of individual over time
Survey
Experimental (which is our focus of this seminar)
IT in Education Symposium 2004

7. Typical characteristics of experimental studies
Identification of independent variables (including any treatment variables) and dependent variable
Active manipulation of treatment variables by the researcher, e.g., adopting a new teaching practice
The use of random assignment of units, typically students, to each treatment type
Measure and analyze any differences between the results, i.e., values of the dependent variable, associated with different treatments
IT in Education Symposium 2004

8. IT in Education Symposium 2004
Caveat 1
Research data may suggest association, but not causal relationship, between or among variables.
IT in Education Symposium 2004

9. IT in Education Symposium 2004
Caveat 2
Too many potential independent variables to choose from
Student factors: age, sex, IT skill, learning ability, accessibility to computer, and family background, etc.
Teacher factors: teaching styles, teaching methods, and IT skill, etc.
School factors: school policy, principal’s leadership, IT equipment availability and accessibility, etc.
IT in Education Symposium 2004

10. IT in Education Symposium 2004
Caveat 3
It is difficult to set up a proper control for comparison purpose in educational research.
A random sampling strategy, especially when the sampling size is small, cannot warrant a good control.
Pre-test may be required to demonstrate that control and experimental groups are properly established.
IT in Education Symposium 2004

11. IT in Education Symposium 2004
Caveat 4
Ensure that you only change one independent variable at a time in a selected treatment or experimental group.
If you cannot ensure that only one independent variable is changed in a treatment group due to some practical constraints, do discuss it and argue that the values of those “uncontrolled” independent variables are pretty much unchanged.
IT in Education Symposium 2004

12. Differences between research in natural science and education
Hard to set up control experiments for comparison purpose in educational research
Too many potential independent variables which are not controllable
Experimental results may not be reproducible
Same experiments cannot be applied to the same subjects without affecting the validity of the research results.
IT in Education Symposium 2004

13. Typical research tools
Survey
Direct observation
Interview
Focus group
IT in Education Symposium 2004

14. A remark on research tool
It is often difficult to distinguish between qualitative and quantitative studies at the level of research technique or data collection procedures.
The distinction between qualitative and quantitative studies lies on how collected data are analyzed.
IT in Education Symposium 2004

15. Some common mistakes in research design
Unclear or non-measurable research goals
Setting over-ambitious, vaguely defined or too many (sometimes even unrelated) research goals
Depth is more important than breadth
Fail to pose significant questions that can be investigated empirically
Examples of significant questions
Questions that address unexploited research areas
Refining or refuting a prior relevant theory
IT in Education Symposium 2004

16. IT in Education Symposium 2004
Caveat 5
Research goals must be measurable.
Goals-> Perspectives -> Indicators
E.g., if you want to study whether student motivation can be improved after adopting ICT, you have to ask yourself in which aspects of learning that student motivation would exhibit and how signs of student motivation can be measured in each of those aspects.
IT in Education Symposium 2004

17. Some common mistakes in research design (continued)
Doing your own research
Be aware of relevant prior research, if any, by doing a proper literature review so that
you won’t reinvent the wheel
you would have a better idea of how to justify the value of your research among other similar research
you can discuss your work in light of others, especially relevant theory
IT in Education Symposium 2004

18. IT in Education Symposium 2004
Caveat 6
Arguments must be substantiated.
If you cannot “prove” it (from the data you collected), then “steal” it (from other research).
The legitimate way of “stealing” ideas is referred to as “proper citation”.
IT in Education Symposium 2004

19. Some common mistakes in research design (continued)
Interpreting statistics in your own ways
Beware of statistical error
Be aware of any assumptions of a statistical test, e.g., normality of data for parametric statistics
Computing means of ordinal data can be misleading
IT in Education Symposium 2004

20. Some common problems in writing a dissertation
Too eager to tell the audience all you know
Convey a coherent message in your dissertation
Use appendix if necessary
Too “close” to your work
Logical gap in arguments could be difficult to detect by yourself
Distance “yourself” from your work before re-reading
IT in Education Symposium 2004

21. Some common problems in writing a dissertation (continued)
Telling your research results only
Are your research results in line with other similar research?
Too little time for the write-up process
Teachers assess students’ dissertations largely based on their dissertation.
A poorly written dissertation can be “lethal”.
IT in Education Symposium 2004

22. IT in Education Symposium 2004
Caveat 7
Negative results can be as meaningful as positive results.
Provided that there is nothing wrong with the research design, negative results often reflect possible inadequacies in current theories and may give hints at future research direction.
IT in Education Symposium 2004

23. IT in Education Symposium 2004
Thank You!
IT in Education Symposium 2004