1. Copyright © Allyn & Bacon 2008 Intelligent Consumer Chapter 14 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: Any public performance or display, including transmission of any image over a network; Preparation of any derivative work, including the extraction, in whole or in part, of any images; Any rental, lease, or lending of the program.

2. Copyright © Allyn & Bacon 2008 Discussion Topics The purpose of the text Questions for quantitative studies Questions for qualitative studies Reviewing three studies

3. Copyright © Allyn & Bacon 2008 Text Purpose Present and explain the fundamental principles of educational research – The quality of published educational research varies greatly – There is a need to have the knowledge and skills to critically evaluate educational research Rephrasing the consumer tips from each chapter into questions to provide guidelines for the evaluation of research articles

4. Copyright © Allyn & Bacon 2008 Quantitative Criteria Topics for evaluation questions for quantitative studies – Research problem – Review of the literature – Research hypotheses – Selection of participants – Instrumentation – Design – Results – Discussion and conclusions

5. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies Research problem – What are the independent and dependent variables? – Is the problem researchable? – Is the problem significant? Will the results have practical or theoretical importance?

6. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies – Is the problem stated clearly and succinctly? – Does the problem communicate whether the study is descriptive, experimental, or non- experimental? – Does the problem indicate the population studied? – Does the problem indicate the variables in the study?

7. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies Review of the literature – Does the review of literature seem comprehensive? Are all important previous studies included? – Are primary sources emphasized? – Is the review up-to-date? – Have studies been critically reviewed, flaws noted, and results summarized?

8. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies – Does the review emphasize studies directly related to the problem? – Does the review explicitly relate previous studies t the problem? – If appropriate, does the review establish a basis for research hypotheses? – Is the review well organized?

9. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies Research hypotheses – Is the hypothesis stated in declarative form? – Does the hypothesis follow from the literature? – Does the hypothesis state expected relationships or differences? – Is the hypothesis testable? – Is the hypothesis clear and concise?

10. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies Selection of participants – Are the participants clearly described? – Is the population clearly defined? – Is the method of sampling used? If so, is it proportional or disproportional? – What is the return rate in a survey study? – Are volunteers used? – Is there an adequate number of participants?

11. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies Instrumentation – Is evidence for validity and reliability clearly stated and adequate? Is the instrument appropriate for the participants? – Are the instruments clearly described? If an instrument is designed for a study by the researchers, is there a description of its development? – Are the procedures for gathering data clearly described? – Are norms appropriate if norm-referenced tests are used?

12. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies – Are standard-setting procedures appropriate if criterion- referenced tests are used? – Do the scores distort the reality of the findings? – Do response sets or faking influence the results? – Are observers and interviewers adequately trained? – Are there observer and interviewer effects?

13. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies Design – Non-experimental If descriptive, are relationships inferred? Do graphic presentations distort the findings? If comparative, are criteria for identifying different groups clear? Are causative conclusions reached from correlational findings?

14. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies Is the correlation affected by restriction in the range and reliability of the instruments? If predictions are made, are they based on a different sample? Is the size of the correlation large enough? If causal-comparative, has the causal condition already occurred? How comparable are the subjects in the groups being compared?

15. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies – Experimental Is there direct manipulation of an independent variable? Is the design clearly described? What extraneous variables are not controlled in the design? Are the treatments very different from one another? Is each replication of the treatment independent of the other replications? Is the number of subjects equal to the number of treatment replications?

16. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies – Single subject Is the measurement of the target behavior reliable? Is the target behavior clearly defined? Are there enough measures of the behavior to establish stability? Are procedures, subjects, and settings described in detail? Is there a single treatment? Are there experimenter or observer effects?

17. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies Results – Is there an appropriately descriptive statistical summary? – Is statistical significance confused with practical significance? – Is statistical significance confused with internal or external validity? – Are appropriate statistical tests used?

18. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies – Are levels of significance interpreted correctly? – How clearly are the results presented? – Is there a sufficient number of subjects to give valid statistical results? – Are data clearly and accurately presented in graphs and tables?

19. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies Discussion and conclusions – Is interpretation of the results separate from reporting of the results? – Are the results discussed in relation to previous research, methodology, and the research problem? – Do the conclusions address the research problem? – Do the conclusions follow from the interpretation of the results?

20. Copyright © Allyn & Bacon 2008 Questions for Quantitative Studies – Are the conclusions appropriately limited by the nature of the subjects, treatments, and measures? – Is lack of statistical significance properly interpreted? – Are the limitations of the findings reasonable? – Are the recommendations and implications specific? – Are the conclusions consistent with what is known from previous research?

21. Copyright © Allyn & Bacon 2008 Qualitative Criteria Topics for evaluation questions for qualitative studies – Introduction and problem – Review of the literature – Methodology – Discussion and conclusions

22. Copyright © Allyn & Bacon 2008 Questions for Qualitative Studies Introduction and problem – Are the researcher’s background, interests, and potential biases clear from the outset? – Does the researcher have the skill and training needed to conduct the study? – Is the problem feasible?

23. Copyright © Allyn & Bacon 2008 Questions for Qualitative Studies – Is the problem significant? – Is there a clear conceptual and theoretical framework for the problem? – Does the introduction include an overview of the design of the study? – Is the purpose of the study clearly stated?

24. Copyright © Allyn & Bacon 2008 Questions for Qualitative Studies Review of the literature – Is the review preliminary? Does it indicate that the researcher is knowledgeable about previous work in the area? – Is the review up-to-date? – Does the review establish an adequate background and theoretical framework for the study? – Is the review well organized? – Is the literature analyzed as well as summarized?

25. Copyright © Allyn & Bacon 2008 Questions for Qualitative Studies Methodology – Is the method of selecting participants clear? – Is the selection of participants biased? – Will the participants selected provide a credible answer to the research question? – How involved is the researcher in the setting? Will the researcher’s involvement affect the findings?

26. Copyright © Allyn & Bacon 2008 Questions for Qualitative Studies – How detailed are the field notes? – Are data collectors properly trained? – Are multiple methods of data collection used? – Is the study long enough?

27. Copyright © Allyn & Bacon 2008 Questions for Qualitative Studies Discussion and conclusions – Are descriptions clearly separate from interpretations and researcher’s opinions? – Is the credibility of the findings addressed in terms of reliability, internal validity, an external validity? – Are the results discussed in relation to previous research?

28. Copyright © Allyn & Bacon 2008 Questions for Qualitative Studies – Do the conclusions follow from the interpretation of the results? Are the conclusions consistent with what is known from previous research? – Are appropriate limitations indicated? – Are appropriate recommendations and implications indicated?

29. Copyright © Allyn & Bacon 2008 Evaluation of Studies Evaluation of a quantitative, non-experimental research article – Article 1 – Investigating High School Students’ Science Experiences and Mechanics Understanding by Phillips, K. A. & Barrow, L. H. – See the text Evaluation of a quantitative, experimental research article – Article 2 – The Effect of Reciprocal Peer Counseling in the Enhancement of Self-Concept Among Adolescents by Egbochuku, E.O., & Obiunu, J.J. – See the text

30. Copyright © Allyn & Bacon 2008 Evaluation of Studies Evaluation of a qualitative research article – Article 3 – How Elementary School Counselors Can Meet the Needs of Students with Disabilities by Frye, H.N. – See the text