1. Understanding the Research Process & Research Reports: Knowing What To Look for—and Why!
Glenn M. Hymel, EdD, LMT
Department of Psychology
Loyola University New Orleans, &
Massage Therapy Foundation
AMTA; Cincinnati, OH; 28 September 2007

2. PaPa Glenn’s Advisory Council:
Master Jason, Master Andrew, & Master Sean

3. Orientation to Session’s Materials & Methods
Description, Content Outlines, & Performance Objectives for the Session--Handout
Chapter 3/Unit 1 from Hymel (2006)--Handout
Chapter 4/Unit 2 from Hymel (2006)--Handout
PowerPoint Slides & Overhead Transparencies
Didactic Presentation (with Attempts at Humor)
Analyses of Three Published Massage Therapy Research Studies
Ongoing Questions & Discussions
Occasional “Exercises in Accountability”--Handouts
Planning for Session Follow-Ups

4. A Few Guiding Thoughts . . .
“I never faced a problem which was more than the eternal problem of finding order” (B. F. Skinner, in the context of discussing scientific practice).
“Everything should be made as simple as possible, but not simpler” (Albert Einstein).
“Live as if you were to die tomorrow; learn as if you were to live forever” (Gandhi).
“Blessed are the flexible, for they shall not be bent out of shape” (Michael McGriffy, MD)

5. “Our Most Important Research Questions
May Be ‘Hiding’ in Our Unchallenged Solutions”
___________________________________________________________
From Harris, S. (1991). “You want proof? I’ll give you proof!”: More cartoons from Sidney Harris. New York: W. H. Freeman and Company.

6. Introduction & Overview: Research Universe
Quantitative Research
Qualitative Research
Integrative Research

7. Introduction & Overview: Research Continuum (General to Specific)
Research Category (Most General)
Research Strategy
Research Method
Research Design
Research Procedure (Most Specific)

8. Introduction & Overview: Quantitative Research Category
Difference-Oriented Research Strategy
True Experimental or RCT Method
Quasi-Experimental Method
Single-Case Experimental Method
Nonexperimental Comparative Groups Method
Association-Oriented Research Strategy
Correlational Method
Predictive Method
Descriptive-Oriented Research Strategy
Single-Case Quantitative Analysis Method
Survey method
Naturalistic/Structured Observational Method
Case Report Method

9. Introduction & Overview: Chapter 3/Unit 1— An Overview of the True Experimental/RCT Research Method
Chang, M. Y., Wang, S. Y., & Chen, C. H. (2002). Effects of massage on pain and anxiety during labor: A randomized controlled trial. Journal of Advanced Nursing, 38(1),

10. Clarification of Terms
Measurement
Statistics
Research
Evaluation

11. “Our Species Had to Start Somewhere in the
Quest for Measurement Sophistication”
________________________________________________
From Harris, S. (2004). Einstein simplified: Cartoons on science (Rev. ed.). New Brunswick, NJ: Rutgers University Press.

12. True Experimental, or Randomized Controlled Trial, Research Method
First Example from the Massage Therapy Research Literature: Chang, Wang, & Chen (2002)
Abstract (p. 24 in Chapter 3 Handout)

13. “A Warm-Up Exercise in Flowcharting
To Prepare for the Very Next Section!”
_______________________________________________
From Harris, S. (1991). “You want proof? I’ll give you proof!”: More cartoons from Sidney Harris. New York: W. H. Freeman.

14. “Jumping Ahead”— Schematic Illustration of the Chang et al
“Jumping Ahead”— Schematic Illustration of the Chang et al. (2002) Study
Flowchart of Sorts (p. 32 in Chapter 3 Handout)
Reflecting Only Select Features of, & a Slight Modification in, the Study by Chang et al. (2002)

15. Schematic (cont.) of Chang et al. (2002) Study
Research Question & Professional Literature Review
Population & Sample
Randomization: Random Selection & Random Assignment

16. Schematic (cont.) of the Chang et al. (2002) Study: Variables
Independent Variable
Dependent Variable
Extraneous Variable
Control Variable
Confounding Variable
Intervening Variable

17. “Yet Another Type of Control Group?”
Another Humble Hymelian Observation: The professional literature seems to be silent on the actual existence of this type of control group distinction; therefore, please exercise caution if and when an author refers to an alleged “out of control” group!
_________________________________________________________________
From Mueller, P. S. (nd). P. S. Mueller cartoons.

18. Schematic (cont.) of the Chang et al. (2002) Study: Hypotheses
Research Hypothesis
Null (or Statistical) Hypothesis)
Alternative Hypothesis

19. Schematic (cont. ) of the Chang et al
Schematic (cont.) of the Chang et al. (2002) Study: Parameter & Statistic
Parameter = Characteristic of a Population
Statistic = Characteristic of a Sample
Parameter : Population :: Statistic : Sample
Relationship to Research Hypothesis, Null Hypothesis, & Alternative Hypothesis

20. Schematic (cont. ) of the Chang et al
Schematic (cont.) of the Chang et al. (2002) Study: Statistical Analysis
Statistical Testing & Inference
Alpha (α) Level (or, Probability of a Type I Error)
P Value (or, Level of Significance)
Statistical Power (or, Power Analysis)
Confidence Interval
Effect size

21. “Off on a Tangent”: A Simple Example of What’s Happening in Null Hypothesis Significance Testing
Claim: Hester Sue Has ESP
Options: Take It on ”Faith” or Empirically Investigate
Null Hypothesis: Hester Sue Does Not Have ESP
Research Hypothesis: Hester Sue Does Have ESP
Task & Observations: Hester Sue Is Asked To Predict the Outcomes of 10 Flips of a Coin, & We Observe the Results
Analysis: Compare Observed Results with Expected Probabilities (Jaccard & Becker, 2002, p.170, next slide)
Decision: With α = .05, Reject Null Hypothesis If Observed Results Have a Probability of Happening < .05

22. “Off on a Tangent”: A Simple Example of What’s Happening in Null Hypothesis Significance Testing
(cont.)
Expected
Probability of
Indicated Number
Number of Expected or More of
Correct Predictions Probability Correct Predictions
p= p=.001
p= p=.011
p= p=.055
p= p=.172

23. Schematic (cont. ) of the Chang et al
Schematic (cont.) of the Chang et al. (2002) Study: Experimental Validity
Internal Validity
Issue: Nature of Relationship between Independent Variable & Dependent Variable
Driven by Random Assignment & Manipulated Independent Variable
External Validity
Issue: Generalizability
Driven by Random Selection

24. Schematic (cont. ) of the Chang et al
Schematic (cont.) of the Chang et al. (2002) Study: Experimental Research Design Notation
See p. 31 in Chapter 3 Handout
“Shorthand” for Identifying Research Category, Strategy, Method, Design, & Procedures

25. Experimental Research Design Notation for the Chang et al. (2002) Study
RA EG X OPost
RA CG ● OPost

26. Measurement & Statistics As Research Tools
Valid & Reliable Measuring Instruments: A Necessity
Measurement Scale & Research Issues: Determinants of Statistical Usage
Descriptive Statistics & Inferential Statistics: Distinctions

27. Major Ethical Issues in Behavioral & Health Science Research
Considerations, e.g., in the Chang et al. (2002) Study
Role of Institutional Review Board (IRB)
Informed Consent
Confidentiality
Latitude To Exit Study without Prejudice
Control Group Entitlement If Possible
Debriefing

28. Where Do We Go from Here?
Concepts, Principles, & Procedures Considered in Greater Detail
Analyses of Two Additional Massage Therapy Research Studies

29. Introduction & Overview: Chapter 4/Unit 2— A More Focused View of the True Experimental/RCT Research Method
Van den Dolder, P. A., & Roberts, D. L. (2003) A trial into the effectiveness of soft tissue massage in the treatment of shoulder pain. Australian Journal of Physiotherapy, 49(3),
Preyde, M. (2000). Effectiveness of massage therapy for subacute low-back pain: A randomized controlled trial. Canadian Medical Association Journal, 162(13),

30. True Experimental/RCT Method & the Research Category-to-Procedures Continuum
Quantitative Research Category
Difference-Oriented Research Strategy
True Experimental/RCT Research Method
Randomized, Control-Group, Posttest-Only Research Design (i.e., Chang et al., (2002)
Random Assignment & Manipulated IV As Prerequisite Research Procedures for Possible Cause-and-Effect Conclusions

31. A “Closer Look” at the True Experimental/RCT Research Method
Second Example from the Massage Therapy Research Literature: Van den Dolder & Roberts (2003)
Abstract (p. 39 in Chapter 4 Handout)

32. A “Closer Look” at Van den Dolder & Roberts (2003)
Research Question & Literature Review
Population, Randomization, & Sample
Variables
Hypotheses
Measurement & Statistics
Research Design
External Validity & Internal Validity
Major Ethical considerations

33. A Framework for Reading a Research Article
A Research Article’s Structure (“Anatomy”), Function (“Physiology”), & Implied Criteria for Evaluation
A Research Article’s Sections Implying Structure & Function
Preliminary Section: Title Page & Abstract
Introductory Section
Method Section
Results Section
Discussion Section
Concluding Section: References & Beyond

34. Content of the Article’s Preliminary Section
Title page suggesting type of study being reported, major variables investigated, and participants who were the focal point of the research effort
Abstract that synthesizes the study’s introduction, method, results, & discussion sections

35. Content of the Article’s Introductory Section
General literature review of available research in the study’s problem area
Specific literature review providing greater detail and depth of coverage
Purpose statement via an explicit statement of the research question
Rationale or justification for the research hypothesis
Explicit statement of the research hypothesis

36. Content of the Article’s Method Section
Participants & sampling procedures, with emphasis on inclusion & exclusion criteria
Research strategy, method, & design
Variables operationalized
Instrumentation used, with emphasis on reliability & validity issues

37. Content of the Article’s Results Section
As implied, outcomes of the study are presented
Reliance on descriptive statistics & inferential statistics
Results of analyses conveyed in context of null hypothesis testing, with implications for the confirmation/disconfirmation of the research and alternative hypotheses

38. Content of the Article’s Discussion Section
Reflections on manner in which the study was conducted, inclusive of its limitations & delimitations (boundaries)
Elaboration on interpretation of study’s findings begun in the results section
Acknowledgment of study’s significance and its relationship to earlier research findings previously cited
Theorizing as to why the study’s results are what they are
Suggestions of recommended areas of further research as logical sequels to the study

39. Content of the Article’s Concluding Section
References via bibliographic citations of works actually cited in the body of the report
Possibly appendixes, author notes, and footnotes as needed

40. A Framework for Reading a Research Article (cont.)
Implied Criteria for Critiquing a Research Article: Reflections of Structure & Function
Critique Questions (n=29) To Ask in Each Section of a Research Article (pp in Chapter 4 Handout)

41. Criteria, or Critique Questions, for the Preliminary Section
1. Does the title of the study provide a basis for identifying the type of study, major variables, & participants?
2. Does the abstract synthesize the main body of the report (i.e., the introduction, method, results, & discussion) with a particular focus on the research question, research hypothesis, participants, research method & design, major variables, instruments, statistical techniques, principal findings, & conclusions?

42. Criteria, or Critique Questions, for the Introductory Section
3. Is the reader introduced to the relevant professional literature bearing on the study being reported by way of a general overview of the research problem area as well as a more specific coverage of individual studies?
4. Is the purpose of the study identified by way of the research question being formulated at an operational level?

43. Criteria, or Critique Questions, for the Introductory Section (cont.)
5. Is a rationale or justification, based on various features of the professional literature, presented as a context or framework for the study’s research hypothesis?
6. Do the authors state the study’s research hypothesis in such a way that the predicted answer to the study’s research question is clear & unambiguous?

44. Criteria, or Critique Questions, for the Method Section
7. Are the study’s participants clearly characterized along with the inclusion & exclusion criteria used for identifying them?
8. Did the researchers justify the number of participants constituting the sample size by way of a power analysis?
9. Was an accessible population of potential participants acknowledged along with an indication of how the sample was derived from such a population, be it through random selection or some other procedure?

45. Criteria, or Critique Questions, for the Method Section (cont.)
10.Did the authors specify the manner in which the participants were assigned to the two or more comparison groups, be it through random assignment or some other means?
11. Was any clarification provided as to how the ethical aspects of the study were governed, particularly in reference to the protection of the participants, the overall integrity of the research, & the earlier approval of the study by an IRB?
12. Was the nature of the research effort adequately characterized in terms of its position in the research continuum (i.e., its position regarding research category, strategy, method, design, & defining procedures)?

46. Criteria, or Critique Questions, for the Method Section (cont.)
13. Were the study’s variables operationalized in a comprehensive fashion so that their manipulation or measurement could be replicated?
14. Did the authors clearly specify the equipment or instruments used in the study for variable manipulation or measurement purposes, along with a documentation of the technical characteristics of such including validity & reliability?

47. Criteria, or Critique Questions, for the Results Section (cont.)
15. Were the data analysis techniques used in the study identified and justified?
16. Were the results of the study communicated by an appeal to descriptive and/or inferential statistics consistent with the nature of the research question as well as the research method & measurement scales used?
17. Were the results of the data analysis related to an appropriate decision regarding the study’s null or statistical hypothesis?

48. Criteria, or Critique Questions, for the Results Section (cont.)
18. Was the decision regarding the null hypothesis acknowledged as a basis for inferring decisions concerning the alternative & research hypotheses?
19. If hypothesis testing was performed, were the analyses augmented with other statistical techniques such as confidence interval estimation or effect size calculations?
20. Were tables & figures appropriately used so as to render the data analyses more comprehensible?

49. Criteria, or Critique Questions, for the Discussion Section (cont.)
21. Did the researchers reflect on the manner in which the study was designed and conducted regarding any limitations and/or delimitations (boundaries)?
22. Did the authors elaborate on the interpretation of the study’s findings beyond the interpretation that was begun in the results section?
23. Did the researchers address the significance of the study and its findings, particularly as they relate to earlier studies in the problem area investigated?

50. Criteria, or Critique Questions, for the Discussion Section (cont.)
24. Did the discussion section address possible intervening variables in the study that might explain why the results obtained were indeed forthcoming?
25. Were recommendations suggested to the reader regarding needed follow-up studies that might replicate, fully or partially, or at least augment, the study?

51. Criteria, or Critique Questions, for the Concluding Section
26. Does the list of references accurately reflect each of the sources cited in the research report, with a consistent bibliographic citation style used?
27. Does the research report contain any appendixes that augment in greater detail information provided earlier in the article?

52. Criteria, or Critique Questions, for the Concluding Section (cont.)
28. Is there any information in the form of author notes providing insight regarding funding support for the study, contact directives for communicating with the authors as a follow up, or collegial assistance in completing the study?
29. Are any footnotes provided that elaborate on one or more aspects of the study that would have been misplaced or distracting if they had been embedded in the main body of the report?

53. An Even “Closer Look” at the True Experimental/RCT Research Method
Third Example from the Massage Therapy Research Literature: Preyde (2000)
Full-Text Article (pp in Chapter 4 Handout)
Detailed Analysis of Article Provided Via 29 Critique Questions (pp )

54. Where Do We Go from Here? Other Options in the Quantitative Research Category
Difference-Oriented Research Strategy
True Experimental or RCT Method
Quasi-Experimental Method
Single-Case Experimental Method
Nonexperimental Comparative Groups Method
Association-Oriented Research Strategy
Correlational Method
Predictive Method
Descriptive-Oriented Research Strategy
Single-Case Quantitative Analysis Method
Survey method
Naturalistic/Structured Observational Method
Case Report Method

55. Where Do We Go from Here. (cont
Where Do We Go from Here? (cont.) Options in the Qualitative Research Category
Contextual/Interpretive-Oriented Research Strategy
Case Study Method
Phenomenological Method
Grounded Theory Method
Ethnographic Method
Example (viz., of Case Study): Anderson, R. (1999). A case study in integrative medicine: Alternative theories & the language of biomedicine. The Journal of Alternative & Complementary Medicine, 5(2),

56. Where Do We Go from Here. (cont
Where Do We Go from Here? (cont.) Options in the Integrative Research Category
Synthesis-Oriented Research Strategy
Traditional Narrative Review Method
Critical Systematic Review Method
Meta-Analytic Systematic Review Method
Best-Evidence Synthesis Method
Qualitative Systematic Review Methods
Qualitative Meta-Synthesis method
Qualitative Meta-Summary Method
Example (viz., of Meta-Analysis): Moyer, C. A., Rounds, J., & Hannum, J. W. (2004). A meta-analysis of massage therapy research. Psychological Bulletin, 130(1), 3-18.

57. Where Do We Go from Here. (cont
Where Do We Go from Here? (cont.) Examples of Most Critical Input Needed from Massage Therapists As Members of a Research Team
Determination of principal anatomical structure(s) & physiological function(s) impacted by the client’s presenting signs & symptoms
Formulation of a research-based & practice-sensitive treatment protocol appropriate to the client’s presenting condition and desired outcomes of the intervention
Ensuring that the treatment protocol is specific & operational enough to allow replication (full or partial) in future studies and/or practice applications
Translating or “transitioning” massage research findings into practice-relevant applications for the benefit of clients (i.e., translational research)

58. Follow-Ups to This Session
Contact at or (504)
Participation in Massage Research Education (MRE) LISTSERV via
Reflections on Needed Curricular Content and Instructional Delivery Methods To Advance MRE
Resources of the Massage Therapy Foundation:
Massage Research Database at
Student & Practitioner Case Report Contests at
Forthcoming (Summer 2008) International Journal of Therapeutic Massage & Bodywork: Research, Education, & Practice
Resources of the National Center for Complementary & Alternative Medicine (NCCAM) at

59. A Parting Thought
“Some people see things as they are and ask ‘Why?’; others dream things that never were and ask ‘Why not?’” (George Bernard Shaw).
Bringing the incredible healing power of touch to others most in need, both physically and emotionally, is a privilege, an honor, and indeed a sacred trust. In that regard, let us always endeavor to count ourselves among those who “dream things that never were and ask ‘Why not?’”
(Hymel, 2006, p. xiv)