1. What is a C2 Review Protocol? Presented by: Harris Cooper University of Missouri-Columbia Larry V. Hedges University of Chicago

2. A Review Protocol is a document that: Sets out the reviewers’ intentions with regard to the topic and the methods to be used in carrying out a proposed review Is meant for inclusion in the Campbell Database of Systematic Reviews

3. How were the Guidelines for a C2 Review Protocol Established?

4. The C2 Protocol Guidelines were: Drafted by the Methods Working Group at the request of the Steering Committee using the Cochrane Collaboration Protocol Guidelines and other sources as guides Reviewed (via e-mail) by attendees of the C2 Inaugural Meeting, sent to approximately 80 people, 7 of whom provided feedback Reviewed by the Steering Committee Approved by the Steering Committee (December 22, 2000)

5. Why are C2 Review Protocols Necessary? Preparing a review is a complex process that comprises many judgments and decisions. The methods to be used should be established beforehand (to the extent possible) because: The studies to be included are usually identified after they have been completed The results of many of these studies may already be known to the reviewers

6. Why are C2 Review Protocols Necessary? Therefore, it is important to make the review process as well-defined, systematic, and unbiased as possible while maintaining a practical perspective. Requiring detailed protocols is one way to: Allow the editorial referees to provide guidance and advice Prevent problems from occurring during the review process Ensure that final products will meet the standards of the Campbell Collaboration

7. What Should a C2 Protocol Contain?

8. A protocol for a Campbell Review should consist of the following sections: 1. Cover Sheet 2. Background for the Review 3. Objectives for the Review

9. A protocol for a Campbell Review should consist of the following sections: 4. Methods Criteria for inclusion and exclusion of studies in the review Search strategy for identification of relevant studies Description of methods used in the component studies Criteria for determination of independent findings Details of study coding categories Statistical procedures and conventions Treatment of qualitative research

10. A protocol for a Campbell Review should consist of the following sections: 5. Timeframe 6. Plans for Updating the Review 7. Acknowledgements 8. Statement Concerning Conflict of Interest 9. References 10. Tables

11. What Should a C2 Protocol Cover Sheet Contain?

12. The cover sheet of the protocol should include: The title of the review The names of the reviewers Contact information for the lead reviewer Sources of support

13. What should a C2 Protocol Background Section Contain?

14. The background section of the protocol should present: An overview of the theoretical, conceptual, and/or practical issues surrounding the research problem A general description of prior reviews, the controversies these reviews have created or left unresolved, and which of these will be the focus of the new review effort

15. The background section: Sets out the context of an already formed body of knowledge Provides the rationale for the review Explains why the questions being asked are important Sets the stage for the empirical results that follow

16. The background section should contain: A conceptual discussion of the research problem A brief overview of the research question including its theoretical, practical, and methodological history The qualitative and historical debates surrounding the research question A discussion of previous reviews of the research topic

17. The background section should answer questions such as: From where does the problem, approach, and/or intervention in the research come? Do debates exist surrounding the meaning of the problem or utility of the intervention? Do theories predict how the major variables involved in the review will be related to one another? Do different theories or philosophies of treatment yield conflicting predictions?

18. What Should a C2 Protocol Objectives Section Contain? All Campbell Collaboration Reviews are undertaken to gather, summarize and integrate empirical research so as to help people understand the evidence.

19. Within this overarching framework, reviews can, for example, be meant to: Produce general statements about relationships and treatment effects through the synthesis of individual study results Find reasons for conflicting evidence Explain variations in practice

20. Within this overarching framework, reviews can, for example, be meant to: Answer questions, using variations in studies, that could not have been answered in the individual component studies Review the evidence on the subjective experience of an intervention Build connections between related areas of research

21. In setting out the objectives, reviewers should keep in mind that Campbell Reviews must: Help people make practical decisions about social behavioral interventions and public policy Address the choices (practical options) people face when deciding about whether or not to adopt a policy or practice Relate to outcomes that are meaningful to people making decisions about public policy

22. What should a C2 Methodology Section Contain? The methods section should describe operationally how the review will be conducted.

23. What should a C2 Methodology Section Contain? The methods sections of a C2 review will differ considerably from that of a primary research study.

24. What should a C2 Methodology Section Contain? Most methods section will need to address several separate issues: Criteria for inclusion and exclusion of studies in the review Search Strategy for identification of possibly relevant studies Description of methods used in primary research

25. What should a C2 Methodology Section Contain? Criteria for determination of independent findings Details of study coding strategies Statistical procedures and conventions that may be used How qualitative research will be treated in the review

26. How should a C2 Protocol Address Questions of Study Inclusion/Exclusion? The protocol should explain the criteria that will be applied to determine relevance of studies uncovered by the search. It is critical that these criteria be explicated in advance.

27. How should a C2 Protocol Address Questions of Study Inclusion/Exclusion? This section of the protocol should answer question such as: What characteristics of studies will be used to determine if a study was relevant to the topic of interest? What characteristics of studies will lead to exclusion?

28. How should a C2 Protocol Address Questions of Study Inclusion/Exclusion? Will decisions be based on: Report tile? Abstracts? Full Reports? Who will make the relevance decisions? How will reliability of relevance decisions be assessed?

29. How should a C2 Protocol Address Questions of Study Inclusion/Exclusion? Examples of studies that would be included and excluded should be given.

30. How should a C2 Protocol Address Questions of Study Inclusion/Exclusion? C2 reviews can include evidence from studies of implementation. This evidence can derive from a broad range of qualitative and quantitative evidence.

31. How Should a C2 Protocol Address Questions of Search Strategy? Reviewers should present details of their search strategy, such as: Reference databases used Hand searches of specific journals Personal contacts

32. How Should a C2 Protocol Address Questions of Search Strategy? It should also include a rationale for the search strategy, bearing in mind that the ultimate goal is to reduce the differences between the body of retrieved studies and those that go undiscovered.

33. How Should a C2 Protocol Address Questions of Search Strategy? Reviewers need to report the keywords, years searched, and search strategy used to guide the search of reference databases and bibliographies.

34. How Should a C2 Protocol Address Questions of Search Strategy? Reviewers also need to indicate mechanisms they will use to retrieve documents, especially unpublished ones.

35. How Should a C2 Protocol Address Questions of Search Strategy? As far as possible, evidence considered should not be restricted by nationality of investigators or by language unless there is a good justification to do so.

36. How Should a C2 Protocol Address Descriptions of Primary Research Methods The protocol should describe the methods most commonly used in the primary research covered in the review.

37. How Should a C2 Protocol Address Descriptions of Primary Research Methods This should focus more on designs actually realized than on theory, e.g. Participant sampling procedures Research designs Measurement methods

38. How Should a C2 Protocol Address Descriptions of Primary Research Methods Reviewers should identify a few studies that illustrate the methods used in primary research and present the details of these studies.

39. How Should a C2 Protocol Address Criteria for Determining Independence on Findings? Reviewers need to describe how they will handle studies that produce multiple findings of effectiveness based on the same data.

40. How Should a C2 Protocol Address Criteria for Determining Independence on Findings? This can happen when: Several types of outcome are measured on the same subjects The same outcome is measured at several points in time Several treatment groups are compared to the same control group Several different studies (publications) use data from the same people

41. How Should a C2 Protocol Address Criteria for Determining Independence on Findings? If more than one outcome is used from the same study (or from different studies using the same people) reviewers need to explain how they will assure that the different outcomes will be statistically independent.

42. How should a C2 Protocol Describe Coding Categories? Characteristics of studies that will be coded and examined for potential use as moderators of study outcomes should be described.

43. How should a C2 Protocol Describe Coding Categories? All retrieved characteristics should be mentioned, even if they are not eventually used.

44. How should a C2 Protocol Describe Coding Categories? If some outcomes or moderators are excluded, a rationale should be given for that exclusion.

45. How should a C2 Protocol Describe Coding Categories? Information about quality assurance of coding (e.g., coding reliability assessment and monitoring) should be included in this section.

46. How Should a C2 Protocol Describe Statistical Procedures? This section should describe the procedures and conventions used to carry out the quantitative analysis of results.

47. How Should a C2 Protocol Describe Statistical Procedures? It should explain: The effect size measure used Any adjustments that are made to effect sizes to reduce bias The techniques that will be used to combine evidence How missing data will be handled

48. How Should a C2 Protocol Describe Statistical Procedures? How statistics describing the overall literature will be presented What techniques will be used to assess variability of results What techniques will be used to explain variation in results What sensitivity analyses will be carried out

49. How Should a C2 Protocol Describe Statistical Procedures? A rational for each of the choices above should be given.

50. How Should a C2 Protocol Describe Statistical Procedures? If no quantitative synthesis will be carried out, rationale should be given for the alternative technique.

51. How should a C2 Protocol Address Treatment of Qualitative Research?

52. Qualitative Studies Can Be Part of C2 Reviews Qualitative studies can assist in: Defining interventions more precisely The choice of relevant outcome measure The development of valid research questions Interpretation heterogeneous results

53. Qualitative Studies Can Be Part of C2 Reviews When a review contains relevant qualitative research, reviewers should operationally describe for qualitative studies: Criteria for inclusion and exclusion of qualitative studies Methods used in these primary research studies Criteria for determining independent findings Characteristics of included studies in the same detail as they do for quantitative research.

54. What Should a C2 Protocol Timeframe Contain?

55. Examples of some benchmarks to be used in setting targets are the anticipated dates for completion of: Searches for published and unpublished studies Pilot testing of inclusion criteria Relevance assessments Pilot testing of study codes and data collection Extraction of data from research reports Statistical Analysis Preparation of report

56. What should a C2 Protocol Timeframe Contain? Reviewers together with the editors of their Collaborative Review Group, must determine and appropriate timeframe for a specific review. Targets may vary widely from review to review depending on its scope and complexity, as well as the resources available.

57. What Should a C2 Protocol Plan for Updating the Review Contain?

58. Updating Plans should include: Specifications for how the review, once completed, will be updated Information on who will be responsible for updates The frequency with which updates can be expected

59. What Should a C2 Protocol Acknowledgements Section Contain? Acknowledgment should be made of all individuals contributing to the preparation of the protocol who were not listed on the cover sheet.

60. What Should a C2 Protocol Statement Concerning Conflict of Interest Contain? It is a matter of Campbell Collaboration policy that direct funding from a single source with a vested interest in the results of the review is not acceptable.

61. Reviewers should report any conflict of interest capable of influencing their judgments including: Personal Political Academic Financial

62. Campbell Reviews should be free of any real or perceived bias introduced by: The receipt of any benefit in cash or kind Any hospitality Any subsidy derived from any source that may have or be perceived to have an interest in the outcome of the review Financial conflicts of interest cause the most concern. They can and should be avoided, but must be reported if there are any.

63. Campbell Reviews should be free of any real or perceived bias It is impossible to abolish conflict of interest, since the only person who does not have some vested interest in a subject is somebody who knows nothing about it.

64. Campbell Reviews should be free of any real or perceived bias Disclosing a conflict of interest does not necessarily reduce the worth of a review and it does not imply dishonesty.

65. Campbell Reviews should be free of any real or perceived bias Reviewers should include statements in their protocol about potential conflicts even when they are confident that their judgments will not be influenced.

66. Campbell Reviews should be free of any real or perceived bias Editors may decide that disclosure is not warranted or they may decide that readers should know about such a conflict of interest so that they can make up their own minds about how important it is. Decisions about whether or not to publish such information should be made jointly by reviewers and editors.

67. What Should a C2 Protocol References Section Contain? The protocol should include complete references of all cited works. References and other stylistic considerations should follow the guidelines of the American Psychological Association: American Psychological Association. (1994). Publication Manual of the American Psychological Association (4 th Ed.). Washington, DC: Author

68. What Should a C2 Protocol Tables and Figures Contain? Any tables and figures included in the protocol should appear after the text.

69. How is a C2 Protocol Registered? Once a proposed protocol has been completed it should be sent to the appropriate Collaborative Review Group editors.

70. How is a C2 Protocol Registered? When the editors are satisfied that the protocol meets the standards of the Campbell Collaboration they will include it in the Review Group’s module for publication in the Campbell Database of Systematic Reviews. Publishing protocols may encourage interested parties to contact the reviewers and may discourage others from undertaking a review on the same topic

71. Editors and reviewers should not include a protocol in a module unless there is a firm commitment: To complete the review within a reasonable timeframe To keep the review up-to-date once it is completed

72. What Happens if the Approved C2 Protocol Undergoes Changes? While every effort should be made to adhere to a predetermined protocol, it is recognized that this is not always possible or appropriate. Changes in the protocol should not be made on the basis of how they affect the results of the review. As a rule, when possible analyses should be performed to show the effect of the change on the results of the review.

73. How should a C2 Protocol be cited? When the protocol is converted into a full review, the fact that this review was preceded by a published protocol should be noted.

74. How should a C2 Protocol be cited? It is Campbell Collaboration policy that protocols that have not been converted into full reviews within: TWO YEARS will be withdrawn from the Campbell Database of Systematic Reviews.

75. What Other Sources of Assistance Can Help Develop a C2 Protocol? In addition to the guidelines presented above, prospective reviewers will find sound advice for proposing and conducting Campbell Reviews in the following works: Clark M. & Oxman A. D. (Eds.).(2000). Cochrane reviewers handbook: Version 4.1. In: Review Manager (RevMan) [Computer program]. Version 4.1 Oxford, England: The Cochrane Center. (Available on-line at: http://www.cochrane.org/cochrane/hbook.htm ). http://www.cochrane.org/cochrane/hbook.htm

76. What Other Sources of Assistance Can Help Develop a C2 Protocol? In addition to the guidelines presented above, prospective reviewers will find sound advice for proposing and conducting Campbell Reviews in the following works: Cooper, H. & Hedges, L. V. (Eds.). (1994). The handbook of research synthesis. New York: Russell Sage Foundation.

77. What Other Sources of Assistance Can Help Develop a C2 Protocol? In addition to the guidelines presented above, prospective reviewers will find sound advice for proposing and conducting Campbell Reviews in the following works: National Health Service Centre for Reviews and Dissemination. (2000). Undertaking systematic reviews of research on effectiveness. York, England: University of York. (Available at: http://www.york.ac.uk/inst/crd/report4.htm ). http://www.york.ac.uk/inst/crd/report4.htm

78. What Other Sources of Assistance Can Help Develop a C2 Protocol? Reviewers may contact the Campbell Secretariat for further guidance in preparing Campbell Review protocols.