1. Scott Yates and Mike Strauss Office of Scientific Quality Review Surviving the Path to Peer Review Success

2. “1998 Farm Bill”*  ARS research peer- reviewed every 5 years  Most review panelists external to ARS  Satisfactory review before beginning research Why OSQR Review? *Technically the Agricultural Research, Extension, and Education Reform Act of 1998 and not considered a “Farm Bill.”

3. Who Oversees OSQR? Scott Yates, ARS Scientific Quality Review Officer (SQRO) Approves chairs and panelists Certifies project plans Mike Strauss, OSQR Coordinator Schedules panels, trains chairs and panelists Manages the review Scott and Mike: Attend panel meetings Read project plans Read reviews Read and evaluate responses

4. National Action Plan OSQR Review Retrospective evaluation Stakeholder input Input Implement Plan Assess Input Objectives set (PDRAM) Project Plan prepared. Research initiated Annual progress reviews You are here COI Lists

5. “Life after the PDRAM…” FIRST: Review OSQR Handbook and Area/RL expectations -Plan Drafted lead scientist and project team -Review by other colleagues -Review by RL -Revision Revised plan to Area Office for approval (some require proof of outsider review) If needed, plan revised Approved Plan sent by Area to Office of National Programs Validation by National Program Leader Due to OSQR Validated plan returned to Area Revision if Needed (through Area)

6. Some Advice Set a time line When due to the Area Office What does Area require? Some want proof of review outside your group. If not you should still send the plan outside your group for review. Schedule time for: Meet and talk about overall plan BEFORE writing! Members to coordinate plans Writing Compiling a cohesive document REVIEW! Revision after review

7. What Do you Talk About? A suggestion… First Meeting (1 hour, no notes, now writing…just talk) (30 min) What is the goal for each piece of the plan? ONE short (6 words) sentence for each Objective (30 min) What is the goal/intent/focus of the whole PLAN? ONE short (6 words or less) sentence Extra Credit: What is the ONE WORD that characterizes all? Second Meeting 1-3 days later (Group Notes, work ONLY from memory!): -15 min. What is the overall focus of the plan? -15 min. What do you recall are the key elements of this project? - 10 min. Who/what will benefit? -30 min. What are the steps you will take to achieve that? With this you should have base material to write the Summary and Need for Research…and to organize the plan into a coherent document.. Extra Credit: Now do this for each Objective!

8. Who are the Reviewers?

9. How is a Panel Selected? Suggestions/nominations from ONP, others Potential chairs screened for conflicts Candidates interviewed, SQRO approves Chairs Coordinator develop proposed panelists SQRO reviews and approves panel

10. Reviewers NEED to know… What is the problem? Why is it important? Where are you going with it? How are you going to get there? Don’t make them hunt for this! Background – Problem – Solution

11. Your plan should tell a story… A “big” story of the whole plan and “smaller” stories through the objectives and sub- objectives. “I read numerous OSQR plans; the real gems were the ones that told a story, a story I found interesting despite being outside my area of expertise. I enjoyed reading these plans because I was actually learning something new.”

12. Aggregated Plans Some plans have “independent” pieces. Be aware of… (there still can be a “big” story that ties them together) - Clarity…why are all these pieces here? What links them? State, however, that they are independent - Consistency…in level of detail Someone needs to oversee the final product. - Content (flow)…an “easy read?” The general format is not rigid. - “Consensus” All parts should say the same thing! How do you do this? TALK FIRST…

13. Review Products Action Class Score Consensus review comments

14. Project Review Criteria Adequacy of Approach and Procedures Probability of Successfully accomplishing the Project’s Objectives Merit and Significance

15. Title and Investigators..………….page 1 Signature Page……………...........page 2 Table of Contents……….………….page 3 Project summary (250 words)...page 4 Objectives...…………..................page 5 Need for research (1-2 p) Scientific Background (5-7 p) Approach & Procedures (6-15 p) Prior Accomplishments (2 p) Literature Cited Milestone Table (1-3 p) Past Accomplishments of Project Team Members Issues of Concern statements Appendices (letters plus other material) 15 - 30 pages + 4 pages for tables/figures Document Outline

16. Project Plan Components Write this in active voice…And plain English! What are you doing? Why is it important? Earlier work (1-2 sentences)? What will you do next? So what!? Make it compelling! Project Summary – 250 words

17. Project Plan Components Express need scientifically AND in the context of NP Action Plan. Be concise in statement of research purpose. Discuss potential benefits and anticipated products. Briefly note the overall approach (e.g., …using microarray technologies we will elucidate…” DON’T repeat, the overview! NEED FOR RESEARCH: 1-2 pages Where are you going?

18. Project Plan Components Why are all these pieces here? How do they relate? Include a figure. - Objectives and sub-objectives - Personnel - Outcomes - Related projects Objectives: 1-2 pages

19. Project Plan Components Gap Analysis Demonstrate understanding and gaps. Not an exhaustive. Rationale for the work 1/3 of project plan length Note similar projects within and outside ARS and how your past work prepares for or leads to this (provide details in the Prior Accomplishments section…but say enough to convince reviewers you know the area). Cite preliminary data from your projects, if available Scientific Background: 5-7 pages

20. Project Plan Components Prior Accomplishments: 1-2 pages Prior project terminated within two years Major objectives and accomplishments Prior project investigators Impact of prior work (science, technology, users) Pertinent publications A table or chart of past data can be very helpful.

21. Project Plan Components Approaches & Procedures: 6-12 pages Approaches & Procedures: 6-12 pages How are you going to get there? Experimental design Approaches and methods any why they are appropriate. Advantages and limitations (important if “risky”). Who will do what, how, when Collaborations Letters in Appendix need to confirm what you say! For SCAs, a copy of the agreement is sufficient. Management, evaluation, and contingencies. What is your path to success? How will you monitor it?

22. MILESTONES AND OUTCOMES Project Plan Components Dynamic over the project lifecycle

23. Project Title Project No. National Program (Number: Name) Objective NP Action Plan Component NP Action Plan Problem Statement Subobjective Goal/Hypothesis SY TeamMonthsMilestone Anticipated ProductProgress/Changes 12 24 36 48 60 Goal/Hypothesis SY TeamMonthsMilestoneAnticipated Product Progress/Changes 12 24 36 48 60 See the OSQR Handbook for this new format for Milestones Table This column for management after review.

24. Readability and narrative flowReadability and narrative flow Connection between parts (a diagram)Connection between parts (a diagram) Appropriate roles for allAppropriate roles for all Appropriate expertise on team or from collaboratorsAppropriate expertise on team or from collaborators Grammar/spelling/proofreadingGrammar/spelling/proofreading Appropriate detail in ApproachAppropriate detail in Approach Clear, proper, milestonesClear, proper, milestones Real contingenciesReal contingencies Does the plan instill confidence in this team’s abilities? Project Plan Checklist

25. Lack of connection--How/why do the parts of your plan relate? Or if part does not, why is it there? Uneven presentation—Edit for consistency Context of plan—How does this fit with other similar work within and outside ARS? Proofreading—You may not fail for this but it can show a lack of attention and undermine confidence. Statistically sound—Are replicates sufficient? How will you analyze…“We’ve always done it this way” is not sufficient. Some Frequent Criticisms

26. How will it get done?—Who does what? What other resources are there? (postdocs, technicians, students…include in human and physical resources) Vagueness that prevents real analysis—If the information is confidential say why you can’t tell them but say enough to allow some level of analysis. Risk without justification--Risk can be good but ONLY if it’s apparent you are aware of the challenge and have justified it. Data accumulation without analysis—It’s not enough to gather data, what will you do with it? Vacancies—Guidance on Web DON’T say you are PLANNING to do something you have already done and published! …More

27. To keep in mind… The reviewers need to see the logical “thread” through your work…the story. Don’t make readers “search” for what you are doing! Be clear, accurate, and correct. Don’t assume reviewers know you and your work…(a poor plan may not be saved even if they do!)

28. Some hints to success… Proofread Your Plan Ask a nontechnical person to read your plan Ask someone who hasn’t seen it to read and proofread your plan Ask a highly critical colleague to read it thoroughly. Are collaborations documented appropriately? Treat this the same care you would a competitive proposal. The reviewers will!

29. Hypotheses Don’t avoid them if appropriate but don’t force them if they are not. Must be falsifiable and testable. Not restatements of objectives. A GOAL may be better for some work like breeding or germplasm characterization…but explain that! EITHER hypothesis or a clear goal is essential.

30. What Happens After Review? No, Minor or Moderate Revision Lead Scientist responds to comments. Scientific Quality Review Officer certifies compliance with recommendations. Major Revision or Not Feasible Lead Scientist revises and responds to comments. Panel performs a second review assessing response to their comments and assigns a new Action Class Score. If still Major or Not Feasible, project is returned for administrative action. No further review. Projects are reviewed no more than two times (There are no page limits for revised plans)

31. The Impact of Vertically Striped Voles (VSV) on Wheat, Rye, and Egg Production R. U. Kidding1321-38000-123-00D1/5/2006 Frontiers of Vole Biology and Relativity Theory What Happens After Review?

32. Can I disagree with the panel? This is a dialogue If you really disagree…put it away for a few days! Then… Honestly consider panel opinions. Be polite but if you disagree say why DON’T skip changes to plan DON’T insult or impugn panelists DO provide justification for your alternative view Panels are NOT perfect…they are colleagues

33. How not to disagree Q: The panel does not see any [expertise] in this plan. A: “I disagree.” [no explanation] Q: Can you provide some preliminary data to support this idea. A: “Yes, we have preliminary data but can’t/won’t show it to you.” Q: The panel suggests you try this approach. A: But that’s just too difficult. A: We’re not allowed to alter this project in any way. [not true!] A: We’ve done it our way for [x] years and see no reason to change. A: The panel 5 years ago approved this so you can’t change it. Q: This is not a hypothesis. Fix it or change to a goal statement. A: I looked at Tom’s plan and Bill’s and their panel didn’t make them do this so I don’t think I should have to do it. Q: Did you do a power analysis? A: No we did not, but we’ve always done it this way before. A. Yes. It said we needed more so we ignored it.

34. Good to know… Reviewers may (on rare occasion) comment on Objectives…we discourage this, if they do, we ask that they not consider it in their scoring. A final copy of your responses is sent to the panel (for their information) after it is certified. For plans scoring Moderate or higher, OSQR reviews the responses and may return them for additional work before certification if needed. The Officer can decline certification if, after several attempts, it is judged that the researchers have not or cannot adequately address reviewer comments (i.e., your plan does not “pass” until it is certified). For plans scoring Major Revision or below, OSQR only briefly checks to see if the responses complete…review and scoring for these plans is by the panel.

35. Last Words Proofread Seek Review then proofread and seek more review And lastly Proofread and Seek Review However…

36. Your Plan Should Tell a Story `It is the story of your research…want to know how? OSQR has worked with Randy Olson, a marine scientist turned filmmaker and will send you a copy of his recent book “Houston we have a Narrative”

37. Questions?