Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.archivedOER Public Archive Home Page

Pilot-Online Review of R21 Applications Daniel F. McDonald, Ph.D. Chief, Musculoskeletal Oral & Skin Sciences and Renal & Urological Sciences IRGs Center for Scientific Review, NIH

I. Objectives To assess alternate review paradigm given increasing overall CSR review burden and ultimately limited human/financial resources To relieve review burden on standing skeletal biology study sections

II. Meeting Mechanics Choose application pool Administrative review ( SRA ) Recruit reviewers Make assignments Critique prep ( reviewers ) IAR submission Phase “Triaging” process Initiation of chat room Monitoring of discussions Establishment of final levels of enthusiasm Procurement of individual reviewer votes

Chatroom Setup Application ID Preliminary Critiques Discussion Final Scores

III. Post-meeting Activities Critique revision ( reviewers via IAR ) Score/Code entry/release Summary Statement generation/release Process assessment > CSR personnel (SRA, IT support, senior staff) > Reviewers > Program staff

Summary of Review Load 52 R21, 4 R03 and 2 R15 applications52 R21, 4 R03 and 2 R15 applications 194 assignments for 46 reviewers194 assignments for 46 reviewers 21 of the 58 applications went unscored21 of the 58 applications went unscored

Number of Reviewers Participating 137 assignments to 37 scored applications137 assignments to 37 scored applications Average of 3.3 reviewers assigned to each scored applicationAverage of 3.3 reviewers assigned to each scored application

Summary of Comment Source Average of 3.7 members commented on a given scored application in electronic panelsAverage of 3.7 members commented on a given scored application in electronic panelsvs. For a regular study section, average number of members participating in a discussion per application = 4.6For a regular study section, average number of members participating in a discussion per application = 4.6 [N = 75 study sections, with a mean of 19.7 members per study section; Unpublished observations made in DRG study sections in 1989]

Summary of Comment Source (cont’d) Average of:Average of: –5.9 comments per application from assigned reviewers –0.8 comments from unassigned reviewers [almost always the Chair] –6.8 total comments per application

Summary of Scoring Profile 7 of 37 scored applications were scored by all reviewers7 of 37 scored applications were scored by all reviewers The other 30 applications were scored by all but 1 or 2 reviewersThe other 30 applications were scored by all but 1 or 2 reviewers Conflicts counted as scores to differentiate from those that didn't score at allConflicts counted as scores to differentiate from those that didn't score at all

Data Analysis Suggests: Number of comments tended to increase with range of initial scores.Number of comments tended to increase with range of initial scores. Once the initial score spread from the assigned reviewers gets above some figure between 0.5 and 1.0, there is an increase in the number of comments that are made.Once the initial score spread from the assigned reviewers gets above some figure between 0.5 and 1.0, there is an increase in the number of comments that are made. The additional comments are all from the assigned reviewers, not the unassigned reviewers.The additional comments are all from the assigned reviewers, not the unassigned reviewers.

Data Analysis Suggests (cont’d): Final score not closely related to number of comments. Range of final scores from assigned reviewers was much narrower than their initial scores. Final score range from assigned reviewers narrower than from all reviewers. Initial range of scores from assigned reviewers broader than final range from all reviewers.

IV. Preliminary Post-meeting Reflections (SRA) ADVANTAGES Maximum flexibility in construction Recruiting tool Convenience for potential reviewers Substantial reviewer cost savings SHORTCOMINGS Unfamiliar approach to reviewers Added security vulnerabilities No real-time face-to-face interaction Restricted involvement of subgroup of panel members in any particular evaluation More time-consuming for all parties (SRA/Chair/panelists) Meeting size constraints