1. The Proposal Review Process Matt Germonprez Mutual of Omaha Associate Professor ISQA College of IS&T

2. Logistics Reviewers are asked to take a look at a large number of proposals – One page abstracts The reviewers rate the proposals they would like to review The spreadsheet is returned to the program director Reviewers then receive approximately 10 proposals to review in about 4 weeks. The review panel is 1-2 days with the addition of ad hoc reviewers – Trend toward virtual review sessions

3. The Review Each proposal is a 15 page narrative Data management plan Prior funding Important contributions Many pages of budget Letters of commitment Each takes about 8 hours to review and write-up – That said, review times vary Proposals are rated Excellent to Poor

4. The Finer Details of the Review Intellectual Merit Broader Impacts

5. Intellectual Merit Should this be done? – How important is the proposed activity to advancing knowledge and understanding within its own field or across fields? – To what extent does the proposal suggest and explore creative, original, and potentially transformative concepts? Can this be done? – How well qualified is the proposer (individual or team) to conduct the project? – How well conceived and organized is the proposed activity? – Is there sufficient access to resources (equipment, facilities, etc.)?

6. Broader Impacts Would this be good for society? – How well does the activity advance discovery and understanding while promoting teaching, training, and learning? – How well does the proposed activity broaden the participation of underrepresented groups gender, ethnicity, disability, geographic, etc.? – To what extent will it enhance the infrastructure for research and education, such as facilities, instrumentation, networks, and partnerships? – Will the results be disseminated broadly to enhance scientific and technological understanding? – What may be the benefits of the proposed activity to society? – Post doc mentoring plan

7. Merit and Impacts Together The following elements should be considered in the review for both criteria: 1.What is the potential for the proposed activity to: – Advance knowledge and understanding within its own field or across different fields (Intellectual Merit)? – Benefit society or advance desired societal outcomes (Broader Impacts)? 2. To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts? 3. Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success? 4. How well qualified is the individual, team, or organization to conduct the proposed activities? 5. Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities?

8. Inside the Review Roundtable discussion with about 20 people There are 3-4 people assigned to the proposal (having reviewed and posted their reviews prior) There is one scribe who did not review the proposal but takes notes Conflicts of interest leave the room but everyone else can stay and can comment

9. The Review One person begins the review with a brief summary of the proposal. They give their reaction to the merit and impacts This discussion lasts for 10-15 minutes. Sometimes longer, sometimes shorter Fundamentally, each reviewer is basing the discussion the intellectual merit and broader impacts of the proposal

10. Rating Following discussion, the panels are rated as a group into – Highly competitive – Competitive – Not competitive Not too many end up in highly competitive. Most in competitive. Many are on the border but they need to end up in one category

11. Decision to Fund Not made by the review team Review team never knows budget or target number Recommendations are provided to the program officer Reviews and scribe report are uploaded Ratings are finalized Reviewers are done

12. Honest tips: Merit Have a strong and simple narrative. This sounds obvious but many of the reviewers are NOT from your field of study. Be explicit on your questions and make them questions that reviewers can relate to: Why do for-profit organizations participate in open source? Vs. What is the changing nature of design in open source engagements? Be about something tractable: Open Source. Citizen Science. Social Inclusion. Public Transit.

13. Honest Tips: Method Be clear in your methodology across years – Year 1: Interviews – Year 2: Ethnography – Year 3: Data Analysis To be honest, there is not a lot of hang up on this… I’ve never seen a proposal with strong merit and impact be sunk on method. Many of the methods are very engaged with their respective communities – Citizen Science? Citizen Science Alliance – Disaster Recovery? American Red Cross – Open Source? Linux Foundation

14. Honest Tips: Support Use letters of commitment to signal external support – They are very useful and very important – A strong letter of commitment can go a long way Research team can make a difference too – Co-PIs – Consultants

15. Honest Tips: Broader Impacts Our university is 50% women therefore we have the potential to have a broader impact on women in the field of IT These statements are very poor and very common Broader impacts is a place where many proposals fail and is, interestingly, an area of increasing importance in the reviews process Be explicit on the courses you will teach. The students you will connect with. The initiatives you will work with Take a long look around you and think how you can connect broadly at your college, university, community, society

16. Honest Tips: Broader Impacts I believe that broader impacts is the one area to focus on in an effort to improve granting success. Without a strong broader impacts, the proposal can have a ring of being self-serving.

17. Final Observations Reviewers can be hard to sway University affiliation does not matter – Proposal are not rejected based on biases Mixed rankings do not matter Review if you can, it is invaluable END