1. Proposal Review Process Mock Review Webinar
Louis Everett, Scott Grissom & Don Millard
Don
Division of Undergraduate Education
National Science Foundation

2. Caution
Most of the information presented in this session represents the presenter’s opinion and is not an official NSF position

3. Webinar Agenda Introduction (5min)
Overview of the review process (5min)
Instructions on producing a panel review (5min)
Organization of participants into teams and designation of panel chairs (5min)
Local teams discuss strengths and weaknesses (20min)
Teams locally report/discuss results (10min)
Reporting to virtual group with ratings – note: facilitators will be asked to select an individual to report (10min)
BREAK (15min)
Individuals consider ways to improve the proposal (5min)
Local teams discuss suggestions for improvement (10min)
Teams locally report/discuss results (5min)
Report back to virtual group - note: facilitator picks an individual to report (5min)
PD commentary on responses (10min)
Think  – What have I learned today that I will use in preparing my next proposal? (5min)
Share your thoughts with local participants (5min)
Facilitator reports results back to virtual group (5min)
Wrap-up Q&A (10min)

4. Webinar Goals - Expected Outcomes
Help participants to:
Become more familiar with the proposal review process
Better understand the TUES-specific criteria
Better understand the use of intellectual merit/broader impact criteria in reviewing proposals
Develop more competitive proposals that effectively meet the expectations of the TUES program
Please note: A number of the following slides are provided for informational purposes – we will not be going through all of them

5. TUES Program

6. TUES vs. CCLI
Title changed to emphasize the special interest in projects that have the potential to transform undergraduate STEM education 
Review criteria was modified to emphasize the desire for projects that:
Propose materials, processes, or models that have the potential to
Enhance student learning
Be adapted easily by other sites
Involve a significant effort to facilitate adaptation at other sites
Institutionalize the approach at the investigator's college or university as appropriate (e.g., for the Type)
Have the potential to contribute to a cultural shift in undergraduate STEM education

7. TUES Program
Vision: Excellent STEM education for all undergraduate students
Reflects national concerns about producing:
Skilled STEM professionals (including K-12 teachers)
Citizens knowledgeable about STEM and how it relates to their lives
Seeks to build a community of faculty committed to improving undergraduate STEM education
Encourages projects with potential to advance and transform undergraduate STEM education

8. TUES Project Components
Creating Learning Materials and Strategies
Guided by research on teaching and learning
Incorporate and be inspired by advances within the discipline
Implementing New Instructional Strategies
Contribute to understanding on how existing strategies:
Can be widely adopted
Are transferred to diverse settings
Impact student learning in diverse settings
Developing Faculty Expertise
Enable faculty to acquire new knowledge and skills in order to revise their curricula and teaching practices
Involve a diverse group of faculty

9. TUES Project Components (cont)
Assessing and Evaluating Student Achievement:
Develop and disseminate valid and reliable tests of STEM knowledge
Collect, synthesize, and interpret information about student understanding, reasoning, practical skills, interests, attitudes or other valued outcomes
Conducting Research on Undergraduate STEM Education:
Explore how:
Effective teaching strategies and curricula enhance learning and attitudes
Widespread practices have diffused through the community
Faculty and programs implement changes in their curriculum

10. Instructional Materials and Methods Projects
Projects developing instructional materials and methods should:
Be based on how students learn
Consider transferability and dissemination throughout the project's lifetime
Involve efforts to facilitate adaptation at other sites in more advanced projects

11. Type 1 Projects Expect to award approximately 10%
Total budget: up to $200,000 for 2 to 3 years
$250,000 when 4-year and 2-year schools collaborate
Typically involve a single institution & one program component – but there are exceptions
Contribute to the understanding of undergraduate STEM education
Informative evaluation effort based on the project's specific expected outcomes
Institutionalized at the participating colleges and universities
Deadlines:
May 26, 2011 (A-M)
May 27, 2011 (N-W)

12. Type 2, 3, and CRP Projects Type 2 Projects Type 3 Projects
20 to 25 awards expected
Total budget: up to $600K for 2 to 4 years
Type 3 Projects
3 to 5 awards expected
Budget negotiable, but not to exceed $5M over 5 years
Tues Central Resource Projects
1 to 3 awards expected
Budget negotiable, depending on the scope and scale of the activity
Small focused workshop projects -- 1 to 2 years & up to $100K
Large scale projects -- 3 to 5 years & $300K to $3M
Deadline:
January 13, 2012

13. Review Process
Ning

14. NSF Peer Review Process
Reviewers are solicited by program directors
For example - A Typical TUES Type 1 Panel Review:
4 panels/program director (60-70 proposals/PD)
6-8 reviewers/panel
17 proposals/panel – not all read by every panelist
Approximately 130 engineering reviewers
Reviewers assign individual ratings and prepare written comments on Intellectual Merit & Broader Impacts prior to coming to the panel meeting
Panel meeting is held in DC area – over a 1.5 day period

15. Panel Review Meeting
Panel Chair establishes order of proposal review process
Proposals are discussed individually
A “scribe” is designated to capture all of the points brought up in discussion and produce a summary review – called the “Panel Summary”
All reviewers return on day 2 to approve all the Panel Summaries

16. Audience for Reviews NSF program directors Applicants
Informs recommendations relative to funding
Guides pre-award negotiations
Applicants
If proposal is funded:
Provides suggestions for improving project
If proposal is not funded:
Provides information to guide a revision of the proposal

17. NSF Review Criteria Intellectual merit Broader impacts
All proposals are evaluated using the NSB-approved review criterion
Intellectual merit
Broader impacts
The TUES Solicitation provides two sets of suggested questions to help define these criteria
Standard NSF set
TUES specific set

18. Caution Regarding Suggested Questions
Suggested questions are only a guide for considering intellectual merit and broader impacts
Suggested questions are NOT:
A complete list of “requirements”
Applicable to every proposal
An official checklist

19. NSF Suggested Questions for Intellectual Merit
Will the project:
Include activities important in advancing knowledge?
Involve qualified proposer(s)?
Contain creative and original concepts?
Have a well conceived and organized plan?
Include sufficient access to resources?

20. NSF Suggested Questions for Broader Impacts
Will the project:
Advance discovery - promote teaching & learning?
Broaden participation of underrepresented groups?
Enhance the infrastructure?
Include broad dissemination?
Benefit society?

21. TUES Suggested Questions for Intellectual Merit
Will the project:
Produce one or more of the following:
Exemplary materials, processes, or models that enhance student learning and can be adopted by other sites
Important findings related to student learning?
Build on existing knowledge about STEM education?
Have explicit and appropriate expected measurable outcomes integrated into an evaluation plan?
Include an evaluation effort that is likely to produce useful information?
Institutionalize the approach at the investigator's college or university (as appropriate for the Type)

22. TUES Suggested Questions for Broader Impacts
Will the project:
Involve a significant effort to facilitate adaptation at other sites?
Contribute to the understanding of STEM education?
Help build and diversify the STEM education community?
Have a broad impact on STEM education in an area of recognized need or opportunity?
Have the potential to contribute to a significant advancement and cultural shift in undergraduate STEM education?

23. Writing the Review

24. Review Material The Entire Proposal is Used to Inform Reviewers
Project Summary
Project Description
Biographical Sketches
Budget
Supplementary Documentation
During talk need to give the appropriate directive relative to the need to read or use Supplementary Documentation 24

25. Overview Rating and text should be consistent
A review should indicate an opinion on the merit of the project
The rating should indicate an overall evaluation of the proposal’s merit
Excellent, Very Good, Good, Fair, and Poor
Describe positive aspects
Not just list them -- Provide details
Identify concerns (or weaknesses)
Offer suggestions for improvement
Rating and text should be consistent

26. Characteristics of Informative Reviews and Panel Summaries
Uses appropriate style
Contains adequate details
Contains understandable, specific, and complete statements
Relates strengths and weakness to review criteria
Indicates why an item is a strength or weakness
Justifies the proposal rating in the written critique
A reader should be able to guess the rating from the written text

27. Specific and Complete Comments
Identify a strength or weaknesses
“The evaluation plan is a strength.”
Identify a strength or weaknesses and indicate why it is one
“The evaluation plan includes a competent, independent evaluator...”
“The background discussion is well referenced, shows a good understanding of the prior work, supports the proposed work...”

28. Specific and Complete Comments (cont.)
Identify a strength or weaknesses and indicate why it is one and why it is important
“The evaluation plan includes a competent, independent evaluator, … and it will guide the investigators as the project evolves and provide a measure of its effectiveness at the end.”
Identify a strength, indicate why it is one, why it is important, and how it could be improved
“The evaluation plan is a strength because it includes a competent, independent evaluator, … and this will guide the investigators as the project evolves and provide a measure of its effectiveness at the end. It could be improved by adding …”

29. Other Important Ideas Try to be constructive in your written comments
Provide suggestions to help applicants improve their proposals
Do not be overly critical in your ratings
Most fundable proposals have some weaknesses
Some are correctable through negotiations

30. Strengths & Weakness
Don

31. Team Activity Process
Local teams discuss the strengths and weaknesses identified in the individual reviews
Team “scribe” takes notes during discussion
Teams locally report/discuss results with facilitator’s guidance
Facilitators guide discussion and selection of an individual to report to the full virtual group
Participating organizations will be called upon by webinar moderator – PLEASE BE READY TO REPORT BACK
Program Directors will offer comments on reports

32. Reporting Redundancy is OK – indicative of common perception
Responses will be solicited as follows:
Intellectual Merit
Strengths
Weaknesses
Broader Impacts
Overall Perception
Is this a proposal worthy of funding?

33. ONE MINUTE Until Reporting Begins

34. Reporting Redundancy is OK – indicative of common perception
Responses will be solicited as follows:
Intellectual Merit
Strengths
Weaknesses
Broader Impacts
Overall Perception
Is this a proposal worthy of funding?

35. Questions?

36. BREAK (15 min)

37. ONE MINUTE Until Next Session

38. Suggestions for Improvement
Ning

39. Improvements Activity Process
Individuals consider ways that the proposal could be improved – create a list
Local teams discuss the suggestions for improvement
Team “scribe” takes notes during discussion
Teams locally report/discuss results with facilitator’s guidance
Facilitators guide discussion and selection of an individual to report to the full virtual group
Participating organizations will be called upon by webinar moderator – PLEASE BE READY TO REPORT BACK
Program Directors will offer comments on reports

40. Reporting Try to minimize redundant responses Identify approaches for:
Building on strengths
Overcoming weaknesses
Responses should include potential improvements to:
Idea
Project infrastructure
Project implementation plan
Evaluation plan
Dissemination plan
Proposal participation/involvement

41. ONE MINUTE Until Reporting Begins

42. Reporting - Reminder Try to minimize redundant responses
Identify approaches for:
Building on strengths
Overcoming weaknesses
Responses should include potential improvements to:
Idea
Project infrastructure
Project implementation plan
Evaluation plan
Dissemination plan
Proposal participation/involvement

43. Sample Proposal Review
Comments Regarding the Proposal’s Intellectual Merit, Broader Impacts and Weaknesses
Don

44. Sample Proposal Review Comments
The following comments reflect a combination of the proposal’s panel review, individual reviewers and inputs provided by NSF program officers

45. Intellectual Merit (1)
The proposal is the evolution of the successful studio pedagogy to a mobile studio pedagogy.
The proposal builds on a instructional methodology that is accepted in the STEM literature.
The proposal is student-focused and grounded in the STEM knowledge base.
A primary innovation is the development of a low cost virtual instrument board that, when coupled with a computer/laptop, provides students with a portable experimental platform

46. Intellectual Merit (2)
This experimental platform serves as a low cost replacement for the laboratory equipment found in the original studio classrooms and provides the students with an "anywhere, anytime" experimental platform. The PIs are well qualified to carry out the work.
The proposal was responsive to the cyclic model of knowledge creation contained in the TUES/CCLI request for proposals.
The panel felt the proposal would be strengthened by a more detailed evaluation plan with clearly stated, measurable outcomes.

47. Broader Impacts
The inclusion of an historically black college or university and a community college should magnify the impact of the proposed pedagogy.
The broader impacts of the proposed work include the potential to significantly impact a large number of under- served students by providing a low cost, portable experimental platform.
The partnership was made concrete through an instructor exchange program between the institutions.

48. Weaknesses (1)
While the evaluation team appears to have worked together with the PIs for several years, some basic information about the qualifications of those individuals should have been included in the proposal.
The proposal could be strengthened with more definition of the role of each partner.
It is not clear that it will positively impact student learning; just because a student has the means to perform an experiment 24/7, does not mean that they will.

49. Weaknesses (2)
The proposal could benefit from clarifying or identifying the role of laboratory experiences where actual industrial-quality laboratory equipment is utilized. Too much reliance on the proposed pedagogy might also leave the student with limited experience in the use of the real industrial grade and scale laboratory equipment.
Moving from the original studio to the mobile studio essentially means that the new work is now based on an untested model. The proposal could be strengthened by addressing this issue.

50. Weaknesses (3)
It has not been demonstrated that the mobile studio is as effective as the original studio with its significant faculty involvement and structure.
This proposal would have been much stronger if some preliminary data on efficacy was included.
The proposal mentions a pilot test that was done at Howard University but no details are provided.
The proposal fails to explicitly address how it will focus on components of the cyclic model for innovation in STEM education.

51. Practical Aspects of Review Process
Ning

52. Practical Aspects of Review Process
Reviewers have:
Many proposals
Ten or more from several areas
Limited time for your proposal
20 minutes for first read
Different experiences in review process
Veterans to novices
Different levels of knowledge in proposal area
Experts to outsiders
Discussions of proposals’ merits at panel meeting
Share expertise and experience

53. Practical Aspects of Review Process
Write down a list of suggestions (guidelines) that you would suggest that a colleague should follow - to deal with practical aspects of the Review Process
2 minutes

54. ONE MINUTE

55. Considering the Practical Aspects of the Review Process
The following comments reflect perspectives provided by NSF program officers

56. Proposal Writing Guidelines
Use good style (clarity, organization, etc.)
Be concise, but complete
Write simply but professionally
Avoid jargon and acronyms
Check grammar and spelling
Use sections, headings, short paragraphs & bullets (Avoid dense, compact text)
Reinforce your ideas
Summarize; Highlight (bolding, italics)
Give examples

57. Proposal Writing Guidelines
Use tables, figures – where it makes sense
Reinforce your ideas
Summarize; Highlight (bolding, italics)
Give examples

58. Proposal Writing Guidelines
Provide appropriate level of detail
Pay special attention to Project Summary
Summarize goals, rationale, methods, and evaluation and dissemination plans
Address intellectual merit and broader impacts
Explicitly and independently
Three paragraphs with headings:
“Summary”
“Intellectual Merit”
“Broader Impacts”

59. Proposal Writing Guidelines
Follow the solicitation and GPG
Adhere to page, font size, and margin limitations
Use allotted space but don’t pad the proposal
Follow suggested (or implied) organization
Use appendices sparingly (check solicitation to see if allowed)
Include letters showing commitments from others
“Support letters” are not allowed
Avoid form letters

60. Proposal Writing Guidelines
Prepare credible budget
Consistent with the scope of project
Clearly explain and justify each item
Address prior funding when appropriate
Emphasize results
Sell your ideas but don’t over promote
Proofread the proposal
“Tell a story” and turn a good idea into a competitive proposal

61. A Successful Proposal Good idea + need Right people + infrastructure
Assessment of outcomes that measure effect on student learning (with goals/objectives linked to evaluation)
Active dissemination plan
Efforts to broaden participation of underrepresented groups

62. Reflection
Don

63. Reflection Activity
Individuals consider: What have I learned today that I will be able to used in preparing my next proposal?
Create a list (5min)
Share your list with local participants (5min)
Facilitators report back to virtual group (5min)
Participating organizations will be called upon by webinar moderators – PLEASE BE READY TO REPORT BACK

64. Closing Comments

65. Toward Transformation…
Model good practices that increase/improve learning
Facilitate direct experience with the methods and processes of inquiry/experimentation
Empower the student
Enhance ability to produce and evaluate innovative results
Products
Solutions to problems
Metrics on outcomes
Establish a community that will help inform, disseminate and sustain engineering education efforts

66. A Successful TUES Proposal…
Good idea + need
Right people + infrastructure
Can be readily adopted at other sites
Assessment of outcomes that measure effect on student learning (with goals/outcomes linked to evaluation)
Active dissemination plan
Shows promise for institutionalization
Efforts to broaden participation of underrepresented groups

67. Questions?

68. Thanks for participating
Thanks for participating! Please respond to the survey located at:
Louis Everett –
Scott Grissom –
Don Millard –