1. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Improving Undergraduate STEM Education Program (IUSE: EHR) Program Solicitation: NSF 14-588 IUSE Reviewer Orientation

2. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Outline  IUSE: EHR Overview  Panel Review Process Expectations  Ratings & Panel Summary Criteria  Panel Review Process  COI/Confidentiality & Implicit Bias

3. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation IUSE: EHR OVERVIEW

4. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation IUSE: EHR supports 2 NSF strategic goals: 1. Transform the Frontiers of Science and Engineering: Supporting fundamental, high-risk and potentially transformative research in STEM Supporting fundamental, high-risk and potentially transformative research in STEM Education of the next generation of the STEM workforce to continue this transformation Education of the next generation of the STEM workforce to continue this transformation

5. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation IUSE: EHR supports 2 NSF strategic goals: 2. Stimulate Innovation and Address Societal Needs through Research and Education: supports research and development on STEM education supports research and development on STEM education prepare a diverse, globally competent STEM workforce and a STEM-literate citizenry prepare a diverse, globally competent STEM workforce and a STEM-literate citizenry

6. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation IUSE: EHR Program Goals Improve STEM Learning & Learning Environments – Improve the knowledge base for defining, identifying, and innovating effective undergraduate STEM education teaching and learning Improve STEM Learning & Learning Environments – Improve the knowledge base for defining, identifying, and innovating effective undergraduate STEM education teaching and learning Broaden Participation & Institutional Capacity for STEM Learning – Increase the number and diversity of undergraduate students Broaden Participation & Institutional Capacity for STEM Learning – Increase the number and diversity of undergraduate students Build the Professional STEM Workforce for Tomorrow – Improve the preparation of undergraduate students Build the Professional STEM Workforce for Tomorrow – Improve the preparation of undergraduate students

7. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Educational Practice and Research Cycle Educational Practice Questions Ideas Educational Research Answers Insights Which help improve... That results in... Which lead to... Identifies and motivates... STEM Learning Environment

8. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation IUSE: EHR 2 tracks Engaged Student Learning Institutional and Community Transformation Research, Design, and Development studies that involve creation, exploration, and implementation of tools, resources, or models Projects that use innovative approaches to substantially increase the propagation of highly effective methods of STEM teaching and learning in institutions of higher education

9. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Panel Review Process Expectations

10. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Who reviews - What and How? In our panels we include reviewers that are In our panels we include reviewers that are – Experts in fields of STEM education, STEM content, methodology, cognitive science, and administration – Panelists read 10 – 12 proposals, but panels may include more than 12 proposals and thus you will not read every proposal assigned to a panel – IUSE is using a mix of virtual and face to face panels. Often the more complex Exploration proposals are being discussed face to face in panels here in the DC area.

11. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation The Proposer Receives: Reviews Panel Summary (if applicable) Context statement & Award/Declination letter $ NATIONAL SCIENCE FOUNDATION 4201 Wilson Boulevard Arlington, Virginia 22230 Dear Dr. Doe, The National Science Foundation hereby awards a grant of... NATIONAL SCIENCE FOUNDATION 4201 Wilson Boulevard Arlington, Virginia 22230 Dear Dr. Doe, I regret to inform you that the National Science Foundation is unable to support your proposal referenced above... & anonymous

12. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation NSF Merit Review Criteria Guiding Principles All NSF projects should be of the highest quality and have the potential to advance, if not transform, the frontiers of knowledge. All NSF projects should be of the highest quality and have the potential to advance, if not transform, the frontiers of knowledge. Meaningful assessment and evaluation of NSF-funded projects should be based on appropriate metrics, keeping in mind the likely correlation between the effect of broader impacts and the resources provided to implement projects. Meaningful assessment and evaluation of NSF-funded projects should be based on appropriate metrics, keeping in mind the likely correlation between the effect of broader impacts and the resources provided to implement projects.

13. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation NSF Merit Review Criteria NSF Merit Review Criteria Required Both criteria are to be given full consideration during the review and decision-making processes; each criterion is necessary but neither, by itself, is sufficient. Intellectual Merit: The Intellectual Merit criterion encompasses the potential to advance knowledge Intellectual Merit: The Intellectual Merit criterion encompasses the potential to advance knowledge Broader Impacts: The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes Broader Impacts: The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes http://www.nsf.gov/pubs/policydocs/pappguide/nsf14001/gpg_3.jsp#IIIA

14. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Merit Review Considerations The following elements should be considered in the review for both Intellectual Merit & Broader Impacts: What is the potential for the proposed activity to: What is the potential for the proposed activity to:  Advance knowledge and understanding within its own field or across different fields (Intellectual Merit)? and and  Benefit society or advance desired societal outcomes (Broader Impacts)? To what extent does the proposed activity suggest and explore creative, original or potentially transformative concepts? To what extent does the proposed activity suggest and explore creative, original or potentially transformative concepts?

15. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation (Merit Review Considerations cont.) The following elements should be considered in the review for both Intellectual Merit & Broader Impacts: Is the plan for carrying out the proposed activities well- reasoned, well-organized, and based on a sound rationale? Is the plan for carrying out the proposed activities well- reasoned, well-organized, and based on a sound rationale? Does the plan incorporate mechanisms to assess success? Does the plan incorporate mechanisms to assess success? How well qualified is the individual, team, or institution to conduct the proposed activities? How well qualified is the individual, team, or institution to conduct the proposed activities? Are there adequate resources available to the PI (either at the home institution or through collaborations) to carry out the proposed activities? Are there adequate resources available to the PI (either at the home institution or through collaborations) to carry out the proposed activities?

16. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Common Guidelines for Education Research & Development: NSF13-126 Offer guidance on building the evidence base in STEM learning Offer guidance on building the evidence base in STEM learning Are not a “requirement” for IUSE: EHR proposals, but may be helpful to proposers and reviewers Are not a “requirement” for IUSE: EHR proposals, but may be helpful to proposers and reviewers We encourage you to be familiar with them We encourage you to be familiar with them

17. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Should be both “knowledge -using” and “knowledge-producing.” Should be both “knowledge -using” and “knowledge-producing.” – Using: Is the project building on prior work? – Producing: What will “we” learn from the planned conduct of this project? *Perspective: It is OK for IUSE: EHR projects to serve the PI’s institution, but they should also serve all of us by providing useful knowledge for broad constituencies of educators Basic perspectives from the Common Guidelines for IUSE: EHR Projects

18. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Applied work (e.g. development) is fine as long as it adheres to the principles of the preceding slide. [NSF budgets are not large enough to support simple local improvement projects.] Applied work (e.g. development) is fine as long as it adheres to the principles of the preceding slide. [NSF budgets are not large enough to support simple local improvement projects.] While we call for innovation, replication is also acceptable and an important part of science. [How do we know that something is effective if has been tested at only one place?] While we call for innovation, replication is also acceptable and an important part of science. [How do we know that something is effective if has been tested at only one place?] (Basic Perspectives cont.)

19. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation IUSE: EHR accepts projects of different size. They should be judged relative to their funds request. IUSE: EHR accepts projects of different size. They should be judged relative to their funds request. Institutional transformation projects by their very nature seek major local impacts. These also need to provide insights about how to accomplish the transformation, and thus the proposal should deal explicitly with the theory of change that underlies its promise, and provide an applied research design. Institutional transformation projects by their very nature seek major local impacts. These also need to provide insights about how to accomplish the transformation, and thus the proposal should deal explicitly with the theory of change that underlies its promise, and provide an applied research design. (Basic Perspectives cont.)

20. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation As you explore the project budget request and compare this with the scope of the project, please note that: Budgets that seem out of line with the proposed project can be negotiated by the program officer if the project itself seems promising. Budgets that seem out of line with the proposed project can be negotiated by the program officer if the project itself seems promising. Hence, we request that you don’t assign a “killer rating” due to budget alone. [A review might provide a rating conditional on budget reforms.] Hence, we request that you don’t assign a “killer rating” due to budget alone. [A review might provide a rating conditional on budget reforms.] The Project Budget & Summary Rating

21. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Reviewer Ratings and Panel Summary Criteria

22. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Use the Entire Proposal to Inform Your Review: Project Summary Project Summary Project Description Project Description Biographical Sketches Biographical Sketches Budget Budget Letters of Commitment Letters of Commitment Project Data Form Project Data Form Current & Pending Support Current & Pending Support

23. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Individual Review Process: Rating the Proposal Assign only ONE rating: E, V, G, F, P Assign only ONE rating: E, V, G, F, P Do NOT assign split ratings, such as E/V or V/G Do NOT assign split ratings, such as E/V or V/G Your rating MUST be consistent with your Your rating MUST be consistent with your written review written review

24. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Within FastLane: In your own words, write aspects you find to be compelling and those that you believe will benefit from improvement; Do NOT cut and paste text from the proposal into your review In your own words, write aspects you find to be compelling and those that you believe will benefit from improvement; Do NOT cut and paste text from the proposal into your review Be sure that your written review is consistent with your rating Be sure that your written review is consistent with your rating It is OK to modify reviews during the panel meetings, including a change of rating It is OK to modify reviews during the panel meetings, including a change of rating Be sure any modifications to reviews are recorded in FastLane! - MUST be done BEFORE the panel is closed. Be sure any modifications to reviews are recorded in FastLane! - MUST be done BEFORE the panel is closed. Note that many MS Word symbols do not translate in FastLane - check your work after submitting Note that many MS Word symbols do not translate in FastLane - check your work after submitting Individual Review Process: Writing Your Review

25. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Characteristics of Informative Reviews Include a section for IM, BI and summary Include a section for IM, BI and summary Use appropriate style (positive tone) Use appropriate style (positive tone) Contain adequate details Contain adequate details Write understandable, specific, and complete statements Write understandable, specific, and complete statements Relate strengths and weaknesses, supported by detailed explanation, associated with review criteria Relate strengths and weaknesses, supported by detailed explanation, associated with review criteria If an item would have helped to strengthen the proposal, a clear statement should be included as to why that would be the case. If an item would have helped to strengthen the proposal, a clear statement should be included as to why that would be the case.

26. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Panel Review Process

27. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation During the panel: Each proposal will typically have four reviews Each proposal will typically have four reviews Each proposal will have a reviewer assigned as the Scribe/Lead Discussant (called the primary panelist) Each proposal will have a reviewer assigned as the Scribe/Lead Discussant (called the primary panelist) The Scribe/Lead Discussant will write the panel summary as well as write an individual review The Scribe/Lead Discussant will write the panel summary as well as write an individual review Panel members assigned to review the proposal are called secondary panelists. Panel members assigned to review the proposal are called secondary panelists. Panel members not assigned to review the proposal are simply identified as “panelists.” Panel members not assigned to review the proposal are simply identified as “panelists.”

28. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Proposal Discussion The Scribe/ primary panelist is the lead discussant who presents an overview of the project being proposed. The Scribe/ primary panelist is the lead discussant who presents an overview of the project being proposed. The Scribe then provides the “highlights” of her/his own review The Scribe then provides the “highlights” of her/his own review Each assigned “secondary” panelist will provide his/her individual “highlights” while the scribe takes notes Each assigned “secondary” panelist will provide his/her individual “highlights” while the scribe takes notes Additional discussion will include all panelists providing the scribe with sufficient information to be able to compose the Panel Summary Additional discussion will include all panelists providing the scribe with sufficient information to be able to compose the Panel Summary

29. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Panel Summaries For each proposal: Initially framed by primary panelist who serves as scribe Initially framed by primary panelist who serves as scribe Should reflect the panel discussion (not just restate individual reviews) Should reflect the panel discussion (not just restate individual reviews) Include short, clear comments to help PIs improve their projects Include short, clear comments to help PIs improve their projects Should be written in 3 rd -person, as the panel, and will be read and agreed to by all panelists Should be written in 3 rd -person, as the panel, and will be read and agreed to by all panelists

30. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Characteristics of Informative Panel Summaries Includes a section for IM, BI and summary Includes a section for IM, BI and summary Uses appropriate style (positive tone) Uses appropriate style (positive tone) Contains adequate details Contains adequate details Contains understandable, specific, and complete statements Contains understandable, specific, and complete statements Relates strengths and weaknesses, supported by detailed explanation, associated with review criteria Relates strengths and weaknesses, supported by detailed explanation, associated with review criteria If an item would have helped to strengthen the proposal, a clear statement should be included as to why that would be the case If an item would have helped to strengthen the proposal, a clear statement should be included as to why that would be the case

31. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation COI/Confidentiality & Implicit Bias

32. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Primary purpose is to remove or limit the influence (or appearance of influence) of ties to an applicant institution or investigator that could affect reviewer advice Primary purpose is to remove or limit the influence (or appearance of influence) of ties to an applicant institution or investigator that could affect reviewer advice Second purpose is to preserve the trust of the scientific community, Congress, and the general public in the integrity, effectiveness, and evenhandedness of NSF’s peer review process Second purpose is to preserve the trust of the scientific community, Congress, and the general public in the integrity, effectiveness, and evenhandedness of NSF’s peer review process Conflict of Interest (COI)

33. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Statement on Confidentiality NSF receives proposals in confidence and protects the confidentiality of their contents NSF receives proposals in confidence and protects the confidentiality of their contents Proposals should not be discussed before the review process or after the review process with anyone outside of the panel Proposals should not be discussed before the review process or after the review process with anyone outside of the panel You must not copy, quote from, or otherwise use or disclose (to anyone) material from any proposal you are asked to review You must not copy, quote from, or otherwise use or disclose (to anyone) material from any proposal you are asked to review The fact that you are reviewing IUSE proposals now should also be kept confidential The fact that you are reviewing IUSE proposals now should also be kept confidential

34. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Implicit Bias Implicit bias toward a group (“schemas”) Implicit bias toward a group (“schemas”) – Non-conscious hypotheses/stereotypes, often about competence Implicit institutional or geographic bias Implicit institutional or geographic bias Lack of critical mass leads to greater reliance on schemas Lack of critical mass leads to greater reliance on schemas – Fewer women and minorities in sciences Accumulation of disadvantage Accumulation of disadvantage – Accumulation of small biases in the same direction has large effects over time – Very small differences in treatment can have major consequences in salary, promotion, and prestige (Valian 1998) https://implicit.harvard.edu/implicit/

35. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Implicit Bias Schemas are: Widely shared culturally: Widely shared culturally: – All people, including members of underrepresented groups, hold schemas about these groups – People are often not aware of them Applied more under circumstances of: Applied more under circumstances of: – Lack of information – Stress from completing tasks – Time pressure – Lack of critical mass Fiske (2002). Current Directions in Psychological Science, 11, 123-128 Fiske (2002). Current Directions in Psychological Science, 11, 123-128.

36. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Questions?Comments?

37. Division of Undergraduate Education Directorate of Education and Human Resources National Science Foundation Thank You!