1. Panel Briefing CAREER Panel

2. CISE Organization and Core Research Programs CISE Cross-Cutting Programs Cross-Foundation Programs 30% 70% CISE Core Programs

3. CAREER Program The Faculty Early Career Development (CAREER) Program National Science Foundation's most prestigious awards in support of junior faculty who exemplify the role of teacher-scholars through outstanding research excellent education and the integration of education and research within the context of the mission of their organizations.

4. CAREER Proposal All CAREER proposals must include –A Summary explicitly addressing intellectual merits and broader impacts –An integrated research and education plan at their core. –A description of the proposed research project covering Background and significance Specific objectives Preliminary results where appropriate Methods and procedures to be used Anticipated results –A description of the proposed educational activities, including plans to evaluate their impact on students and other participants; –A description of how the research and educational activities are integrated with one another; and –Results of prior NSF support, if applicable.

5. Supplementary Information Letter from the PIs department head –Must support the proposed integrated research and education plan –Evaluated as part of the assessment of the overall merits of the proposal Data management plan Postdoctoral mentoring plan (if requesting support for postdocs)

6. Panel Objectives Provide informative feedback to the PIs –Even the best proposals can benefit from advice –The weaker the proposals are – the more crucial the feedback –Informative and constructive feedback is especially important in the case of CAREER proposals Provide advice to NSF Program Directors for making award recommendations Provide basis for decision making by the NSF Division Director in approving (or not) funding or declination recommendations

7. Conflicts of Interest Purpose remove or limit the influence (or appearance of influence) of ties to an institution or investigator preserve the trust of the scientific community, Congress, and the general public Institutional (statutory) –Current, previous (12 months) or possible future employment –Spousal or off-spring employment or enrollment –Received payment for participation in advisory committees, honorarium, etc. (12 months) Personal (regulatory) –Thesis advisor or student relationship –Coauthor, research collaborator within past 48 months –Family member or close friend of the PI Panelists must not be present in the room when proposals are discussed with which they have a COI.

8. Confidentiality Yo ur participation is confidential! –OK to say you participated in an NSF panel Granularity: IIS Division, year 2009 –Not OK to say which panel, which day! Panel recommendations are confidential! –Do not discuss recommendations and panel proceedings, panel membership, proposals, … Proposals contain sensitive information –Proposals contain sensitive information; not in public domain –Do not copy, distribute or quote from them –Leave them and any notes in the room after the panel Panel makes recommendations, not decisions

9. Proposal Evaluation NSF Evaluation Criteria: –Intellectual Merit –Broader Impacts CAREER proposals: –Integration of Research and Education –Letter from the department chair (required) Data Management Plan Postdoctoral Mentoring Plan (if applicable) Results from Prior NSF Support (if applicable) Human Subjects Considerations (if applicable) Facilities and Resources Budget

10. Standard NSF Evaluation Criteria: Intellectual Merit Importance of proposed activity –To advance knowledge and understanding –Within the field and across fields How well conceived and organized is the proposed activity? –Soundness and feasibility of approach, evaluation, research plan Significance of expected contributions Creative, original, potentially transformative research Qualifications of the investigator Data Management Plan Facilities and Resources –Equipment, facilities, etc –Requested Support (budget)

11. Advance discovery and understanding while promoting teaching, training, and learning Broaden the participation of underrepresented groups –Computer Science education, computer systems workforce –Gender, ethnicity, disability, geographic, age, etc. Enhance the infrastructure for research and education? Disseminate results (data, software, …) to enable further research, education, technology transfer Societal benefits Standard NSF Evaluation Criteria: Broader Impacts

12. Integration of Research and Education All CAREER proposals must have an integrated research and education plan at their core. Can be at any level – K-12, undergraduate, graduate, professional practice, community at large Examples of activities –Development of innovative curricula, pedagogy –Outreach (community, profession) –Mentoring –Broadening participation –Enquiry-based learning –Integration of a research component into undergraduate courses Must advance the PIs own disciplinary and educational interests and goals, as well as the needs and context of his or her organization

13. CAREER: Integration of Research & Education There is a broad range of possible activities –Innovative curriculum and pedagogy –Outreach and mentoring –Students or the general public –At any level: K-12, undergraduate, graduate –Designing new materials and practices Activities must be integrated

14. Writing Good Reviews Imagine yourself in the role of the PI Write the kind of review that you would like to receive on your own proposals Reviews should be –Informative, substantive –Non-inflammatory –Anonymous –Substantiated –Unbiased –Constructive Avoiding bias –Beware of sources of implicit bias –Evaluate the proposal as written –Focus on strengths and weaknesses under each review criterion –Weigh the strengths and weaknesses to arrive at an overall rating

15. Writing Good Reviews Fill out all sections Rating (use single rating if possible; if you must use a double rating, say why: e.g., I rate the proposal VG on intellectual merits but E on broader impacts) Proposal summary Intellectual Merits Strengths & Weaknesses Educational Activities, including Integration of Research and Education Strengths & Weaknesses Broader Impacts Strengths & Weaknesses Summary statement Justify the rating (E, VG, G, F, P) in terms of your overall assessment of intellectual merits, broader impacts AND integration of research and education activities

16. Writing Good Reviews You are on the panel because you expertise relevant to the review of proposals being reviewed –Do not say: I am not qualified to review this proposal... –Such comments prompt PIs to question the entire review: Reviewers qualifications to conduct a proposal review Assignment of proposals to panels Assignment of reviewers to proposals Avoid unsubstantiated statements -Do not say There is nothing novel about this work -Instead, you might say The novelty of this work relative to the work of XYZ et al. (see ) is unclear Avoid inflammatory language

17. Implicit bias toward a group (schemas): Non-conscious hypotheses/stereotypes, often about competence Lack of critical mass leads to greater reliance on schemas Few women and minorities in sciences Accumulation of disadvantage Small bias in same direction has large effect (in salary, promotion, and prestige) over time [Valian1998] Minimizing Bias in Evaluation

18. Panelist grades: E, V, G, F, P Avoid being overly harsh (I never give an E) or overly generous. Be discriminative & use the entire spectrum P.. E Panel recommendations: Highly Competitive (HC): Solid proposal, deserves funding. Competitive (C): Good proposal, but some portions unconvincing. Low Competitive (LC): PI is encouraged to revise and resubmit Not Competitive (NC): PI is discouraged from resubmitting Panel recommendation is based on insights gained during discussion Funding a project with F or P rating and declining one with E rating requires explanation by PD. Grades & Recommendations

19. Please follow the format in writing panel summaries. Do not merely list reviewers comments. Should reflect the whole discussion to be useful to NSF and PI Crisp comments to help unsuccessful PIs improve their proposals for the next competition Comments in reviews and panel summaries should be constructive, informative, non-inflammatory, and non- discriminatory Ignore letters of endorsement except from organizations contributing funds, data/traces, use of facilities, etc. Panel Summaries