1. Office of Research and Development November 7 th, 2008 Randy Wentsel National Program Director Moving Toward Research Excellence Presentation to ORD Managers

2. Office of Research and Development 1 Overview of Briefing Role of NPDs Research excellence Decision analysis tools Evaluation of our research Communication of our research Accountability Q & A

3. 10/24/20152 Role of National Program Directors Lead planning and development of research programs Cross laboratory view for shifts in program Primary interface with program office and regions Scientific leader - responsible for scientific direction for specific areas of the research program Report to DAA for Science

4. 10/24/20153 Planning and Development of Research Programs Lead Research Coordination Team (RCT) or similar group Lead annual research planning Lead drafting of Multi-year Plans (MYPs) Lead Board of Scientific Counselors reviews Lead PART review Lead science discussions with SAB

5. 10/24/20154 How ORD Evolves its Research Program NPDs Decide What Research Area-Specific Work We Do and When We Do It Planning the Program Decision Inputs Programs and Regions (RCTs) EPA Strategic Plan Administration’s priorities Congressional mandates BOSC Reviews SAB, NAS, other external advice Stakeholders NPDs, SC, MC, EC ORD Executive Council Corporate Decisions on What We Do... and... How We Do It Evaluation Program and Regional Office Feedback BOSC Program Evaluations NAS, NAPA, and other advisory bodies PART Reviews Implementing the Program L/C Directors Decide How ORD Produces its Research Products NPDs Responsible for Communicating Products to Clients L/C Directors Responsible for Developing ORD’s Research Products

6. 10/24/20155 ORD Programs and NPDs www.epa.gov/ord Goal 1 - Air (Dan Costa) NAAQS Air Toxics Goal 2 - Water Drinking Water (Audrey Levine) Water Quality (Chuck Noss) Goal 5 - Stewardship Sustainability & Biofuels (Alan Hecht) Goal 3 - Land (Randy Wentsel) Land Protection & Restoration Goal 4 – Safe Communities and Healthy Ecosystems Homeland Security (Greg Sayles) Human Health (Sally Darney) & Ecosystems (Rick Linthurst) Human Health Risk Assessment (John Vandenberg) Computational Toxicology (Robert Kavlock) Endocrine Disrupting Chemicals (Elaine Francis) Global Change (Joel Scheraga) Pesticides & Toxics (Elaine Francis) Nanotechnology (Jeff Morris)

7. 10/24/20156 The Formula for Research Quality Quality Scientists/Engineers Creative, Innovative Broadly Trained Motivated Quality Facilities/Equipment Conductive to Quality Output Attracts High Quality S&E’s Quality Leadership/Management Environment for Innovation Stress Team-Work Provides Relevance and Timelines Quality Research Sustained Quality Leads to a Brand or Go To Status Potential to Make a Real Difference

8. 10/24/20157 The Formula for Relevant Research Meets Needs of Decision Makers Multiple levels of communication Provides high quality science Informs decisions Timely Knowledge Transfer Meets client schedule Responsive to requests for assistance Enables Tech Transfer Strong Leadership/Management Environment for collaboration with customers Focus on outcomes Relevant Research Sustained Relevance Leads to a Brand or Go To Status Makes a Real Difference

9. 10/24/20158 Research Excellence = f(Technical Quality x Relevance) Increasing excellence Job shopping Ivory tower Relevance Technical Quality Research Excellence Research Excellence – How To Measure It?

10. 10/24/20159 Implementing an ORD Evaluation Framework Research Activities Research Outputs Short-Term Outcomes Intermediate Outcomes Long-Term Outcomes Resources Strategic Goals Strategic Objectives Research ProgramEnvironmental Outcomes and Results Clients Independent Expert Evaluation: Focus:Outcome-oriented progress; R&D investment criteria Evidence: MYPs; synthesis products; performance data; client feedback Attribution: Sphere of influence that includes ORD, EPA clients & partners Policy Development

11. 10/24/201510 BOSC Program Evaluations  Help answer the questions: Are we doing the right science? Are we doing the science right?  Provide guidance for evolving the research program  Provide evidence for OMB evaluations using the Performance Assessment Rating Tool

12. 10/24/201511 Program Assessment Rating Tool (PART) – Required program “audit” by OMB is likely to change, but not go away – Emphasis has been on 4 areas: Purpose/Design, Strategic Planning, Program Management, and Program Results – Ratings/scores based largely on meeting annual and long-term performance goals with emphasis on outcomes, and feedback from external program evaluations like BOSC

13. 10/24/201512 Decision Analysis Tools Value of Information is a form of decision analysis that evaluates the benefits of information for reducing uncertainty in a decision making context Multi-Criteria Decision Analysis reduces complex problems to a singular basis in order to select a preferred alternative These tools may be applied to research planning and/or communicating the value of our research to BOSC, OMB, Congress 4

14. 10/24/201513 Communication of our research accomplishments Performance Measurement and Evaluation –Suite of metrics for quality (bibliometrics), relevance (client citation), and performance (client use) –Outputs and outcomes, annual and long-term –Partner surveys = feedback and areas for improvement –Independent peer review = evaluation and improvement strategies –Informs decisions = impact on human health and the environment

15. 10/24/201514 Communication Challenges Issues –Complete database of ORD publications that partners can use –Demonstrating use/impact of research – faster, cheaper, better science –How do we document impacts? –Value of ORD research To who? –Scientific community –Partners (POs, regions, user community) –Congress, OMB, policymakers

16. Performance Management Challenges – How can we improve our effectiveness as a team (line mgt & NPDs)? – What are methods to improve our communication to: the scientific community, partners (POs, regions, user community), Congress, OMB, and policymakers? – Death by Accountability--GPRA/BOSC/PART/Division-Reviews- -How can we measure and evaluate performance, and ensure that ALL individuals are working toward the mission? – If ORD is to place emphasis on integrated, multidisciplinary programs, what 2-3 specific actions can division directors and NPDs take together to foster such programs?