1. Evaluating R&D Programs: R&D Evaluation Planning Concept Mapping and Logic Modeling Presentation to the Washington Research Evaluation Network June 2008 William Trochim Cornell University wmt1@cornell.edu

2. Questions What new useful and applicable approaches and tools for ex-ante evaluations have been uncovered? Which agencies have used these tools? How can these tools be matched with different levels of decision making? Can systems-level analysis address gaps and assist in science and technology decision making at the federal level?

3. Overview of Evaluation Planning Process Concept Map Logic Model Evaluation Questions Measurement Development Evaluation Design  Implement Evaluation

4. Where Has Approach Been Used NIH/NCI Transdisciplinary Tobacco Use Research Centers (TTURCs) CDC, Prevention Research Centers (PRCs) NIH/NIAID/DMID, Research Centers of Excellence (RCEs) NIH/NIAID/DAIDS, AIDS Clinical Research Networks NIH/NCRR/CTSA, Weill Cornell Clinical and Translational Science Center

5. DMID Research Centers of Excellence (RCEs) PLANNING: Identified the focus prompt and participants “In order to foster excellence in advancing the biodefense and infectious disease research enterprise, the RCE program and its Centers should...” IDEA GENERATION: Eliciting knowledge and opinion Brainstorming : 279 stakeholders were invited to participate. Participants generated 248 statements with 117 unique visitors hitting the site. 100 statements were derived from the Literature Review. The planning group edited statement list for relevance and representativeness. Planning Group approved final set of 94 ideas. STRUCTURING: Data collection to build the conceptual framework Sorting and Rating of Ideas 279 participants were invited to rate the 94 ideas on relative Importance and Expected Progress from 2003-present 74 people (27%) contributed their input on the ratings. 33 people invited to organize the 94 ideas into conceptually similar categories. 24 people (72%) participated in the conceptual sorting of the ideas. ANALYSIS: Series of multivariate analyses

6. The point map shows all the elements in relation to one another. value and reward original ideas and innovative technologies (4) “In order to foster excellence in advancing the biodefense and infectious disease research enterprise, the RCE program and its Centers should...”

7. Conceptually similar ideas are in close proximity value and reward original ideas and innovative technologies (4) demonstrate its flexibility in addressing emerging infectious disease research needs. (19) have the capacity to rapidly move into areas of growing importance. (56)

8. Conceptually different ideas are further apart value and reward original ideas and innovative technologies (4) create training that is relevant to and integrated with the goals of the strategic plans. (21) define who the stakeholders are once product development moves into the pipeline (72) “In order to foster excellence in advancing the biodefense and infectious disease research enterprise, the RCE program and its Centers should...”

9. Collaboration and Communication The cluster map organizes ideas into groups

10. Each cluster retains the details: For example, Training and Recruitment promote and encourage opportunities for mentoring junior faculty/researchers in the field of biodefense and emerging infectious disease research. (7) encourage the participation of post-docs, students, and other non-PI scientists at each RCE's regional meeting. (8) recruit qualified students, trainees, women and minorities. (10) help new investigators obtain funding. (16) create training that is relevant to and integrated with the goals of the strategic plans. (21) develop more standardized training programs across RCEs by looking at successful programs at each RCE and developing some best practices. (26) develop an expanded cadre of new researchers, clinicians, and technical personnel who can help lead the national biodefense mission. (45) supplement RCE activities by applying for direct training supports through T, K and F awards. (55) establish training programs with pharmaceutical company partners to allow training of scientists in drug discovery and non-clinical development. (68) develop policies, criteria and processes for selecting career development candidates, including special efforts to recruit qualified women and minorities. (76) bring new people into the area of biodefense and emerging infectious disease research. (80)

11. DMID RCE Final Conceptual Framework …contains all the details and provides a conceptual framework. Integration and Synergy Funds Management Management Innovation and Flexibility Capacity and Infrastructure Collaboration and Communication Research Mission Training and Recruitment Translational Activities Emergency Response

12. Training and Recruitment Integration and Synergy Funds Management Management Innovation and Flexibility Capacity and Infrastructure Collaboration and Communication Research Mission Translational Activities Emergency Response Outcomes Activities Outputs Concept Map with Regions

13. Logic Model for RCE Program RCE Program Funding Manage Funds flexibly and efficiently Establish Management structures & processes Innovative & Flexible response to changing needs and opportunities Synergy between Centers (Integration & Synergy) Capacity to serve as a regional resource Expanded cadre of investigators (Recruit and Train staff) Leveraged funds from elsewhere (Integration & Synergy) Readiness to Respond in an Emergency Better response to infectious diseases that threaten humans Translate and Apply Science to Practice Scientific Knowledge on Category A,B,C organisms (Research ) Leadership with expertise (Integration & Synergy) Biodefense Priorities Build Infrastructure Recruit & Train staff InputsOutputs Outcomes Activities Impact Collaboration & Communication Conduct Research on Category A,B,C organisms Research results on Category A,B,C organisms Product Development (Translational Activities)

14. Structure OutcomeProcess Scientific Integration Communication Professional Validation Collaboration Health Impacts Recognition Collaboration Training Transdisciplinary Integration TD Research Institutionalization Health Outcomes Improved Interventions Translation to Practice Policy Implications Science & Models Methods Publications Communications TTURC Concept Map

15. Communication Collaboration Scientific Integration Professional Validation Health Impacts Collaboration Training Transdisciplinary Integration Interventions Health Outcomes Methods Science & Models Translation To Practice Policy Implications Communication External Recognition And Support Internal Recognition And Support Publications TTURC Logic Model Intermediate Markers Immediate Markers Long-Term Outcomes

16. CDC PRC Concept Map Engage the Community Diversity & Sensitivity Relationships & Recognition Active Dissemination Technical Assistance Training Research Methods Research Agenda Core Expertise & Resources

17. From Map to Logic Model InputsOutputsOutcomesActivities Engage the Community Diversity & Sensitivity Relationships & Recognition Active Dissemination Technical Assistance Training Research Methods Research Agenda Core Expertise & Resources

18. Draft logic model National, Regional or Local Health Priorities and Health Disparities Community Advisory Board PRC Capacity Core expertise & resources Diverse faculty/ staff Faculty/staff sensitive to community issues Facility Communication & data systems Conduct Core & Other Research Using Sound Research Methods Conduct Training Test Innovative Strategies Skilled Workforce Disseminate Research Findings Translation of Research To Practice Provide Technical Assistance Widespread Knowledge of Effective Interventions Trust Relationships With Community Partners, Other PRCs, and CDC Improved Community Health and Reductions in Health Disparities Widespread Use of Effective Interventions Engage the Community Establish Research Agenda Recognition Research and Other Publications InputsOutputsOutcomesActivities

19. INPUTS National, Local, or Regional Health Priorities And Health Disparities Engage the Community Establish Research Agenda Conduct Core and Other Research Using Sound Research Methods Train/Mentor/P rovide Technical Assistance to: Researchers Practitioners Students Community Members PRC Community Committee PRC Capacity Human resources: core expertise, diversity, sensitivity Facility Communication & data systems Administrative capacity Evaluation expertise Relationships with State & Local Health Departments, Community Partners, University Partners, Other PRCs, and CDC Motivating Conditions for Developing and Maintaining Relationships (e.g., Trust) Research/ Evaluation Findings Communicated & Disseminated in: Publications Presentations Media Reports Trainees and Technical Assistance Recipients Programs/ Interventions OUTCOMES Improved Community and Population Health and Elimination of Health Disparities Enhanced Community Capacity for Prevention Translation of Research to Practice and Policy Widespread Use of Effective Programs and Policies Skilled Public Health Professionals PROGRAM ACTIVITIES OUTPUTS Evaluation Expanded Resources Recognition Logic Model

20. AIDS Clinical Research Networks Map DAIDS Policies and Procedures Operations and Management Resource Utilization Community Involvement Communication, Collaboration, Harmonization Scientific Agenda-setting Biomedical objectives Relevance to Participants

21. Appropriate human and infrastructure Resources are in place DAIDS Policies and Procedures InputsOutputs Short Term Outcomes Activities Longer Term Outcomes/ Impact Collaboration and Communication (Communication, Collaboration, Harmonization) Operations and Management processes are in place Community Involvement Scientific agenda is Relevant to Participants High quality scientific results and increased knowledge (Biomedical Objectives) Treatment and prevention measures (Biomedical Objectives) Community input (Community Involvement) Scientific Agenda Setting processes Harmonized systems and procedures (Communication, Collaboration, Harmonization) Scientific research plan and priorities (Scientific Agenda Setting) Increased capacity of developing world sites (Resource Utilization) Recruitment and retention goals met (Community Involvement) Results Published (Biomedical Objectives) Biomedical Objectives: HIV/AIDS mortality and morbidity reduced Integrated use of developing world sites (Resource Utilization)

22. Conclusions Participatory and Collaborative Integrative Mixed Methods Systems-Based Approach Comprehensive Conceptual Model Foundation for eEvaluation

23. Relevant Literature Trochim. W., Marcus, S.E., Mâsse, L.C., Moser, R.P., Weld, P. (2008). The Evaluation of Large Research Initiatives: A Participatory Integrative Mixed-Methods Approach, American Journal of Evaluation, 29, 1, 8-28. Anderson, L. A., Gwaltney, M. K., Sundra, D. L., Brownson, R. C., Kane, M., Cross, A. W., et al. (2006). Using concept mapping to develop a logic model for the prevention research centers program. Preventing Chronic Disease: Public Health Research, Practice and Policy, 3(1), 1-9. Trochim, W. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12(1), 1-16. Trochim, W. and Kane, M (2005). Concept mapping: An introduction to structured conceptualization in health care. International Journal for Quality in Health Care, 17, 3, June 2005, 187-191. http://www.socialresearchmethods.net/research/cm.htm

24. Further Information William Trochim –wmt1@cornell.eduwmt1@cornell.edu –http://www.socialresearchmethods.net/http://www.socialresearchmethods.net/ Concept Systems Incorporated –http://www.conceptsystems.com/http://www.conceptsystems.com/