1. PUTTING A VALUE ON BIOMEDICAL RESEARCH PROGRAMS: ADAPTING THE RESEARCH PAYBACK FRAMEWORK FOR APPLICATION IN THE UNITED STATES JACK E. SCOTT, MARGARET BLASINSKY and MARY DUFOUR The Madrillon Group Inc., Vienna, Virginia G. STEPHANE PHILOGENE Office of Behavioral and Social Sciences Research National Institutes of Health, Bethesda, Maryland and Rachel J. Mandal National Cancer Institute, Bethesda, Maryland American Evaluation Association, Anaheim, California November 5, 2011

2. OVERVIEW OF PRESENTATION The Mind-Body Interactions and Health Program Overview of the Payback Framework Design and Methods of the MBIH Evaluation Findings from the Evaluation Lessons Learned from Applying the Payback Framework 2

3. MIND-BODY INTERACTIONS AND HEALTH PROGRAM Rationale: Cognitions, emotions and stress can affect physical & mental health Some mind-body interventions can improve health Mechanisms of these effects not understood MBIH Program: Established by Congress in 1999 Ten-year, trans-NIH program directed by OBSSR 12 partnering NIH Institutes and Centers $175 million dollars 15 MBIH research centers 44 investigator-initiated grants 3

4. THE RESEARCH PAYBACK FRAMEWORK Developed in mid-1990s by Martin Buxton, Stephen Hanney, and Health Economics Research Group (HERG) at Brunel University, London Two components: –Multidimensional categorization of research benefits –Input-process-output-outcome model of the research process 4

5. TYPES OF RESEARCH BENEFITS BENEFIT CATEGORYEXAMPLES OF INDICATORS Knowledge Productivity Publications Oral Presentations Research Targeting and Capacity Development New Grants Career Development New research tools, methods, models, and infrastructure Informing Policy and Product Development Uptake of research findings in policy formulation New products Health and Health Sector Improvement in patients’ outcomes, quality of life Changes in healthcare delivery Broader Economic and Social Impacts Benefits to society—decreased costs of care, reduced disability days 5

6. Stage 0: Topic or Issue Identification Interface A: Project Specification & Selection Stage 1: Inputs Stage 2: Processes Stage 3: Primary Output s Interface B: Dissemination Stage 4: Secondary Outputs: Policy and Product Developme nt Stage 5: Adoption by Clinicians & Public Stage 6: Final Outcomes Health & Health Sector and Broader Economic & Social Impacts Internal Reservoir of Knowledge & Expertise (Center Level) Stock or Reservoir of Knowledge Direct Feedback Paths Direct Impact from Processes and Primary Outputs to Adoption by Clinicians and Public Direct Impact from Processes and Primary Outputs to Adoption by Clinicians and Public PAYBACK FRAMEWORK RESEARCH LOGIC MODEL 6

7. MBIH PROGRAM EVALUATION DESIGN & METHODS Cross-sectional mixed-methods design Focus on overall achievements: –Program as a whole –MBIH research centers –MBIH investigator-initiated research projects Six broad evaluation questions Data sources: program documents, NIH databases, semi-structured interviews, bibliometric analyses 7

8. MBIH RESEARCH CENTER INTERNAL STUDIES PILOT STUDIES 209 pilot studies (14/15 centers) Smaller studies led by post- doctoral fellows or junior faculty 1-2 years duration Predominantly clinical research (79 percent) versus basic research (11 percent) or both (10 percent) SUB-PROJECTS 78 sub-projects (11/15 centers) Larger in scope and scale (similar to R01) led by established investigators 3-4 years in duration Clinical research predominant (48 percent), but more basic science (32 percent) and both (20 percent) 8

9. PAYBACK BENEFIT CATEGORIES KNOWLEDGE PRODUCTIVITY Published articles identified from NIH e-SPA database between January 1, 2000 and December 31, 2009 429 publications (336 peer-reviewed research articles and 93 non-research articles as defined by Pub-MED) 171 oral presentations Comparison of publications and presentations by pilot studies and sub-projects: –Pilot studies: 64 percent of studies led to oral presentation and 47 percent led to a publication –Sub-projects: 47 percent of studies led to oral presentation and 69 percent led to publication 9

10. PAYBACK BENEFIT CATEGORIES RESEARCH TARGETING & CAPACITY DEVELOPMENT Research Targeting: NIH Spin-off Grants 100 of 287 studies (pilots + sub-projects) led to a new verified NIH spin-off grant Spin-off grants =$184,781,090 in new NIH funding Represented $1.95 in new NIH funds for each dollar spent on MBIH centers Spin-off ratio for 13 centers ranged from $0.07 to $7.93 Research Targeting: Non-federal Grants Could not collect & verify numbers and amounts of non- federal grants 13/15 centers obtained funding from non-federal sources 10/13 obtained funds from foundations 9/13 obtained funds from host institutions 10

11. PAYBACK BENEFIT CATEGORIES RESEARCH TARGETING & CAPACITY DEVELOPMENT CAREER DEVELOPMENT Investigator awards and honors (2/15) Graduate and post-graduate training (8/15) Faculty promotions including tenure (8/15) Recruitment of new faculty (9/15) Formation of new collaborations (15/15) TECHNICAL CAPACITY New research tools, methods and models (11/15) –Used by other investigators at host institution (10/11) –Used by other investigators at other institutions (7/11) New research infrastructure (10/15) –New equipment or instruments –New services 11

12. PAYBACK BENEFIT CATEGORIES INFORMING POLICY AND NEW PRODUCT DEVELOPMENT Policy considered broadly (both administrative and clinical) Three indicators: –Research influenced policy formulation/development (9/15) –Research influenced clinical guidelines or treatment recommendations (6/15) –Research influenced medical or healthcare professional education or training (13/15) Examples: –“White coat” hypertension –National curriculum for mind-body medicine in family residencies 12

13. PAYBACK BENEFIT CATEGORIES HEALTH AND HEALTH SECTOR Requires some degree of adoption of new practices or behaviors by clinical practitioners Three indicators: –Adoption by clinical practitioners (9/15) –Changes in healthcare service delivery (9/15) –Improvements in health outcomes and quality of life (6/15) More difficult for PIs to report because they do not usually track this 13

14. PAYBACK BENEFIT CATEGORIES BROADER ECONOMIC AND SOCIAL IMPACTS Most difficult type of benefit to measure –Cost analyses are often not available –Time it can take for a finding to affect clinical practice and health –Attributing an impact to a specific single research project 4/15 PIs believe that their MBIH research will produce broader economic and social benefits 14

15. COMPARING RESEARCH CENTERS ACROSS PAYBACK CATEGORIES Centers can be compared in terms of individual payback benefit categories, but more interesting to compare centers on all five categories at once Case study scoring system –Indicators rated on scales scored from 0 to 5 –Ratings on 7 indicators conducted by members of Madrillon evaluation team (82 percent agreement on ratings) –Some indicators combined to represent benefit category –Final ratings for five domains ( Research targeting; capacity development; effects on medical or healthcare professional training; effects on policy; and adoption by clinical practitioners) 15

16. COMPARISON OF TWO MBIH CENTERS USING RADAR GRAPHS MBIH Research Center DMBIH Research Center H 16

17. LESSONS LEARNED FOR FUTURE APPLICATIONS OF THE PAYBACK FRAMEWORK Payback Framework well-suited to evaluations of NIH biomedical research programs MBIH research centers had important impacts in all five benefit categories Greatest challenge is the case study scoring methodology As more Payback Framework evaluations are completed, it may become possible to establish benchmarks 17