-- BUSINESS PROPRIETARY --© 2007 viaSim 1 Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.archivedOER Public Archive Home Page

A New Model for Predicting the Number of NIH Grant Applications viaSim J. Chris White April 19, 2007

-- BUSINESS PROPRIETARY --© 2007 viaSim 3 Overview  It is fundamentally important for all of NIH to improve the ability to forecast budget needs and improve planning related to manpower requirements and paylines that are linked to the number of applications received.  Calculating manpower based on number of applications is straightforward, so the key issue is forecasting the incoming application stream.  CSR is the triage organization for incoming NIH proposals and reviews approximately two-thirds of these proposals.  Problem: Results from common statistical forecasting models are sometimes unsatisfactory.  Objective: Develop a forecasting model using a simulation approach called system dynamics.  Major Tasks: Interview research institutions to understand how they make decisions Interview NIH personnel to understand how NIH makes funding decisions Develop system dynamics (SD) model based on interviews Compare and contrast the traditional statistical forecasting approach with the SD structural forecasting approach

-- BUSINESS PROPRIETARY --© 2007 viaSim 4 Statistical Forecasting Example

-- BUSINESS PROPRIETARY --© 2007 viaSim 5 Previous CSR Statistical Forecasting Model

-- BUSINESS PROPRIETARY --© 2007 viaSim 6 Meyers’ Statistical Forecast

-- BUSINESS PROPRIETARY --© 2007 viaSim 7 Two Fundamental Types of Forecast Modeling  Statistical modeling attempts to project future results by extrapolating historical process data into trends: Highly dependent on the quality of data used to generate trends Focuses on results (sometimes using correlations with inputs), with no consideration for the activities Does not project well when future is significantly different from past  Structural modeling attempts to project future results by simulating the activities or operations: Does not depend on data to create the model because the flow of activities is independent of data Data is used to tailor and calibrate the model (as inputs) Focuses on activities, which generate results Activities Results Inputs

-- BUSINESS PROPRIETARY --© 2007 viaSim 8 Model Scope NIH Budget Apps Submitted Desired Proportion NIH Funds Institution Staff + + Success Rate + _ + Completed Grants + NIH Review Capacity + Apps Reviewed + Apps Unfunded + ++ Required Funds + Funding Gap +– _ _ + Funded Grants + + __

-- BUSINESS PROPRIETARY --© 2007 viaSim 9 Example Feedback Loop: An increase in grants funded leads to a decrease in applications submitted NIH Budget Apps Submitted Desired Proportion NIH Funds Institution Staff + + Success Rate + _ + Completed Grants + NIH Review Capacity + Apps Reviewed + Apps Unfunded + ++ Required Funds + Funding Gap + – _ _ + Funded Grants + + __

-- BUSINESS PROPRIETARY --© 2007 viaSim 10 Dr. Zerhouni’s “Perfect Storm” NIH Budget Apps Submitted Desired Proportion NIH Funds Institution Staff + + Success Rate + _ + Completed Grants + NIH Review Capacity + Apps Reviewed + Apps Unfunded + ++ Required Funds + Funding Gap + – _ _ + Funded Grants + + __

-- BUSINESS PROPRIETARY --© 2007 viaSim 11 Simulation Scope NIH Budget Apps Submitted Desired Proportion NIH Funds Institution Staff + Success Rate + _ + Completed Grants + Apps Reviewed + Apps Unfunded + + Required Funds + Funding Gap + – _ _ + Funded Grants + + _

-- BUSINESS PROPRIETARY --© 2007 viaSim 12 Simulation Model Example Institutions Work Sector Feedback Loop: An increase in grants funded leads to a decrease in apps submitted

-- BUSINESS PROPRIETARY --© 2007 viaSim 13 Previous Simulation Results: Apps Reviewed by CSR

-- BUSINESS PROPRIETARY --© 2007 viaSim 14 New Baseline Simulation Results

Demonstration

-- BUSINESS PROPRIETARY --© 2007 viaSim 16 Next Steps  Train NIH staff in use of simulation tool.  Finalize any reporting requirements.  Model enhancements: Expand to include more internal NIH processes and policies (e.g., review processes). Expand to include more operational processes at institutions (e.g., facility expansions). Integrate with PI simulation model.

Appendix System Dynamics Methodology

-- BUSINESS PROPRIETARY --© 2007 viaSim 18 Why System Dynamics (SD)? Ability to influence results Organizational structure, IT systems, business processes, financial processes, etc. Structure Corporate growth, inventory oscillations, labor force oscillations, etc. Behavior Stock out, excess inventory, layoff, etc. Results In any system, structure guides behavior, and behavior determines results.

-- BUSINESS PROPRIETARY --© 2007 viaSim 19 System Dynamics Methodology 2. Water accumulates in tub Good analogy: Water in bathtub 3. Water flows out through drain 2. Money accumulates in “Cash Balance” 1. Water flows in through faucet 3. Money flows out through “Expenses” 1. Money flows in through “Revenue”

-- BUSINESS PROPRIETARY --© 2007 viaSim 20 System Dynamics Methodology  Reinforcing loops push a variable to continue to grow (or decrease): Compound interest Radioactive decay  Balancing loops attempt to bring a variable to some target value: Adjusting personnel based on budget available Thermostat in house Body temperature Number of personnel drives expenses Number of personnel drives work completion Work completed drives revenue Available money drives hiring (or not hiring) of new personnel Balancing Feedback Loop  Arrows represent connections between variables.  Interconnections create “feedback loops”.

-- BUSINESS PROPRIETARY --© 2007 viaSim 21 What Makes SD Different?  SD creates an “operational”, working model of activities.  SD captures “feedback loops”, which are the function of management. Ex: Cutting back when overbudget Ex: Adjusting the workforce to meet demand  SD captures time delays and non-linear relationships that can significantly impact performance. Ex: Full impact of new policies not realized until several years later. Ex: Doubling of labor force does not always double throughput.