1. From Republic of Science to Audit Society Irwin Feller Professor Emeritus, Economics, Pennsylvania State University Assessing impact of public policies - Assessing impact of public research. Which complementarities ASIRPA Paris, France June 13, 2012

2. Presentation’s Focus Good and Bad Metrics/Methodologies for Evaluating the Impacts of Research Good and Bad Use(s) of Assessments of Research

3. Outline Perennial S&T Policy Decisions Political Economy of Governance of Science Use/Nonuse/Misuse of Evidence Choice of Methods/Metrics

4. The Times They are A-changin Then you’d better start swimming or you’ll sink like a stone.

5. What Has Been Lost (As seen by some) “The age of chivalry is gone. That of sophisters, economists and calculators has succeeded” Edmund Burke, Reflections on the Revolution in France

6. Numerous Review Articles/Methodology Handbooks “Measuring the Returns to R&D”, Hall, Mairesse, Mohen (NBER, 2009) Evaluation in National Research Funding Agencies: Approaches, Experiences, and Case Studies, European Science Foundation

7. Context, Context, Context (Historical, Path Dependent) Context of National Science/Innovation Systems (Current Political/Policy) Context (Decision/Situational) Context (Linguistic/Disciplinary) Context

8. Assessment Assessment is an historical, integral component of patron-based scientific activity.

9. Assessment Framework Who is Assessing Whom Using What Criteria/Performance Measures/Methods to Document and Value What Outcomes, with What Impacts on the Vitality of the Scientific Enterprise and the Position(s) of Those Engaged in Scientific Activity?

10. Pre-New Public Management Assessment Paradigm Republic of Science (M.Polyani) Peer (Expert) Review Social Contract

11. Social Contract for Science “Government promises to fund the basic science that peer reviewers find most worthy of support, and scientists promise that the research will be performed well and honestly and will produce a steady stream of discoveries that can be translated into new products, medicines or weapons” (Guston and Keniston, 1994: “The Social Contract for Science”)

12. New Public Management Paradigm Accountability Deregulation Competition Performance Measurement

13. What is New Performance measures are increasingly mandated components of appropriation decisions and oversight reviews Dominant ethos is that better measures will lead to better (evidence-based) decisions Stream of new data sets and analytical techniques Increased political/policy trends towards performance based budgeting

14. The Evidence-based Decision Making Imperative “Agencies should demonstrate the use of evidence throughout their FY2014 budget submissions. “….comparative cost-effectiveness of agency investments: allocation of funding across agency programs or within programs” OMB: Use of Evidence and Evaluation in the 2014 Budget (May 18,2012)

15. Tensions Among Accountability, Efficiency and Autonomy Fine line between improved, evidence-based decision-making –”Wanted Better Benchmarks” and increased influence on the direction and inner workings of the scientific enterprise- ”Asking Scientists to Measure Up”

16. Low-Stakes/High-Stakes Assessments Low Stakes –Reputational Surveys: National Research Council Assessments of Graduate Programs; Shanghai Academic Ranking of World Universities High Stakes ‒ Performance-based University Research Funding Systems: UK-Research Assessment Exercise; Germany University Excellence Competition

17. 3 Faces (Purposes) of Evaluation Learn about a program’s operations (Does it Work?; How can it be made better?) Control the behavior of those responsible for program implementation (Modify objectives; reallocate resources; reassign responsibilities) Influence the responses of outsiders in the program’s political environment (create the appearance of a well managed program; preemptively set metrics and methodologies)

18. Generic Science Policy Questions The major issues in science policy are about allocating sufficient resources to science, to distribute them wisely between activities, to make sure that resources are used efficiently and contribute to social welfare” (Lundvall, B. and S. Borras, 2005, p. 605)

19. Promises of Research Performance Assessment Objectives provide useful baseline for assessing performance. Performance measurement focuses attention on the end objectives of public policy, on what’s happened or happening outside rather than inside the black box. Well defined objectives and documentation of results facilitate communication with funders, performers, users, and others.

20. Limitations of Research Performance Measurement Returns/Impacts to research are uncertain, long-term, and circuitous Impacts are typically dependent on complementary actions by agents outside of Federal agency control Benefits from “failure” are underestimated Specious precision in selection of measures Distortion of Incentives Limited (public) evidence of contributions to improved decision making

21. Assessment as Lever for Structural change “ Given that science is changing, the institutions that are efficient in supporting science at one point in time may be less appropriate at a later point of time. On precise dimensions, a failure to continually re-tune science policy may therefore impede scientific progress. B. Jones (2010) As Science Evolves, How Can Science Policy?

22. Performance Metrics Metrics Abound: Generic List of 37, with New Ones Constantly being Proposed Most Programs Have Multiple Objectives-- Select Metrics most relevant to Decisions “Cherry Pick” Metrics (Strategic Retreat from Objectives)

23. All Performance Measures Can be Gamed “Once STI indicators are made targets for STI policy, such indicators lose most of the informational content that qualify them to play such a role” (Freeman and Soete, 2009)

24. Overview of Evaluation Methodologies METHODBRIEF DESCRIPTIONEXAMPLE OF USE Analytical conceptual modeling of underlying theory Investigating underlying concepts and developing models to advance understanding of some aspect of a program, project or phenomenon. To describe conceptually the paths through which spillover effects may occur. SurveyAsking multiple parties a uniform set of questions about activities, plans, relationships, accomplishments, value, or other topics, which can be statistically analyzed. To find out how many companies have licensed their newly developed technology to others. Case study – descriptive Investigating in-depth a program or project, a technology, or a facility, describing and explaining how and why developments of interest have occurred. To recount how a particular joint venture was formed, how its participants shared research tasks, and why the collaboration was successful or unsuccessful. Case study - economic estimation Adding to a descriptive case study quantification of economic effects, such as through benefit-cost analysis. To estimate whether, and by how much, benefits of a project exceed its costs. Econometric and statistical analysis Using tools of statistics, mathematical economics, and econometrics to analyze functional relationships between economic and social phenomena and to forecast economic effects. To determine how public funding affects private funding of research. Sociometric and social network analysis Identifying and studying the structure of relationships by direct observation, survey, and statistical analysis of secondary databases to increase under-standing of social organizational behavior and related economic outcomes. To learn how projects can be structured to increase the diffusion of resulting knowledge.

25. Taking the “Con” Out of Econometrics Leamer (1986): “Hardly anyone takes the data analysis seriously”. Distressing lack of robustness to changes in key (‘whimsical’) assumptions

26. Credibility Revolution in Empirical Economics “Empirical microeconomics has experienced a credibility revolution, with a consequent increase in policy relevance and scientific impact…. “Primary engine driving improvement has been a focus on the quality of empirical research designs” Angrist and Pischke (J. Economic Perspectives, 2010)

27. BeforeAfter Treatment Group S Т Comparison/ Control Group S C τ τ + 1 τ Issue: Before/After Design Shows changes “related” to policy intervention, but does not adjust for “intervening” factors. (Threats to internal validity) Reframe Analysis: Did policy “cause” change(s) in treatment group different from those observable in a comparison/control group

28. Trends in U.S. Agricultural Productivity

29. Impact of R&D on U.S. Agricultural Productivity

30. Benefit -Cost Analysis Steps Conduct Technical Analysis Identify Next Best Alternative Estimate Program Costs Estimate Economic Benefits Determine Agency Attribution Estimate Benefits of Economic Return RTI 2010

31. Benefit-Cost Estimates of Returns to Health Research An average 45 year old in 1994 had a life expectancy 4 ½ years longer than in 1950 because cardiovascular disease mortality had decreased. “…(U)nambiguous conclusion …that medical research on cardiovascular disease is clearly worth the cost” (2002; p.113). In benefit-cost terms, this increase is estimated to yield a 4 to 1 return for medical treatment and a 30-to-1 return for research and dissemination costs related to behavioral change. (Cutler and Kadiyala, 2002)

32. Benefit-Cost Estimates of DOE-EERE Geothermal Technology Studies Metric PDC (thousands $2008) Binary (thousands $2008) TOUGH (thousands $2008) Cement (thousands $2008) Total Benefits$15,823,141$379,870$928,369 $7,828 Program Cost $108,886 $195,469$101,807 $137,223 Net Benefits $15,714,255 $184,401$826,562 -$129,395 NPV of Net Benefits @3% $8,147,636 $74,642$418,997-$106,679 NPV of Net Benefits @7% $3,850,834 $3,380$173,667-$86,579 Benefits/Costs Ratio 145.31.9 9.10.06 Benefits/Costs Ratio @3% 94.41.5 6.10.03 Benefits/Costs Ratio @7% 56.41.0 3.70.012 Internal Rate of Return 91%7% 19%NA RTI: 2010

33. Econometric Approach Manufacturing Extension Partnership VariableClient MeanNon-Client Mean N155915,982 Age, 199215.9716.04 Employment, 1992170.2171.70 Employment Growth Rate, 1987-19920.013-0.088 Sales, 199230,797,19913,418,587 Sales Growth Rate, 1987-19920.052-0.085 Sales Growth Rate, 1982-19870.4270.338 Annual wage, 199228,07225,013 Production Worker Share, 19920.6990.724 Value Added Per Worker, 198753,04250,853 Value Added Per Worker, 199256,70952,797 Labor Productivity Growth Rate, 1987-19920.2150.203 Labor Productivity Growth Rate, 1982-19920.0520.010 # of Extension Projects3.82NA Total Project Costs63,787NA R. Jarmin Measuring the Impact of Manufacturing Extension Summary Statistics

34. “Dominant” U.S. Methodology is Expert Panels “The most effective means of evaluating federally funded research programs is expert review. Expert review-which includes quality review, relevance review, and benchmarking should be used to assess both basic research and applied research programs” (National Academies, Evaluating Federal Research Programs, 1999, p. 5)

35. Bibliometrics: US Added to reputational surveys in NRC assessments Patent to Citation Linkage to Document Impacts of Basic Research Increasingly used by departments/colleges No use of performance-based funding Little evidence (to date) of impacts on Federal funding of academic research

36. Use of Bibliometric Data to Allocate Resources Across Fields Over the last 3 decades, even as the US position in the life sciences has remained strong, its world share of engineering papers has been cut almost in half, from 38% in 1981 to 21% in 2009, placing it below the share (33%) for the EU27. Similar declines in world share are noted for mathematics, physics, and chemistry. If bibliometric performance is a function of resource allocation, a nation gets what it funds. Formulation begs questions if what it’s producing is what it most needs, and if what it’s producing is being produced in the most efficient manner.

37. Is Anyone Listening? “The ideas of economists and political philosophers, both when they are right and when they are wrong, are more powerful than is commonly understood. Indeed the world is ruled by little else. Practical men, who believe themselves to be quite exempt from any intellectual influence, are usually the slaves of some defunct economist. Madmen in authority, who hear voices in the air, are distilling their frenzy from some academic scribbler of a few years back” (J.Keynes)

38. Is Anyone Listening? Yes Continuing Impact of “Academic Scribblers” on Men in Power in Setting Policy Worldview –Solow-Abramovitz-Romer –Arrow-Nelson –Mansfield

39. BIG “3” FEDERAL SCIENCE QUESTIONS QUESTION How much should be allocated to Federal research? How much to spend across missions/agencies/fields of science? Which performers; what allocation criteria? ROLE OF PERFORMANCE MEASURES Measures do not provide a basis for determining if, say, 3% is too high, too low, or just right. Measures/methodologies provide multiple answers, leading to multiple possible decisions Potentially of considerable value, but underutilized

40. Using Social Rates of Return to Guide Resource Allocations “But it is evident that these studies can provide very limited guidance to the Office of Management and Budget or to the Congress regarding many pressing issues. Because they are retrospective, they shed little light on current resource allocation decisions, since these decisions depend on the benefits and costs of proposed projects, not those completed in the past”. (Mansfield, 1991, p. 26).

41. Use, Non-Use, Misuse of Program Assessments Use: NIH Benefit-Cost Studies Nonuse: ATP (Program terminated); DOE-ERRE (Budget slashed) Misuse: Overgeneralization of findings to different decision/policy settings “Bunkum”-Worthless, mundane, incompetently done assessments

42. Asymmetrical Impacts of Well Done Evaluations If program is not working, kill it because it is ineffective If program is working, prima facie evidence that the private sector would engage in it where it not being crowded out

43. The Ever Lurking (Ideological) Counterfactual:ATP “ATP’s defenders claim that these subsidies generate greater technological innovation. They point out all the technologies on the market that ATP funded. Of course, ATP grants have funded some successful products. But the key question is whether the market would have produced those products even without ATP. Both economic theory and practice say, “Yes.” Brian Riedl, Testimony before the Homeland Security and Government Affairs Committee, United States Senate, May 2005

44. Thank you