Discussion From Republic of Science to Audit Society, Irwin Feller S. Charlot ASIRPA, Paris, France; June 13, 2012

Outline New Questions/Issues & What’s at Stake How They’re Answered? Validity of Performance Metrics and Methodological Choice(s)  Econometrics Use, Non-Use and Misuse of Research Assessments

Pre-New Public Management Assessment Paradigm Republic of Science (M.Polyani) Peer (Expert) Review Social Contract

New Public Management Paradigm Accountability Deregulation Competition (among different uses of public funds) Performance Measurement (for evaluating research uses)

Promises of Research Performance Assessment Objectives provide useful baseline for assessing performance. Performance measurement focuses attention on the end objectives of public policy, on what’s happened or happening outside rather than inside the black box. Well defined objectives and documentation of results facilitate communication with funders, performers, users, and others.

Limitations of Research Performance Measurement Returns/Impacts to research are uncertain, long-term, and circuitous Specious precision in selection of measures Impacts are typically dependent on complementary actions by agents outside of Federal agency control Limited (public) evidence of contributions to improved decision making Benefits from “failure” are underestimated Distortion of Incentives : opportunistic behavior (young researchers to be employed and elder researchers to catch future funds) First comment/issue + Role of creativity/very innovative ideas in science progress (i. e. “scientific revolutions”)

Overview of Evaluation Methodologies METHODBRIEF DESCRIPTIONEXAMPLE OF USE Analytical conceptual modeling of underlying theory Investigating underlying concepts and developing models to advance understanding of some aspect of a program, project or phenomenon. To describe conceptually the paths through which spillover effects may occur. SurveyAsking multiple parties a uniform set of questions about activities, plans, relationships, accomplishments, value, or other topics, which can be statistically analyzed. To find out how many companies have licensed their newly developed technology to others. Case study – descriptive Investigating in-depth a program or project, a technology, or a facility, describing and explaining how and why developments of interest have occurred. To recount how a particular joint venture was formed, how its participants shared research tasks, and why the collaboration was successful or unsuccessful. Case study - economic estimation Adding to a descriptive case study quantification of economic effects, such as through benefit-cost analysis. To estimate whether, and by how much, benefits of a project exceed its costs. Econometric and statistical analysis Using tools of statistics, mathematical economics, and econometrics to analyze functional relationships between economic and social phenomena and to forecast economic effects. To determine how public funding affects private funding of research. Sociometric and social network analysis Identifying and studying the structure of relationships by direct observation, survey, and statistical analysis of secondary databases to increase under-standing of social organizational behavior and related economic outcomes. To learn how projects can be structured to increase the diffusion of resulting knowledge.

Second comment/question Complementarities between methodologies Econometric modeling needs analytical conceptual modeling of underlying theory to be pertinent Econometric analysis also needs to take into account the policy design, context… to be pertinent  Survey, case studies.. No econometric identification of impacts without these components in the evaluation model

Complementarity second example Benefit -Cost Analysis can be made by econometric model Conduct Technical Analysis Identify Next Best Alternative Estimate Program Costs Estimate Economic Benefits Determine Agency Attribution Estimate Benefits of Economic Return RTI 2010

BeforeAfter Treatment Group S Т Comparison/ Control Group S C τ τ + 1 τ Issue: Before/After Design Shows changes “related” to policy intervention, but does not adjust for “intervening” factors. (Threats to internal validity) Reframe Analysis: Did policy “cause” change(s) in treatment group different from those observable in a comparison/control group Microeconometrics of policy evaluation

Third comment/question Econometric enhancements: Non parametric analysis  no a priori constraint on relationship between outcome (whatever the outcome chosen) and R&D spending or funding  No knowledge production function a priori Taking into account the effect of non observable characteristics or time varing characteristics on outcomes  context, context, context

Fourth comment/question “Dominant” U.S. but also European Union Methodology is Expert Panels Problem of network effects Same issue as peer evaluation and bibliometrics  Only issue for « low impacts » (publications..) but not for high impacts???

Is Anyone Listening? My small experience (one evaluation report): no one is listening as a researcher I agree that “Doing good may not make you happy, but doing wrong will certainly make you unhappy” But for a novice at evaluating policy what are arguments not to stop this kind of intellectual exercise? (except publishing or funding researches) What type of advices? For ASIRPA?

Thank you