Evaluating and monitoring complex systems Pier Francesco Moretti JPI to Co-Work – 19 December Warsaw

Alice in Wonderworld: “if you do not know where you are going, how do you know when you get there…” A traveler needs both a map and a compass. BUT… But…

Evaluation (why) Justification of spending public funds/accountability Learning process/policy advice/ adaptation Evaluation (what) Process (tactical) Results (operational) Evaluation (how) Correlation/ causality Monitoring Data collection Relevance Responsibility Quantitative (what is happening) Qualitative (why is happening) in complex systems… Parameters Indicators Impacts (strategic) Parameters Indicators

Complex systems: understanding the process is crucial! Dynamic complex systems are inherently chaotic and unstable, but, they usually evolve into one of a number of possible steady states. These steady states are called "attractor basins". It is by causing dynamic complex systems to switch between attractor basins that control can be exercised.

From 1 to which of the 2s? 1 2a 2b 2c JPI addresses societal challenges and deal with complex systems. According to OECD, social indicators cannot be used to evaluate the effectiveness of particular social program. JPIs are not properly social programmes… But, how much can actions influence/control the evolution of the system? Contribution can come from different stakeholders! Evaluate to understand, but identify what you control (How much the action you adopt is correlated to the result you expect? What happened if you did not adopted that action?) Any system is linear at short timescales… how short? Warning: the evolution timescales of the system can be shorter than those of the actions! Are we driving the process? Evaluation is close linked to Foresight,ex ante, governance! Impact can be defined as a change of a variable A attributed to an input B (  A/  B): should we monitor its continuity? Better focusing on structural interventions (monotonous trends and not targets)? Are only funds are (and how much) impacting?

Indicators linked to objectives’ hierarchy, with their timescales: project monitoring, process evaluation, impact assessment. Set of process/impact indicators: context, inputs (including expenditure), outputs. leading, coincident, lagging (timescales) But what constitutes success? If targets for indicators are not met, does it imply lack of success (can it depend on missing data collection or identification) ? Be careful to use /working towards numerical metrics: it can distract from the objective! Criteria for process/impact indicators: Evaluating: why-what-how. Always together… Learning process…statistics to evaluate the “best action”: is it possible, where? Is there the possibility to compare realizations/experiments? From JPND/Voluntary guidelines ERAC-GPC: relevance, effectiveness, efficiency, utility, sustainability. Moreover: simple to operate, cost effective, credible, flexible… From Evaluation Journal…: validity, relevance, appropriateness, robustness, manageability.

From the “voluntary guidelines” ERAC-GPC : risks and uncertainties of research, time lag, Correlation/causality, information circulation, ex-ante, costs/administrative burden… Open issues BUT also: cross-border joint actions…variable geometry! …flexible (case by case) and dynamic indicators… based on past experience: is enough? New tools…. Dynamics of concept based clustering

Evaluation of projects: a long established experience in the principles of the evaluation procedure, criteria, scoring, reports, panels. See as an example the guide for evaluators as this is published on the web: /bonus_calls/bonus_call_2012_viable_ecosystem/ /bonus_calls/bonus_call_2012_viable_ecosystem/ Open issues also in the projects’ evaluation? inter disciplinary panels, ranking in the virtual common pot…