New Frontiers? New frontiers in Evaluation April 24th and 25th, Vienna Dorothea Sturn

Seite 2 The FFG (Austrian Research Promotion Agency) founded September 2004 Out of four forerunning organisations –Forschungsförderungsfonds für die gewerbliche Wirtschaft (FFF) –Technologie Impulse Gesellschaft (TIG) –Austrian Space Agency (ASA) –Büro für internationale Technologiekooperationen (BIT) As a result of an evaluation?! FFF should be merged into a broader innovation agency. The proposed merger with TIG, BIT and ASA appears to be a reasonable option for achieving this although other configuration would also be possible. (FFF/FWF evaluation) Mio Fundig Budget 182 people employed

Seite 3 Geschäftsführung RevisionStrategie Basis- programme (BP) Struktur- programme (SP) Thematische Programme (TP) Europäische und internationale Programme (EIP) Interne Services Agentur für Luft- und Raumfahrt (ALR) Organigramm

Seite 4 Managers Ignore all strategic arguments Feel bothered Defend projects Policy Makers Ignore results from evaluators Overworked decide intransparent Evaluators Ignore questions from policy Behave as scientists Traditional world of ignorance

Seite 5 Did anything change? YES Rise of formative and interactive approaches Policy questions are taken into account Evaluators give important inputs in decision-making NO ALL parties are overworked and busy, higher level of policy makers is absent Other influences and interests are stronger Arguments and results of evaluations are both complex and difficult to implement (please give me an evaluator with only one hand)

Seite 6 Examples The linear model Portfolio analysis and Goal overload Additionality

Seite 7 The linear model What evaluators tell us: The linear model is a rare case What we do: For each Euro we spent must be declared whether it is basic research or applied research

Seite 8 Why? Small enterprise Medium-sized enterprise Large enterprise Fundamental research100% Industrial research65%60%50% Experimental development35%30%20% Simple answer: Commission wants it More complex answer: Difficult distribution of funding among different interest groups Public opinion: who gets how much money out of the organisation

Seite 9 Portfolio Evaluation What evaluators tell us: Do not only look at a single programme – you have to optimise the whole portfolio What we do: We are introducing a new programme twice a year. As a result, we have an enormous amount of small and byzantine programmes with overlapping goals

Seite 10 Why? It is nearly impossible not to include the usual catchwords e.g. SMEs, technology transfer, excellence It is nearly impossible to let a programme peter out It is nearly impossible not to jump on a new thrilling topic, if other countries do so It is nearly impossible not to do something new, since newspapers more and more like the research topic

Seite 11 Additionality What evaluators tell us: Additionality is complex to assess What we ask: Where we get the best value for our money?

Seite 12 Policy maker: I want to make it faster Expert: It is the motor which is responsible for the speed Policy maker: next generation car will consist of at least 80% motor Is it possible to concentrate on measures showing high additionality?

Seite 13 How we can improve? Evaluation should be: Support for strategic management A context dependent managerial activity (Philipe Larédo)

Seite 14 What evaluators can do: Combine formative and summative parts Integrate elements that contribute to the actual decision-making and policy design (without loosing independency) Organise the evaluation process as a process of interaction Be brave – even if it means biting the hand which feeds you – but don't be too cynical, understand that there are multiple and conflicting objectives, interests and stakeholder

Seite 15 What funding agencies / policy makers can do: Increase the organisation's strategic intelligence, absorptive capacity and evaluation know-how at all levels (including CEO) Monitoring and information system: Collect all data you use and use all data you collect Contact evaluators at the beginning of a process of change Use experts but do not abuse them Let evaluation become part of an active learning process