Draft Evaluation Methodology Futurreg 4 th Steering Committee Logroño, 28 th November 2006

E valuation: “The evaluation of public intervention consist of judging its value in relation to explicit criteria, and on the basis of information that has been specially gathered and analysed.” I ntrinsic features: Description Regulatory dimension Instrumental dimension O thers features: Scope of public policy evaluations. Controlling public expenditure and improving administration. Evaluation should be more than just an instrument of control to measure complex features. Key actors involved in the process.

¿ W hy evaluate? Verify programme effectiveness and efficiency Learn Aid reflection To see what is happening? Aid transparency

+ Futurreg Foresight as a technique for decision making in organisations. Evaluation strengthens the stated purpose of the project to provide a learning tool for European regions. Evaluate to ensure transfer is as good as possible. Foresight must be evaluated in context. Evaluate whether the results and conclusions are used and/or taken into account in the region.

¿ W hich kind of evaluation is required? On going supplements the monitoring system and broadens the capacity to manoeuvre. Ex post attempts to measure project effectiveness and efficiency.

E valuation structure Context Methodology Data collection Data analysis Global analysis and selection of cases for study Summaries of results and/or impacts Communication of results and recommendations

E valuation Methodology Analysis of the project in two dimensions: Analysis of the project as a whole Analysis of foresight applications performed in Futurreg. Applications grouped into different categories. Indicators Indicators concerning the project as a whole, its objectives, toolkit and foresight methods. Three questionnaires for different actors involved in the project Partners (7), Application managers (14-15) and Participants in foresight applications (case of studies)

A nalize the project as a whole (I) As Futurreg is part of the INTERREG IIIC programme, its contribution to the programme objectives through this project needs to be evaluated. Importance to evaluate aspects like cooperation and project partner inter-communication, partner experience exchange, the toolkit, etc. Project as learning experience for participating regions and as a methodological guide for future experiences in other European regions in the futures and foresight field

A nalize the project as a whole (II) At this level, the main questions for evaluation are: Were the objectives well defined? Have they been achieved? Have the expected results been obtained? Was the workplan carried out? Have the deadlines and planned costs of the project been respected? Have the results been transferred from the project to other regions?

A nalyze the project as a whole (III) Indicators: Project activity and working plan indicators (objective indicators) Result indicators Others indicators

Indicators (I)

Indicators (II)

Indicators (III)

E valuating applications (I) Critical areas (1) were used in classifying applications: S ocial, factors influencing human beings, society and lifestyle T echnological, factors stimulated by science and its applications E conomic, factors affecting industry and wealth creation E nvironment, factors impacting on the physical world we living P olitical, factors relating to government and administration [1] Preliminary version of the UPGRADE blueprint has been produced for the dissemination conference “Building the future on knowledge”. Brussels, September 23, 2004.

S ocial T echnological E conomical E nvironment P olitical Aristotle University of Thessaloniki Main theme: Innovation, new product development. Type of application: Strategy development towards technology and knowledge transfer.. Galway-Mayo Institute of Technology Main theme: Innovation in SME Type of application: Product Development and Innovation Planning. Institute of Technology, Sligo Main theme: Higher education Type of application: Develop strategy Municipality of Thermi Main theme: Digital Research Centre: Digital Cities, e- goverment. Type of application: regional policies, organizational restructure. ADER, INESCOP and AICCOR Main theme: Innovation, research and technology. Type of application: Identification of Future Technological trends in the footwear Sector. Malta enterprise Main theme: Innovation. Type of application: Development of a regional innovation strategy. University of Malta Main theme: Knowledge Transfer Type of application: development of knowledge transfer strategy within the University´s ten year vision. Loimaa Region Development Centre Main theme: Economic development (entrepreneurship) Type of application: Create networks and star up new joint projects among entrepreneurs. Regional Council of Satakunta Main theme: manages the responsability of the long term regional development, defines objectives of the long term development and provides guidelines for the member comunities of land uses. Type of application: preparation of a new Regional strategic plan Countryside Council of Wales Main theme: Conserving tha natural enviroment and helping people to enjoy it and earn from it. Type of application: Developing the organisation´s strategic direction in medium term. Menter a Busnes Main theme: Economic value of the Welsh Language. Type of application: Company development and strategic contribution to policy framework and programme for action. Fondation Rurale de Wallonie Main theme: Support for rural development operations and on going education.. Type of application: diagnosis on future needs of rural territories of Wallonia in order to define the new strategy of the organisation FRW. Support existing working groups of the organisation. SCRL Main theme: Territorial development, tourism and patrimony, agriculture and sustainable energies. Type of application: improve the use of foresight in rural areas of Wallonia. Athone Institute of Technology Main theme: Innovation/Sustainable development. Type of application: Development of a strategy/Organizational development.

E valuating applications (II) Case study selection criteria: Operational level and situation of exercise. Design of foresight exercise. Relevance of issues on which the foresight exercise is carried out. Territorial balance according to types and regions involved in the project. Novelty.

E valuating applications (III) The main questions (1) the evaluation considers at this level are: 1. Introduction of foresight in the region and complementarity to other regional actions. Are the defined objectives consistent with the needs of the environment? Are they defined in detail? Has foresight implementation design taken into account regional characteristics and problems? Is foresight integrated and complemented by other types of regional actions, with which it shares common objectives? 2. Target hits Has the project fulfilled its general objectives? Were results as expected? (1) The questions are a result of the collective work of the partners and of the conclusions propoused in the last up skilling workshop

E valuating applications (IV) 3. Foresight as a useful tool Has foresight proved its use as a tool for partners and regional organizations? What type of method was used? Were the methods used appropriate for the targets set? To what extent was the foresight guide useful in the development of applications? 4. Foresight as a participatory process What is the role of private and public agents in the development of the applications? Was civil society involved? Was the foresight practice carried out as an open, participatory process? 5. The role of experts, stakeholders and external consultants What role did foresight experts, stakeholders and external consultants play in the performance of the project and applications?

E valuating applications (V) 6. Application of results Have the results of the foresight applications been applied in the regions? 7. Diffusion of regional foresight results Have other European regions been informed of the programme results? Do regional agents know about regional foresight applications and their results? 8. Extension and sustainability of the foresight culture In view of the foresight experience in the region, has the foresight culture developed in the region? Are regional organizations showing interest in maintaining and using this tool?

E valuating applications (VI) Indicators: Indicators on Futures toolkit Indicators on foresight methods

Indicators (I)

Indicators (II)

Indicators (III)

Indicators (IV)

Q uestionnaries Presented below are three kind of questionnaires for agents taking part in the Futurreg project: Questionnaire for partners The questionnaire is designed to obtain information on how the working plan developed and the measures worked, on the information generated, what type of difficulties were founded, etc. Questionnaire to application agents Sent to the agents of all regional applications. It will provide outstanding information on the management of applications, their design, how they worked, the methods used, the measures developed once foresight had concluded, the role of the experts, etc. Questionnaire to participants in applications This questionnaire will only be sent to participants in regional applications selected as case studies.

Evaluation methodology:  Presentation of Draft. 28 November  Deadline for comments. End of December  Methodology evaluation document. 15 of January. On-going evaluation:  Evaluation criterion. 28 November  Indicators. 28 November  Evaluation of applications (selection of a specific number of applications). 30 November  Questionnaire 28 November  Data processing. 31 January  Draft ongoing evaluation report. 31 May  Ongoing evaluation report. 29 June Evaluation Ex-post evaluation:  Evaluation criterion. 31 August  Indicators. 31 August  Evaluation of applications (selection of a specific number of applications). 28 September  Questionnaire. 28 September  Data processing. 31 October  Draft ex post evaluation report. 30 November  Ex post evaluation report. 28 December C hronogram