1. Evaluating Research Dynamics Using Network Analysis (in the context of EU funded R&D) Frank Cunningham DG Information Society and Media European Commission and Caroline Wagner George Washington University CES/AEA Conference, Toronto October 27 2005

2. Shift Towards Evidence Based Policy Making in EU  Commission has become more an ‘executive arm’ of the European Institutions  Manages a budget of €100 B /year  Rising (possibly) to €1000 B in period 2007-13  Shift towards ‘evidence-based’ policy development reinforced by enlargement to 25 Member States

3. EU R&D Framework Programmes  Currently in the 6 th Framework Programme  Total budget of €17B over 4 years (2003-6)  Information Society Technologies (IST) €3.8B  Annual EU investment in IST of €1B  EU investment about 3% of total IST-RTD in Europe.

4. Current Evaluation and Impact Assessment Methodologies  Commission has considerable experience in evaluation and monitoring of effectiveness of EU RTD investments over the last 15 years.  Focused mainly on ‘outputs’ rather than impacts.  Focused mainly on ‘project’ level and on the ‘private-return’ to RTD participants.  Now increased attention is given to results- based assessment and macro-economic and social impacts of EU RTD.

5. Gaining a better Understanding of Impact of EU RTD Funding  Work under way in DG INFSO since 2002  Established that EU funding has created an intensely linked network of European research collaborations.  New tools of ‘data mining’ and ‘link analysis’ were applied to EU IST databases.  Confirmed that network of collaborations is ‘small world’ with strong clustering  High degree of connectivity dominated by a small number of ‘hubs’

6. New Evaluation Methodologies  Innovative firms do not innovate in isolation but in interaction with other ‘actors’  In ‘Innovation Systems’ the links between innovation actors are crucial  Collaborative R&D Programmes are one important way of strengthening linkages  Partnerships through collaboration networks helps to achieve ‘critical mass’ of dispersed EU R&D resources.

7. Series of Evaluative Studies  DG INFSO is commissioning a series of evaluative studies to answer evaluative questions about the networking aspects and dynamics of:  IST RTD  Innovation & Deployment  Part of our strategy for network evaluation and impact assessment

8. Series of Evaluative Studies  1. ERAnets Study (RAND Europe)  (subject of this presentation)  2. Evaluation of Progress towards ERA for IST (CESPRI)  (subject of following presentation)  3. Networks of Innovation in IS Development and Deployment in Europe  (tenders in evaluation process)

9. ERAnets Study (RAND Europe)  Main Objectives were to Assess:  The degree to which IST Researchers in Europe collaborate.  How integration of IST Research in ERA changed with introduction of new FP6 Instruments.  Carried out from Jan 2004 to March 2005

10. Collaboration Has Grown Significantly Europe - 1990

11. Collaboration Has Grown Significantly Europe - 2000

12. ERAnets Study Breaking New Ground in Evaluation  An Evaluative study supporting the Evaluation and Monitoring Unit of DG Information Society  Analysed the networks in IST Research and compared the current networks emerging in the Framework Programme with those emerging in other frameworks and communities  Providing insight into the structuring effects of FP6 on European IST RTD  Developed, and refined tools and datasets that can be used by the INFSO Evaluation and Monitoring Unit and others

13. FP6 Networks of Excellence Produce Highly Clustered Network NoEs, call 1 retained, projects as cliques, component 1, diameter 3, average path length 2.1 (colour is number of steps away from the centre, from 1-3)

14. ERAnets Study – Main Findings (1)  Main Findings show that IST RTD funding at the European level is most effective in:  Connecting universities and business  Connecting research in different themes and disciplines  Integrating new Member States  Integrating key patent-holders; and  Integrating small businesses

15. ERAnets Study – Main Findings (2) Main Findings also show that:  FP6 network participants are more tightly interconnected than previous FPs  FP provides a major integrating function …however  Large institutes and companies act as ‘gate-keepers’  SMEs have been ‘crowded out’

16. Next steps: 2005-6  Broadened RTD network analysis to the “ERA in the Global context”: CESPRI (in 2005)  Extending network analysis from RTD to include innovation and deployment : to be contracted in 2006.  Clarifying the “causality links” between IST- RTD and FP and “Lisbon” objectives : Technopolis (10/04-8/05)  Initiating the systematic use of key indicators for monitoring the output and impact of IST-RTD at the Strategic Objective-level;  Identification and assessment of the capabilities of “system dynamic” model and tools for macro- economic/social impacts of investments in IST- RTD and ICT deployment : to be contracted;

17. Further Information DG INFSO Evaluation Activities http://europa.eu.int/comm/dgs/information_society/evalua tion/index_en.htm IST RTD (FP 6 Thematic Priority) http://www.cordis.lu/ist/ Communication: i2010 - A European Information Society for growth and Employment http://europa.eu.int/information_society/eeurope/i2010/ind ex_en.htm

18. Further Information Frank Cunningham DG INFSO Evaluation and Monitoring Unit European Commission Tel : + 32-2-296-8119 Office BU 31 0/52 Ave de Beaulieu 31 1160 Bruxelles Belgium Email: frank.cunningham@cec.eu.int