Monica Brezzi – Aline Pennisi - Simona De Luca Data analysis and indicator systems: our work with the Italian National Statistical Institute

Objectives of regional development written in Clarify, focus, measure public policies monitor and evaluate Socio-economic context describe Indicators = a set of quantitative variables useful for planning, project selection and monitoring and evaluation activities INDICATORS AND PUBLIC POLICY Community Support Framework (CSF) National Economic documents

a)Support implementation of policies and checking their progress b)Producing information useful for general knowledge on the programme and for evaluation purposes …..guaranteeing transparency on public policies USE OF INDICATORS AND MONITORING DATA POLICY MAKERS, MANAGING AND PAYMENT AUTHORITIES EVALUATORS, INSTITUTIONAL PARTNERS, CITIZENS Aggregate data on programme priorities Project level data

a)Indicators for the general objectives: macroeconomic model and breakthrough variables; b)Indicators of the potential of the areas concerned with development policies: context indicators and regional public accounts; c)Indicators of implemented projects : outcome, output and result indicators. WHICH ARE THE INDICATORS AND MONITORING DATA DEFINED IN THE ITALIAN CSF ?

FAILURE OF PREVIOUS EXPERIENCE ON MONITORING WHY DID ITALY CHOOSE THESE INDICATORS? (1) Member State’s and Regions’ needs not fully expressed and attitude of compliance; Evaluation demand weak; Difficulty to build a coherent framework of monitoring indicators; Difficulty to measure indicators chosen;  IMPOSSIBILITY OF “FILLING” THE INDICATORS CHOSEN;  INCAPABILITY OF MONITORING THE PROGRESS;  LITTLE KNOWLEDGE ON TERRITORIAL EXPENDITURE (BY THEME, PRIORITY, LEVEL OF GOVERNMENT)

EX ANTE EVALUATION OF THE CSF WHY DID ITALY CHOOSE THESE INDICATORS? (2) “Interpret” the EC framework to create a national framework; Provide definitions and methodological support; Build a quantified framework; Choose both bottom-up and top-down procedures (according to the nature of indicators); Great care on available indicators  SYSTEMS OF INDICATORS (AND LINKS AMONG THEM);  LEARNING PROCESS ON HOW TO DEFINE/ MEASURE INDICATORS AND MONITOR RESULTS;  TO VERIFY ADDITIONALITY

BREAKTHROUGH VARIABLES AND CONTEXT INDICATORS During PSM and ex-ante evaluation of CSF – written in CSF and ROP;  Central & Local Evaluation Units not yet established or not fully active; REGIONAL PUBLIC ACCOUNTS During PSM and ex-ante evaluation of CSF  Central Evaluation Unit and 21 Regional Units set up for this purpose; OUTCOME, RESULTS & IMPACTS INDICATORS After CSF and OP;  Central & Local Evaluation Units established and in most cases fully active. WHEN DID ITALY IDENTIFY ITS INDICATORS?

Indicators with the necessary characteristics were not always readily available Partnership with official statistics producers (co-decision) to search existing available information, collect and update information To define adequate indicators in specific sectors (or territorial level) not previously available Involving stakeholders and interested users (bottom-up) to build information at the appropriate level for monitoring and evaluation to extract local information otherwise not available To make regional administrations aware of using secondary information for policy planning and evaluation Formal agreement between DPS and ISTAT to produce statistical information to support regional development policies (around 7 mln euro) HOW DID ITALY GET THESE INDICATORS?

Typical difficulties in the availability of territorial data: insufficient timeliness the administrative subdivisions (Regions, Provinces, Municipalities) are usually preferred to functional breakdowns (such as employment basins, protected areas, other important areas for development policies, etc.) relevant sectors may not be covered… DPS – ISTAT Agreement to promote a greater production of statistics within the official national statistical system: Reliability, high quality, transparency and public dissemination Shared and standard methodologies, to ensure data comparability over time and space Easily accessible DPS-ISTAT AGREEMENT TO PROMOTE A WIDER PRODUCTION OF STATISTICS WITH TERRITORIAL BREAKDOWNS

PRODUCTS OF THE DPS-ISTAT AGREEMENT Type of projects financed under the DPS-Istat Agreement: Increased timeliness Extension of surveys to obtain regional / sub-regional breakdowns Territorial and multi-thematic databases Pilot studies Anticipation of regional economic accounts Household consumption survey: regional data on poverty since 2002; Survey on ICT use in enterprises; regional indicators since 2003 Survey on innovation in enterprises: regional indicators in 2006 Water supply and consumption census: regional indicators in 2006 CONTEXT INDICATORS DATABASE Database on infrastructure in italian provinces Database Health for All at province level INCIPIT Database on natural, cultural and quality-product resources at various levels of aggregation Extracting data from Business Register Regional environmental accounts Regional indicators on industrial production …..

THE CONTEXT INDICATOR REGIONAL DATABASE well-being; desired characteristics of regional economic systems (productivity, environment, labour market); quality of services; supply of infrastructure; demography and education A quantified description of disparities, gaps, potential and territorial competitiveness of Italian regions in terms of Database from official statistics containing 100 indicators with yearly values (starting from 1995) for each region (NUTS2 level) and macro area. The database is of public domain on the website and updated versions are released three times a year

Some examples of context indicators

What do we use context indicators for? 1.To pin down regional weaknesses and strengths. 2.To help focusing the objectives and to provide some direction for policy. 3.To increase the accountability for policy beneficiaries. How can these three tasks be addressed?  Using many indicators to describe the same phenomenon;  Partnership with the Regional Administrations in defining the indicators;  Measure the policy effort through setting targets for a small subset of indicators (by regional policy-makers); What we call a “soft use”

Year Source: Elaboration from NSO data How to address policy focusing? Percentage of families under the poverty line (regions ranked by quartiles)

Percentage of families with difficulties to access to health services year Source elaborations from NSO data Per head expenditure for investments in the health sector year Source RPA database Potential for policy focusing

Kg per capita of urban waste produced Potential for policy focusing Source, elaborations from APAT Multidimensional approach Total expenditure per waste produced (euro per tons) Source, elaborations from RPA database

How does the South perform with respect to Italy? What can policy do to fill the gap? Families perceiving irregularities in water distribution (as % of total families): Italy and South; Southern regions with target values Are there significant differences among regions? What can regional policy makers do to reduce the gaps? Ex-ante evaluation and definition of targets

Comparing the monitored impact with the national objective …

“New” indicators on information society Since 2001 national level surveys started in EU Member states coordinated by Eurostat. Now, since 2003, regional data is available too. Enterprises that have their own website/homepage - % (source: Enterprise survey on ICT usage)

 Context indicators were chosen when defining the CSF ;  At that time Regional Evaluation Units not yet established or fully active;  Authorities of the ROP selected the indicators and chose targets mainly to comply with EU requirements; USING CONTEXT INDICATORS  Indicators and targets have not been truly monitored by Regions;  Only recently Regions have started using them for evaluation;  The impact of context indicators on the public arena is still inadequate (although the OECD World Forum on Key Indicators…)  Economic and social partners have not extensively used the results of context indicators to pin down regional performances.  Media do not convey these data. …As a result

 Not enough partnership in selecting indicators (with administrations in charge of policies);  Not enough partnership (and accountability building) in choosing the targets;  “Cultural” difficulty to use quantitative information and to relate to measurable policy objectives;  Database and indicators were not easily accessible / downloadable;  Timeliness of the updating of the set of indicators not always in line with the official sector-publication on the data;  Analysis or evaluation based also on context indicators were not available at the beginning or in the past. MAIN CAUSES FOR A LIMITED USE OF CONTEXT INDICATORS FOR EVALUATION

 Indicators available on the web site of DPS and ISTAT and tables can be downloaded in Excel formatweb site of DPS ISTAT  Meta information available on the same websites  Newsletter produced after each database update Newsletter  Sharing experience in the use of indicators for evaluation  Context Indicators project presented in international meetings  Context indicators have been used in the update evaluations both from the Ministry of Economy and Regional Administrations  Analysis based also on context indicators are regularly included in the Annual Report of the DPS and in official documents of Ministry of EconomyAnnual Report of the DPS SOME INITIATIVES TO SUPPORT DISSEMINATION AND USE OF CONTEXT INDICATORS

STRENGTHS  Resources (human and financial) to produce the indicators, beyond the structural funds and OPs ;  Growing interest of “experts” (researchers, socio-economic institutions, evaluators etc.) WHAT CAN STILL BE IMPROVED AS REGARDS CONTEXT INDICATORS? WEAKNESSES  Policy makers, regions, managing authorities, social and economic partners are not sufficiently involved in selecting indicators and setting targets;  Some indicators are not satisfactory: a) too far from the actual policy actions (e.g. productivity of tourism industry, variation in the value of exports), b) proxies of what we would like to measure (e.g., household perception of the quality of services in public transport, water, etc.) c) breaks in time-series / changes of definitions to be handled

Objectives of regional development written in Clarify, focus, measure public policies monitor and evaluate Socio-economic context describe Indicators = a set of quantitative variables useful for planning, project selection and monitoring and evaluation activities INDICATORS AND PUBLIC POLICY Community Support Framework (CSF) National Economic documents National Strategic Reference Framework National Economic documents Lisbon and Goteborg strategy

Review of the current database – some “cleaning up” :  Intensify the link between the selected indicators and “new” policy priorities  Include regional quantification of Lisbon strategy structural indicators, where relevant  Specific NSRF work group discussing the possibility of including “hard” indicators on the provision and quality of a small number of “essential” public services (on social inclusion, waste management, water supply and education) New desirable features for the indicators and for the database:  Easy interpretation: policy makers must immediately understand how to read an improvement in the indicator’s trend  Comparable definitions across regions but also with other countries  Achieve a continuous on-line updating of the database  Use a thematic rather than a priority axis classification WHAT CAN BE DONE FOR THE NEXT PROGRAMMING PERIOD ?

Selection of indicators and targets:  Try to have policy makers suggest the indicators and targets (and use the technicians/experts only to guide the choice of measurement tools and check consistency)  Select a limited number of indicators with targets at the NSRF level  At the OP level agree on this small list; have each OP fix its own targets and specify the course-of-action / strategy expected to achieve such targets  Continue to refer to a more extensive list of context indicators for more general purposes WHAT CAN BE DONE FOR THE NEXT PROGRAMMING PERIOD ?

INCLUDING REGIONAL LEVEL DATA FOR LISBON STRUCTURAL INDICATORS (March 2006)

FOR FURTHER INFORMATION Website of Department for Development Policies (DPS) DPS Annual Reports, 2004 Annual Report on Actions in Under-Utilised Areas, available at Context Indicators database on the National Statistical Office’s website Newsletter on context indicators F. Barca, M. Brezzi, F. Terribile, F. Utili (2004), Measuring for decision making: soft and hard use of indicators in regional development policies, Materiali uval,n.2 available at