1. Funding: STEER – INTELLIGENT ENERGY EUROPE (IEE) Grant Agreement IEE/10/290/SI2.589420 Full title: ECOSTARS Europe Acronym: ECOSTARS Project start date: June 2011 End date: May 2014 www.ecostars-europe.eu WP6 Evaluation State of play 06/03/2013 | 4th Consortium Meeting| Parma| Presented by: Michela Fioretto, ISIS

2. 17 September 2015 1 What have we done so far D6.1 – Project Evaluation Plan revised to: incorporate some missing information on Local Schemes (e.g. Ostrava); better clarify how to define the baseline measurements and which are the different interlocutors process evaluation questions are referring to; validate evaluation reference points of each local site. D6.2 – Engagement and Progress Database further developed to: be completely coherent with impact and process indicators reported in Evaluation Plan; allow scheme auditors to have comprehensive tool to be used for fleet audit; have more flexibility in upload functionality.

3. 17 September 2015 2 The Impact Evaluation Approach Key to impact evaluation is the identification and measurements of appropriate performance indicators that need to: describe the main characteristics of the situation at hand; be measured or estimated both before and after the scheme implementation. A comprehensive yet limited set of indicators should be defined and utilized by each city to assess the most relevant impacts in the following areas: Energy efficiency Environmental sustainability Operational management efficiency Engagement program effects Motivation and image perception Each city has defined its own set of indicators and the relative time scale of data collection.

4. 17 September 2015 3 Process evaluation The process evaluation intends to assess why the local scheme performed as they did and in what way internal or external factors played a role. The main insights of process evaluation concern the identification of drivers (or factors of success) and barriers that facilitate or hinder the successful implementation of the ECOSTARS scheme. The methodology relies primarily on questionnaires and interviews with the auditors and those involved in the engagement effort → questionnaire for process evaluation assessment will be addressed – with different questions – both to operators and scheme managers.

5. 17 September 2015 4 Impact indicators: cities timescale Today the Baseline indicators should be already collected!

6. 17 September 2015 5 The database was subject to slight delay due to the need of improving database functionalities to: – be completely coherent with impact and process indicators reported in Evaluation Plan; – allow scheme auditors to have comprehensive tool to be used for fleet audit; – have more flexibility in upload functionality. The online database is now ready (link ECOSTARS database) used by:link ECOSTARS database  the auditors to store information on the engagement of operators their progress of operators overtime.  the evaluators to store data on impact and progress measurement. After an e-mail of request, each partner receives a login and password to access the private area. Each partner is asked to inform ISIS about which are the operators already enrolled and to inform us about any newly operator involved. D6.2 Engagement and Progress Database

7. 17 September 2015 6 The ECOSTARS Database The database consists of 2 main sections: the auditing part the evaluation part The auditing section comprises 3 possible actions: 1.the vehicle audit 2.the fleet audit 3.the upload of the Audit Excel file The evaluation section allows to insert the indicators values for the baseline, the intermediate (if foreseen) and the final measurements as reported in the local evaluation plans of each city.

8. 17 September 2015 7

9. 17 September 2015 8 The Audit Excel file The Audit Excel file should contain the data of all the vehicles belonging to an operator fleet. By uploading such a file for each audited operator, the information concerning every single vehicle is automatically inserted in the database (avoiding the manual data entry for each vehicle through the so-called ‘vehicle audit’ action) and so the auditor can directly go to the fleet audit section. The Excel file should refer to 1 operator only. You need to create a file for each fleet audited. ISIS is taking care to upload the already existing Fleet Audit files  please, send the files to be uploaded on the database

10. Fleet Operators contacted Operators having joined the scheme N. of vehicles audited Mean number of stars received Rotterdam 634943.82 Parma 1968743.37 Basque Region 50199753.62 Swedish Regions 4073044.7 Edinburgh 120272’8284.29 Ostrava 8233411.6 South Yorkshire 350525’8253.92 17 September 2015 9 Local Scheme Audits state of play

11. 17 September 2015 10 The next Steps Each Local Auditor is asked to: Send to ISIS the Excel files of the already audited fleet in order to be uploaded on the database Send to ISIS the Road Map of each fleet operator audited in order to be attached online to the fleet audit of pertinence Collect and send to ISIS the set of baseline indicators (please, refer to the list reported in the Local Evaluation Plans chapters of D6.1) ISIS will : Give assistance to auditors for the use of the database; Give assistance to site evaluation managers engaged in the impact and process evaluation activities; Maintain and improve the database. please ask whenever you are in doubt!

12. ActionResponsible partiesDue by Send to ISIS the list of operators that joined the scheme Local Scheme Auditors15 th March 2013 Send to LSA user & password to access the DB ISIS As soon as the operators list has been received Send to ISIS the Excel file of fleet audits already done together with the Roadmap Local Scheme Auditors15 th March 2013 Upload the fleet audits and the related Roadmap on the DB ISIS29 th March 2013 Send to ISIS the consolidated impacts indicators list together with the baseline data collected (concerning the audited fleets) Local Scheme Auditors/Managers 29 th March 2013 17 September 2015 11 Actions Time Plan

13. 17 September 2015 12 Thank you Mario Gualdi –Michela Fioretto mgualdi@isis-it.commgualdi@isis-it.com–mfioretto@isis-it.commfioretto@isis-it.com Istituto di Studi per l'Integrazione dei Sistemi – ISIS Via Flaminia, 21 I-00196 Rome Italy Web: www.isis-it.comwww.isis-it.com

14. 17 September 2015 13 A before data collection is necessary to establish the qualitative-quantitative baseline against which compare the results of the ECOSTARS scheme implementation. Since the operators recruitment is an ongoing process as far as a new operator will be involved a new baseline for indicators ex-ante measurement will be set up. The Baseline measurement concept M12 ★★ Baseline Operator 1 Intermediate Operator 1 Baseline Operator 2 M23 M18 ★ ★ Intermediate Operator 2 M25 ★