1. Methodology for impact assessment of ESS.VIP projects (cost-benefit)
Christine Kormann Task Force Integrated Planning
Susanne Taillemite Finance
Sorina Vâju Social statistics- modernisation and coordination
Karo Nuortila Goods- production and international trade
17 June
RDG

2. Background Methodology builds on the work of two groups (2013):
Task Force "Impact analysis of ESS.VIP projects” (AT, EE, DE, IE, IT, NL, SE)- conducted a pilot (PRIX)
Sponsorship on Standardisation (business cases for European standardisation initiatives)
First presented to the ESSC in February 2013, refined and agreed by the ESS RDG and the DIME in June 2013
Today: information on recent and ongoing experience
In December: proposals for adaptations to the methodology to the ESS RDG

3. Principles of the methodology
Methodology conceived to assess ESS-VIP projects before they are launched.
Combines qualitative assessment based on a SWOT analysis and quantitative assessment of costs
Combines assessment at national and European level
Involves statistical and financial experts

4. Main steps

5. Qualitative analysis Strengths Weaknesses Opportunities Threats
Identify the different work packages that deserve a specific analysis
Establish the main criteria : fixed SWOT items (saving of resources, quality, burden…)
… and flexible SWOT items adapted to specific project needs, identified by a focus group
Scoring: all 24 SWOT items are assessed by each NSI on relevance and impact

6. Template SWOT rating (1)
Cost analysis feeds in item 2a

7. Template SWOT rating (2)
Study on burden feeds into item 3a

8. Quantitative analysis
Focus only on business processes affected by change
Identify cost drivers (impact of changes on the costs)
Estimate current, development and future costs of production
Cost categories: staff, IT solutions, overhead, other costs
If available, build on concrete experiences from countries
Costs for Eurostat + costs for NSIs
Conclusions of the quantitative analysis feed into the SWOT analysis.

9. Experience so far
Methodology conceived to assess ESS-VIP projects before they are launched
ESS.VIP ADMIN: implemented qualitative analysis; some suggestions for improvement of the methodology
ESS.VIP REDESIGN of Intrastat: Comparison of different options ongoing, qualitative and quantitative
Implementation of the methodology is a resource- intensive exercise

10. Experience so far
ESS.VIP ADMIN

11. Background Purpose: Contributes to ESS Vision 2020 key areas:
facilitate the increased use of administrative data sources in the ESS
without compromising on output quality
Contributes to ESS Vision 2020 key areas:
New data sources
Quality
Robust statistical process
Users and stakeholders
Adopted by ESSC in February 2015

12. Average scores (21 replies)
S W O T

13. Experience – focus group
Positive
High level of involvement
Consensus
Negative
Understanding of the SWOT methodology
Difficulty to classify SWOT items
Not enough time to agree on the full list or provide the explanations

14. Experience – apply and process questionnaire
Positive
Negative
Attempt of standardised assessment
No standard time frame
Explanations create partial overlaps between items
Fixed items
Deployment vs execution?
Disagreement with classification
Graph
Impact on own NSI or the ESS?
Important items may be missing
Understanding impact and relevance
Importance of comments for improving the business case

15. Recommendations 1. Improve the clarity of the instructions
Impact on their own NSI not ESS
Respondents justify their score
Explain relevance and impact
2. Allow more time for the workshop (3 hours)  

16. Recommendations 3. Reflect on possible changes to the methodology
Include more flexibility in the SWOT classification, the 12 items of strengths and weaknesses (internal to the project) could be put together and scored negative or positive.
Reflect on if an how the scope of the evaluation can be standardised

17. ESS.VIP REDESIGN of Intrastat
Experience so far
ESS.VIP REDESIGN of Intrastat

18. Background
In November 2011, the Council called upon ESS " … to take effective measures ensuring a substantial reduction of the response burden by redeveloping Intrastat, while maintaining a sound level of quality needed".
Launch of the SIMSTAT (ESS.VIP) project decided by ESSC in May 2012
Complementary project REDESIGN (ESS.VIP) decided by ESSC in November 2014
REDESIGN tasks
Options for re-designing Intrastat
Administrative burden reduction and cost-benefit assessment of each option
Key legal and methodological issues of each option

19. Identification of options
Q1/2015 – Q2/2015
Three options chosen to be assessed
SIMSTAT
Revised Intrastat
Single Flow
Options built around their main principles to make them distinctive enough to assess them

20. Identification of SWOT items
REDESIGN workshop
Group discussions on how the options are going to be assessed and to define SWOT items
Various aspects suggested to consider
Methodology originally defined to assess different projects, not options within a project => issues to be solved
Fixed SWOT items: some not suitable for the purpose of the project
Internal (S/W) and external items (O/T)
Quality items: depending on the options can be strengths or weaknesses

21. Identification of SWOT items
Proposal prepared by three Member States (DK, IE, UK)
Solutions
Fixed SWOT items: some not included in the list
Internal items: internal to Intrastat or inherent to the options
External items: aspects affecting stakeholders
Quality items: allocation to either strengths or weaknesses, some further work needed

22. Strengths Weaknesses a. Saving of resources
b. Improved comparability over space
c. Enhanced flexibility
d. Enhanced relevance of statistics
e. Ensured comparability over time
f. Enhanced re-usability of data
a. Costs (of development, transition, support and maintenance)
b. Complexity of changes
c. Lack of compatibility with national data needs
d. Lower accuracy of detailed output
e. Impact on timeliness
f. Dependency on other ITGS compilers

23. Opportunities Threats
a. Reduction of administrative burden upon respondents b. Enhanced standardisation c. Meeting stakeholder perceptions and expectations d. System durability/sustainability e. Compliance with the needs of internal users and stakeholders f. Demonstration of Vision 2020 implementation
a. Lack of consistency with national policies
b. Lack of support from financers
c. Disproportionate impact on smaller Member States
d. Timeline to implement the option
e. Reduction in trust/acceptance by users over quality of the data
f. Reduction in trust by PSIs over data security

24. Quantitative analysis
For different options
1. As-is situation
3. To-be situation
2. Development costs and adaptation costs
Current operating/ production costs
Operating costs after implementation
Evolutive maintenance costs

25. Roadmap quantitative analysis for "Redesign of Intrastat"
Steps
Roadmap
3.1.1 Determine the timeline of the impact analysis √
3.1.2 Identify the main areas of the GSBPM model affected by change
3.1.3 As-is situation
(To be finalised 30 June)
3.1.4 Grouping of countries
3.1.5 Identification of cost drivers
3.1.6 Development costs and adaptation costs
Deadline: 15 July
3.1.7 To-be state
September
3.1.8 Summary report
October

26. TO DO FOR 15 JULY: Estimate development costs and adaptation costs related to the different options
A distinction has to be made between development costs of the project at EU-level (i.e. not to be borne by each NSI), and costs for adaptation of the local system in each NSI.
Given the purpose of the ESS.VIP projects, main developments for the different options may be carried out by Eurostat or a small group of NSIs and are assessed by Eurostat in cooperation with the relevant NSIs.
Adaptation costs for each of the options have to be estimated for NSIs in each group based on the identification of the cost drivers.

27. State of play and next steps
Proposal discussed and welcomed by the ITGS SG beginning of June
Some work to be done for rewording some items and stressing the quality items
Written consultation in Q3/2015 to adopt the list of items
Assessment of the options Q4/2015
Summary and synthesis report Q1/2016