Presentation is loading. Please wait.

Presentation is loading. Please wait.

Eurostat 1 7a. Practical use case 1: Pesticides Use Project Blanaru Cristina Eurostat Unit B5: “Central data and metadata services” SDMX Basics course,

Published byBarnard Burns Modified over 8 years ago

Similar presentations

Presentation on theme: "Eurostat 1 7a. Practical use case 1: Pesticides Use Project Blanaru Cristina Eurostat Unit B5: “Central data and metadata services” SDMX Basics course,"— Presentation transcript:

1 Eurostat 1 7a. Practical use case 1: Pesticides Use Project Blanaru Cristina Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October 2015

2 Eurostat PROCESS PHASES - PESTICIDES PROJECT 2 PHASE 1 PHASE 2PHASE 3PHASE 4

3 Eurostat Phase 1: Preparation The legal acts: Regulation (EC) No 1185/2009 of the European Parliament and of the Council of 25 November 2009 concerning statistics on pesticides Commission Implementing Regulation (EU) No 1264/2014 of 26 November 2014

4 Eurostat The parts involved: ESTAT+ NSIs; Data providers: NSIs; The collection and transmission of data for every 5 year period; One single file per country, covering the whole 5 years period; 4 Phase 1: Preparation

5 Eurostat The first deadline for transmission: December 2015 The format of the file: xml, send via eDAMIS; The dataset name: AEI_PESTUSE_5; The use of the MDT(internal database) like production system; 5 Phase 1: Preparation

6 Eurostat June 2015:Settle on the matrix and the associated code lists; a)Re-use of existing code lists and cross-domain code list: CL_FREQ, CL_OBS_STATUS, CL_UNIT_MULT, CL_DECIMALS; b) Create specific code lists: CL_PESTICIDES, CL_CROPS, CL_MEASURE_PES; 6 Phase 2: Compliance

7 Eurostat c) common standards for the collection, processing and dissemination (not always possible) The purpose of statistical standardisation: data about the same concepts are collected and communicated in the same way facilitates data comparisons to be made across domains and over time. 7 Phase 2: Compliance

8 Eurostat Explanation of Pesticides Matrix - Overview Sheet ● summarises all concepts and code lists used 8 Concept IDConcept NameDescriptionRoleLevelUsa ge Stat us Code list / format FREQFrequencyFrequency of the series (e.g. A, Q, M). Frequency dimension SDMX+CL_FREQ+2.0 REF_AREAReference area Reporting Country in ISO code (The country, or geographical/political group of countries that the measured economic phenomenon relates to) Dimension ESTAT+SCL_GEO_EUEFTACC+1.0 TIME_PERIODTime PeriodThe time period of the data. Dimension ObservationalTimePeriod PESTICIDESPesticidesHarmonised classification of substances Dimension ESTAT+CL_PESTICIDES+1.1 CROPSCrops Dimension ESTAT+CL_CROPS+1.1 OBS_VALUEValueThe value of the dataPrimary measure FLOAT OBS_STATUSStatus flag Status of the observation, such as normal, estimated or provisionalAttributeObservationMSDMX+CL_OBS_STATUS+2.0 CONF_STATUSConfidentiality flagConfidentiality status of the observationAttributeObservationCSDMX+CL_CONF_STATUS+1.1 TITLETitle AttributeSeriesCSTRING UNITUnit AttributeObservationMESTAT+SCL_UNIT+1.1 UNIT_MULTUnit multiplier AttributeDatasetCESTAT+CL_UNIT_MULT+1.0 DECIMALSDecimals AttributeDatasetCESTAT+CL_DECIMALS+1.0 MEASUREMeasureCross sectional measureMeasure Dimension ESTAT+CL_MEASURE_PES+1.0 QUANTITYQuantity Cross-sectional measure AREA_TREATEDArea treated Cross-sectional measure

9 Eurostat Explanation of Pesticides Matrix – Matrix Sheet 9 Concepts FREQ REF_AREA PESTICIDES CROPS OBS_STATUS CONF_STATUS UNIT UNIT_MULT DECIMALS MEASURE QUANTITY AREA TREATED Legend: # … Code list fully used in concept %… Code list partly used in concept (see detailed sheet) (blank) … Concept not used in table (code) … fixed single code from code list Datasets Dataset description AEI_PESTUSE_5%##%#%%### STATISTICS ON AGRICULTURAL USE OF PESTICIDES

10 Eurostat Explanation of Pesticides Matrix – Matrix Sheet Shows the relationship between the dataset(s) and the concepts Each concept has a hyperlink pointing to the corresponding code list sheet The cells link a dataset (row) to a concept (column) 10

11 Eurostat Explanation of Pesticides Matrix – Matrix Sheet The cells contain: A # sign if the code list is fully used in the dataset; A % sign if the code list is partiallly used in the dataset 11

12 Eurostat Explanation of Pesticides Matrix – Code list Sheets Showing the contents of each of the code lists used: CL_FREQ sheet CL_PESTICIDES sheet CL_CROPS sheet CL_OBS_STATUS sheet……… 12

13 Eurostat drawn up the guidelines (B5 provides the draft to the business unit: E1) Reviewed the guidelines by the business unit; Develop a cross-sectional DSD: 2 values reported: for QUANTITY and AREA_TREATED Provide the matrix to the IT production team (for their preparatory work) 13 Phase 2: Compliance

14 Eurostat Phase 3: Implementation Finalize the guidelines (unit B5); Prepare test files for the package and test the DSD (unit B5); SDMX DSD (USE_PESTICI) is uploaded into the Euro Registry - https://webgate.ec.europa.eu/sdmxregistry/ https://webgate.ec.europa.eu/sdmxregistry/ Send the SDMX package to the pilot countries (Portugal and UK) 14

15 Eurostat The Conversion demo INPUT FILE FORMAT (CSV) Data must be structured according to DSD The header of the CSV file should look like this 15 FREQ;REF_AREA;PESTICIDES;CROPS;MEASURE;TIME_PERIOD;OBS_VALUE;OBS_STATUS; CONF_STATUS;UNIT;UNIT_MULT;DECIMALS A; DE; F01_01; L1000; QTY;2 010;1000.0;A;D;KG;0;0 A;DE;F01_01;L1000;ATRT;2010;2000.0;P;C;HA;0;0 A;DE;F01_01_04;L3000;QTY;2011;1001.0;E;N;KG;0;0 A;DE;F01_01_04;L3000;ATRT;2011;2001.0;F;R;HA;0;0

16 Eurostat The Conversion demo OUTPUT FILE FORMAT (xml) 1.Generate SDMX-ML files directly from the production system 2. Generate SDMX-ML files from files in other formats (e.g. CSV), using the SDMX Converter application 16

17 Eurostat The conversion 17

18 Eurostat Conversion – steps: Select Input file :(1) Select Input format: CSV (2) Select Output file (3) Select Output format: CROSS_SDMX (4) Select DSD: downloaded from the Euro Registry "USE_PESTICI+ESTAT+1.1.xml" (5) Select Header file (6) or create the header (7) 18

19 Eurostat Conversion – steps: Select Default Mapping (8) Select CSV delimiter: normally ";" (9) Select CSV Header Row value: USE_COLUMN_HEADER (10) Convert (11) 19

20 Eurostat Phase 3: Implementation Received the test files from Portugal and UK via eDAMIS; Validation of the test files by MDT Team; 20

21 Eurostat Issues on the testing phase: 1. the header unmarked in the Converter application 21

22 Eurostat Issues on the testing phase: 1. The solution 22

23 Eurostat Issues on the testing phase: 2. aggregation problem 23

24 Eurostat Issues on the testing phase: 3. the wrong extension of the file sent (unk instead of xml); 24

25 Eurostat Phase 4: Production SDMX compliant data exchange used in production; Send the package to all countries; 25

26 Eurostat 26 Thank you for your attention. I am happy to take questions. For further information please contact: ESTAT-SUPPORT-SDMX@ec.europa.eu

Download ppt "Eurostat 1 7a. Practical use case 1: Pesticides Use Project Blanaru Cristina Eurostat Unit B5: “Central data and metadata services” SDMX Basics course,"

Similar presentations

Survey Data Management and Combined use of DDI and SDMX DDI and SDMX use case Labor Force Statistics.

Survey Data Management and Combined use of DDI and SDMX DDI and SDMX use case Labor Force Statistics.
Overview of SDMX: Statistical Data and Metadata eXchange Technical and Content Standards for Statistical Data Ann McPhail, Division Chief Statistics Department,

Overview of SDMX: Statistical Data and Metadata eXchange Technical and Content Standards for Statistical Data Ann McPhail, Division Chief Statistics Department,
1 Annual National Accounts  1. Situation of OECD annual national accounts database  2. New features of the joint OECD-Eurostat questionnaire  3. COFOG2.

1 Annual National Accounts  1. Situation of OECD annual national accounts database  2. New features of the joint OECD-Eurostat questionnaire  3. COFOG2.
Implementation of SDMX for data and metadata exchange SDMX Basics Course 12 April 2013 Daniel Suranyi Eurostat B5 Management of statistical data and metadata.

Implementation of SDMX for data and metadata exchange SDMX Basics Course 12 April 2013 Daniel Suranyi Eurostat B5 Management of statistical data and metadata.
13-Jul-07 Implementation of SDMX for data and metadata exchange Balance of Payments Working Group 2-3 April 2012 Daniel Suranyi Eurostat B5 Management.

13-Jul-07 Implementation of SDMX for data and metadata exchange Balance of Payments Working Group 2-3 April 2012 Daniel Suranyi Eurostat B5 Management.
1 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX Basics Core Elements Information Model Data Structure Definition.

1 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics – October 2011 SDMX Basics Core Elements Information Model Data Structure Definition.
Slide 1 Eurostat Unit B3 – Statistical Information Technologies CoRD Meeting – 4 June 2007 Agenda Item 8 Preliminary ideas for a 2011 census hub Giuseppe.

Slide 1 Eurostat Unit B3 – Statistical Information Technologies CoRD Meeting – 4 June 2007 Agenda Item 8 Preliminary ideas for a 2011 census hub Giuseppe.
Situation of ANA data and metadata in 2008 Transmission to the OECD : progress report.

Situation of ANA data and metadata in 2008 Transmission to the OECD : progress report.
A Data Structure Definition for Eurostat's short-term business statistics Jan Planovsky, Eurostat SDMX Global Conference Bangkok, 28-30 September 2015.

A Data Structure Definition for Eurostat's short-term business statistics Jan Planovsky, Eurostat SDMX Global Conference Bangkok, September 2015.
Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services”

Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services”
Eurostat achievements and challenges Emanuele Baldacci, Director European Commission - Eurostat Director Methodology; Corporate statistical.

Eurostat achievements and challenges Emanuele Baldacci, Director European Commission - Eurostat Director Methodology; Corporate statistical.
Eurostat 4. SDMX: Main objects for data exchange 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October.

Eurostat 4. SDMX: Main objects for data exchange 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October.
SDMX IT Tools Introduction

SDMX IT Tools Introduction
SDMX and Metadata SDMX Basics Course 12 April 2013 Daniel Suranyi Eurostat B5 Management of statistical data and metadata.

SDMX and Metadata SDMX Basics Course 12 April 2013 Daniel Suranyi Eurostat B5 Management of statistical data and metadata.
2.An overview of SDMX (What is SDMX? Part I) 1 Edward Cook Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015.

2.An overview of SDMX (What is SDMX? Part I) 1 Edward Cook Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October 2015.
1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015.

1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October 2015.
SDMX IT Tools SDMX use in practice in NA

SDMX IT Tools SDMX use in practice in NA
7b. SDMX practical use case: Census Hub

7b. SDMX practical use case: Census Hub
Implementation of SDMX for Balance of Payments Balance of Payments Working Group 9-10 April 2013 BP-13-11 Daniel Suranyi Eurostat B5 Management of statistical.

Implementation of SDMX for Balance of Payments Balance of Payments Working Group 9-10 April 2013 BP Daniel Suranyi Eurostat B5 Management of statistical.
Eurostat 1 3.An overview of the SDMX implementation process Edward Cook Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29.

Eurostat 1 3.An overview of the SDMX implementation process Edward Cook Eurostat Unit B5: “Central data and metadata services” SDMX Basics course,

Similar presentations

Ads by Google