1. 9. Practical use case 3: Pesticides Use Project
Blanaru Cristina
Eurostat
Unit B5: “Data and metadata services nsad standards”
SDMX Basics course, March 2016

2. PROCESS PHASES - PESTICIDES PROJECT
PHASE 2
PHASE 3
PHASE 4
Each SDMX project has 4 phases.

3. Phase 1: Preparation The legal acts:
Regulation (EC) No 1185/2009 of the European Parliament and of the Council of 25 November 2009 concerning statistics on pesticides
Commission Implementing Regulation (EU) No 1264/2014 of 26 November 2014

4. Phase 1: Preparation The parts involved: ESTAT+ NSIs;
Data providers: NSIs;
The collection and transmission of data for every 5 year period;
One single file per country, covering the whole 5 years period;

5. Phase 1: Preparation
The first deadline for transmission: December 2015
The format of the file: xml, send via eDAMIS;
The dataset name: AEI_PESTUSE_5;
The use of the MDT(internal database) like production system;

6. Phase 2: Compliance
June 2015:Settle on the matrix and the associated code lists;
Re-use of existing code lists and cross-domain code list: CL_FREQ, CL_OBS_STATUS, CL_UNIT_MULT, CL_DECIMALS;
Create specific code lists: CL_PESTICIDES, CL_CROPS, CL_MEASURE_PES;

7. Phase 2: Compliance
c) common standards for the collection, processing and dissemination (not always possible)
The purpose of statistical standardisation:
data about the same concepts are collected and communicated in the same way
facilitates data comparisons to be made across domains and over time.

8. Explanation of Pesticides Matrix - Overview Sheet ● summarises all concepts and code lists used
Concept ID
Concept Name
Description
Role
Level
Usage Status
Code list / format
FREQ
Frequency
Frequency of the series (e.g. A, Q, M).
Frequency dimension
SDMX+CL_FREQ+2.0
REF_AREA
Reference area
Reporting Country in ISO code (The country, or geographical/political group of countries that the measured economic phenomenon relates to)
Dimension
ESTAT+SCL_GEO_EUEFTACC+1.0
TIME_PERIOD
Time Period
The time period of the data.
ObservationalTimePeriod
PESTICIDES
Pesticides
Harmonised classification of substances
ESTAT+CL_PESTICIDES+1.1
CROPS
Crops
ESTAT+CL_CROPS+1.1
OBS_VALUE
Value
The value of the data
Primary measure
FLOAT
OBS_STATUS
Status flag
Status of the observation, such as normal, estimated or provisional
Attribute
Observation M
SDMX+CL_OBS_STATUS+2.0
CONF_STATUS
Confidentiality flag
Confidentiality status of the observation C
SDMX+CL_CONF_STATUS+1.1
TITLE
Title
Series
STRING
UNIT
Unit
ESTAT+SCL_UNIT+1.1
UNIT_MULT
Unit multiplier
Dataset
ESTAT+CL_UNIT_MULT+1.0
DECIMALS
Decimals
ESTAT+CL_DECIMALS+1.0
MEASURE
Measure
Cross sectional measure
Measure Dimension
ESTAT+CL_MEASURE_PES+1.0
QUANTITY
Quantity
Cross-sectional measure
AREA_TREATED
Area treated

9. Explanation of Pesticides Matrix – Matrix Sheet
Concepts
FREQ
REF_AREA
PESTICIDES
CROPS
OBS_STATUS
CONF_STATUS
UNIT
UNIT_MULT
DECIMALS
MEASURE
QUANTITY
AREA TREATED
Legend: # … Code list fully used in concept %… Code list partly used in concept (see detailed sheet) (blank) … Concept not used in table (code) … fixed single code from code list
Datasets
Dataset description
AEI_PESTUSE_5 % #
STATISTICS ON AGRICULTURAL USE OF PESTICIDES

10. Explanation of Pesticides Matrix – Matrix Sheet
Shows the relationship between the dataset(s) and the concepts
Each concept has a hyperlink pointing to the corresponding code list sheet
The cells link a dataset (row) to a concept (column)

11. Explanation of Pesticides Matrix – Matrix Sheet
The cells contain:
A # sign if the code list is fully used in the dataset;
A % sign if the code list is partiallly used in the dataset

12. Explanation of Pesticides Matrix – Code list Sheets
Showing the contents of each of the code lists used:
CL_FREQ sheet
CL_PESTICIDES sheet
CL_CROPS sheet
CL_OBS_STATUS sheet………

13. Phase 2: Compliance
drawn up the guidelines (B5 provides the draft to the business unit: E1)
Reviewed the guidelines by the business unit;
Develop a cross-sectional DSD: 2 values reported: for QUANTITY and AREA_TREATED
Provide the matrix to the IT production team (for their preparatory work)

14. Phase 3: Implementation
Finalize the guidelines (unit B5);
Prepare test files for the package and test the DSD (unit B5);
SDMX DSD (USE_PESTICI) is uploaded into the Euro Registry -
Send the SDMX package to the pilot countries (Portugal and UK)

15. The Conversion demo INPUT FILE FORMAT (CSV)
Data must be structured according to DSD
The header of the CSV file should look like this
FREQ;REF_AREA;PESTICIDES;CROPS;MEASURE;TIME_PERIOD;OBS_VALUE;OBS_STATUS;CONF_STATUS;UNIT;UNIT_MULT;DECIMALS
A; DE; F01_01; L1000; QTY;2 010;1000.0;A;D;KG;0;0
A;DE;F01_01;L1000;ATRT;2010;2000.0;P;C;HA;0;0
A;DE;F01_01_04;L3000;QTY;2011;1001.0;E;N;KG;0;0
A;DE;F01_01_04;L3000;ATRT;2011;2001.0;F;R;HA;0;0

16. The Conversion demo OUTPUT FILE FORMAT (xml)
Generate SDMX-ML files directly from the production system
Generate SDMX-ML files from files in other formats (e.g. CSV), using the SDMX Converter application

17. The conversion

18. Conversion – steps: Select Input file :(1)
Select Input format: CSV (2)
Select Output file (3)
Select Output format: CROSS_SDMX (4)
Select DSD: downloaded from the Euro Registry "USE_PESTICI+ESTAT+1.1.xml" (5)
Select Header file (6) or create the header (7)

19. Conversion – steps: Select Default Mapping (8)
Select CSV delimiter: normally ";" (9)
Select CSV Header Row value: USE_COLUMN_HEADER (10)
Convert (11)

20. Phase 3: Implementation
Received the test files from Portugal and UK via eDAMIS;
Validation of the test files by MDT Team;

21. Issues on the testing phase: 1
Issues on the testing phase: 1. the header unmarked in the Converter application

22. Issues on the testing phase: 1. The solution

23. Issues on the testing phase:
2. aggregation problem

24. Issues on the testing phase:
3. the wrong extension of the file sent (unk instead of xml);

25. Phase 4: Production SDMX compliant data exchange used in production;
Send the package to all countries;

26. Thank you for your attention. I am happy to take questions.
For further information please contact: