1. Data collection of 2012: Data transmission standards and tools
ESTP course on Waste Statistics, April 2012
Hartmut Schrör, Eurostat

2. Contents Data transmission: What is changing?
What is SDMX? Why use it?
Tools for SDMX compliant data transmission
Webforms
SDMX converter
How to revise data for years ?
Reference documents and help
24/04/2012
Data transmission standards and tools

3. 24/04/2012
Data transmission standards and tools

4. Data transmission: phase-out of Excel tool
Outdated metadata in “Edawaste”
codes used for waste, waste operations, NACE are no longer standard
use of “standard code lists” in WStatR data
transcoding would be necessary
Labour intensive maintenance
country specific regional NUTS 2 breakdown
one Excel file required for each reporting country
Possible incompatibility with local PC installation
Version of Excel
authorisation to run macros required
24/04/2012
Data transmission standards and tools

5. Data transmission: phase-out of Excel tool
Macros for export of XML files
bugs in program code => duplicates
problems with special characters
Prevalidations in Excel sometimes not used
Member States sent Excel files rather than exported XML files
=> Prevalidations not performed
Standard tools available for SDMX
eDAMIS web forms
SDMX Converter
=> duplication of data entry tools should be avoided
24/04/2012
Data transmission standards and tools

6. Contents Data transmission: What is changing?
What is SDMX? Why use it?
Tools for SDMX compliant data transmission
Webforms
SDMX converter
How to revise data for years ?
Reference documents and help
24/04/2012
Data transmission standards and tools

7. Statistical Data and Metadata Exchange
Common data transmission standard supported by Eurostat, ECB, OECD, UN, World Bank, IMF, BIS
Purpose: easy exchange of statistical data by using
Harmonised metadata (codelists)
An open file format (XML)
“SDMX-ML”: SDMX compliant XML format
=> different IT systems (databases, programs...) understand the same “language”.
More explanations in the Manual on Waste Statistics, chapter 5
24/04/2012
Data transmission standards and tools

8. What is a DSD (data structure definition)?

9. Example of a DSD – waste codes
24/04/2012
Data transmission standards and tools

10. DSDs… define for each dataset
the dimensions composing it
e.g. GENER: nace_r2, geo, time, waste, hazard
the codes and labels of all elements in a dimension
the valid codes in a particular dataset
describe each dataset separately (GENER, TREATM, REGIO)
are noted in XML files
are stored in the SDMX Registry
24/04/2012
Data transmission standards and tools

11. The past: simple “CSV” file format
geo,time,waste,hazard,nace_r2,obs_value,obs_status,obs_conf
BG,2010,TOTAL,TOTAL,A,7022,,
BG,2010,TOTAL,TOTAL,C,,M,
BG,2010,TOTAL,TOTAL,C10-C12,639,,
BG,2010,TOTAL,TOTAL,C13-C15,2213,,
BG,2010,TOTAL,TOTAL,C16,1234,,C
BG,2010,TOTAL,TOTAL,C17_C18,4321,,C
BG,2010,TOTAL,TOTAL,C19,2222,,C
BG,2010,TOTAL,TOTAL,C20-C22,5432,,C
BG,2010,TOTAL,TOTAL,C23,7777,,D
BG,2010,TOTAL,TOTAL,C24_C25,1234,,D
BG,2010,TOTAL,TOTAL,C26-C30,4741,,
BG,2010,TOTAL,TOTAL,C31-C33,3560,E,
BG,2010,TOTAL,TOTAL,E,3026,,
BG,2010,TOTAL,TOTAL,E36_E37_E39,5090,, …
24/04/2012
Data transmission standards and tools

12. The present / future: SDMX-ML format
24/04/2012
Data transmission standards and tools

13. How to create these files?
24/04/2012
Data transmission standards and tools

14. Contents Data transmission: What is changing?
What is SDMX? Why use it?
Tools for SDMX compliant data transmission
Webforms
SDMX converter
How to revise data for years ?
Reference documents and help
24/04/2012
Data transmission standards and tools

15. web forms - overview Where to find them
functionality – how to enter data, etc.
limitations
advantages
24/04/2012
Data transmission standards and tools

16. Where to find the web forms
Go to eDAMIS Web Portal
logon with CIRCA user ID
go to “Transmission”, “web form entry”

17. Where to find the web forms
Select web form
WASTE_GENER_A2
WASTE_TREATM_A2
WASTE_REGIO1_A2 (incineration and recovery)
WASTE_REGIO2_A2 (landfills)

18. web form “GENER”

19. Web forms - functionality
typing in data
numeric values
flags (status, confidentiality)
comment on the table
locking headers (like “freeze panes” in Excel)
saving and re-opening
data are not transmitted when saved
“draft” status until web form is transmitted to Eurostat
copying and pasting e.g. from Excel
table layout must be identical
test before use
24/04/2012
Data transmission standards and tools

21. Web forms - functionality
Export / import using CSV format
export to CSV possible
editing in Excel
re-import into the web form
export to various other formats
24/04/2012
Data transmission standards and tools

22. Web forms – calculations and checks
Calculations of totals performed in the web forms
hazardous + non-hazardous
waste (EWC-Stat items)
NACE activities
treatment operations
No totals in REGIO tables on treatment facilities and capacities
on purpose, because regional data may differ from national data
sums to be entered manually
24/04/2012
Data transmission standards and tools

23. Web forms – checks numeric values only no negative values
warning (not ‘error’) if a data cell is empty
if it should be empty because the value is not available
leave it empty
set the “M” flag
justify warning
clear distinction between real zero and “missing” / “not available”
24/04/2012
Data transmission standards and tools

24. Why two web forms for “REGIO”?
Table on facilities and capacities
breakdown into NUTS 2 regions
energy recovery
incineration without energy recovery
other recovery
all landfills
no NUTS 2 breakdown for types of landfills
hazardous waste
non-hazardous waste
inert waste
Difficult to display in one web form
24/04/2012
Data transmission standards and tools

25. Why two web forms for “REGIO”?
treatment item number 1 2 3 4
Treatment categories 
Energy recovery (R1)
Waste incineration (D10)
Recovery (R2 — R11)
Landfilling (D1, D5, D12)
Population served by collection 3a
3 b
landfills for haz. waste
landfills for non-haz waste
landfills for inert waste
landfills total
Regions, NUTS 2 level
no. of facilities
capacity t/a
rest capacity m³
closed %
Region 1
Region 2
Region 3 … ….
National total

26. two web forms for “REGIO”
REGIO1 (all data with NUTS 2 breakdown)
energy recovery
incineration
other recovery
REGIO2
Landfills (total at NUTS 2 level, detail at national level)
“population served by collection” not to be reported in web forms but in quality report
coverage is not a waste operation => does not fit in any codelists used for table definition
only one value
24/04/2012
Data transmission standards and tools

27. web forms – limitations
Column width varies according to header, but cannot be changed.
long header => wide column
no word wrapping in header
Calculated totals in GENER and TREATM cannot be modified
but totals to be entered in REGIO
Only one column for flags
flag for “status” (estimated, etc.) and “confidentiality” combined in one column (drop-down list).
Reduced set of flags to comply with SDMX standard
24/04/2012
Data transmission standards and tools

29. flags – valid combinations
status flags
M: missing
E: estimated
P: provisional
No “R” for revised (rarely used).
confidentiality
C: primary confidentiality (regardless of what type)
D: secondary confidentiality (hidden to protect value with C flag)
No “A”, “B” flags
valid: M, E, EC, ED, P, PC, PD
24/04/2012
Data transmission standards and tools

30. web forms – advantages
Transmission of the dataset directly from the web form
no file to be saved on your PC or elsewhere
no separate manual transmission of the tables via eDAMIS
No trouble with
correct use of codes
file formats
local PC settings and installed software
only a web browser with Java is required
=> SDMX-ML files are generated and sent off automatically when clicking on “official transfer”.
24/04/2012
Data transmission standards and tools

31. SDMX Converter – what does it do?
Conversion of a text file to SDMX-ML format
source: CSV
target: XML file using SDMX standard
24/04/2012
Data transmission standards and tools

32. CSV file format
geo,time,waste,hazard,nace_r2,obs_value,obs_status,obs_conf
BG,2010,TOTAL,TOTAL,A,7022,,
BG,2010,TOTAL,TOTAL,C,,M,
BG,2010,TOTAL,TOTAL,C10-C12,639,,
BG,2010,TOTAL,TOTAL,C13-C15,2213,,
BG,2010,TOTAL,TOTAL,C16,1234,,C
BG,2010,TOTAL,TOTAL,C17_C18,4321,,C
BG,2010,TOTAL,TOTAL,C19,2222,,C
BG,2010,TOTAL,TOTAL,C20-C22,5432,,C
BG,2010,TOTAL,TOTAL,C23,7777,,D
BG,2010,TOTAL,TOTAL,C24_C25,1234,,D
BG,2010,TOTAL,TOTAL,C26-C30,4741,,
BG,2010,TOTAL,TOTAL,C31-C33,3560,E,
BG,2010,TOTAL,TOTAL,E,3026,,
BG,2010,TOTAL,TOTAL,E36_E37_E39,5090,, …
24/04/2012
Data transmission standards and tools

33. SDMX-ML format
24/04/2012
Data transmission standards and tools

35. What is required to use SDMX converter?
software “SDMX converter” + “Java Runtime Environment”
standard code lists used for WStatR
CSV input files
using standard codes, or
using other codes that are mapped to standard codes in SDMX converter
DSDs for WStatR
available in the SDMX Registry
24/04/2012
Data transmission standards and tools

36. Where to get SDMX converter?
download the latest version (currently 2.7.2) from CIRCA
Java based software and documentation
install on your PC
ask your local co-ordinator / IT support
24/04/2012
Data transmission standards and tools

37. What to do with the generated files?
Send XML files to Eurostat as before via eDAMIS
Send quality report along with data files
24/04/2012
Data transmission standards and tools

38. Contents Data transmission: What is changing?
What is SDMX? Why use it?
Tools for SDMX compliant data transmission
Webforms
SDMX converter
How to revise data for years ?
Reference documents and help
24/04/2012
Data transmission standards and tools

39. Revisions of data 2004 to 2008 Edawaste data entry tool?
pre-filled and reusable in many cases, but…
recoding would be necessary for
NACE (for 2004, 2006 to NACE Rev. 2)
waste (mapping to to 2010 waste breakdown)
all codes because of SDMX and new standard code lists
Update procedure is cumbersome and may lead to errors.
24/04/2012
Data transmission standards and tools

40. Revisions 2004-2008: suggested solution
Eurostat
extracts the country / table / year from our database into Excel
sends the Excel file to the country for editing
Country
edits the Excel table (without changing the layout)
sends the Excel file back to Eurostat via eDAMIS
copies and pastes the table contents back into the database
24/04/2012
Data transmission standards and tools

43. Contents Data transmission: What is changing?
What is SDMX? Why use it?
Tools for SDMX compliant data transmission
Webforms
SDMX converter
How to revise data for years ?
Reference documents and help
24/04/2012
Data transmission standards and tools

44. Reference documents and help
CIRCA directory with
guidance documents
quality report template
links to eDAMIS, SDMX Registry…
Ask your local eDAMIS co-ordinator
24/04/2012
Data transmission standards and tools

45. Thank you for your attention
24/04/2012
Data transmission standards and tools