1. Editing Building Block (EBB) Validation Tool for FDI and ITS Balance of Payments Working Group 02 April 2012 Unit B4, IT for Statistical Production Georges Pongas

2. Project Scope The current implementation covers the validations for the domains: Foreign Direct Investment International Trade in Services

3. EBB Introduction EBB allows users to import data, perform a set of predefined checks and operations on the imported datasets and export data resulted from the processing operation. The target of EBB is to be flexible and support virtually any set of validation rules (EBB Scripting Language). EBB can be integrated with 3 rd party databases (read/write) EBB can be integrated with 3 rd party applications by exposing the full EBB API as Web Services. 3

4. EBB Operations Validations / Computations –Record –Vertical –Hierarchical Dataset Operations –Copy dataset, Merge, Alter, Aggregate, etc. 4

5. Technology Overview Web Based Interface –Unified interface for both the local version and the server deployment –EUROSTAT Look & Feel –Light interface, simplified workflows RDBMS (Oracle and PostgreSQL) ECAS or local authentication 5

6. Local Deployment All components running locally on the user’s machine Tomcat server PostgreSQL database Local authentication 6

7. EBB Installation Launch EBB Installer Installation wizard guides the user through the process Enter destination folder for the installation Enter destination Start Menu EBB is installed in the designated location Shortcuts are created in the Start Menu 7

8. User Workflow Data Import Job Execution Job Results Data Export 8

9. User Module Functionality Change Password –Allows users to change their password Dataset Import/Export –Allows users to import and export data to and from the System as well as monitor any ongoing import/export processes Job Execution –Allows users to execute programs on imported datasets and view the results of the execution 9

10. Data Import EBB supports the following formats: –CSV –FLR –GESMES (supports time-interval data) The file structure is checked at the time of the data import step –number of fields –length –data types The import step is fault tolerant, a maximum number of errors (specified by the user) can be ignored by the process, resulting in partial data import

11. Validation Job In order to create a validation job the user must specify: –a validation program (must be pre-loaded in the system) –the imported dataset on which the validation needs to be performed The validation job checks for: –correct use of codes –duplicate records –integrity rules for different breakdowns Currently, the validation job does not report errors for missing information (incomplete breakdown information; except ITS flows)

12. Validations – codes lookup and duplicates The following lookup operations are performed in order to prevent invalid codes from being specified: –NACE rev 1.1 & 2 codes (FDI) –Reporting Country codes (FDI and ITS) Duplicate rows are checked based on the following set of columns: –frequency, reference area, adjustment indicator, data type bop, bop coded item, currency breakdown, counterpart area, currency, resident economic activity, non resident economic activity, time period, time format

13. Validations – ITS integrity rules The rules have been extracted from the Vademecum document Because missing data is not validated, all requests can be processed with the same program Counterpart Breakdown Examples –A1 = D3 + D5 (World) –D7 = DZ + EG + IL + JO + LB + MA + PS + SY + TN + TR (Mediterranean countries for years 2007..2009) –D7 = AL + BA + DZ + EG + HR + IL + JO + LB + MA + ME + MR + PS + SY + TN + TR (Mediterranean countries for year 2010)

14. Validations – ITS integrity rules Coded Item Breakdown Examples –206 = 207 + 208 + 209 (Sea transport) –981 = 245 + 249 + 253 + 260 + 262 + 266 + 268 + 287 + 291 (Other services) Flow Breakdown Examples –check for missing flows –4 = 2 – 3 (Net, credit, debit) –2 >= 0

15. Validations – FDI integrity rules The rules have been extracted from the Vademecum document Because missing data is not validated, all requests can be processed with the same program Counterpart Breakdown Examples –A1 = D3 + A5 + E2 + E4 + E7 + F2 + F7 + 7Z + Z8 (World for years until 2006) –A1 = V1 + A5 + E2 + E4 + E7 + F2 + F7 + 7Z + Z8 (World starting with year 2007)

16. Validations – FDI integrity rules NACE1 Breakdown Examples –2295 = 1805 + 2205 (total textiles/wood,paper) –6400 = 6410 + 6420 (post and telecom) NACE2 Breakdown Examples –C1895 = C1405 + C1805 (total textiles/wood,paper) –M6900 = M6910 + M6920 (legal/accounting activities) Coded Item Breakdown Examples –505 = 506 + 530 (assets) –555 = 556 + 580 (liabilities)

17. User interface – Data Import

18. User Interface – Select Program

19. User Interface – Create Job

20. User Interface – Job Results

21. User Interface – Error Report Statistics

22. Detailed Statistics Report

23. Testing Feedback The number of rows for lists with pagination should be customizable; to be added this year Specifying a custom tolerance (LHS = RHS +/- 3); currently a tolerance of 1 is added for each member of the expression; this can be changed to a parameter as a possible enhancement Rule E7 >= B2 + B7 + CA + US should take into account the YEAR of the data (Cuba has been added recently to both B2 and B7); fixed for the next release

24. EBB Rules Example Lookup Check CONDITION inLookup(REF_AREA, countryCodes, "CODE"); ERRMSG "Invalid country code" SEVERITY "Error" (REF_AREA); Check missing flows CONDITION countMissing(OBS_VALUE['4'],OBS_VALUE['2'],OBS_VALUE['3']) = 0; ERRMSG 'Data type (flow) missing' SEVERITY 'Error' (OBS_VALUE['4'],OBS_VALUE['2'],OBS_VALUE['3']);

25. 25 THANK YOU FOR YOUR ATTENTION QUESTIONS? The validation support team for the standalone version of EBB is reachable via the functional mailbox: ESTAT-VALIDATION@ec.europa.eu EBB?