24/5/2011 MSIS 2011 – 23-25 May 2011, Luxembourg 1 Statistical data editing near the source using cloud computing concepts George Pongas, Christine Wirtz.

Slides:

Advertisements

Similar presentations

Chapter 1: The Database Environment

Advertisements

Building Portals to access Grid Middleware National Technical University of Athens Konstantinos Dolkas, On behalf of Andreas Menychtas.

Welcome to Middleware Joseph Amrithraj

Web Services for an Intelligent Tutoring System that Operates as a Virtual Reality Game Maria Virvou, George Katsionis Department of Informatics University.

Programming Basic Concepts © Juhani Välimäki 2003.

1 OBJECTIVES To generate a web-based system enables to assemble model configurations. to submit these configurations on different.

OPERATIONAL METADATA FOR FEDERATING STATISTICAL REFERENCE SYSTEMS AT EUROSTAT G. Pongas, F. Vernadat EC Eurostat B2.

Distributed Database Management Systems

1 CS 502: Computing Methods for Digital Libraries Lecture 22 Repositories.

Fundamental System Concepts Asper School of Business University of Manitoba Systems Analysis & Design Instructor: Bob Travica Updated: September 2014.

Introduction to Web Applications Instructor: Enoch E. Damson.

© 2003, Prentice-Hall Chapter Chapter 2: The Data Warehouse Modern Data Warehousing, Mining, and Visualization: Core Concepts by George M. Marakas.

Software Engineering for Cloud Computing Rao, Feng 04/27/2011.

Metadata driven application for aggregation and tabular protection Andreja Smukavec SURS.

9 Feb 2004Mikko Mäkinen & Saija Ylönen Joint UNECE/Eurostat/OECD work session on statistical metadata (METIS) Geneva, 9-11 February 2004, Topic (ii): Metadata.

WP.5 - DDI-SDMX Integration

Geneva, 30 October 2009 Giuseppe Sindoni, Istat, Italy An online system for multi-channel, register-based census data collection.

1 Meeting on the Management of Statistical Information Systems (MSIS 2010) (Daejeon, Republic of Korea, April 2010) NIS ICT Strategy in the Production.

C Copyright © 2009, Oracle. All rights reserved. Appendix C: Service-Oriented Architectures.

Cloud Computing. What is Cloud Computing? Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable.

Metadata management and statistical business process at Statistics Estonia Work Session on Statistical Metadata (Geneva, Switzerland 8-10 May 2013) Kaja.

SDMX and DDI Working Together Technical Workshop 5-7 June 2013

Cloud computing.

IPlant Collaborative Tools and Services Workshop iPlant Collaborative Tools and Services Workshop Overview of Atmosphere.

2 Systems Architecture, Fifth Edition Chapter Goals Describe the activities of information systems professionals Describe the technical knowledge of computer.

NOAA Center for Tsunami Research Tweb: A Web-Based Tsunami Model Forecast Tool System Architecture and Development Eugene F. Burger 1, Linus Kamb 2, Tracey.

Marakas: Decision Support Systems, 2nd Edition © 2003, Prentice-Hall Chapter Chapter 10: The Data Warehouse Decision Support Systems in the 21 st.

File Systems and Databases Lecture 1. Files and Databases File: A collection of records or documents dealing with one organization, person, area or subject.

Eurostat Expression language (EL) in Eurostat SDMX - TWG Luxembourg, 5 Jun 2013 Adam Wroński.

Plan  Introduction  What is Cloud Computing?  Why is it called ‘’Cloud Computing’’?  Characteristics of Cloud Computing  Advantages of Cloud Computing.

Metadata driven application for data processing – from local toward global solution Rudi Seljak Statistical Office of the Republic of Slovenia.

Francesco Rizzo (ISTAT - Italy) Stefano De Francisci (ISTAT – Italy) An integration approach for the Statistical Information System of Istat using SDMX.

Editing Building Block (EBB) Validation Tool for FDI and ITS Balance of Payments Working Group 02 April 2012 Unit B4, IT for Statistical Production Georges.

Electronic data collection System in CSB of Latvia By Karlis Zeila, Vice President, CSB of Latvia UN ECE Work Session on Statistical Data Editing, 16 –

Electronic data collection System in CSB of Latvia By Karlis Zeila, Vice President, CSB of Latvia IT DG meeting, October , Eurostat.

SDMX IT Tools Introduction

Open GSBPM compliant data processing system in Statistics Estonia (VAIS) 2011 MSIS Conference Maia Ennok Head of Data Warehouse Service Data Processing.

1 The EDIT System, Overview European Commission – Eurostat.

EDIT – Eurostat’s editing tool

An Enterprise Clinical Data Search Solution. is Designed for: Informatics professionals, clinicians, statisticians, data managers and process/quality.

5.8 Finalise data files 5.6 Calculate weights Price index for legal services Quality Management / Metadata Management Specify Needs Design Build CollectProcessAnalyse.

Metadata Driven Integrated INFORMATION SYSTEM of CSB of LATVIA Version Central Statistical Bureau of Latvia April 5 – 9, 2008 / Luxembourg Presentation.

IT Directors’ Group Meeting October 2010 Sharing data validation tools in the ESS Christine WIRTZ – Head of Unit B3 Georges PONGAS – Unit B3 Daniel.

15-16 December 2010 CGST Meeting 1 IT Developments TRIS 1 – TRIS 1 / TRIS 2 Item 7.1 on the agenda 1 TRIS = TRansport Information System.

Eurostat Sharing data validation services Item 5.1 of the agenda.

Database application development 1. Chapter 8 © 2013 Pearson Education, Inc. Publishing as Prentice Hall OBJECTIVES  Define terms  Explain three components.

TRITON - An event driven SOA architecture MSIS Jakob Engdahl, Statistic Sweden

IT 5433 LM1. Learning Objectives Understand key terms in database Explain file processing systems List parts of a database environment Explain types of.

Administrative Data and Official Statistics Administrative Data and Official Statistics Principles and good practices Quality in Statistics: Administrative.

CLOUD ARCHITECTURE Many organizations and researchers have defined the architecture for cloud computing. Basically the whole system can be divided into.

Software Testing.

Basic 1960s It was designed to emphasize ease of use. Became widespread on microcomputers It is relatively simple. Will make it easier for people with.

Hardware and Software Hardware refers to the physical devices of the computer system e.g. monitor, keyboard, printer, RAM etc. Software is a set of programs,

Chapter 9: The Client/Server Database Environment

Introduction to Cloud Computing

Introduction to Enterprise Systems

EUROSTAT Unit B3 IT for statistical production Ewa Stacewicz

Tomaž Špeh, Rudi Seljak Statistical Office of the Republic of Slovenia

Lecture 1 File Systems and Databases.

Chapter 1: The Database Environment

The Database Environment

Sharing data validation activities in the ESS.

Cloud Computing: Concepts

Fundamental Concepts and Models

Data validation handbook

Data processing German foreign trade statistics

Statistical data editing near the source using cloud computing concepts George Pongas, Christine Wirtz -Eurostat.

European Census Hub: a cooperation model for dissemination of EU statistics Paper prepared by Ioannis Xirouchakis Presentation: Christine WIRTZ, Eurostat.

EDIT data validation system Ewa Stacewicz EUROSTAT VALIDATION TEAM

Validation Activities in the ESS What you will hear today…

Presentation transcript:

24/5/2011 MSIS 2011 – May 2011, Luxembourg 1 Statistical data editing near the source using cloud computing concepts George Pongas, Christine Wirtz -Eurostat MSIS 2011 – May 2011, Luxembourg

24/5/ Editing near the source Accelerates speed of final delivery to users and institutions Checks and imputations are near the respondent Data knowledge is frequently more profound in the primary collector institutions Logical proximity is better than physical: Data and application sharing

24/5/ Cloud and SOA in few Lines Separates ownership and usage of data storage computer power and application development and execution (cloud) Cloud variants are IaaS, PaaS, SaaS Cloud architectures are: – Public –Private –Mixed –Community Based on web technologies and independent software components to interlink on demand (SOA)

24/5/ Data Editing in Eurostat High volume of arrivals (> per year) Format heterogeneity Data checking absorbs substantial volume of human resources Erroneous data imply communications with MS Eurostat as a rule does not Impute… Interest to have a Common distributed solutions

24/5/ Eurostat’s web enabled system for editing (Editing building block (Ebb) Completely Metadata Driven Exists in 2 versions: –PC version –Web-based version Technologies used: –ANTLR –Java –Tomcat or Weblogic –Hibernate –Postgres or Oracle

24/5/ EBB Information Flow

24/5/ Implementation Details EBB is written using a set of Web services of the following types: –Administration –Program –Job

24/5/ EBB functionalities Support of categorical, text and numeric variables Separation of programmer and user interfaces Conditional and unconditional rules Multi-record rules Deterministic imputation Use of auxiliary data File operations Special functions (unicity, duplication checks...) Outliers (HB, Sigma Gap, Terror) Input/output of data/metadata Reporting

24/5/ Usage until now Embedded in SAS (for microdata editing) To distribute to data providers as standalone version –FDI (foreign direct investments) –ITS (international trade in services) –SBS (structural business statistics) –CVTS (continuous vocational training survey), –AES (adult education survey)