1. 18-20 Mayl-09 A distributed architecture for.. 1 A DISTRIBUTED ARCHITECTURE FOR STATISTICAL DATA PROCESSING AND DISSEMINATION G. Pongas and A Wroński Eurostat Meeting on the Management of Statistical Information Systems, MSIS 2009, Oslo

2. 18-20 Jul-09 A distributed architecture … 2 Outline Our idea of a distributed system System components Collaboration levels Conclusion

3. 18-20 Jul-09 A distributed architecture … 3 Purpose of the paper Present the architecture of a distributed statistical system able to –Promote collaboration among SA –Decrease overall burden on SAs –Increase data quality –Improve timeliness –Increase efficiency in data exchange Key message All the functions and data components must be described through metadata

4. 18-20 Jul-09 A distributed architecture … 4 Such a system should –Evolve from / integrate existing systems –Permit different collaboration levels –Not enforce a unique system Assumptions

5. 18-20 Jul-09 A distributed architecture … 5 Ultimate Distributed System Requirements Fast Data Access Distributed Process Automation Common Logical Data Model Acceptance Access Security Autonomy of Statistical Agencies’ Systems

6. 18-20 Jul-09 A distributed architecture … 6 System Components Distributed Agents (DA) Central Agents (CSA) Local Systems of the Statistical Agencies Web Clients Firewalls

7. 18-20 Jul-09 A distributed architecture … 7 Distributed Agent (DA): –Provide agreed processing on top of the common logical model –Provide connection between the common logical model and the Statistical Agency data model –Ensure secure access to the data of the Statistical Agency Central Agent (CSA) –Connects to Distributed Agents –Provides for data unification by using: Common model Unique data identification Global metadata and the metadata of the DAs. Component functionality

8. 18-20 Jul-09 A distributed architecture … 8 Local System –Provision of Data –Production Functions as usual Firewall –Intermediary between DAs and Local Systems. Provides security and filtering Web Client –Dissemination clients –Data editing and imputation clients –Complex (analysis oriented) clients –Content maintenance client Component functionality, continued

9. 18-20 Jul-09 A distributed architecture … 9 Data Dispatch Collaboration Data transmission occurs according to predefined formats and content Use of common transmission software SA1 SA2 SA3 Architecture options presented from the simplest to the ultimate

10. 18-20 Jul-09 A distributed architecture … 10 SA provides access to selected files stored in the DAs database (download facility) SA provides query software on top of the DAs database The above corresponds to the classical dissemination schema Pull Data Collaboration SA1SA2SA3 DA1DA2DA3 Client2 Client1

11. 18-20 Jul-09 A distributed architecture … 11 SA provides through the DA : Data and metadata database (warehouse) Query facilities Subscription facilities for automating data distribution Push Data Collaboration SA1SA2 SA3 DA1DA2 DA3 SA4 subscribed to DAs

12. 18-20 Jul-09 A distributed architecture … 12 Requires an intermediary level between DAs and Clients (The CAs) Unifies selected subsets of data coming from different DAs Can offer processing services using data coming from different DAs Avoids unnecessary data replication Advanced (Ultimate) Collaboration CSA1 SA1SA2SA3 DA1DA2DA3 CSA2 Web client1Web client2

13. 18-20 Jul-09 A distributed architecture … 13 Summary of functionality FunctionalityDispatchPullPushAdvanced Data dispatch√√√√ Download√√√ Web Services√√√ Queries√√ Computations√ Data combinations√ Computational services√ Collaboration Type

14. 18-20 Jul-09 A distributed architecture … 14 Conclusions Data federation can be gradual (à la carte) Data federation can coexist with the current legacy systems Data federation is based on common logical model and common / shared metadata Not only data and metadata can be shared but also functionalities Key message All the functions and data components must be described through metadata