1. 11 CORE Architecture Mauro Bruno, Monica Scannapieco, Carlo Vaccari, Giulia Vaste Antonino Virgillito, Diego Zardetto (Istat)

2. 2 Outline Overview of CORE architecture Design & implementation issues Data Model IAPIs Repository GUIs and Web application 2

3. CORE Objective Provide a unique environment for: – Designing Statistical processes in terms of abstract services Exchanged data and metadata – Running Designed processes by invoking existing (wrapped) tools

4. CORE Design: Services Abstract services: specify a well-defined functionality in a technology-independent way An abstract service can be implemented by one or more concrete services, i.e. IT tools Examples: sample allocation, record linkage, estimates and errors computation, etc.

5. CORE Design: Services GSBPM classification – Documentation purpose – Provided that a CORE service can be linked to several IT tools, GSBPM tagging enables to perform searches retrieving, for instance “all the IT tools implementing the 5.4 Impute subprocess of GSBPM proposal”

6. CORE Design: Services Service inputs and outputs – Specified by logical names – Characterized wrt their “role” in data exchanges Non-CORE: if they are not provided by/to other services of the process, but are only “local” to a specific service CORE: otherwise, i.e. they are passed by/to other services and hence they do need to undergo CORE transformations

7. CORE Design: Data and Metadata They are specified as service inputs and outputs – Logical names link them to previously specified services – Non-CORE data do not need any further specification but the file system path where they can be retrieved

8. CORE Design: CORE Data 3 elements concur to the specification of CORE data – Domain descriptor – CORE data model – Mapping model

9. Domain descriptor model Entity Like Entity Relationships entities Entity properties Like Entity Relationships attributes Very simple (meta-)model: can easily describe “in fieri” models like GSIM

10. Example of Domain Descriptor

11. o1o1 Domain Descriptor: role Role of the domain descriptor: from service-to-service data mapping to service-to-global data mapping S1S1 i1i1 S2S2 i2i2 o2o2 i2i2 o1o1 O 1 mapped to i 2 Via ad-hoc mapping DD o1o1 i2i2 O 1 mapped to i 2 Via DD

12. 12 CORE Data Model Rectangular data set CORE tag: Data set level (mandatory) Column level (optional) Rows level (optional) Data set kind Column kind 12

13. CORE Data Model: role Specified once and valid for all processes Extensible, i.e. core tag, data set kind, column kind can be modified Adds more semantics to data – Example of usage: mapping to other models

14. 14 Mapping model Rectangular data assumption Mapping is intended to be specified wrt Domain Descriptor Columns are to be mapped to properties of an entity It contains the specification of how CORE data model concepts are associated to data 14

15. 15 CORE Logical Architecture GUI CORE Repository Integration APIs Process Engine Runtime SERVICES … 15

16. 16 CORE GUIs Process design – Ad-hoc customization of an existing tool (Oryx) Service design – Set of interfaces for the definition of services and related data flow Data design – Set of interfaces for the specification of domain descriptors and mapping files

17. Integration APIs Purpose: making a tool a CORE service – Translates inputs and outputs of the tool in a completely transparent and automatic way CORE Service

18. Repository Processes and their instances Services with their GSBPM and CORE classification Tools and their runtime features Data with their logical classification within CORE processes

19. 19 Process Engine Official statistics processes can be viewed from two perspectives: – Functional: they are data-oriented, reflecting a common feature of scientific workflows – Organizational: they are workflow-oriented, have the complexity of real production lines, with the need of harmonizing the work of different actors

20. 20 Process Engine Hence our process engine has two layers DATA FLOW CONTROL SYSTEM WF ENGINE Complex control flows Syncronizing constructs, cycles, conditions, etc. E.g.: Interactive multi-user editing imputation Simple control flows Sequence of tasks is composed by connecting the output of one task to the input of another Data intensive operations

21. 21 Implementation issues Java web application implementing: – GUIs – CSV-CORE Integration API – Data flow control system Layered design firmly based on frameworks: – Hibernate: database mapping – Struts2: model-view-controller approach Repository implementation: MySQL dbms

22. 22 Web Application Design Entities Model Data access DAOsServices View (GUI) Forms Input validation Controller Actions Struts2 Hibernate Business Logic

23. Architecture Deployment Web-based architectured centered on a centralized component – CORE Environment Different CORE deployments can co-exist – Intra- or Inter- organization Services can be remotely executed – Support is needed in the form of a distributed component for tool execution and data transfer

24. Types of service runtime Batch – Tool executed by a command line call – Can be automated Interactive – User interact with the tool through a tool-provided GUI – Cannot be automated Web service – No tool – procedure distributed on a web service actived by a programming language call – Can be automated

25. CORE Distributed Deployment GUI Definition Repository Integration APIs Process Engine Runtime CORE Environment Web service client Remote activation Runtime Runtime agent Batch-Interactive runtime Web service runtime Web container

26. Conclusions and Possible Future Work CORE implementation is a proof-of-concept prototype showing: – Real implementation of industrialized (standardized and automated) statistical processes – Reuse of IT tools possibly developed on different platforms and by different NSIs – GSBPM-aware services implementation – A unique common data model enabling integration of heterogeneous data exchanged between services – Openess to evolving statistical information models (e.g. GSIM) through a dedicated slot

27. SUMMARIZING: What we will see in the DEMO