1. EUDAT operations perspective and initial achievements Johannes Reetz EGI Technical Forum 2012 EGI/EUDAT/PRACE operations workshop Prague, 18 Sep 2012 eudat.eu

2. raw data description registration preservation citable publication analysis enrichment referable data temporary data increasingly automatic processes as well as crowd sourcing increasing volumes Knowledge Creation Cycle is changing 2

3. Lossless separation of content and carrier in the digital domain changes the world - a revolution comparable with the invention of book printing Data creators are not known personally to data users anymore - we need to solve the trust problem Science and research are one of the primary drivers for the data tides Data accessibility changes nature, pace and direction of research Diversity in many dimensions is the dominant feature of many scientific information and this is expected to increase due to the inherent innovation forces Technology allows to involve the citizens in different roles – not just as consumers, also as contributors of valuable information (crowd sourcing), increasing volume and complexity Increasing pressure towards open access Setting the Scene (1) 3

4. Repurposing data within and across disciplinary domains requires significant efforts on integration and interoperability (e.g. wrt meta data), as well as agreement on standards and sharing practices. It is not just the sheer volume of data, but its increasing complexity and diversity - even though the increasing amount of valuable data needs to be tackled too Data re-usage in an open data infrastructure implies that data creators, managers and users must rely on each other’s quality of work. Advanced mechanisms are necessary to establish trust between all stakeholders. This requires input and feedback from the communities. Setting the Scene (2) 4

5. 5 Collaborative Data Infrastructure (HLEG)

6. The Case for Multi-Disciplinary Services 6

7. 7 The EUDAT project

8. EUDAT Communities on board 8

9. 9 Communities ↔ Common Data Centres

10. 1. Capturing Communities Requirements (WP4)  Understand how data is organised + collect requirements starting with the EUDAT initial communities: EPOS, LIFEWATCH, CLARIN, ENES, VPH  Further interviews have been conducted and analysis documents are available about CESSDA, DARIAH, DiXa, ECRIN, EMSO, ICOS, PanData and WLCG 2. Building the (generic) services (WP5)  Technology appraisal  Development of services (via taskforces) 3. Deploying services and operating the collaborative infrastructure (WP6)  Co-designing the collaborative (federated) data infrastructure with interfaces for federated operations  Integrating and coordinating resource provision, service provision, operations and support Service Delivery Approach 10

11. Service Handover 11

12. Classification of Services 12 1.Collaborative Services wiki, task tracker, versioning system, webserver, user documentation produce and publish 2.Operational Services monitoring, site registry, service registry, resource registry, resource provisioning coordination 3.EUDAT Core Services rule-oriented data mngmt system, file transfer service, credential conversion EUDAT Coordinated Core Services persistent identifier service, credential delegation 4.Community Service Housing EUDAT site provides resource, community admin deploys/operate the service 5.External Community Service external site, e.g., community center provides resources and services 1.Collaborative Services wiki, task tracker, versioning system, webserver, user documentation produce and publish 2.Operational Services monitoring, site registry, service registry, resource registry, resource provisioning coordination 3.EUDAT Core Services rule-oriented data mngmt system, file transfer service, credential conversion EUDAT Coordinated Core Services persistent identifier service, credential delegation 4.Community Service Housing EUDAT site provides resource, community admin deploys/operate the service 5.External Community Service external site, e.g., community center provides resources and services

13. Classification of Services 13 1.Collaborative Services wiki, task tracker, versioning system, webserver, user documentation produce and publish 2.Operational Services monitoring, site registry, service registry, resource registry, resource provisioning coordination 3.EUDAT Core Services rule-oriented data mngmt system, file transfer service, credential conversion EUDAT Coordinated Core Services persistent identifier service, credential delegation 4.Community Service Housing EUDAT site provides resource, community admin deploys/operate the service 5.External Community Service external site, e.g., community center provides resources and services 1.Collaborative Services Confluence, Jira, SVN, Apache+DRUPAL, Zopyx Produce & Publisch Authoring Environment (?) 2.Operational Services Nagios+SVN, GOCDB, Resource Coordination Tool (under development), … 3.EUDAT Core Services iRODS, gridFTP, XSEDE file manager, Contrail federation software (?) EUDAT Coordinated Core Services EPIC PID service (MoU), Globus myproxy 4.Community Service Housing e.g. CLARIN WebLicht (service for the automatic annotation of text corpora) 5.External Community Service …

14. 14 Dynamic replication to HPC workspace for processing

15. Attribute Provider AuthZ either community-managed or ( ) attributes provided by user’s home IdP are reused Identity credential conversion AtP 1 AtP 2 AtP 3 zoned credential conversion service unique user Ids, project-wise mapped to attribute based access control information Different types of Identity Providers AuthN * consolidated credentials IdP A IdP B IdP D IdP C eID shib OpenID x.509 *

16. Community center EUDAT center EPOS CLARIN ENES VPH Lifewatch 16 Towards Safe Replication

17. rule-oriented replication CLARIN 17

18. raw data description citable publication analysis enrichment EUDAT business domain - two generic data services 18 registration preservation temporary data referable data Data Staging Safe Replication

19. Generic Service Model Fundamental Core Services meet strongly overlapping service requirements Extended Core Services are mainly community-supported, community requirements are typically overlapping between some disciplines Collaboration between Teams Fundamental Core Services are operated and supported by an Operations Team which collaborates across the participating centres. Extended Core Services and other joint multi-disciplinary service must be community-supported, the requirements are overlapping between a specific subset of disciplines WP6 task breakdown structure 19

20. Operational approach 20 Coordinated provisioning of network, storage, and compute Resources Coordinated provisioning of Services on top of these resources, including a registry for all the services deployed and operated and service status monitoring Coordination of Operations via EUDAT teams, integration support for (new) resource/service providers, issue tracking, change management, user support with interfaces to community support teams Coordinated security and quality management CSIRT, security and quality management, certification of repositories and centres (Data Seal of Approval) Operations

21. 21 Site A Services Compute Storage Deputy Security SUPPORT Site A Network Site A Services Compute Storage Deputy Security SUPPORT Site B Network Site A Services Compute Storage Deputy Security SUPPORT Site C Network Site A Services Compute Storage Deputy Security SUPPORT Site D Network Service Provisioning Resource Provisioning Qlty Assurance, Security Coordination & Support

22. 22 EUDAT Operationsteam (1st Project Year) bi-weekly meetings

23. Organisation of EUDAT Operations 1st User Forum - Barcelona, 7/8. March 2012 23 Support Security ResourcesServices Operations Quality Assurance Operations

24. Coordination Quality Assurance Coordinated Service Provisioning 1st User Forum - Barcelona, 7/8. March 2012 24 Support Security Resources Core services AAI, PID, data management, … Community services e.g. workflow, metadata registries Operational services e.g.site/service reg., monitoring, TTS, Collaborative services e.g. wiki, task tracker, svn, webpage, Services Operations

25. Coordination Quality Assurance Coordinated Resource Provisioning 1st User Forum - Barcelona, 7/8. March 2012 25 Support Security Services Resources network service hosting platforms storage Operations

26. Coordinated Support 1st User Forum - Barcelona, 7/8. March 2012 26 Security ResourcesServices Operations Coordination Quality Assurance Operations Support community support (tbd) internal support, issue tracking (done) user documentation (tbd) data centres integration support (tbd) Support

27. Coordinated Security 1st User Forum - Barcelona, 7/8. March 2012 27 Support ResourcesServices Operations Coordination Quality Assurance Support Security risk analysis (started) security management plan security policies, guidelines security assessment operational security (started) (Computer Security Incident RT) Operations

28. Coordinated Quality Assurance 1st User Forum - Barcelona, 7/8. March 2012 28 Support Security ResourcesServices Operations Coordination Support Quality Trusted (certified, e.g. DSA) data management long-term preservation compliance with security policies, data privacy policies. Operations

29. 1st User Forum - Barcelona, 7/8. March 2012 29 Operations Coordinated Operation Operations Team EUDAT centre and community deputies administration of site registry config./change management problem solving, escalation Operations Support Security ResourcesServices Quality Assurance

30. 30 Nagios based monitoring

31. 31 GOCDB@RZG as site registry https://registry-eudat.esc.rzg.mpg.de/view_portal/index.php?Page_Type=View_Sites

32. 32 GOCDB as service registry https://registry-eudat.esc.rzg.mpg.de/view_portal/index.php?Page_Type=View_Service_Groups

33. Helpdesk TTS (RT@CINECA) Aim of the EUDAT TTS Provide a contact point for support activity Provide a tool to manage: –Internal Request/Issues –External Request/Issues Integrated with the Resource/Service Provisioning Coordination Tool 33

34. TTS configuration Approach on a “per site” basis: –Three different types of ticket which results in 3 different generic queues: Support Resources Services Security? –A queue for each site to allow local support collaboration on a single request/issue 34

35. Access to the service Web: https://helpdesk.eudat.eu using the Crowd SSO credentialshttps://helpdesk.eudat.eu Mail: 35 Queue NameQueue TypeMail address to open ticket Mail address notified SupportRouting queuettseudat- support@cineca.it Helpdesk responsible ResourcesRouting queuettseudat- resources@cineca.it c.manzano@fz- juelich.de ServicesRouting queuettseudat- services@cineca.it m.morgotti@cineca.i t

36. Achievements in Operations 36 Initial operations team is working, bi-weekly meetings Services for collaboration and communication available and accessible (currently using CROWD SSO for authN/authZ)  Confluence Wiki, Jira, SVN, VC, Web server, TTS (Request Tracker) Service Hosting Framework is provided via the initial centres  Coordinated provisioning of service hosting resources with storage  Resources are provided in a coordinated way, many are currently used for test environments EUDAT-EPIC MoU in preparation; EPIC PID service run by SARA will be provided as coordinated core service GOCDB has been tested using a separate instance. It was shown to be useful as a site and service registry, it is used by the central monitoring. Central monitoring service makes progress (central instance available) – demo in June 2012 Security plan available for EUDAT (site security contacts, initial CSIRT) Quality assessment document is available (based on the Data Seal of Approval), sites are now starting with the assessment process.

37. Plan for the next project year 37  TTS (RT@CINECA) and Site Registry (GOCDB@RZG): Transition from preproduction to production  User documentation: Produce and Publish Authoring Framework (tbd), Editorial Team  Guidelines for integrating additional centres and external services in the 2 nd year  Service Provisioning  advance the Nagios monitoring infrastructure and further integrate production services  promote existing services to communities via EUDAT website with WP3  get communities services integrated (community service housing) with WP4  Resource Provisioning (T6.3)  Complete the Resource coordination framework: DB with web frontend  collect further resource (storage) requirements from the communities  Security and QA  impove acceptable user policy (wrt to privileged external admins)  advance the security plan for EUDAT and its implementation  Centers proceed with the DSA assessment

38. Welcome to the 1st EUDAT Conference! 38 22-24 October 2012, Barcelona International event with keynotes from Europe and US A forum to discuss the future of data infrastructures Project presentations and poster sessions 2nd EUDAT User Forum Training tutorials