1. SRB 1 & iRODS 2 Arcot Rajasekar Reagan Moore Mike Wan SDSC/UCSD Pathways to OOI-CI CyberData Architecture 1 Storage Resource Broker 2 integrated Rule Oriented Data Management Systems

2. OOI-CI Data Capability  Data Network: Federated data,metadata and its preservation via data streams, repositories and catalogs

3. Data Virtualization with SRB FTP System at JHU Database At UCLA File System at BWH User Application

4. Data Virtualization with SRB Common naming convention and set of attributes for describing digital entities User Application Logical name space Location independent identifier Persistent identifier Collection owned data Access controls Audit trails Checksums Descriptive metadata Replication Managed I/O Inter-realm authentication Single sign-on system Uniform Access Interface FTP System at JHU Database At UCLA File System at BWH

5. SRB Federation  Enables an SRB Network to recognize the presence of another SRB Network and be able to interact with it.  The overall federation can still be viewed at one single namespace whereby each participating Zone is a child node of the root node.  This keeps the learning curve very low and travelling from one Zone to another is like changing directory. DB SRB DB SRB DB SRB Trust Relation Trust Relation Trust Relation Zone B Zone C Zone A

6. BIRN: Biomedical Information Research Network

7. NOAO Zone Architecture Cerro Tolelo, Chile Kitts Peak, AZ Tuscon, AZ La Serena, Chile UIUC, IL (Archive)

8. NOAO Data Flow

9. UK eScience: e-minerals/e-materials

10. ROADNet Sensors

11. ROADNet:Virtualization of Sensor Access VORB = SRB + Antelope ORB

12. User Base & Diversity of Applications  Collections at SDSC: >1 PetaBytes, >150 Million files Multi-disciplinary Scientific Data  Astronomy, Cosmology  Neuro Science, Cell-Signalling & other Bio-medical Informatics  Environmental, Geological & Ecological Data  Educational (web) & Research Data (Chem, Phys,…)  Archival & Library Collections  Earthquake Data, Seismic Simulations  Real-time Sensor Data Growing at 1TB a day Supporting large projects: TeraGrid, NVO, SCEC, SEEK/Kepler, GEON, ROADNet, JCSG, AfCS, SIO Explorer, SALK, PAT, UCSDLibrary, …

13. iRODS : A Rule-Oriented Data Management System  Based on the SRB experience User Feedback Modularity and Ease of Extensibility Code from experience – better structures  AIM: To make a flexible data management system Easy to customize at finer level  Example: Can we add additional post processing on ingestion  Example: Can we use workflows for server-side data management  Example: Can we provide queued and batch processing  Example: Can we provide customizable management policy  Solution: iRODS Uses rule-based architecture to provide flexibility with server-side workflow services.

14. iRODS Overview RuleEngine DataTransport MetadataCatalog ExecutionControl MessagingSystem ExecutionEngine Virtualization ServerSideWorkflow PersistentStateinformation Queuing/Scheduling PolicyManagement ControlCommunication Data Movement & Management SRB Functionalities

15. iRODS System Client InterfaceAdmin Interface Current State Rule Invoker Micro Service Modules Metadata-based Services Resources Micro Service Modules Resource-based Services Service Manager Consistency Check Module Rule Modifier Module Consistency Check Module Engine Rule Confs Config Modifier Module Metadata Modifier Module Metadata Persistent Repository Consistency Check Module Rule Base

16. How iRODS Works:  Micro-services compact “functions” that have a well-defined semantics as well as recoverability on failure.  Rules policy definitions possibly with alternatives having triggering conditions/events  Server-side Workflows chains and combinations of micro-services executed on demand, scheduled/queued, remotely, sequentially, parallelly, …  Peer-to-peer distributed agents execution and management environment with a well-integrated data-control message system varying capabilities can be published and supported  Complete Transparency – Logical Naming Conventions persistent and state-ful system client-agnostic – policy enforcement independence

17. How iRODS Works: An Example  Micro-services define a compact “function” that has a well-defined semantics as well as recoverability on failure. ComputeChecksum, Replicate, CreateThumbnail,… We have defined close to 100 micro-services Some micro-services interface to web-services  Rules define a policy (say, for ingestion into a particular collection) Rules can invoke other rules and micro-services Rules can be event-based or recurring/periodic OnIngestion: If Collection == /home/*/sio/seabed computeChecksum, replicate, sendEmail(orcutt,…)  On An Event: The rules form a workflow chain (with fail-over alternates) of underlying micro-services and are executed  These micro-services/chains can be executed immediately, delayed, remotely, parallelly, …

18. Mapping to OOI  Online Data Repository Immediate Access for Distributed Data - Global Naming Spaces Autonomous Data Stewardship – seamless federated access Replication and Fault-tolerance Infrastructure Independence – Vendor & System agnostic Transparent Policies for Ingestion, Discovery, Access, Management  Persistent Archive Service Long-term availability Technology Obsolescence and Migration Integrity, Authenticity, Chain of custody  Aggregation Service Categorization, Classification - Multiplt Viewspoints Logical Collections  Attribution Service User-defined Metadata, Domain-specific Ontology Annotations, Qualifications Querying and Discovery  User Interface Suite Uniform Access – Ingestion, Attribution, Discovery, Access Web Portal, GUI, scripting, command-line, APIs and Bindings Interfaces to external systems: OPeNDAP, THREDDS, Kepler,… Process interface: Compute Grids, Web Services, User Systems

19. Path Forward  iRODS fills the basic requirements of data architecture of OOI-CI  Extensions are needed to customize iRODS for OOI-CI Sensor data  MBARI SSDS/PUCK, VORB/ARTS, … Streaming (large volume data - video) Enabling CyberPoPS Integration with CEI, COI, and other modules of OOI-CI User/Service/Portal/Process Interfaces Importantly: Designing/Implimenting Policy-based rules and micro-services suited for OOI-CI & autonomous observatories

20. Questions? irods.sdsc.edu www.sdsc.edu/srb {sekar,moore,mwan}@sdsc.edu