Ocean Observatories Initiative OOI Cyberinfrastructure Overview Matthew Arrott VMware Presentation March 5, 2010

OOI CI LCO Review, Feb Goals for this Presentation

OOI CI LCO Review, Feb Overview The OOI Integrated Observatory OOI Network Architecture OOI Logical Architecture Conclusions

OOI CI LCO Review, Feb OOI Program Overview

OOI CI LCO Review, Feb OOI Integrated Observatory

OOI CI LCO Review, Feb Integrated Observatory

OOI CI LCO Review, Feb Science & Education Operations Interactive Ocean Observing Interactive Ocean Modeling & Data Assimilation Automated Data Product Generation Discipline-Driven Semantic Organization of Data Interactive Instrument Network Integrated Observatory Management User-Driven Integration of Resource Observatory Activity Model

OOI CI LCO Review, Feb Scope Release 1: Data Distribution Network (2011) –“Ingest” data (products) from external data sources; both streaming and cataloged –Characterize data sources with their metadata attributes (format, structure, meaning) –Transform external data into internal OOI data formats, for stream-based distribution and persistence –Distribute data via streaming and DAP servers to data analysts and numerical modelers –Provide initial instrument integration, control and sensor data acquisition –Provide instrument and platform direct access –Provide service integration and execution platform

OOI CI LCO Review, Feb Scope of Future Releases Release 2: Managed Instrument Network (2012) –Fully managed instrument activation and control; ready for deployment on OOI moorings and cable infrastructure –Qualified data products based on automated data processing Release 3: On-Demand Measurement Processing (2013) –Observatory management and resource scheduling –On demand and event driven measurements –Data processing workflows Release 4: Integrated Modeling Network (2014) –Interactive analysis and synthesis –Numerical model integration Release 5: Interactive Ocean Observatory (2015) –Full control of data, processes, models, instruments –Closed loop, adaptive sensing and platform control

10 Goal after Release 5

OOI CI LCO Review, Feb Release Schedule

OOI CI LCO Review, Feb Network Architecture

OOI CI LCO Review, Feb Network Architecture

OOI CI LCO Review, Feb International Research Networks

OOI CI LCO Review, Feb National Research Networks

OOI National Network OOI CI LCO Review, Feb

OOI CI LCO Review, Feb Network Deployment

OOI CI LCO Review, Feb OOI Logical Architecture

OOI CI LCO Review, Feb Release-1 OOI CI Functions Instrument and platform interfaces Instrument and platform control from the observatory Direct access to instruments and platforms Data acquisition from sensors and external observatories Management of science data and other observatory information Data inventory and metadata annotation Data ingestion, i.e. canonicalization and metadata annotation Data discovery, cataloging and presentation Data distribution, replication, streaming Data persistence, assurance, retrieval Process and service execution Virtualization of computing Elastic scaling to demand High availability services Service-orientation Reliable message based asynchronous communication Security: identity and policy management Multi facility system integration Sensing & Acquisition Data Management Common Operating Infrastructure Common Execution Infrastructure Interface Adapters Application Infrastructure

OOI CI LCO Review, Feb The Capability Container Ingredients: Capability: function, service, component Plumbing: Messaging, security Container: drop it all in as needed Packaging: label it, tape it shut… done!

OOI CI LCO Review, Feb Secure Reliable Messaging Capability Container Capability Container Capability Container Capability Container Capability Container Capability (Service) Capability (Service) Capability (Service) Capability (Service) Capability (Service)

OOI CI LCO Review, Feb The “Exchange” Applications communicate through Exchange Spaces Exchange Spaces are namespaces of “communicators” Applications need to enroll in Exchange Spaces Governance applies within Exchange Spaces

OOI CI LCO Review, Feb Exchange Space and Points Exchange Points are the message routing and queuing resources of Exchange Spaces Communicators play the role of Producer, Consumer, and Distributer (Broker)

OOI CI LCO Review, Feb Exchange Space Exchange Space is comprised of –Distributed Application Facility (DAF) –Distributed IPC Facility (DIF)

OOI CI LCO Review, Feb Messaging Abstraction Capability Container Capability Container Capability Container Send Message From: “name2” To: “name4” In: Exchg-Space1 Action: “invoke-service”

OOI CI LCO Review, Feb Common Message Format Capability Container –Provides message handling through interceptors After a service sends a message, before it receives a message Message signing and validation Policy enforcement Governance tracking –Provides a common message format for all CI messages Based in ACL FIPA message structure Content, encoding, ontology

OOI CI LCO Review, Feb Services and Resources as Agents Resource Agent Resource Agent Proxy Agent Capability Container Capability Container Capability Container Capability Container

OOI CI LCO Review, Feb Scenario An instrument, a physical resource, is represented by an agent to the system and its users Users request control of the instrument Capabilities are projected into another domain of authority by a proxy agent

OOI CI LCO Review, Feb Back-End Infrastructure

OOI CI LCO Review, Feb Exchange Points and the DIF

OOI CI LCO Review, Feb Message Brokers over DIF

DIF Member Process OOI CI LCO Review, Feb

OOI CI LCO Review, Feb COI Infrastructure Services The Common Operating Infrastructure (COI) is the integration & communication environment for all the other subsystem services

OOI CI LCO Review, Feb Integration Strategy Capability Container Secure reliable messaging “Exchange” Service-oriented architecture Non-central governance COI Infrastructure Services

OOI CI LCO Review, Feb CI Interfaces and Subsystems

OOI CI LCO Review, Feb Preparing Services for Deployment Instrument Point Acquisition Point Ingest Point Application Point Storage Point Access Portal

OOI CI LCO Review, Feb Deployment Scenario

OOI CI LCO Review, Feb Deployment Plan and Strategy Deployment of capability containers in multiple execution environments by leveraging virtualization –The CEI will make it work CyberPoP (Cyberinfrastructure Point of Presence): physical deployment of one or multiple capability containers –Different types

OOI CI LCO Review, Feb Available, Elastic Execution

OOI CI LCO Review, Feb CEI Service Interactions Planner Controller (“Executive”) Provisioner Fault Monitoring and Compensation Registry Resource (Taskable) OV2 CEI

OOI CI LCO Review, Feb Scenario: Planner and Controller The pattern Planner- Controller-Resource can repeat itself at multiple (nested levels) OV2 CEI

OOI CI LCO Review, Feb CEI Architecture Overview OOI CI LCO Review, Feb EPU EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) HA Service (OOI Application) VM (Deployable Unit) VM (Deployable Unit) VM (Deployable Unit) Application Software (Deployable Type)

OOI CI LCO Review, Feb CEI Architecture Overview OOI CI LCO Review, Feb

OOI CI LCO Review, Feb CEI Service Interaction OOI CI LCO Review, Feb Planner ControllerProvisioner Monitoring Information Deployable Type Registry Context Broker IaaS AdapterIaaS Exchange VM info Proc Info Other Info

OOI CI LCO Review, Feb Elastic Processing Unit Elastic Processing Unit (EPU): High- Availability Services with policy based scaling to demand OOI CI LCO Review, Feb EPU Controller Register HA service EPU Worker N1 EPU Worker N1 Msg to N EPU Worker N2 EPU Worker N2 EPU Worker NX EPU Worker NX …

OOI CI LCO Review, Feb EPU: Prototype 1 EPU design and prototype: –Policy-configurable planner/scheduler –Scaling policy: size of work queue –All components integrated with Magnet/AMQP OOI CI LCO Review, Feb

OOI CI LCO Review, Feb Governance

OOI CI LCO Review, Feb Motivating Governance Administering collaborations –Based on framing normative relationships among peers –Abstracting away from low-level details OOI, broadly: many stakeholders; many resources; longevity of decades Exchange spaces, narrowly: abstractions for communicating; assembly of multiple topologies for messaging; analogous to traditional enterprise integration patterns

OOI CI LCO Review, Feb Elements of a Service Engagement Enactment: doing the domain work – what the end user cares most about Administration: captured via contracts –Partnerships –Rules of encounter Identity Enforcement

OOI CI LCO Review, Feb What is Governance? Broadly, administering service engagements IT Governance: How IT resources are administered SOA Governance: How services are created, deployed, removed, … Currently, governance is manual –Low productivity –Poor scalability for fine-grained, real time governance decisions –Hidden, implicit considerations yield low confidence and poor maintainability

OOI CI LCO Review, Feb Why Governance? Stakeholders using resources to best serve individual and collective needs –Share resources in a controlled manner –Configure and reconfigure dynamically –Enable unanticipated uses for resources –Respect human organizational needs In a nutshell, stakeholders administer themselves

OOI CI LCO Review, Feb Separation of Concerns Protocol: specifying the interactions among autonomous parties Policy: specifying the decision making of each autonomous party as it participates in various protocols Behavior: specifying the implementation that realizes the interactions

OOI CI LCO Review, Feb Exchange Space Use Case OOI CI LCO Review, Feb

OOI CI LCO Review, Feb Messaging View of Enrollment OOI CI LCO Review, Feb

OOI CI LCO Review, Feb Community Affiliation Use Case OOI CI LCO Review, Feb

OOI CI LCO Review, Feb Combined Scenario, Schematically OOI CI LCO Review, Feb

OOI CI LCO Review, Feb IdM Architecture Overview

OOI CI LCO Review, Feb Conclusions

OOI CI LCO Review, Feb Summary

OOI CI LCO Review, Feb Next Steps

OOI CI LCO Review, Feb Thanks !