RDA 9th Plenary Breakout 3, 5 April 2017 16:00-17:30 Joint meeting: IG Brokering, IG Data Fabric “Advancing Data Fabric with Brokering services” Wrap-up and next steps (Stefano Nativi, Bridget Almas, Peter Wittenburg, Jay Pearlman, Larry Lannom, ….)

Why are we here? Data Fabric Core Components, such as the PIT and DTR, provide essential pieces of infrastructure for managing and sharing Research Data But in order to be useful, they must be connected to provide functionality in research workflows Brokers have traditionally served this role in SOA architectures, providing facade interfaces, mediation and abstraction

GAP Composition Abstractions and Implementations PID Abstract Composition Data Brokers Executable Workflow(s) Objet Types Processing Brokers PID and Object typing help fill the gap Brokering services (e.g. Data services brokering and Processing services brokering) help fill the gap.

From Human-Controlled Process to Type-Triggered Automatic Processing End-to-end architecture Community Services architecture N M N + M N M N x M Data Server Client applications Client applications Harmonization task Mediation task Intermediation services Community (Third-party) Human Controlled Processing (HCP) Type-Triggered Automatic Processing (T-TAP) Web-as-a-Platform Mediation task Mediation task Data Servers 4

Client-Broker-Server architecture Community Services architecture Data Server Client applications Community services (Third-party) Web-as-a-Platform Intermediation services Broker(s)

Useful (architectural and Software) engineering Patterns Separation of concerns (SoC) separate architecture into distinct components, such that each component addresses a separate concern Information Expert Assign a responsibility to the component that has the information needed to fulfill it Low coupling Implement low dependency between the components

Brokering Services Provide the intermediation functionalities to pass from Abstraction to Implementations and vice versa Apply Service-Oriented Architecture approach Utilize Web-as-a-Platform (Web 2.0 patterns) Intermediation-as-a-Service Third-party services: not managed by Client/Server organizations

Brokering Services Benefits Challenges Multi-purpose, Reusable, Participate in Composition Sustainable through ability to evolve Flexible, Configurable, Extensible Challenges Trust, Governance New cultural and business model Requires specialist knowledge Types Discovery and Access Processing and Transformation

Future: Data and workflow integration Goal: Make the life easier for scientists who are no experts in programming and handling data Portals: integrate data and compute workflows Data Workflow Result Preview: Set 1 ✔ Set 2 ✔ Set 3 Workflow diagram: Diana tools – Taverna Heatmap: wikipedia … Data: <PID> 9

Future: Data and workflow integration Label data with PID Label (parts of) workflows with PIDs #Load data mRNA = W1(D1) miRNA = W1(D2) #Analysis res1 = W2(mRNA, miRNA) res = W3(res1) #PID for result file = writeToFile(res) create_PID(file) #Plot for preview Plot(res) PID D1 PID D2 PID D3 PID W1 PID W2 Portal builds the script and executes it based on PIDs for data and PIDs for code snippets. PID W3 PID W4 10

Matching in federation environment Repository metadata of DO bit sequences of DO PID Resolver MD bs PID record 1 DTR 5 PID Rights DB Type Record 3 2 12 4 rights record 6 13 10 Agent 9 Y 8 7 11 Controller WF 14 Matcher 14 14 Broker Profiles Broker Broker

Scalability, efficiency, flexibility, sustainability, close to the Client needs Clients Application Community services & Interface new breed of organizations providing intermediation services to “abstract” current “implementations” and enable composition Composition Intermediation Supply system

PID Centric Data Management and Access Brokering & Mediation services

Biodiversity Use Case

Climate data processing use case (I) Challenge: Worsening ratio of bandwith vs. volume Server-side processing – cultural change in scientific work Solution: Favour support for most common processing tasks over complexity/variety Service innovation rather than technical innovation Understand users, train influencers, reach high quality

Climate data processing use case (II) Motivation: multiple initiatives to build data processing solutions – some exist, some to come variety of data sources: CMIP6, Copernicus, ... some standardization across them, but detail differences brokering as middleware service – not for end- users, but software agents

WPS (final, preprocessing) Climate data processing use case (III) Agent (user need) WPS (final, preprocessing) Data sources Broker Broker component with a specific role: Middleware service catalogs will come; defines broker role Needed: Clear definition for broker component, respecting multiple roles common interface for broker tasking and output format (to be fed into controller)

Broker acting as Controller

Discussion

Thank you !