NaradaBrokering for DS-RT 2005 Grid Tutorial IEEE DS-RT 2005 Montreal Canada Oct Geoffrey Fox CTO Anabas Corporation and Computer Science, Informatics, Physics Pervasive Technology Laboratories Indiana University Bloomington IN

Grid (Web Service) Messaging Build distributed systems from “interoperable” services linked by messages (SOAP) – architect capabilities as services Grids are “just” large scale sets of such services Need to support real time streams and NOT just files (collections of messages) consistent with WS standards (P2P and “central”) Open Source (4 downloads/day) is a scalable distributed pub-sub system supporting multiple standards (JMS, WS) and subscription methodshttp:// Implements “Service Internet” and Notification areas of WS-* Infrastructure Manage messaging for Optimize communication for bad links, firewalls etc Collaboration (multi-cast streams) Fault tolerance with re-transmitted messages and Replicated Services Replay – access any message at any time Virtualize addressing with pub-sub metaphor Performance from protocol (UDP v Parallel TCP..) and representation Heterogeneous Clients – filter to and from PDA’s Candidate for Axis2-MOM (Message Oriented Middleware) infrastructure

NaradaBrokering Stream NB supports messages and streams NB role for Grid is Similar to MPI role for MPP Queues

Multiple protocol transport support In publish-subscribe Paradigm with different Protocols on each link Transport protocols supported include TCP, Parallel TCP streams, UDP, Multicast, SSL, HTTP and HTTPS. Communications through authenticating proxies/firewalls & NATs. Network QoS based Routing Allows Highest performance transport Subscription FormatsSubscription can be Strings, Integers, XPath queries, Regular Expressions, SQL and tag=value pairs. Reliable delivery Robust and exactly-once delivery in presence of failures Ordered delivery Producer Order and Total Order over a message type. Time Ordered delivery using Grid-wide NTP based absolute time Recovery and Replay Recovery from failures and disconnects. Replay of events/messages at any time. Buffering services. Security Message-level WS-Security compatible security Message Payload options Compression and Decompression of payloads Fragmentation and Coalescing of payloads Messaging Related Compliance Java Message Service ( JMS ) 1.0.2b compliant Support for routing P2P JXTA interactions. Grid Feature SupportNaradaBrokering enhanced Grid-FTP. Bridge to Globus GT3. Web Services supportedImplementations of WS-ReliableMessaging, WS-Reliability and WS-Eventing. Traditional NaradaBrokering Features

Features for March—June 2005 Releases Production implementations of WS-Eventing, WS- Notification, WS-RM and WS-Reliability. SOAP message support and NaradaBrokers viewed as SOAP Intermediaries Active replay support: Pause and Replay live streams. Stream Linkage: can link permanently multiple streams – using in annotating real-time video streams Replicated storage support for fault tolerance and resiliency to storage failures. Management: HPSearch Scripting Interface to streams and services Broker Discovery: Locate appropriate brokers

Summary NaradaBrokering provides a fully distributed queue manager where queues buffer streams with overheads of a few milliseconds per broker << 30 ms frame interval << 100’s ms network delay Much faster than using databases or writing files Collaboration is implemented by sharing synchronizing streams Compatible with Grids, Web Services, Java Message Service Streams are “first class entities” with rich set of features Don ‘t open a socket; hand data to NaradaBrokering Software Overlay Network or Message Oriented Middleware

NaradaBrokering Services

Reliable Delivery Service Guaranteed delivery in multiple producer/ consumer settings. Guarantees hold true in the presence of Node/Link Failures Links can lose messages and garble message order. Storage failures: Stores need to recover after failure. Prolonged entity disconnects Exactly-Once and Ordered delivery of events Uses both positive& negative acknowledgements Supports Replay and Fast Recovery from failures Independent of underlying archival system. Was used to enhance fault tolerance in Grid-FTP. Uses “Reliable Storage” to keep messages temporarily

Dealing with large payload sizes To cope with large payloads, the substrate incorporate 2 sets of services. Compression/Decompression service: The substrate incorporate support for zlib based compression and decompression of payloads. Fragmentation/Coalescing Service: These service can break up a large payload into smaller fragments. The coalescing service can take these smaller fragments and coalesce them into the original large payload. This was used to deal with transfer of large payloads (up to 1 GB) in the NB enhanced Grid-FTP application.

Replay Services Replay requestors can specify replays based on several parameters A range of sequence numbers can be specified. Additionally, constraints on an event’s content synopsis can be specified. Based on a specified time range. Replay services have been tested with applications such as Audio/Video conferencing, Whiteboards etc. Essential for recording and replay of collaborative sessions Important special case supports rewind and similar operations on a real-time stream

Buffering Service This service is incorporated into the system to facilitate the buffering of events prior to releasing them. Buffering service time orders events and releases event based on three metrics Number of events in the buffer Size of the buffer Time spent by event in a buffer.

Time Differential Service This service is essential to reduce jitters in large distributed environments. Networks introduce unpredictable delays that increase jitter. Service takes events released by buffering service, and ensures that it preserves time spacing between events. TDS can provide time spacing resolution of up to 1 millisecond between events.

Trans-Atlantic Settings

NaradaBrokering NTP Service NaradaBrokering includes an implementation of the Network Time Protocol (NTP) All entities within the system use NTP to communicate with atomic time servers maintained by organizations like NIST and USNO to compute offsets Offset is the computed difference between global time and the local time. The offset is computed based on the time returned from multiple atomic time servers. The NTP algorithms weighs results from individual time clocks based on the distance of the atomic server from the entity. This ensures that all entities are within 1 ms of each other. The timestamps account for clock drifts on machines Time returned corrects software clocks which can slow down with increased computing load on the machine.

Broker Discovery Service Locates the nearest available broker that a client can connect to. Incorporates specialized nodes – broker discovery nodes – to maintain broker info. Depending on load or security issues, brokers may decide to respond/ignore discovery requests. If available the scheme can exploit IP multicast for discovery. Nearest broker determined by ping times, loss rates and available bandwidth.

Broker Discovery: Brokers at Indianapolis, NCSA, UMN, FSU, IU & San Diego Supercomputing Center. Broker at Indy selects IU, NCSA and UMN for pings.

Broker Discovery: Brokers at Indianapolis, NCSA, University of Minnesota, FSU & San Diego Supercomputing Center. Cardiff selects Indy, NCSA and UMN for pings.

Topic Discovery Service Allows publishers and subscribers to advertise topics. Creator of topic possesses credentials to indicate ownership of the topic. Discovery of topics takes into account credentials of client trying to discover topic. Topic owner may restrict discovery to a limited authorized set of clients. Discovery requests can be made using simple strings or regular expression queries.

Based on Message Level Security Messages organized into topics Each topic has a separate key; Topics can be organized into sessions Security Service

NaradaBrokering Support for SOAP and Web Services

SOAP Support I The broker can receive SOAP messages (over HTTP) from any entity. This removes any client dependence in client-broker interaction The broker can function as an intermediary performing multiple roles which could just be routing but also involve mapping using filters There can be multiple filter-pipelines, each comprising multiple filters, available at the broker node. Some of these would be system filter-pipelines configured statically. Filter-Pipelines can also be configured by users, dynamically, at run-time.

SOAP Support II Multiple roles could be associated with Different servlets hosted by a broker. A given servlet hosted by a broker. Scheme will allow filters to be registered for individual roles. A filter could be part of multiple roles. There is a dedicated filter pipeline per role. This implies that a NaradaBroker can be used as a Web Service container although full container support is not yet available Filters are used internally by NB to implement performance monitoring

The FilterPipeline-Filter model I The filter and filter-chain facilitate many of the interactions that are missing in JAX-RPC handlers. Filters are NaradaBrokering approach to the handlers used in Web Service containers Filters can inject messages at any time These messages can be sent either to the application or over the network. No limit on the number of messages that can be triggered because of a single message from application. Messages can be injected into a Filter Pipeline from either directions. Filters can generate responses automatically. No need to route to application.

The FilterPipeline-Filter model II Applications have access to individual filters and filter-pipeline at all times. Explicitly direct which filters need to be skipped or added. Filters have access to position within Filter Pipeline, and can specify message injection at a specific location. Dynamic reconfiguration possible for Filter Chain. Allow different networking substrates to be registered. This can be dynamically changed. Network substrate is last filter and is responsible ONLY for routing SOAP message.

Web Services Support I Currently we have incorporated support for the following Web Service specifications WS-Eventing (WSE): This is a publish/subscribe based notification framework from Microsoft and IBM. WS-ReliableMessaging WSRM): This is a protocol for ensuring the guaranteed delivery of SOAP messages between 2 Web Service endpoints. This specification is from IBM and Microsoft. WS-Reliability (WSR)- This is a competing specification from Oracle and Sun in the area of reliable messaging between Web Services. These handlers are available for use in Axis1.2 or exploiting NB SOAP Intermediary support without a container Axis1.2 version can be used inside container or as a Proxy

Web Services Support - II We are also working on implementing support for the WS-Notification (WSN) suite of specification that is part of the Web Services Resource Framework (WSRF). WS-Notification explicitly adds brokers to Eventing Note that almost all these specifications leverage the WS-Addressing (WSA) specification. We have incorporated support for all the rules associated with WSA.

NaradaBrokering in Web Services a) WSM WSR WSN WSE support for Axis1.2 which is available as standalone handlers without need for any NaradaBrokers b) The support described in a) implemented as a separate proxy and inside containers c) NaradaBrokers used as SOAP Intermediaries d) NaradaBrokers can support filters in SOAP intermediaries forming limited light-weight containers e) NaradaBrokers can be Brokers defined in WSN Specification f) One can use NaradaBrokers in non-brokered publish-subscribe such as WS-Eventing to make it scalable

OperationMeanStd Dev Std Error OutlierMinMaxMem (Bytes) Create an XMLBeans based Envelope Document Create an Axis based SOAPMessage Convert an EnvelopeDocument to a SOAPMessage Convert SOAPMessage to EnvelopeDocument Create a WS-Addressing EPR (Contains just a URL address) Create a WS-Addressing EPR (Contains WSA ReferenceProperties) Create an Envelope targeted to a specific WSA EPR Create an Envelope targeted to a specific WSA EPR with most WSA message information headers Implementation of WS-Reliable Messaging (WSRM) I

Implementation of WS-Reliable Messaging (WSRM) II OperationMeanStd Dev Std Error OutlierMinMaxMem (Bytes) Parse an EnvelopeDocument to retrieve WSA Headers Create a Wsrm Fault Create a Wsrm SequenceRequest Create a Wsrm SequenceResponse Create a Wsrm SequenceDocument Add a WsrmSequenceDocument to an existing envelope. (Contains sequence identifier and message number) Create a WSRM SequenceAcknowledgement based on a set of message numbers Create a WSRM TerminateSequence

Transport Layer in NaradaBrokering

Transport Layer Support for multiple network protocols such as TCP, UDP, Multicast, SSL, RTP, HTTP and Parallel TCP. Support for both blocking and non-blocking IO in the TCP support. The UDP support manages payloads greater than 64K datagram limit. Also incorporates pinging mechanism to detect connection losses in connectionless setting. Tunnel through firewalls/proxies Microsoft’s ISA, Checkpoint, Apache

hop Transit Delay (Milliseconds) Message Payload Size (Bytes) Mean transit delay for message samples in NaradaBrokering: Different communication hops hop-2 hop-5 hop-7 Pentium-3, 1GHz, 256 MB RAM 100 Mbps LAN JRE 1.3 Linux

Performance of NaradaBrokering in collaborative settings

“GridMPI” v. NaradaBrokering In parallel computing, MPI and PVM provided “all the features one needed’ for inter-node messaging NB aims to play same role for the Grid but the requirements and constraints are very different NB is not MPI ported to a Grid/Globus environment Typically MPI aiming at microsecond latency but for Grid, time scales are different 100 millisecond quite normal network latency 30 millisecond typical packet time sensitivity (this is one audio or video frame) but even here can buffer frames on client (conferencing to streaming) 1 millisecond is time for a Java server to “think” Jitter in latency (transit time through broker) due to routing, processing (in NB) or packet loss recovery is important property Grids need and can use software supported message functions and trade-offs between hardware and software routing different from parallel computing

HPSearch Management Engine HPSearch is an engine for orchestrating distributed Web Service interactions It uses an event system and supports both file transfers and data streams. HPSearch flows can be scripted with JavaScript HPSearch engine binds the flow to a particular set of services and executes the script. HPSearch can access and set NaradaBrokering features (create topics, display performance data) ProxyWebService: a wrapper class that adds notification and streaming support to a remote Web Service. HPSearch is a streaming sensitive workflow engine

WMS GIS service and a data Layer

Data Filter (Danube) Pattern Informatics (Danube)  Accumulate Data  Run PI Code  Create Graph  Convert RAW -> GML GPS Database (Gridfarm001) WMS HPSearch (TRex) HPSearch (Danube) HPSearch hosts an AXIS service for remote deployment of scripts GML (Danube) WS Context (Tambora) NaradaBroker network: Used by HPSearch engines as well as for data transfer Actual Data flow HPSearch controls the Web services Final Output pulled by the WMS HPSearch Engines communicate using NB Messaging infrastructure Virtual Data flow Data can be stored and retrieved from the 3 rd part repository (Context Service) WMS submits script execution request (URI of script, parameters) Workflow (BPEL) Fragment

SensorML and NaradaBrokering OGC defined a set SensorML of specifications indicating how to integrate Sensors with its GIS Services We are using Southern California SCIGN GPS data to prototype this RYO Binary Text GML Sensor Source Filter NaradaBrokering Topics You can access whichever version you want!

NaradaBrokering Futures Support for replicated storages within the system. In a system with N replicas the scheme can sustain the loss of N-1 replicas. Clarification and expansion of NB Broker to act as a WS container Integration with Axis 2.0 as Message Oriented Middleware infrastructure Support for High Performance transport and representation for Web Services Needs Context catalog under development Performance based routing The broker network will dynamically respond to changes in the network based on metrics gathered at individual broker nodes. Replicated publishers for fault tolerance Pure client P2P implementation (originally we linked to JXTA) Security Enhancements for fine-grain topic authorization, multi- cast keys, Broker attacks