HPSearch Design & Development via Scripting Harshawardhan Gadgil Dr. Geoffrey Fox, Dr. Marlon Pierce

Why Script Web Services Scripting Advantages Rapid prototyping Scripting Web Services Manipulate data streams, files and web-services using scripting language. Equivalent to shell and Perl in UNIX. Allows us to control the invocation and execution of disparate services using scripting. Equivalent to programming a workflow Setting up data-flow among components by dynamically joining disparate data streams Rapidly deploy applications (under development) We have developed a tool for binding URIs to a scripting language that manipulates the local / remote resource identified by the URI - HPSearch

HPSearch Binds URI to a scripting language We use Mozilla Rhino (A Javascript implementation, Refer: but the principles may be applied to any scripting language such as Perl, Python etc… Every Resource may be identified by a URI and HPSearch allows us to manipulate the resource using the URI. For e.g. Read from a web address and write it to a local file x = “ y = “file:///u/hgadgil/data.txt”; Resource r = new Resource(“Copier”); r.port[0].subscribeFrom(x); // read from r.port[0].publishTo(y); // write to f = new Flow(); f.addStartActivities(r); f.start(“1”);

HPSearch (contd.) Currently provide bindings for the following file:// socket://ip:port ftp:// topic:// jdbc: Host-objects to do specific tasks WSDL – invoke web-services using SOAP PerfMetrics – Bind NaradaBrokering performance metrics. Store published metrics and allow querying Resource – Every data source / filter / sink is a resource. Flow – To create a data flow between resources. Useful for streaming applications. For more information, visit

NaradaBrokering Event- brokering system designed to run on a large network of co-operating brokers. Guarantees delivery in presence of failures Implements high-performance protocols (message transit time < 1 ms per broker) Order-preserving optimized message transport Interface with reliable storage for persistent events Fault tolerant data transport Support for different underlying transport implementations such as TCP, UDP, Multicast, SSL, RTP

HPSearch uses NaradaBrokering to route data streams Consider the script as follows Resource a = new Resource(“ResourceA”); a.port[0].subscribeFrom(“file:///u/hgadgil/data.txt”); a.port[0].publishTo(“topic:///x”); // Optional Resource b = new Resource(“ResourceB”); b.port[0].publishTo(“file:///tmp/rawData.dat”); b.port[0].subscribeFrom(a.port[0].publish); f = new Flow(); f.addComponents(b); f.addStartActivities(a); f.start(“1”); HPSearch + NaradaBrokering Resource A Resource B HPSearch Shell WFE

Broker Network HPSearch - WSProxy WFE WSProxyClient WSProxy Service WSProxyClient can control the WSProxy using simple Web-service calls WSProxy is a simple web- service that listens for requests from the client ( WSProxyClient ) and steers the service Request Handler WFE HPSearch Shell The WSProxy connects to a broker and handles all the data communication on behalf of the service PipeDB Connector Flow Handler

Flood Modeling Example using Web Service scripting: Components Rainfall Data Inches of rainfall gathered by the sensor network at specific intervals (Simulated to be in intervals of 30 min in our example) WebMap Server OpenGIS based terrain data repository Generates Watershed model Compute Service – Runoff Model Computes the runoff model using the Hydrostatic watershed model and Hydrodynamic rainfall data Visualization Service Shows the visual representation of the runoff model for the terrain in question Each of these components run as a separate Web service (Either direct or wrapped as a web service) Scripting to manage the chain of these disparate services into one complete application Each of these components is wrapped using WSProxy which is then controlled by the Workflow Engine

Crisis Grid - Flood Modeling Compute Service is the basic control On start up, MapService sends the spatial information for the terrain under consideration to the Compute service. Compute Service requests the Rainfall publisher to send the next rainfall reading (pulls the next reading). The Compute service then operates the run off model for flood analysis. The computed results are then sent to a Visualization service which displays the computed predicted model results in form of an animation Map Service Rainfall Publisher Compute Service Visualization

maps = "/CGL/FLOODMODEL/MAPS"; rain = "/CGL/FLOODMODEL/RAINFALL"; visu = "/CGL/FLOODMODEL/VISUAL"; mapServer = "org.hpsearch.demo.floodModel.MapService"; mapServerLoc = " mapSource = new WebServiceHandler(mapServer); mapSource.setEndPointURI(mapServerLoc); mapSource.setParameter("MAPSERVERTOPIC", maps); rainFallPublisher = "org.hpsearch.demo.floodModel.RainfallPublisher"; rainFallPublisherLoc = " rainFall = new WebServiceHandler(rainFallPublisher); rainFall.setEndPointURI(rainFallPublisherLoc); rainFall.setParameter("RAINFALLTOPIC", rain); rainFall.setParameter("VISUALIZATIONTOPIC", visu); computeService = "org.hpsearch.demo.floodModel.ComputeService"; computeServiceLoc = " compute = new WebServiceHandler(computeService); compute.setEndPointURI(computeServiceLoc); compute.setParameter("MAPSERVERTOPIC", maps); compute.setParameter("RAINFALLTOPIC", rain); compute.setParameter("VISUALIZATIONTOPIC", visu); crisisGridFlow = new Flow(); crisisGridFlow.addComponents(compute); crisisGridFlow.addStartActivities(mapSource, rainFall); crisisGridFlow.start("1"); HPSearch Java Script Code for creating and initializing the WMS Component Code for creating and initializing the Rainfall data Component Code for creating and initializing the Compute service Code for registering the components of the flow and starting it

How to initialize… First Define the service to be invoked mapServer = "org.hpsearch.demo.floodModel.MapService"; The location of the service. This should be normally done via a registry lookup. For For E.g. NBDiscover.discover(“MapService”, specs); Its hard-coded as of now mapServerLoc = " Initialize a WebService Handler. This object invokes a WSProxyClient that interacts with the WSProxy which is a wrapper on the actual service mapSource = new WebServiceHandler(mapServer); Set the End-point URI mapSource.setEndPointURI(mapServerLoc); Set the service initialization properties. This is optional, but allows for more customization. This could include QoS properties, security tokens and any other service specific parameters mapSource.setParameter("MAPSERVERTOPIC", maps);

How to Initialize (contd.) Finally we create a new Flow Object. This object helps to serialize the requests for setting up a new Flow. The FlowHandler component of the WorkflowEngine then distributes these requests to other engines and the flow is finally started. crisisGridFlow = new Flow(); We add the components of the flow. All components which generate the data to be processed are startActivities and are started after the other components are started. This enables the components to start processing the data as soon as the startActivities publish the data crisisGridFlow.addComponents(compute); crisisGridFlow.addStartActivities(mapSource, rainFall); Finally we start the flow. The ”1” is for debugging purposes. Ignore it for now crisisGridFlow.start("1");

In Short…(w.r.t. Flood model) Handle file-based data. Read from static files (archived data), databases, For E.g. The terrain data is static Support streaming data sources. For E.g. Rainfall data read in real-time in a stream. In an actual system we may expect the frequency of observation (and hence data generation) to be about 5 to 20 minutes. In our system, we assume just one source for rainfall data but in an actual scenario, multiple data sources possible

Wrapping a Simple Service Two implementations of ProxyWebService RunnableProxyWebService Provides operations START, STOP, SUSPEND, RESUME. Service is provided the required Input and Output streams from where it will read and write. Good for applications which work on a chunk of data at a time Easiest way to create your first service WrapperProxyWebService Provides just the 4 operations as above. The application handles its own communication Example (next slide)

package mypackage; public class StringProcessingService extends RunnableProxyWebService { BufferedReader input; BufferedWriter output; public void initService(InputStream[] in, OutputStream[] out) { input = new BufferedReader(new InputStreamReader(in[0])); output = new BufferedWriter(new OutputStreamWriter(out[0])); } public String[] serviceExceptions() { String[] exceptions = { "java.io.EOFException", "java.io.IOException“}; return exceptions; } Coding a simple string reversal service MUST initialize the service in this fashion Enables the WSProxyClient to add error specific handlers. You may return an empty array also.

Coding a simple string reversal service public void process() throws Exception { String s, newS; try { s = input.readLine(); if (s == null) { serviceFinished(); return; } } catch (IOException e) { throw e; } newS = reverseOf(s); output.write(newS); output.newLine(); output.flush(); } public String reverseOf(String s) { … } Your code that processes the data

Deploy using AXIS Create a deployment descriptor for AXIS Or use one which already exists Deploy the service Finally run the script to start using the Filter

Invoking our simple service service = “mypackage.StringProcessingService”; serviceLoc = “ to WSDL”; stringproc = new WebServiceHandler(service); stringproc.setEndpointURI(serviceLoc); stringproc.setInput(“input URI”); stringproc.setOutput(“output URI”); f = new Flow(); f.addStartActivities(stringproc); f.start(“1”);

Useful links for additional information WSProxy Technical Report on the flood model Timing Analysis etc… (incomplete as of now)

Credits… Dr. Sunghoon Ko (CGL) and Dr. Jin-Yong Choi (Purdue Univ.) for the Flood Demo NB – CGL, Dr. Pierce, Dr. Fox