1. Click to add text © 2013 IBM Corporation August 5, 2013 Streams 3.2 By Mohan Dani

2. © 2013 IBM Corporation 2 Agenda  Toolkit Overview  Architecture  Usage Scenarios  Demo

3. © 2013 IBM Corporation 3 Overview – Rules Toolkit  The Rules toolkit will provide its users capability to execute Business Rules  Current Integration with Streams is with Operation Decision Manager(ODM) V8.5.  Use Case : This Integration need arose because of use cases like CDR mediation & Fraud Detection application (SPRINT) – where rules had to be Authored maintained rule flow to be decided and evaluated really quickly at real time.  Streams - real time capability  ODM- rule authoring, maintenance etc.  ODMRulesetExecutor - Provides Following features –File and Database deployment support –Auto mapping of Primitive data types –Custom mapping of User defined types –Refresh of rules –Control to submit output on multiple ports

4. © 2013 IBM Corporation 4 Operator Overview  The Primary function of this operator is to evaluate ODM Rules on streaming input data on pre- configured ODM (Business Rules Management System) rulesets. –Input port –This ODMRulesetExecutor has one input port, atleast 1 required output port –Output Port –Operator can have multiple output ports if ruleset execution output is required to be send to multiple output ports –1 optional error output port. If an error occurs while executing a ruleset from InputStream(Tuple), operator writes the error string into the error output port (if one is specified). –On successful rules evaluation,operator submits the tuple to the output stream corresponding to the rules evaluated result.

5. © 2013 IBM Corporation 5 Architecture

6. © 2013 IBM Corporation 6 External interfaces  TupleRulesetParameterMapping Interface: This Java interface is located under the package. com.ibm.streams.rules.odm. This interface exposes the methods which user needs to implement once to establish Odm ruleset input parameter mapping with the Streams attributes on input port and Odm ruleset output response mapping with streams attributes on output port.  SimpleParameterMapping Class: This java class provides an implementation of TupleRulesetParameterMapping Interface for the mapping with the Streams attributes on input port and Odm ruleset output response mapping with streams attributes on output port for primitive data types.  RulesetExecutionHandler Interface: This Java interface is located under the package. com.ibm.streams.rules.odm. This interface exposes the methods which user needs to implement if they want to submit the ruleset execution output to multiple output ports.  AbstractRulesetExecutionHandler Abstract Class: This java abstract class provides the implementation of the RulesetExecutionHandler interface. User can optionally chose to extend this class for ease in implemenation of the Interface.  SimpleRulesetExecutionHandler Class: This java class is the default implementation for submission of output on Single output port using primitive datatype mapping.

7. © 2013 IBM Corporation 7 Usage Scenario - File Based deployment  The ODM Developer can choose to deploy the ODM ruleset as file based deployment.  In this kind of deployment the rulesets get deployed to file system in a directory.  In ODMRulesetExecutor Operator user is expected to provide directory location of the deployed ruleApp in following operator parameter –ruleAppDirectory

8. © 2013 IBM Corporation 8 Usage Scenario - Database Based deployment  The ODM Developer can choose to deploy the ODM ruleset as database based deployment.  In this kind of deployment the ruleset artifacts get deployed to database in a tables.  ODMRulesetExecutor operator user is expected to provide following parameters : –driverName –driverPath –databaseURL –userName –userPassword

9. © 2013 IBM Corporation 9 Usage Scenario - Multiple Port Submission  The ODM operator has atleast one required output port and one optional error output port. Operator can have multiple output ports if ruleset execution output is required to be send to multiple output ports.  User Can choose to over ride the default behavior by providing and implementation to RulesetExecutionHandler interface or extending the SimpleRulesetExecutionHandler class.  User is expected to provide the class name and library containing the implementation in following Operator parameters –rulesetExecutionHandlerLibrary –rulesetExecutionHandlerClassName

10. © 2013 IBM Corporation 10 Usage Scenario - Mapping  The ODMRulesetExecutor operator by default provides auto mapping of the attributes or primitive data types.  Auto mapping is done by the operator for the attributes and parameters where name and type matches.  If XOM has Custom Types like “Loan”, “Borrower”. User is expected to provide the Mapping using Interfaces and provide the custom mapping details in following Operator parameters –rulesetExecutionHandlerLibrary –rulesetExecutionHandlerClassName

11. © 2013 IBM Corporation 11 Usage Scenario - Refresh of Rules  Rule refresh is a regular occurrence with Business Rules Management Engines, The Rules are optimised after initial runs or they are fine tuned or additional rules added to accommodate changing business scenarios.  Refresh of rules is the scenario that rules have been updated.ODMRulesetExecutor operator is expected to pick the latest rules if user is specifying the generic path in the rulesetPath for example “/ruleApp/ruleSet” without restarting the running SPL application.  Ruleset should be deployed to the Rule Execution Server Console.  This scenario can be enabled by the user by providing values for –ManagementConsoleHost –managementConsolePort.

12. © 2013 IBM Corporation 12 TimeSeries-What's new  HoltWinters, ARIMA, ARMA, AR.Too many forecasting operators to choose from? AutoForecaster enables automatic algorithm selection based on the characteristics of given input data. “Anyone can do Forecasting”  Compression and comparison: PSAX operator converts large timeseries into string literals to support comparison and compression.  Timeseries synthesizer: generator operator continuously generators timeseries signals of different types.  Control signals support for Holtwinters.  Model streaming and control signal support for GAMLearner and GAMScorer.  And some more interesting features introduced..

13. © 2013 IBM Corporation 13 Control Port and Model Streaming Operator Model/ Co-efficients Control port Input data Model streaming output port Output data  Control-port: send signals to running operator:  Retrain: retrain the internal model with a new set or training parameters  Load: load a new model through a control port  Monitor: flush-out model coefficients through the control port  Suspend: Suspend model learning and prediction operation until resume signal is received.  Resume: Resume the suspended model learning and prediction operation.  Model streaming port: flushing model coefficients  Applies to Holtwinter, GAMLearner and GAMScorer

14. © 2013 IBM Corporation 14  PSAX :  Real-time timeseries compression.  Return string literals representation of input timeseires  Ex : Input timeseries is 33 output is X.  AutoForecastor  Auto selection of best algorithm based on the input data characteristics.  “Anyone can do forecasting”. Only two parameters needs to be set.  Generator  Synthesize different types of timeseries signals for analysis and testing. New operators

15. © 2013 IBM Corporation 15 Streams MQTT Toolkit  Introduction to Messaging toolkit  MQTT Toolkit Applications and Usages  MQTT Toolkit Terminologies explained  MQTT Toolkit Overview Architecture  MQTT Toolkit Operators

16. © 2013 IBM Corporation 16 Introduction to messaging toolkit  A set of edge adapters which help to read/write to/from a queue or topic in various messaging providers. –XMS adapters (XMSSource and XMSSink) connect to WebSphere MQ queue or topic using IBM Message Service Client for C++. –JMS adapters (JMSSource and JMSSink) use standard JMS protocol to connect to two well known JMS providers- WebSphere MQ and Apache ActiveMQ. –MQTT adapters( MQTTSource and MQTTSink) use MQTT client API's to publish/subscribe to a topic on an MQTT Provider

17. © 2013 IBM Corporation 17 Applications and Usages  MQTT adapters- Real time traffic management

18. © 2013 IBM Corporation 18 Real time traffic management  Consider a city with lots of vehicles during peak hours. During peak traffic hours there is a considerable jam which causes inevitable delay. Now if we have an intelligent traffic management system, where each vehicle in traffic have devices attached to them which sends the GPS data for the vehicle to a centralized server. This server based on the current geography, climate and traffic conditions can send update to the vehicle and alert it of possible heavy traffic jam/accidents in its route and suggest an alternate route for its destination. This can save considerable time and energy for the driver of the vehicle. Here the devices attached to the cars can have MQTT clients sending GPS data to the MQTT servers which can be picked up by Streams which can then use geospatial toolkit and information from the local geography and likewise suggest an alternate route.

19. © 2013 IBM Corporation 19 Kaveri Release developments  MQTT adapters use MQTT protocol to connect to MQTT providers. –It is designed as an extremely lightweight publish/subscribe messaging transport useful for connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium.

20. © 2013 IBM Corporation 20 Terminologies Explained  Connection Specification Document  Quality Of Service

21. © 2013 IBM Corporation 21 Connection specification Document  Optional parameter.  Contains details about the connection to the mqtt server.

22. © 2013 IBM Corporation 22 MQTT related terminologies  QOS Quality of Service (QoS). –QoS level 0 (At most once delivery) : The message is delivered according to the best efforts of the underlying TCP/IP network. A response is not expected and no retry semantics are defined in the protocol. The message arrives at the server either once or not at all. –QoS level 1 (At least once delivery) : The receipt of a message is acknowledged by the server by sending back an acknowledgement. If there is an identified failure of either the communication link or the sending device, or the acknowledgement message is not received after a specified period of time, the sender resends the message. The message arrives at the server at least once. A message with QoS level 1 has a Message ID in the message header. –QoS level 2 (Exactly once delivery) : Additional protocol flows above QoS level 1 ensure that duplicate messages are not delivered to the receiving application. This is the highest level of delivery, for use when duplicate messages are not acceptable. There is an increase in network traffic, but it is usually acceptable because of the importance of the message content

23. © 2013 IBM Corporation 23 Brief Architecture Diagram for MQTT adapters

24. © 2013 IBM Corporation 24 Kaveri Messaging Toolkit Operators  MQTTSource  MQTTSink

25. © 2013 IBM Corporation 25  Reads data from a MQTT Provider topic and pushes the data into Streams  Main functionalities: –One input ports, One or Two output ports The first output port contains the data which is read from the MQTT Provider and the second optional output port is the error port –Validates if the application can connect to MQTT Provider during runtime –For Initial/ Transient connection failures, the operator tries to reconnect based on the reconnection policy –On successful read submits the tuple to the output stream corresponding to the message read –Provides capability to know which topic a message was received from when subscribing to one or many topics. –Supports two formats:- bin: This format expects the message to be serialized in binary, using network byte order. Tuple attributes are assumed to be serialized in sequence. block: This format is only applicable when the streams tuple contains only a single attribute of type blob, we copy the raw bytes as is from the Message to the tuple without any parsing or formatting. MQTTSource

26. © 2013 IBM Corporation 26  Publishes data from Streams to a MQTT Provider topic  The main functionality of this operator includes: –One required input port and one optional input port. One output port(optional) The input port contains the data to be published to the MQTT Provider's Topic and optional output port is the error port –Validates if application can connect to MQTT Provider during runtime. –For Initial/ Transient connection failures, the operator tries to reconnect based on the reconnection policy specified by the user –On successful write submits the tuple in the input stream to the corresponding message on the topic –Supports two formats:- bin: In this format the message sent is serialized in binary, using network byte order. Tuple attributes are serialized in sequence. block: This format is only applicable when the streams tuple contains only a single attribute of type blob, we copy the raw bytes as is from the attribute to the message without any parsing or formatting). MQTTSink

27. © 2013 IBM Corporation 27

28. © 2013 IBM Corporation 28