1. Complex Event Processing in Oracle RDBMS Use Case: Order Management & Advanced Pricing
Event Processing Symposium November 8th 2006
Aravind Yalamanchi
Principal Member of Technical Staff
Server Technologies

2. Complex Event Processing in RDBMS : Goals
React to streaming data as well as evolving data in the database
Handle continuous queries capable of identifying new results in real-time and acting on them.
Reorder an item from the supplier when a new purchase order pushes the number of items in the warehouse below a threshold.
Coordinate composite requests involving data distributed over time and application space.
Raise an exception when the purchase order from a gold customer is not shipped within two days of the order placement.
Enable real-time analytics and decision making.
Alert the account manager for a customer when the average size of the cancelled orders in the last 30 days is greater than the 60 day average by 25%.
The queries could directly operate on the transactional changes in the database. For example, an INSERT operation into a table could match a query and perform the prescribed action. There are no messages or queues involved.
One of the goals we are set out to achieve in the database is that, the predictive rules that are output by a Data Mining tool can be formulated as the rules or continuous queries with no human intervention. That is, the rules could be machine generated. For example, if a Data Mining tool produces a predictive rule that states that C happens with 90% probability when A and B happen, this can be formulated as a continuous query to monitor A and B in real-time.

3. Approach: Event-Condition-Action Rules
Rules are processed for events that arise from
Event submissions through programmatic interfaces, message queues, or web-services requests.
Transactional or Non-transactional changes to data
Non-occurrence of some expected change (exception)
Rule conditions are expressed using SQL and XML syntax
SQL-WHERE clause for identifying individual events of interest (filtering) using predicates on Scalar, XML, Spatial, or Text data.
XML to correlate events and create complex event patterns
Action preferences determine the exact action for each rule
Pre-packaged actions can be executed in the DB – Alerts, Enqueue, Mail, Web-service requests etc.,.
Results from rule evaluation can be obtained as a SQL result set by querying a view for application level actions.
CEP in the database allows explicit event submissions as well as implicit events from DML activity or non-occurrence of expected change
For most event associations, a timestamp attribute that is part of the event instance is used. The arrival order/time of the event is significant only in two types of rules: 1) Non-occurrence and 2) Window semantics in event collections.
There is no assumption on the correct order of event arrival. There are no optimizations based on the order of event arrival.

4. Types of Event Relationships in ECA Rules
Conjunctions
All the specified events happened
Sequencing
The events happened in the specified order
Disjunction (any n)
Any n of the m specified events happened
Temporal association
Events happened within n time units of each other
Negation
An event did not happen within a deadline
Aggregation
Collections of events following sliding window semantics
Common types of event correlations used in ECA rule conditions
Check for the existence of the events in any order – or simply a conjunction of events
Or the existence of the specified events in a specific order with a event sequencing constructs.
Simple disjunction of events or disjunctions in sets as in Any n out of m events specified.
Temporal constructs check for events happening within n units of time.
Negation constructs react to the non-occurrence of an event, say before the other events or before a set deadline. For example a rule can react to a shipped order event happening before the payment event is detected.
Aggregation constructs gather collections of events using sliding window semantics and tests some aggregate predicates on the event collections. For example, a condition could test the Average size of the purchase order using last 30 purchase orders.
For most event associations, a timestamp attribute that is part of the event instance is used to form a SQL-like predicate. The arrival order/time of the event is significant only in two types of rules: 1) Negation and 2) Aggregation using Window semantics.

5. Order Management System – Sample Rule
If the purchase order from a Gold customer contains more than one item and the order is partially filled, send a multi-part shipment report to the customer.
ON PurchaseOrder (orderId, itemCount, custId, exptdDate ..) porder, ShipItem (itemId, itemType, orderId, itemCount, ..) sitem
IF <condition> <and join= “porder.orderId = sitem.orderId and order.itemCount > sitem.itemCount”> <object name= “porder”> itemCount > 1 and CustomerType(custId) = ‘Gold’ </object> <object name= “sitem”> itemType = ‘Reusable Container’ </object> </and> </condition>
THEN PerformAction(porder.custId, ‘Notify’, ‘Multi-part Shipment’)
Lets consider a rule example involving a composite event. This will demonstrate the use of XML in forming the rule conditions.
This rule says “If the purchase order from a Gold customer contains more than one item and the order is partially filled, send a Multi-part Shipment report to the customer”. Note that in real-world, these events could happen in different applications at different times.
This rule can be expressed declaratively using the standard notation shown here.
In this case, the ON clause identifies the individual primitive events the rule is based on and the IF clause defines the rule condition that is expressed using XML and SQL syntax and the THEN clause defines the rule action.
This rule example states that the rule action will be executed only if it detects a purchase order that meets its criteria, a container shipment event occurs and also, these two events are related to each other with a common order id as well as the number of items ordered being larger than the items in the container.
Although the XML tags in the rule condition makes it verbose, they provide the much needed flexibility especially for defining complex event scenarios which we will discuss with other examples. Some of these scenarios include sequencing, negation, sliding windows and others which are not possible with pure SQL.
Before getting to the other types of rule constructs, we will discuss how the rules are stored and processed in the database.

6. Rules Managed as Data in Relational tables
OrderMgmt Event Structure
OrderMgmtCBK Action Callback Procedure
OrderMgmt (
porder PurchaseOrder,
sitem ShipItem,
truck TruckAtDock) )
PerformAction(..)
rlm$ruleid
rlm$rulecond
ActionType
ActionParam
PRTL_ORDER
<condition> <and join = “porder.orderId = sitem.orderId”> <object name=“porder”> .. </object> <object name=“sitem”> .. </object> </and> </condition>
Notify
Partial Ship
CNTNR_TRCK
<condition> … </condition>
AlertAgent
ContTracker ..
PurcahseOrder (2345, 8, 23456, …);
Event Instances
ShipItem (‘RC123’,’Reusable Container’,2345, 5, …);
In the earlier slide we saw a sample rule and it could just be one rule in a large set of rules used in a typical order management system.
In this slide, we will see how such rules are represented in the database. The rules are stored as data in a rule class table. All the rules stored in a table operate on some common metadata such as PurchaseOrders, ShipItems and TruchAtDock events in an Order Management system.
Each row in this table represents one rule in the application. So, the sample rule we discussed earlier could be the first row in the table. Although there are tools to define the rules, they could be directly manipulated as any other data stored in the database with direct DML operations.
The column storing the rule condition is of Expression data type and it has special semantics. The metadata associated with the column validates any data stored in this column.
At runtime, the instances of the primitive events embedded in the composite event structure are added one at a time. For example, when an instance of the PurchaseOrder event is added, it identifies the rules that are partially true with this event. If there exists a rule (in the rule class) that is completely true with this event, the action callback procedure is called with appropriate arguments (or Action preferences).
Now when a new ShipItem event is added, this event in combination with other events that were added in the past could evaluate some rules to true and the action callback procedure is called with appropriate arguments for all such rules.
Rule Identifier
Rule Condition
Rule Action Preferences
OrderMgmtRC Rule Class (Table)

7. DB Change Notification
ECA Rules - Evaluation
A rule condition is considered true when a set of primitive events satisfying the predicates on the event data also satisfy the event pattern specification.
Each rule condition acts as state machine that reaches an accepting state when the rule condition evaluates to true.
The outcome of a rule matching a set of primitive events is a higher-level event, or a composite event consisting of all the primitive events.
App Server
Rules Application
App Server
DB Change Notification
Database App 
Message Queues
Message Queues
Event instance
Web Services & Other
Web Services & Other
Event Sources
Event Consumers
Events Repository
Incremental Results
A rule condition for complex event pattern is considered true when a set of primitive events satisfying the corresponding predicates on the event data together also satisfy the event pattern specification such as they should all happen with in 10 minutes of each other etc.,
While processing the rule for the primitive events, each rule condition acts as a state machine that reaches an accepting state when all the necessary primitive events are received.
As it was mentioned earlier in the presentation, the events can be separated by time and space and the rules application should keep an event until all the necessary events are evaluated and it also keeps the incremental results from the partial evaluation of rules for efficiency.
The outcome of a rule matching a set of primitive events is a higher-level event or a composite event consisting of all the primitive events.

8. Non-occurrence of events
Route the order to a high priority queue if there is a chance for it to get delayed.
ON PurchaseOrder (orderId, itemCount, custId, exptdDate ..) porder, ShipItem (itemId, itemType, orderId, itemCount, ..) sitem
IF <condition> <and equal= “porder.orderId, sitem.orderId”> <object name= “porder”> CustomerType(custId) = ‘Gold’ </object> <not by=“porder.exptdDate-2”> <object name= “sitem”/> </not> </and> </condition>
THEN PerformAction(porder.orderid, ‘Redirect’,‘High Priority Q’)
The sample rule we discussed earlier could have been an expression using pure SQL as it simply involves a Join. But the use of XML in formulating the declarative rule allows complex event correlation that are not possible with SQL.
This slide shows a rule that tests for the non-occurrence of an event with a time delay.
The time-delay itself is computed from the other event – the purchase order’s expected date.

9. Composite event with Collections
Mark the order as complete when the number of items shipped equal the number of items ordered.
ON PurchaseOrder (orderId, itemCount, custId, ..) porder, ShipItem (itemId, itemType, orderId, itemCount, ..) sitem
IF <condition> <and equal = “porder.orderId, sitem.orderId” having = "count(sitem.*) = porder.itemCount“> <object name= “porder”/> <collection name=“sitem” groupby="orderId“> itemType != ‘Reusable Container’ </collection> </and> </condition>
THEN PerformAction(porder.orderid, ‘Update Order Status’,‘Complete’)
The collections construct can be used to create an aggregation of events.
In this case we are counting all the items shipped for a particular order. At the same time we are excluding any Reusable Containers in this count.
So, this rule creates composite of a collection of shipitems pertaining to a order and the purchase order to determine when the order is complete.

10. Collection of events – Advanced Pricing Rule
Offer a 15% discount on the current purchase order if the customer has at least 10 orders over dollars in the past year.
ON PurchaseOrder (orderId, itemCount, custId, exptdDate, amount) porder
IF <condition> <collection name = “porder” groupby = “custId” windowlen = “365” having = “count(*) >= 10”> amount > </collection> </condition>
THEN PerformAction(porder.orderid, ‘Discount’,‘15’)
The same order management application may have some advanced pricing rules that determine the discount for the purchase order based on the orders in the last year.
Here we use a collection of purchase orders grouped by the customer id and set the window length to 365 days. We are counting for at least 10 orders each with over dollars.
This pricing rule could be one of many such rules that could conflict with each other.

11. Collection of events – Advanced Pricing Rule
Offer a 10% discount on the current purchase order if the average size of the last 5 orders is over dollars.
ON PurchaseOrder (orderId, itemCount, custId, exptdDate, amount) porder
IF <condition> <collection name = “porder”> groupby =“custId” windowsize = “5” having = “avg(amount) > 10000”/> </condition>
THEN PerformAction(porder.orderid, ‘Discount’, ‘10’)
This is yet another rule that offers a discount. Note that a purchase order may match one or more pricing rules recommending conflicting actions.
So, it is necessary to have policies such as conflict resolution to manage these applications.
The logical grouping of the rules pertaining to one application enables us to enforce these elegantly.

12. Logical Grouping of Rules
Declarative policies for event lifecycle management
Consumption: Specification for event reuse
EXCLUSIVE: At most one matching rule executed with the event
SHARED: Event used for multiple rule executions
RULE: Custom consumption policy on a per rule basis
Conflict Resolution: To control the order of rule execution
SQL ORDER BY Clause involving some event and/or rule attributes
Duration: Lifetime of unconsumed events
TRANSACTION/SESSION: Until the end of database tx./session
CALL: Event valid only at the instance it is added
N Units of Time: Elapsed time since the event is first added
Rules can be grouped into hierarchies to manage event abstractions at different levels.
One of the key requirements for ECA rules applications is the ability to manage the events and the rules declaratively and these include the policies to manage the lifecycle of events. The ECA implementation in the databases once again leverages from the database infrastructure to support these policies.
The policies typically vary from one application to another and they are configured with the set of rules defined for a particular application.
One of the policies is the conflict resolution, which controls the order of rule execution. In the database, this is specified as a SQL order by clause involving some of the attributes from the rule action preferences as well as the event attributes. The results from matching some events with a set of rules are automatically ordered using this order-by clause to determine the order in which the rule-actions are executed.
The consumption policy determines the logic for reuse of events for multiple rule executions. If this policy is set to exclusive, one event can be used for only one rule execution. If this policy is set to shared, an event can be shared for multiple rule executions.
The duration policy for the events determines the lifetime of the unconsumed primitive events. By default the events have infinite lifetime and optionally, the user can set them to expire at the end of the transaction, rule/database session, call or a specified time.
Other benefits of managing the rules as logical groups include, the ability to index them so that the filtering and evaluation costs are minimized.
The rules can be grouped into hierarchies to manage event abstractions at different levels.

13. Conclusions
CEP in the database – Production since July 2005 (Oracle 10.2)
Scalable event detection and correlation with persistent (recoverable) state
Support for long running event scenarios
Support for large and dynamic rules sets
Support for rich data types
Declarative policies for event management
Technologies used
Expression Filter for performance and scalability
Database Change Notification, Triggers, Message queues, and Web service requests as event sources
SQL and XML based rule condition language for expressibility
UI modeling tools for ease-of-use
Rule executions can be audited in the database
Flashback feature of Oracle database can be used to debug and replay the rules.

14. Rules Manager feature of Oracle Database
Processes events on multi-terabyte data sets w/ rules in the database
Applications Generating Events
PL/SQL APIs
SQL
JDBC
SQL*Net WS
ECA Rules & Indexing
Event Policies
Persistent Events State
Events
Storage
Action Callback Procedure
Rules
Results
View
Oracle Database with Rules Manager
Persistent state for incremental evaluation & event coordination
Indexing to identify applicable rules within large rule sets
Any Oracle data type in rule conditions, including XML, spatial
Declarative event relationships: AND, ANY, NOT, sets, order, time
Dynamic, batch and DML events w/ policies to control behavior
Rules Manager is a feature of Oracle Database 10g Release 2 to manage declarative Event–Condition–Action (ECA) rules in Oracle Database for standalone (primitive) and composite events comprising multiple primitive events. Application developers building service oriented applications (SOA) with Java, C and PL/SQL can:
Evaluate rules in the database for events on multi-terabyte data sets to minimize network traffic and application server memory requirements, and make rule changes transparent to an application
Maintain persistent state for composite events (comprising multiple primitive events) to incrementally evaluate rules, and to coordinate the primitive events across multiple applications, event threads and time.
Index rules to identify the applicable rules among many thousands in a rules set
Include any SQL data type in a rule condition, including XML (XMLType) and Spatial (SDO_Geometry)
Declare rules for composite events that identify the individual primitive event conditions (using standard SQL Where clause syntax) and relate them using: AND, ANY and NOT with a set of events, Any n, sequence, time deadline
Respond to dynamic, batched and DML events (an application can submit an event dynamically for processing or batch events in database tables. DML Insert/ Update/ Delete events can occur within a transaction and/or after commit.)
Stipulate policies for event and rule processing behavior: Consumption to control whether an event can be considered in multiple rule executions; conflict resolution to determine ordering of composite events and matching rules; and duration to control the lifetime of an event.
Manage rules and event policies dynamically in Oracle Database with application data using Rules Manager PL/SQL APIs, SQL DML, Oracle Enterprise Manager and a JDeveloper plug-in
Develop action logic in a database action callback procedure, and/or an application triggered by querying a results view of events and satisfied rules. (actions within Oracle Database Util_SMTP & Util_HTTP packages include the ability to send and SOAP messages, and schedule and queue actions; user-defined logic can be written in C, Java, and PL/SQL)

15. Rules Manager Application Order Management System
In the following slides we will discuss the steps involved in creating a rules application using the Rules manager feature.

16. Rules Application - Step 1
Create the Object type(s) that represent the event structure(s).
TYPE PurchaseOrder as OBJECT ( Primitive event type --
orderId NUMBER, custId NUMBER,
exptdDate DATE, itemId NUMBER,
itemCount NUMBER, amount NUMBER);
TYPE ShipItem as OBJECT (
orderId NUMBER, itemId NUMBER,
itemType VARCHAR(30), loadId NUMBER);
TYPE TruckAtDock as OBJECT (
loadId NUMBER, truckId NUMBER,
status VARCHAR2(30), capacity NUMBER);
TYPE OrderMgmt as OBJECT ( Composite event type --
porder PurchaseOrder,
sitem ShipItem,
truck TruckAtDock);
Create the metadata for the rules. The composite event type can be configured to make use of PurchaseOrder, ShipItem and TruckAtDock tables to operate rules directly on the table data.
Alternately, the OrderMgmt type can be configured to make use of PurchaseOrder,
ShipItem, and TruckAtDock tables with corresponding columns.

17. Rules Application - Step 2
Create the Rule Class using the event structure(s) created in step 1 and set the rule class properties that include event management policies.
DBMS_RLMGR.CREATE_RULE_CLASS (
rule_class => ‘OrderMgmtRC’, Rule Class Name --
event_struct => ‘OrderMgmt’, Event structure name --
action_cbk => ‘OrderMgmtCBK’, -- Action callback proc --
actprf_spec => Action pref categories --
‘actionType VARCHAR(40),
actionParam VARCHAR(100)’,
rslt_viewnm => 'MatchedRules', Results view (optional) –
rlcls_prop => Rule class properties ‘<composite
equal="(porder.orderId, sitem.orderId) |
(sitem.loadId, truck.loadId)“
ordering="rlm$rule.rlm$ruleid, porder.orderid”>
<collection type=“PurchaseOrder” ... />
</composite>’ );
The programmatic interface created the rule class (the rule class table to store the rule definitions) and configures it with the appropriate metadata and properties.
The optional results view can be specified to operate the rules manager application in a multi-tier mode, where the rules can be evaluated for some events and the view can be queried (from any tier within the same database session) to obtain the list of matched rules and the recommended actions. With this configuration, the actions can be executed in the middle-tier.

18. Rules Application - Step 3
Implement the action callback procedure’s body to perform the appropriate action for each rule matching a set of primitive events. The skeleton for the procedure is generated by the rule class creation step.
CREATE OR REPLACE PROCEDURE OrderMgmtCBK (
porder PurchaseOrder, primitive events that matched the rule --
sitem ShipItem,
trucl TruckAtDock,
rlm$rule OrderMgmtRC%ROWTYPE) IS rule that matched --
BEGIN
PerformAction(decode(porder.orderId, null, ..),
rlm$rule.actionType,
rlm$rule.actionParam);
END;
The Action callback procedure can be implemented to carry the action for each matching rule. This implementation can rely on the list of primitive events that matched the rule and any information that is stored with the rule, such as the rule action preferences.

19. Rules Application - Step 4
Add rules to the rule class using standard DML statements (or UI tools).
INSERT INTO OrderMgmtRC (rlm$ruleid, actionType, actionParam,
rlm$ruledesc, rlm$rulecond) values
('Delayed Order redirect', 'REDIRECT','HIGH_PRIORITY_QUEUE',
'Route the order to a high priority queue if there is a chance for it to get delayed',
'<condition> <and equal= “porder.orderId, sitem.orderId”> <object name= “porder”>
CustomerType(custId) = ‘Gold’ </object> <not by=“porder.exptdDate-2”>
<object name= “sitem”/> </not> </and> </condition>');

20. Rules Application - Step 5
Process the rules for the primitive events.
dbms_rlmgr.process_rules(
rule_class => ‘OrderMgmtRC’,
event_inst => AnyData.convertObject(
PurchaseOrder(12345, 345, ’11-Nov-2006’, ...)));
ShipItem(12345, 567, ’Reusable Container’, ...)));
The action callback procedure is called when an event causes a rule to reach an accepting state.
Now process the rules in the set using a process_rule call for each primitive event. Even though the above slide shows two process_rules calls within a single PL/SQL block, these two calls could be from two different SQL-Plus sessions and the net result is the same.
When a rule reaches an accepting state with a primitive event just added to the system, the action callback procedure is invoked with appropriate arguments.