1. Internet of Things
Amr El Mougy
Alaa Gohar

2. Complex Event Processing
(CEP)
** O. Etzion, P. Niblett, Event Processing in Action, Manning Publications, 2011.

3. It’s a Big Data World
Real-world systems may generate massive amounts of data
Valuable information is hidden but often difficult to discover
Ex: detecting suspicious activity in an on-line banking system

4. What are Events?
An event is an occurrence within a particular system or domain; it is something that has happened, or is contemplated as having happened in that domain. The word event is also used to mean a programming entity that represents such an occurrence in a computing system.
Events can be processed in a traditional synchronous approach or an event-driven approach
Synchronous: attendant takes order, makes coffee, prepares pastry, takes money, gives order
Event-driven: order is made at cashier, causes different person to make coffee, different person makes pastry, etc.

5. How can we use CEP?

6. What is CEP?
Event processing is computing that performs operations on events. Common event processing operations include reading, creating, transforming, and deleting events.
Distinguished by:
Abstraction: event processing logic can be separated from application logic, making both easier to maintain.
Decoupling: event producers are separate from event consumers. Thus, events can be consumed by different applications without them needing to learn about each other’s existence.
Real-world focus: CEP systems are driven by events that happen, or may happen, in the real-world.

7. Example: Flower Delivery System
Description:
A consortium of local flower stores established an agreement with independent van drivers to deliver flowers.
When an order is made, it is broadcast to drivers within the region of the store, with pick up time and delivery time.
A driver is assigned and the customer is notified of the delivery schedule.
Driver makes pick up and delivery and customer signs upon delivery.
A ranking system is maintained for drivers. Each store can specify a minimum rank and whether drivers will be chosen automatically or receive applications.

8. Example: Flower Delivery System

9. System Parts Phase 1: Bid Request Phase 2: Assignment Phase
Delivery lifecycle starts when a delivery request is made
The system enriches the request by adding minimum driver ranking
Each van is equipped with a GPS system that transmits location events
The system translates these longitudes and latitudes into city regions
The delivery request is matched to the drivers within the region with minimum ranking
A bid request is broadcast to these drivers
Phase 2: Assignment Phase
Driver responds to the bid request by sending a delivery bid, specifying current location.
The assignment process is either manual or automatic.
If its manual, the system produces the top 5 ranking delivery bids that match the bid request. The store choose one and creates an assignment manually.
If the assignment is automatic, the first bidder among the top 5 drivers wins.
In either case, the assignment is sent to the driver.
If there are no bidders within 1 min, an alert is produced.

10. System Parts Phase 3: Delivery Process Phase 4: Ranking Evaluation
When driver picks up flowers a pick-up confirmation event is produced. The event is timestamped.
When flowers are delivered, the customer signs and creates a delivery confirmation event, also timestamped.
A pick-up alert is created if the driver does not pick-up the order after 5 mins of agreed upon time.
A delivery alert is created if the delivery confirmation is not received after 10 mins of the required delivery time.
Phase 4: Ranking Evaluation
Drivers are re-evaluated each time they complete 20 deliveries.
If there are no delivery alerts the rank is increased by one point.
More than 5 delivery alerts causes the ranking to be decreased by one.
If the rank increased while the previous evaluation caused a rank decrease, an improvement note is produced.

11. System Parts Phase 5: Activity Monitoring
The system counts the number of assignments per day for each day the driver is active.
The system creates monthly reports that specify
A permanently weak driver is a driver with fewer than five assignments on all the days on which the driver has been active.
An idle driver is a driver with at least one day of activity that had no assignments.
A consistently weak driver is a driver whose assignments, when active, are at least two standard deviations lower than the average assignments per driver on that day.
A consistently strong driver is a driver whose daily assignments are at least two standard deviations higher than the average number of assignments per driver on each day in question.
An improving driver is a driver whose assignments increase or stay the same day by day.

12. Principles of CEP Events are decoupled from application components
The same event can be consumed in different ways by different components. Ex: the event ‘employee hired’ is used by HR to create contracts and used by security to create access.
Since the produces does not know what the consumer is going to do with the event, it generally does not wait for a response, other than delivery ACK.
However, an event may trigger other system components to generate other events. Ex: delivery request  delivery bid  assignment  etc.
Thus, events are usually one-way. Push notifications are used rather than request/response messages
Publish/subscribe middleware can be used to facilitate communications

13. CEP Architecture
Event producer: an entity at the edge of an event processing system that introduces events into the system.
Event consumer: an entity at the edge of an event processing system that receives events from the system.
Event processing agent: a software module that processes events.
Two types of events
Raw event: an event that is introduced into an event processing system by an event producer.
Derived event: an event that is generated as a result of event processing that takes place inside an event processing system.
Event processing network (EPN) is a collection of event processing agents, producers, consumers, and global state elements connected by a collection of channels.

14. Event Properties
Each event is described using a set of parameters that are included in its header
Parameters include: occurrence time, annotation, certainty, identity and source
Ex: Delivery bid header shown in table
Each event also has a payload that includes different data types
Ex: payload of Delivery bid event shown below

15. Event Properties Delivery Request Payload GPS Location Event Payload
Pickup Confirmation Payload
Manual Assignment Payload
Delivery Confirmation Payload
Delivery Request Cancelation Payload

16. Instance of Delivery Request
Event Properties
Instance of Delivery Request

17. Event Producers in the Flower Delivery System
We have four event producers: Store, driver, GPS sensor, vehicle
The target of each event can either be a transmission channel or another EPA
Store Events
Driver Events
GPS Events
Vehicle Events

18. Event Consumers
An event consumer can be a hardware actuator, a human interaction device, or software (log, data analysis, etc.)
In the flower delivery system there are three event consumers: driver, store, and driver monitoring entity
Driver Event Consumer
Store Event Consumer
Driver Monitoring

19. Types of Event Processing Systems
STATELESS EVENT PROCESSING An event processing agent is said to be stateless if the way it processes one event does not influence the way it processes any subsequent events (one event in/out).
it is also possible to have event processing agents that take collections of events as their input or create one or more new events as their output:
An incoming event may be split into multiple events, each containing a subset of the information from the original event.
A stream of multiple incoming events may be aggregated to produce a stream of derived events. The derived events are typically computed from more than one event from the input stream (for example, an output event might contain a running total of a value contained in the input events).
Two streams of incoming events may be composed to produce a stream of derived events.
EVENT STREAM a set of associated events. It is often a temporally totally ordered set (that is to say, there is a well-defined timestamp-based order to the events in the stream). A stream in which all the events must be of the same type is called a homogeneous event stream; a stream in which the events may be of different types is referred to as a heterogeneous event stream.
STATEFUL EVENT PROCESSING An event processing agent is said to be stateful if the way it processes events is influenced by more than one input event.

20. Types of Processing Agents
Filter agents: These are used to eliminate uninteresting events. A Filter agent takes an incoming event object and applies a test to decide whether to discard it or whether to pass it on for processing by subsequent agents. The Filter agent test is usually stateless, in other words, a test based solely on the content of the event instance.
Ex: a test that discards a transaction reported event if its value is less than $100.
Pattern detect agents: These take collections of incoming event objects and examine them to see if they can spot the occurrence of particular patterns.
Ex: use a Pattern detect agent to discard the first two failed logon attempt events, but pass through subsequent ones. Pattern detect agents can emit derived events that describe the pattern that they have detected instead of, or in addition to, passing through the incoming event objects.
Transformation agents: These modify the content of the event objects that they receive.

21. EPAs and EPNs in the Flower Delivery Example
We have three types of EPAs to design our EPN: filter, transform, pattern detection
We first discuss the EPN for the bid request system

22. Location Service EPN
Assignment System EPN

23. Control System EPN

24. Ranking and Reporting EPN

25. Putting Events into Context
A context is a named specification of conditions that groups event instances so that they can be processed in a related way. It assigns each event instance to one or more context partitions. A context may have one or more context dimensions and can give rise to one or more context partitions.
Context can play a big role in deciding how events should be processed
Ex: an anti-theft system implemented a jewelry store detects one person going into the store. If it is night time then an alerts is produced, if it is morning then this is a customer
Context dimension specifies the aspect of the event that is used to categorize the event.
There are four main dimensions: temporal, spatial, state, segment
Context specification is the set of conditions that make up the context, and the context partition is a group of event instances

26. How is Context Used
A stream, as we have defined it, can comprise an open-ended set of event objects. If you want to perform an operation on the stream you can’t wait until all these event objects have been received. Instead you have to divide the stream into a sequence of context partitions, or windows, each of which contains a set of consecutive events. You can then define the operation in terms of its effect on the events in a window. The rule that determines which event instances are admitted into which window is something we call a temporal context.
A stream could contain events that aren’t particularly connected to one another, even though they might occur close together in a temporal sense. They might, for example, refer to occurrences in different locations, or to occurrences involving different entities in the real world. Suppose you were to process the stream with a stateful agent, such as a simple Aggregate agent that counts the number of events. By default this would count all the events in the stream, but what if you want to see separate totals for each location where the events occurred? To do this you need to have a separate agent, or at least a separate instance of the agent, processing the events for each location. Spatial contexts and segmentation-oriented contexts let you assign related events to separate context partitions. You can then have each partition processed by a distinct instance of the event processing agent, so that events in one context partition are processed in isolation from the events in other partitions.
Context also allows event processing agents to be context sensitive, so that an agent that is active in some contexts may be inactive in others. We refer to this as state-oriented context.

27. Temporal Context
A temporal context consists of one or more time intervals, possibly overlapping. Each time interval corresponds to a context partition, which contains events that happen during that interval.
Ex: an intrusion detection system that produces alerts if someone makes more than three password changes in one day. Here the context partition is 24 hrs.
A system may have several context partitions defined over overlapping periods. Here the event instance can belong to more than one partition
The same EPA can define different partitions for different uses, overlapping or not
The figure shows different types of temporal contexts

28. Spatial Context
Groups event instances according to geographical characteristics
Can be a coordinate (long/lat, x/y, etc.) or a spatial identifier (building address, etc.)
Spatial context can be defined as a fixed location, entity distance, or event distance

29. State-Oriented Context
In many systems, events are processed in different ways according to the current state of the system.
Ex: an airport with a heightened threat level performs close monitoring of all passengers. Thus, smaller incidents may produce alerts
A state-oriented context has a single partition. It is controlled by an external entity, and the decision of whether to include an event in the partition is based on the state of the external entity at the time when the event occurs or is detected.

30. Segmentation-Oriented Context
A segmentation-oriented context assigns events to context partitions based on the values of one or more event attributes, either using the value of these attribute(s) to pick a partition directly, or using predicate expressions to define context partition membership.
Ex: the context partition in a system can be the customer. Thus, the system groups all events related to the same customer.

31. Contexts in the Flower Delivery System
Driver Context
Request Context
Monthly Context
Daily Context
Daily Active Driver Composite Context
Can be used to create composite events
When Active Context