Service Oriented Architecture (SOA) and Complex Event Processing (CEP) – Complementary Views of the Enterprise John Salasin, Ph. D. Defense Advanced Research Projects Agency (DARPA) John.Salasin@darpa.mil, jsalasin@verizon.net

Agenda Objectives Reputed SOA Benefits Conceptual view of an SOA-based system Conceptual architecture of an SOA-based system Reputed CEP Benefits Conceptual view of a CEP system Conceptual CEP Architecture Generalized Event Processing Agent

Objectives To provide: Technical background to justify discussions of similarities and differences; More on CEP than SOA (less familiar) Guidance on the contexts in which the techniques are most useful, and; References to additional information.

Reputed SOA Benefits Core Concept: “Orchestrating” business functions to provide services. Reducing integration expense; Increasing asset reuse; Increasing business agility, and; Reducing business risk. 1) Reducing integration expense by: Using loosely-coupled integration components that reduce the complexity and hence the cost of integrating distributed computing environments; Using standards-based interfaces such as Web Services, and; Replacing multiple function calls for API-based integration with coarser-grained Services that can handle a wider range of interactions in a more flexible manner. 2) Increasing asset reuse -- Service reuse becomes the theme, rather than application integration. ROI derived from: Improved time-to-market, More responsive customer service, Reduction in overall staffing, and; A greater ability to outsource Service creation, implementation, and even composition. 3) Increasing business agility Business users control business process definition and management. Parts of an organization’s business process design, management, and opreation can be delegated to the responsible part of the organization, SOA enables embedding a company’s business processes inside the operations of their suppliers or business partners, or visa versa. 4) Reduction of business risk Increased business agility and control. Improved business visibility in the face of change Ensure compliance with changing regulations. Improved governance offers a reduction of liability. Reduced security risks by establishing corporate-wide security, privacy, and implementation policies, and providing auditable information trails.

Conceptual view of an SOA-based system Requestor – person or another service – is user of service. Registry – is automated broker serving the user (and provider). Provider – where the work is done – service is executed according to agreed on performance and quality standards.

Conceptual architecture of an SOA-based system This is traditional layered architecture view

Reputed CEP Benefits Core concept: specifying and managing conceptual hierarchies, or patterns, of events. Users receive information at the appropriate semantic level (Aggregation); Flexibility in monitoring (run-time specification); Analysis of event patterns (pattern matching against observed patterns or rules/policies), and; Consolidation (correlation across hierarchic levels). users receive information at the appropriate semantic level in terms or concepts that are meaningful to them. We can detect patterns of low level events and generate high level events that correlate and summarize the data in the lower level patterns. Event correlation across hierarchic levels. Flexibility Communicating structures of event processing agents provide rapid reaction and interactive on-the-fly reconfiguration of strategies. Analysis CEP provides drill down diagnostic analysis – given a high-level event, we can determine what lower level events caused it. Consolidation: CEP can correlate a diversity of independent inputs from network-level detectors to application-level sensors (e.g., patterns of events on a manufacturing plant controller to observed defects in products)

Conceptual view of a CEP system NOTE: This has been used to diagram both SOA and CEP.

Conceptual CEP Architecture Event Processing Networks (EPNs) are networks of distributed event processing agents (EPAs) running in parallel . Event service (Similar to Data Service in SOA systems) – 4 components Schema Repository stores Schemas that define the format of events Event Repository Stores records of events templates for processors, and; predefined templates that can be activated to filter, aggregate, or otherwise process events. a consumer registry. (Analogous to routing in an SOA) The consumer registry keeps track of which consumer is interested in which events.

Generalized Event Processing Agent This Figure describes 4 types of Agent. One to start and stop specified event processing activities, a second to filter input patterns and pass only those that meet selection criteria, a third to create an output event on matching one or more input patterns, and a fourth to generate an alert (an output event) when ever a condition (e.g., involving causality, sequencing, or timing) is satisfied Stored templates and schema allow rapid configuration and deployment.

CEP Illustrative Examples (1 of 2) Pattern matching and analysis Match what is actually happening (e.g., a broker “trading ahead” of placing a customer order large enough to change the price of a stock) with pre-defined constraints (e.g. trading ahead is illegal). Causal Tracking Identify the Partially Ordered Set (POSET) of events – at all levels -- related to a specific event of interest (e.g., failure of a machine on a chip fab line) Assure that a sequence of events is correct with respect to policy in complex contexts involving multiparty transactions.

CEP Illustrative Examples (2 of 2) Loan Processing Example -- Causality and Aggregation Events are characterized by: Time – usually sequence Causality – red arrows showing process or service orchestration Multiple (2 in this case) levels of abstraction Aggregation – dashed arrows showing lower level events contributing to the composite event (completed loan) We can, for example: If loan isn’t completed, drill down to discover which lower level event failed Check to ensure that process is followed – e.g., that risk management process for loan X doesn’t start until after income review credit check, and house appraisal Ensure that funding decision (event based on funding service) occurs after risk management – t5 >= t4

When CEP Is Useful (1 of 2) SOA is satisfactory when application is data dependent – where the major concerns are with, e.g.: The magnitude of variables rather than the time at which they are set; Relationships among these magnitudes (e.g., sales / ft3 at a set of stores), and; Triggers are based on magnitude (e.g., order when stock level =< “H”).

When CEP Is Useful (2 of 2) CEP should be added when concern is with patterns of events in real-time. E.G., We: • Know (or suspect) the pattern of events we’re looking for (e.g., trading ahead); • Can filter out a finite set of events from the “cloud” to start the process (e.g., sequences starting with customer requests for a stock trade), or; • Want to respond to the pattern immediately, without the delay in retrospective data searches to see if it occurred.

Potential Added Value of Combining SOA/CEP CEP at Enterprise level could support data marshaling in complex processes Optimize access sequences to allow greatest number (possibly weighted by process priorities or deadlines) of processes to continue. Allow identification of causality or dependency relationships across independent services. A potential problem with enterprise-wide services – conflicts regarding data access and concurrency Since Services can be on indeterminate duration, can’t lock data. => “Optimistic” concurrency control may require Service that’s accessed data to check back to see if it’s been changed since access – if so, may require expensive rollback operation. In a large enterprise, many Services may be trying to access data simultaneously.