Presentation is loading. Please wait.

Presentation is loading. Please wait.

Systemübergreifende Prozessüberwachung mit Event Stream Processing Thomas Dücker, SAP Schweiz Juni, 2013.

Similar presentations


Presentation on theme: "Systemübergreifende Prozessüberwachung mit Event Stream Processing Thomas Dücker, SAP Schweiz Juni, 2013."— Presentation transcript:

1 Systemübergreifende Prozessüberwachung mit Event Stream Processing Thomas Dücker, SAP Schweiz Juni, 2013

2 ©2013 SAP AG or an SAP affiliate company. All rights reserved.2 Agenda Herausforderung und Trends Übersicht Operational Process Intelligence  Übersicht  Beispiele  Szenario Mehrwert für Sie Weitere Informationen

3 ©2013 SAP AG or an SAP affiliate company. All rights reserved.3 Event Stream Processing CaptureDisseminate & DecideResponse & Action

4 ©2013 SAP AG or an SAP affiliate company. All rights reserved.4 Traditional Approaches to Event Stream Processing DB application (OLTP) —Pull not push (i.e. not event driven) or —Requires triggers – performance degradation / maintenance nightmare —latency in seconds Custom application (C/C++ / Java) —Specialized, high development / maintenance cost —Slow to change, unresponsive to the business Application or Dashboard High-Speed Data Queries / Results Database Application High-Speed Data “Black-box” Application Alerts / Actions Custom Application

5 ©2013 SAP AG or an SAP affiliate company. All rights reserved.5 The SAP Approach – Sybase ESP Combines advantages of customized application with ease of visual data flow diagramming or SQL Incoming data is processed as it arrives, according to the model rules and operations Publish streaming results to other models, apps, message bus, dashboard, etc. ? Input Streams PUBLISH Market Data Orders, Trades Console Input Reporting Tools Trading Systems Visual / SQL Authoring Reference Data SAP HANA / Sybase IQ Dashboards Message Bus Output Streams SUBSCRIBE

6 ©2013 SAP AG or an SAP affiliate company. All rights reserved.6 CEP Product Convergence Aleri 3.2 Sybase ESP 5.0 Sybase ESP 5.1 Sybase CEP R4 Coral8 v5.6.5 Sept 2011 Sept 2012 Optional (maintenance only) Sybase ESP 5.1 SP1 Dec 2012

7 ©2013 SAP AG or an SAP affiliate company. All rights reserved.7 Advantages of Sybase ESP Approach Rapid Application Development Reduce/eliminate dependence on specialist programming skills for process Cut implementation/deployment time Broad out of the box connectivity Non-intrusive Deployment Event-driven integration with existing systems Unify existing disparate data models Analyze Events as they occur Continuous Insight Respond Immediately

8 ©2013 SAP AG or an SAP affiliate company. All rights reserved.8 Event Stream Processing Scenarios Capital Markets React to incoming ticks to determine trading strategies, monitor the success of the strategies by calculating PnL in real-time; monitor risk in real-time. Retail Provide clickstream analysis for online customer assistance / alternatives, use web scraping to react to competitor price changes

9 ©2013 SAP AG or an SAP affiliate company. All rights reserved.9 Event Stream Processing Scenarios Energy Monitor and optimize energy usage using Smart Metering to ensure QoS Utilities / Manufacturing Service Status Monitoring and Process Control using SCADA / OSISoft

10 ©2013 SAP AG or an SAP affiliate company. All rights reserved.10 Event Stream Processing Scenarios Telecoms Monitor events for QoS, use DPI to detect fraudulent use Police / Homeland Security ANPR event correlation for traffic monitoring, security

11 ESP Architektur

12 ©2013 SAP AG or an SAP affiliate company. All rights reserved.12 ESP Project Architecture Publish Subscribe Data Streams Authoring Tools Adapters I/O Gateway Security Adapters Stream Processor + Multi-threaded + 64 bit - Suse 11 / Redhat 6 / Solaris 10 (Sparc / i86) Win x64 + Low latency + Optional persistence C+C (XML-RPC) Operational Console Security (PAM, SSL, RSA, Kerberos) Data Streams Mem Store Log Store I/O Gateway Data Streams Log Store Command & Control Studio

13 ©2013 SAP AG or an SAP affiliate company. All rights reserved.13 Out of the Box Connectivity In Process JMS, MQ, TIBCO ATOM (RSS) TCP Sockets Databases via JDBC / ODBC Files MS Excel RT SMTP (out) External Market Data ( Reuters, Bloomberg, IDC, ACTIV, NYSE Tech) FIX OpenAdaptor Java Framework  Replication Server  IQ / HANA PubSub SDK C/C++ Java.NET

14 ©2013 SAP AG or an SAP affiliate company. All rights reserved.14 ESP Cluster Architecture I Controller Manager Project Server Controller Manager Nodes Cache Project Server

15 ©2013 SAP AG or an SAP affiliate company. All rights reserved.15 Manager Node − Deploys project(s) to server(s) − Maintains heartbeat with project(s) to detect failure − Manages failover − In multi node, managers are run in a cache so can failover to manage other controller nodes Controller Node − Basically acts as a launch daemon for Manager to start project(s) in a Container ESP Cluster Architecture II

16 ©2013 SAP AG or an SAP affiliate company. All rights reserved.16 Single node cluster – − Single manager/controller node runs multiple projects on a single machine − Detects when a project fails and tries to restart it − This is the default for project development Multi node cluster − Multiple equi-peer managers and controllers run multiple projects on multiple machines − Used to scale out projects across multiple machines − Allows for server failure – promotes failover recovery and data redundancy − Allows for Highly Available “active:active” deployment of projects within a Data Centre ESP Cluster Architecture III

17 ©2013 SAP AG or an SAP affiliate company. All rights reserved.17 Three clusters created by install script: - Studio Cluster (single node) on 9786 : – − Used for Studio Development − Can run multiple projects − Located in $ESP_HOME/studio/clustercfg User-defined Cluster (single node) on − Located in $ESP_HOME/cluster/nodes/node1 Example multi-node Cluster on − Located in $ESP_HOME/cluster/examples ESP Installed Clusters

18 ESP Key Concepts

19 ©2013 SAP AG or an SAP affiliate company. All rights reserved.19 Analyst-level skills (Excel, VBA) Easy to understand complex models No need to learn language syntax Rapid programming Easy to use language (CCL – derived from SQL) Modular, project based approach Visual Dataflow AuthoringCCL Authoring Authoring Options – Eclipse Plugin

20 ©2013 SAP AG or an SAP affiliate company. All rights reserved.20 Visual Dataflow Authoring

21 ©2013 SAP AG or an SAP affiliate company. All rights reserved.21 CCL Authoring

22 ©2013 SAP AG or an SAP affiliate company. All rights reserved.22 Procedural Logic - SPLASH

23 ©2013 SAP AG or an SAP affiliate company. All rights reserved.23 IMPORT 'CreateCompleteChannel.ccl'; CREATE MODULE CreateCompleteChannel IN MessageIn, CacheIn OUT MessageOut BEGIN // Module Code END; LOAD MODULE CreateCompleteChannel AS Channel_A IN MessageIn = MessageA, CacheIn = CacheA OUT MessageOut = MessageOutA; LOAD MODULE CreateCompleteChannel AS Channel_B IN MessageIn = MessageB, CacheIn = CacheB OUT MessageOut = MessageOutB; ESP Modularity

24 ©2013 SAP AG or an SAP affiliate company. All rights reserved.24 Project Code, Compiled Code & Project Resources Project Source Code.CCL (Continuous Computation Language) − Used for schema, stream, window, method definitions compiled into Project Executable.CCX (Continuous Computation eXecutable) Project Resource File.CCR (Continuous Computation Resource) − An xml file used for in process adaptor, cluster, bindings, failover, HA definitions – allows for easy migration between DEV, TEST and PROD

25 ©2013 SAP AG or an SAP affiliate company. All rights reserved.25 Constructing ESP Projects Build model Connect to data feeds using publish adaptor Connect to output devices via subscribe adaptor ESP Project Container PositionsConsole Trades Tick data feed DB Replicated Data

26 ©2013 SAP AG or an SAP affiliate company. All rights reserved.26 Events are retained as records in memory with a primary key so that further events may modify the state of the record - either update or delete it. A stateful event must have PK attribute(s) set. A stateful event MUST have an associated operation code (opcode), one of INSERT, UPDATE, UPSERT, DELETE. If no opcode is provided, ESP defaults it to UPSERT. When an event occurs it applies the opcode through the data flow model depending on the opcode. An insert in a source stream may become an update to an aggregate or join stream. The full set of unique (insert) events to be retained MUST fit in memory. Records in memory can be removed either by setting a retention window, or by publishing a DELETE event for that key. In ESP, stateful events are implemented using a WINDOW Stateful Events

27 ©2013 SAP AG or an SAP affiliate company. All rights reserved.27 An event is retained only while it is being processed. When an event moves from one stream to another it is removed from the first stream so it cannot be updated. All events are treated as insert. A stateless event ignores any PK attribute(s) and always outputs an insert In ESP stateless events are implemented with a STREAM Stateless Events

28 ©2013 SAP AG or an SAP affiliate company. All rights reserved.28 ESP implements stateful and stateless events using 3 element types WINDOW : used for stateful events. A WINDOW maintains state for a user defined period STREAM : used for stateless events. A STREAM does not maintain or react to state DELTA : used to process stateful events, but without maintaining state All can be INPUT, LOCAL, OUTPUT INPUT subscribe to input adaptors i.e. events are published to an INPUT. They are the entry points for events to enter an ESP project. LOCAL perform processing logic. They can NOT be published or subscribed to OUTPUT are derived elements which perform processing logic which CAN be subscribed to ESP Elements

29 ©2013 SAP AG or an SAP affiliate company. All rights reserved.29 WINDOWS must have a PK set Can have an optional retention period set Maintain state in memory or persisted to disk. i.e. State can be recovered in the event of system failure Can publish to STREAM, WINDOW or DELTA Only a WINDOW can be used for aggregation At lease one side of a join MUST be a WINDOW (unless using SPLASH). The side which maintains state depends on the type of the join. Use WINDOW when developing models as they help debugging Windows (stateful)

30 ©2013 SAP AG or an SAP affiliate company. All rights reserved.30 Stream Cannot be persisted so any data is lost in the event of system failure Can NOT publish to DELTA stream Can publish to a WINDOW, but WINDOW must aggregate the input Delta Can only be LOCAL, OUTPUT A stateless element (a STREAM) that understands and processes state (a WINDOW) Can only use a WINDOW as a source Must have a PK set As it does not store events it can NOT be used for aggregation. Cannot be persisted so any data is lost in the event of system failure Streams (stateless)

31 ©2013 SAP AG or an SAP affiliate company. All rights reserved.31 Example: Simple Position Maintenance AverageTrades Window Symbol* Last Price Weighted Avg Price Last Time Aggregate Positions By Book Stream BookId* Symbol* Current Position Average Position Join Trades Stream Symbol* Time* Price Shares Position Window BookId* Symbol* Shares Held Total Book Window BookId* Current Position Average Position Aggregate

32 ©2013 SAP AG or an SAP affiliate company. All rights reserved.32 MSFT12:05: IBM12:05: CSCO12:06: AveragePrices Window Symbol* Last Price Weighted Avg Price Last Time Aggregate Individual Positions Stream BookId* Symbol* Current Position Average Position Join Position Window BookId* Symbol* Shares Held Book Positions Window BookId* Current Position Average Position Aggregate Trades Stream Symbol* Time* Price Shares MSFT12:06: MSFT :05:23 IBM :05:24 CSCO :06:01 MSFT :06:02 IBM :05:24 CSCO :06:01 Book1MSFT3000 Book1IBM700 Book2MSFT1000 Book1MSFT Book1IBM Book2MSFT Book Book Book1MSFT Book1IBM Book2MSFT Book Book Example: Simple Position Maintenance

33 ESP Solution Architektur

34 ©2013 SAP AG or an SAP affiliate company. All rights reserved.34 Event Processing Solution Architecture Events don’t happen in isolation Most Event Stream Scenarios need access to historic data Pattern Detection Risk Calculation Optimisation algorithm To ensure sub millisecond response times use in memory e.g. HANA Many Event Stream Scenarios need access to master data Data written directly to OLTP system Updates to reference, semi-static data For sub second updates without polling use log trawler e.g. Replication Server

35 ©2013 SAP AG or an SAP affiliate company. All rights reserved.35 In-database Algorithms Analytical data Historic data Reference data Real-time ANPR Alerting Framework Event Stream Processor CEP Engine Input Adapter Output Adapter HANA SQL Monitoring BusinessObjects DataService External Databases Business Inteligence BO Predictive Analysis Predictive models (Ad-hoc analysis) OCR Radars

36 ©2013 SAP AG or an SAP affiliate company. All rights reserved.36 Real-time Retailing Framework ESP HANA Detect Event Validate Event Check Results Output Run Optimization Algorithm Manage Exception Phase 1: Event Detection & Validation Phase 2: Processing Phase 3: Validation & Exception Phase 4: Output Straight Through Processing Is competitor price<95% of our price? Is optimum price<95% of current our price? GUI

37 ©2013 SAP AG or an SAP affiliate company. All rights reserved.37 Integration with Hana I Generic input / output ODBC adapter Data can enter via a source stream

38 ©2013 SAP AG or an SAP affiliate company. All rights reserved.38 Integration with Hana II Can be called from a FlexStream using SPLASH for event driven query: - ON TRADE { if (TRADE.Price <> optimalPrice) { [ string s; | ] rec; vector (typeof(rec) v := new vector(typeof(rec)); getData(v,'HanaDSN','CALL DPTL.sp_price_opt (?, ?, ?, ?)', TRADE.Id, TRADE.Datetime, TRADE.Sector, TRADE.Price); if (not (isnull(v))) { newrec := v[0]; } // rest of method }

39 ©2013 SAP AG or an SAP affiliate company. All rights reserved.39 Trading Systems Showing: Concentration limits Clustering Outliers Etc… Trading Systems Market Data Trading Systems Risk Sensitivities Bus Cleansing Normalisation Enrichment Real-time Aggregation of Risk and P&L CEP Continual Mark t o Market HANA Scenario Shifts VAR calculations Analytics: Heatmaps Drill-downs Reports By: Instrument Industry Country Debt Rating Counterparty VaR Time Horizon Etc… Transformation of granular data Real-time Risk Framework ASE Reference Data Replication Server

40 ©2013 SAP AG or an SAP affiliate company. All rights reserved.40 Oil & Gas - Energy Efficiency Reporting Framework SOI2 Replication flow Event flow RS HANA ODBC

41 ©2013 SAP AG or an SAP affiliate company. All rights reserved. Besten Dank Kontaktinformation: Thomas Dücker SAP Schweiz

42 ©2013 SAP AG or an SAP affiliate company. All rights reserved.42 © 2013 SAP AG or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Please see for additional trademark information and notices.http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark

43 ©2013 SAP AG or an SAP affiliate company. All rights reserved.43 © 2013 SAP AG oder ein SAP-Konzernunternehmen. Alle Rechte vorbehalten. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden. Einige der von der SAP AG und ihren Distributoren vermarkteten Softwareprodukte enthalten proprietäre Softwarekomponenten anderer Softwareanbieter. Produkte können länderspezifische Unterschiede aufweisen. Die vorliegenden Unterlagen werden von der SAP AG und ihren Konzernunternehmen („SAP-Konzern“) bereitgestellt und dienen ausschließlich zu Informationszwecken. Der SAP-Konzern übernimmt keinerlei Haftung oder Gewährleistung für Fehler oder Unvollständigkeiten in dieser Publikation. Der SAP-Konzern steht lediglich für Produkte und Dienstleistungen nach der Maßgabe ein, die in der Vereinbarung über die jeweiligen Produkte und Dienstleistungen ausdrücklich geregelt ist. Keine der hierin enthaltenen Informationen ist als zusätzliche Garantie zu interpretieren. SAP und andere in diesem Dokument erwähnte Produkte und Dienstleistungen von SAP sowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und verschiedenen anderen Ländern weltweit. Weitere Hinweise und Informationen zum Markenrecht finden Sie unter


Download ppt "Systemübergreifende Prozessüberwachung mit Event Stream Processing Thomas Dücker, SAP Schweiz Juni, 2013."

Similar presentations


Ads by Google