Stream Processing with Tamás István Ujj

A database is nothing but our conception of it; what is man to say it differs from a stream in nature… Lambda Architecture

Customer Relationship Management Business Process Management Software Quality Management Application Development Manufacturing Support Business Intelligence Machine Learning Big Data

Telecommunications Manufacturing Financial Sector Our Customers

A real-time data architecture

I want to do complex calculations on large amounts of data. You need a batch processing system.

Staging Area New Data Transformation Logic Results New data is written to a temporary staging area. A scheduled batch job executes the transformation logic.

We changed the logic. Let’s recalculate the previous results, too. Recomputation will cost you extra.

Staging Area New Data Transformation Logic Results ETL Master Dataset Transformation Logic (New) Transformation Logic Master Dataset: an immutable, append-only set of raw data. Results (New) Results can be recomputed from historical data.

Why do I have to wait hours for the updated results?! We’ll have to reengineer the system for low latency.

Nathan Marz: Big Data Principles and best practices of scalable real-time data systems

Staging Area New Data Transformation Logic Batch Results Batch Results ETL Master Dataset Transformation Logic (Streaming) Stream Engine Real-Time Results Real-Time Results The batch layer calculates the results with high latency. The speed layer calculates the results on the most recent data in real-time. The batch layer calculates the correct results with high latency. The speed layer calculates the approximate results on the most recent data in real-time.

Your architecture costs me a fortune! This is the price of Big Data.

You don’t need the batch layer. Interesting. That’s half the costs. Stream processing isn’t reliable on its own!

A well-designed streaming system provides exactly-once semantics, even in case of failure. Receiving the data Kafka is a reliable source. Tracking the offsets in checkpoints. Transforming the data Repeatable transformations. Pushing out the data Idempotent updates. Transactional updates. (Saving results and offsets.) Offset

Staging Area Ne w Dat a Transformation Logic Batch Results Batch Results ETL Master Dataset Transformation Logic Stream Engine Real- Time Results Real- Time Results Offset Transformation Logic (New) Offset (New) Real- Time Results Real- Time Results Kafka retains incoming data. Recomputation: processing data from the beginning of the stream with a parallel streaming job.

How can I stream data from my databases? A stream is an ever- growing, immutable set of events. Under the hood, a database is also a stream of events: creates, updates and deletes.

A database is a view over this stream of events. Create Update Delete Create Update Delete Update Delete Database Let’s capture this internal stream.

A consistent snapshot of the entire database contents at one point in time. A real-time stream of changes from that point onward. PostgreSQL and Oracle support both. The technique is called Change Data Capture.

And all this with a single computational model, without code duplication. Complex asynchronous transformations… …with low latency. And fault-tolerance through recomputation.

The SMACK stack Spark for Micro-Batch Processing Mesos for Cluster Management Akka for Event Processing Cassandra for Persistence Kafka for Event Transport

Event ProcessingMicro-Batch Processing LatencySub-secondSeconds to minutes PowerSimple triggersComplex transformations A trade-off between latency and computational power. Responding to single events in real-time or a general analysis over the stream.

Some other alternatives: Storm, Flink, Samza. Event ProcessingMicro-Batch Processing LatencySub-secondSeconds to minutes PowerSimple triggersComplex transformations Akka Streams Reactive Streams with back pressure. Kafka Streams

Event ProcessingMicro-Batch Processing LatencySub-secondSeconds to minutes PowerSimple triggersComplex transformations SQL Machine Learning Graph Analytics Functional API

Cluster Management with YARN Hadoop and related components. Job request comes in, YARN places the job. MESOS Any application. Job request comes in, MESOS offers resources, job accepts or rejects.

Downstream Applications Upstream Sources An architecture for converting large amounts of raw data into vauable information in real-time.

Tamás István Ujj Business Intelligence Inspiration: Nathan Marz, Jay Kreps, Tyler Akidau, Martin Kleppmann, Dean Wampler