1. LEO Kinesis More Kafka-like Blaine Nielsen blaine@leoplatform.io
Open Source Project

2. Data Pipeline Monitoring
LEO
Data Pipeline Monitoring
Feedback via App

3. Business Intelligence
Data Needs Hierarchy
Big Data AI
Real Time
Product Changes
Predictive Analytics
Deep Learning
Custom Data Apps
Data Science
Search
Customer BI
Recommendation
Engine
Data Lake
CEO BI
Business Intelligence
Integrations
Data Warehousing
Data Visualization
Commission
Fraud App
Dashboards
Evolve
Simple Machine Learning
Start at bottom - data in different systems, for good reason, each system has a reason and purchase it was bought. But creates the issue of keeping track of it all. Triggers between systems.
BI - make sense of it all, identify trends, drive biz results
Data Science - harder questions at the data
Custom Data Apps - Sell custom software for more if smarter
Predictive - not only know trends, but predict the future
AI - real-time decision making
Aggregate
Understanding
Explore
Collect/Clean/Store

4. AWS Data Tools AI Machine Learning Custom Apps Data Science
Rekognition
Lex
Polly
Machine Learning
Machine Learning
Cloud providers are giving us tools to address each need on the hierarchy. How fast can we adopt? How robust is our data management
Custom Apps
Lambda
Data Science
Athena
Business Intelligence

5. Data Pipelines Legacy System Data Pipeline Data Pipeline Data Pipeline
Machine Learning
Data Pipeline
Using the new tools requires that we feed it the data the service needs. The services are only as good as we feed it.
I see all the time. Data Scientists waiting on the org to provide access to the data.
Data Pipelines take time, require maintenance. Anyone that has every worked with data knows, data changes or outside agreed upon formats. It breaks, stops flowing. So monitor to address quickly.
Legacy
System
Data Pipeline
Data Pipeline

6. Data Pipeline Spaghetti
Legacy
System
Machine Learning
Pretty soon, we can end up with spaghetti integrations. Maintenance nightmares.
Buffering, handling failure, back pressure, scaling, etc
all get pushed into pipeline responsibilities

7. Reusable Data Pipeline
What if?
Reusable Data Pipeline

8. Lets Try
Event Driven

9. Event-driven pattern PaymentCompleted Events Payments Service
Listen and React
Do and publish
Payments
Service
OrderPlaced Events
Listen & React to Events

10. “Shared Event Stream” Legacy System OrderCreated OrderCreated
Machine Learning
OrderCreated
Legacy
System
Event Stream
OrderCreated
OrderCreated
Faster Adoption
Much easier maintenance
OrderCreated

11. Shared Event Stream? X X X X X X X RabbitMQ
Pub/Sub
# of
Subscribers
Replay
Durable
In-Cloud
Serverless
Visibility X
RabbitMQ
unlimited X X X
RabbitMQ - messaging is not a great fit
Kafka - Wonderful, but really hard to get-up and running and maintenance is a pain
Kinesis - in-cloud serverless is fantastic, but missing some core features
Kafka
unlimited
Forever X X
Kinesis 2 X
7 days

12. (# of Subscribers, where in log)
EventStream
Kafka
AWS Cloud
Communication Stream
Kinesis
Service Catalog
(# of Subscribers, where in log)
DynamoDB
Notification & Broker
Lambda
Data Store
S3, DynamoDB
RabbitMQ - messaging is not a great fit
Kafka - Wonderful, but really hard to get-up and running and maintenance is a pain
Kinesis - in-cloud serverless is fantastic, but missing some core features

13. Introducing LEO Streams
Apache2.0 OpenSource License
Dev Friendly
docs, SDKs, APIs, Connectors
Pub/Sub
unlimited subscribers
Robust
bursting, backoff, gzip, checkpointing, back pressure, auto retry
Replay
rehydrate as needed, A/B Testing
AWS Bill
AWS economics best practice built in
Durable Store
Replay
Monitoring
Pub/Sub
LEO Streams

14. EventStream w/ AWS Producer Stream Data Store Service Catalog Consumer
200 ms
1 min
Machine Learning
Trigger
Notify
SDK read
from Data Store
SDK write
mass
Tech Map
apps…..
archive
Data Lake
Athena Compatible
Choose speed
w/ Auto detection
and mass path based on data flow
Redundancy
Durable
Unlimited Subscribers
Bursting
Backoff/retry
Write
Stream
KAFKA-LIKE (LEO Additions)
Data Store
Service Catalog
Broker logic
Checkpoint management
Tigger
Broker notification
Read
retry
Low Cost
Unlimited topics
on 1 Kinesis Stream

15. Stream Adoption In-App Search Mobile App LEO Streams Search UIs Old
New - 100x improvement
2M avg lag
300 MS avg lag V1
Custom APIs
Legacy System
In-App Search
Loading Scripts
LEO Streams
Pub/Sub
Durable Store
Replay
Monitoring

16. Stream Adoption In-App Search Mobile App LEO Streams Search UIs
New - 100x improvement
Old
300 MS avg lag
2M avg lag V1
MicroService
Custom APIs
MicroService
Legacy System
Change
Log
In-App
Reporting
In-App Search
Loading Scripts
LEO Streams
Pub/Sub
Durable Store
Replay
Monitoring

17. Stream Communication LEO Streams eCommerce Search
Legacy System
MicroService
eCommerce
Search
how fast for the next micro services and SaaS system
LEO Streams
Pub/Sub
Durable Store
Replay
Monitoring
Legacy System
MicroService
Commission
Engine
History
Log File
MicroService
MicroService
In-App
Reporting
Data
Warehousing

18. Monitoring UI Load Enrich offload streamline publishing
Streams
Enrich
Streams
offload
Streams
streamline
publishing
durable store of
data events, forever
subscribe with ease

19. Exception Based Dashboard

20. Event Details
Details
Wrapper for display
and audit purposes

21. to new or existing subscribers
Replay
to new or existing subscribers

22. Trace Events

23. Alarms & Alerts

24. https://github.com/LeoPlatform/Nodejs
SDKs
Stream Events