Presentation is loading. Please wait.

Presentation is loading. Please wait.

REAL-TIME NETWORK ANALYTICS WITH STORM Mauricio Vacas Fausto Inestroza Sonali Parthasarathy.

Similar presentations


Presentation on theme: "REAL-TIME NETWORK ANALYTICS WITH STORM Mauricio Vacas Fausto Inestroza Sonali Parthasarathy."— Presentation transcript:

1 REAL-TIME NETWORK ANALYTICS WITH STORM Mauricio Vacas Fausto Inestroza Sonali Parthasarathy

2 Mauricio Vacas Big Data Architect Sonali Parthasarathy Real-Time Processing Fausto Inestroza Big Data Architect Anita Mehrotra Data Scientist Susie Lu Visualization Krista Schnell Visualization Rick Drushal Engineering Lead John Akred Product Lead The Team

3 WHY REAL-TIME?

4 Distributed Analytics Real-Time Data Ingestion Model Prototyping Exploratory Analytics Real-Time Rule Execution PROCESS UNDERSTAND REACT

5 Accenture Cloud Platform Recommender as a Service … … Network Analytics Services Big Data Platform

6 Drivers consumer devices video usage Issues Operational Costs Understanding service quality degradation Inefficient capacity planning

7 INGEST PROCESS VISUALIZE ANALYZE STORE

8 WHY STORM?

9 Scalability Reliability Data types, size, velocity Mission critical data Processing, computation, etc. Time series / pattern analysis Fault-tolerance What do we need? Multiple use cases

10 How do we get this from Storm? Processing guarantees Low-level Primitives Parallelization Robust fail-over strategies Scalability Reliability Fault-tolerance Processing, computation, etc.

11 PRIMITIVES

12 Stream Spout Bolt Topology Suboptimal network speed, geospatial analysis Request info (IP, user-agent, etc) Pull messages from distributed queue Sessionization, speed calculation Tuple

13 PARALLELISM

14 Nimbus Zookeeper Supervisor W TT W TT W TT W TT

15 Topology Worker Process Task Executor

16 FAULT TOLERANCE

17 Nimbus Supervisor W TT W TT W TT W TT W T W T T T T T T T

18 RELIABILITY

19 IP3 IP2 IP3 IP1 A

20 IP3 IP2 IP3 IP1 A

21 SUBOPTIMAL NETWORK SPEED TOPOLOGY AN EXAMPLE

22 Kafka Spout Pre-processSessionize Calculate N/W Speed per Session Update Speed per IP Identify Suboptimal Speed Store in Cassandra Cassandra Tuple (ip 1)

23 Cassandra Kafka Spout Pre-processSessionize Calculate N/W Speed per Session Update Speed per IP Identify Suboptimal Speed Store in Cassandra Tuple (ip 2) Tuple (ip 1) Parallelism Tuple (ip 1) Tuple (ip 2)

24 Cassandra Kafka Spout Pre-processSessionize Calculate N/W Speed per Session Update Speed per IP Join Compare Speed Store in Cassandra Speed by Location Stream 1 Stream 2 Kafka Spout Tuple (ip 1) Branching and Joins Tuple (ip 1/NY ) Tuple (NY)

25 RULE EXECUTION

26 Drools METHOD 1 Storm METHOD 2 Storm + Drools

27 Kafka Spout Pre-processSessionize Calculate N/W Speed per Session Update Speed per IP Identify Suboptimal Speed Store in Cassandra Cassandra Drools Storm + Drools

28 Copyright © 2012 Accenture All rights reserved. 28 Integration with Cassandra Cassandra Optimal for time series data Near-linear scalable Low read/write latency Custom Bolt Uses Hector API to access Cassandra Creates dynamic columns per request Stores relevant network data

29 Copyright © 2012 Accenture All rights reserved. 29 Lessons Learned Rebalance Topology Tweak Parallelism in bolt Isolation of Topologies Use TimeUUIDUtils Log4j level set to INFO by default

30 Copyright © 2012 Accenture All rights reserved. 30 DEMO

31 Copyright © 2012 Accenture All rights reserved. 31 Next Steps Trident Externalizing Rules Predictive Models Real-Time Notifications


Download ppt "REAL-TIME NETWORK ANALYTICS WITH STORM Mauricio Vacas Fausto Inestroza Sonali Parthasarathy."

Similar presentations


Ads by Google