1. 9/18/2018
Big Data Analytics with HDInsight Module 6 – Storm Essentials Asad Khan Nishant Thacker Principal PM Manager Technical Product Manager Microsoft Microsoft
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Storm Essentials What is Storm Storm Use Cases Introduction to Storm
Storm Topologies
Storm on HDInsight
Storm Essentials
Tools

3. Storm Essentials What is Storm Storm Use Cases Introduction to Storm
Storm Topologies
Storm on HDInsight
Storm Essentials
Tools

4. What is Apache Storm? Streaming data analysis Easy to program Fast
A distributed real time processing platform
Fast
Clocked at 1M+ messages per second per node
Scalable
Thousands of workers per cluster
Fault Tolerant
Failure is expected, and embraced
Reliable
Guaranteed message delivery Exactly-once semantics

5. Storm Essentials What is Storm Storm Use Cases Introduction to Storm
Storm Topologies
Storm on HDInsight
Storm Essentials
Tools

6. Store in query able store
Storm use case on HDInsight— Connected cars
Queuing service
Get data
Store in blob
Get reference data
Do machine learning
Store in query able store
Live dashboard
Cloud gateways
Queuing Service

7. Store in query able store
Storm use case on HDInsight— Connected cars
Azure Blob
HBase
Azure ML
DocumentDB
Queuing Service
Event Hubs
PowerBI
Queuing service
Get data
Store in blob
Get reference data
Do machine learning
Store in query able store
Live dashboard
Cloud gateways
Event Hubs

8. Storm customer use cases
9/18/2018
Storm customer use cases
Input
Operators (Examples)
Side Lookup
Output
Programming Language
Connected Cars
Event hubs
Window based aggregation, Join stream/split stream
HBase, ML
DocumentDB
C# hybrid, Java
ETL
Event Hubs
Partitioning/ organize
N/A
WASB
Java
IoT
Window based aggregation
Hbase, ML
DocumentDB, HBase
Fraud detection
ServiceBus Queue
Filter ML
Hbase
C# hybrid
Social analytics
Twitter
Groupby/trending topics
Realtime dashboard (BI)
Trident
Network monitoring
Kafka
Split (on success/ failure)
SQL
Log Search
Storage Queue/ Event Hub
Parsing & index
Elastic Search
Mobile engagement
Eventhub
Count
HBase
SignalR
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Apache Storm and Event Hubs
Tech Ready 15
9/18/2018
Apache Storm and Event Hubs
Event producers
Collection
Event Queuing System
Transformation
Long-term storage
Presentation and action
Azure ML
Storage adapters
Stream processing
Applications
Web and social
Devices
Sensors
Live Dashboards
Search and query
Data analytics (Excel)
Web/thick client dashboards
Devices to take action
Event hub
Apache Storm on HDInsight
Apache HBase on
HDInsight
DocumentDB
Solr Azure Search
MongoDB SQL
Cloud gateways (web APIs)
Field gateways
Kafka/RabbitMQ/
ActiveMQ
Event hubs
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Storm Essentials What is Storm Storm Use Cases Introduction to Storm
Storm Topologies
Storm on HDInsight
Storm Essentials
Tools

11. Storm introduction {…} Tuple Stream Spout Core unit of data
Immutable set of key/value pair
Unbounded sequence of Tuples
Source of streams
Wraps a streaming data source and emits Tuples

12. Spout API Lifecycle API public interface Ispout extends Serializable {
void open(Map conf, TopologyContext context,
SpoutOutputCollector collector);
void close();
void activate();
void deactivate();
void nextTuple();
void ack(Object id);
void fail(Object id); }
Lifecycle API

13. Spout API Core API public interface Ispout extends Serializable {
void open(Map conf, TopologyContext context,
SpoutOutputCollector collector);
void close();
void activate();
void deactivate();
void nextTuple();
void ack(Object id);
void fail(Object id); }
Core API

14. Spout API Reliability API
public interface Ispout extends Serializable {
void open(Map conf, TopologyContext context,
SpoutOutputCollector collector);
void close();
void activate();
void deactivate();
void nextTuple();
void ack(Object id);
void fail(Object id); }
Reliability API

15. Using Bolts
Write to a data store
Read from a data store
Perform arbitrary computation
(Optionally) Emit additional streams
{…}
Compute
Core functions of a streaming computation | Receive tuples and do stuff

16. Bolt API Lifecycle API public interface IBolt extends Serializable {
void prepare(Map sormConf, TopologyContext context,
OutputCollector collector);
void cleanup();
void execute(Tuple input); }
Lifecycle API

17. Bolt API Core API public interface IBolt extends Serializable {
void prepare(Map sormConf, TopologyContext context,
OutputCollector collector);
void cleanup();
void execute(Tuple input); }
Core API

18. Storm Essentials What is Storm Storm Use Cases Introduction to Storm
Storm Topologies
Storm on HDInsight
Storm Essentials
Tools

19. Topologies

20. Stream groupings {…} Stream Grouping Description Shuffle
9/18/2018
Stream groupings
Stream Grouping
Description
Shuffle
Sends tuples to bolts in random, round robin sequence
Fields
Sends tuples to a bolt based on one or more fields in the tuple
All
Sends a single copy of each tuple to all instances of a receiving bolt
Custom
Customized processing sequence. Use to get maximum flexibility of topology processing based on factors such as data types, load, and seasonality.
Global
Sends tuples generated by all instances of a source to a single target instance
Stream groupings determine how Storm routes Tuples between tasks in a topology ?
{…}
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Topology deployment
Supervisor
Zookeeper

22. What is Trident? Fluent Merges and joins High level abstraction
Stream-oriented API
Merges and joins
High level abstraction
Built on Storm’s core primitives
Built for
Aggregation, groupings, functions, & filters

23. Trident API—fluent and stream oriented
TridentTopology topology = new TridentTopology();
FixedBatchSpout spout = new FixedBatchSpout(…);
Stream stream = topology.newStream(“words”, spout);
stream.each(…, new Myfunction())
.groupBy()
.each(…, new MyFilter())
.persistentAggregate(…);

24. Storm Essentials What is Storm Storm Use Cases Introduction to Storm
Storm Topologies
Storm on HDInsight
Storm Essentials
Tools

25. Apache Storm on HDInsight
9/18/2018
Apache Storm on HDInsight
Managed services
Event Hub
Open source platform
Visual Studio
Analyse millions of messages per second
Azure
Scale-up and scale-down
HBase, SQL Database, DocumentDB
Speed
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. Topology
:56:59 Here's to the crazy ones, the misfits, the rebels, the troublemakers, the round pegs in the square [Lang] en…
:56:59 Here's to the crazy ones, the misfits, the rebels, the troublemakers, the round pegs in the square holes... es
mobile
for ja
iphone es
mobile ja
iphone
mobile 4 1
iphone 5
@stevejobs [Lang] en…
Twitter tweets
Ignore word Bolt
Word count Bolt
Splitter Bolt

28. Get started today!
For more information visit:

29. © 2014 Microsoft Corporation. All rights reserved
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.
MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.