1. Messaging in Azure Event Hubs, Service Bus, and Relay
CLD342a
@DanRosanova

2. What is “messaging”?
It’s not this…
But it’s not all that different

3. Azure Messaging Services
Microsoft Ignite 2016
12/7/2017 1:53 AM
Azure Messaging Services
Service Bus
Event Hubs
Relay
Reliable asynchronous message delivery
Distributed data streaming
Secure two way communication without changes to your network
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. Where can I use messaging?
Microsoft Ignite 2016
12/7/2017 1:53 AM
Where can I use messaging?
Financial Services
Utilities & Telecom
Logistics
Insurance
Healthcare
Government
Inter-Bank transfers
Order fulfillment & trading
Settlement
Reconciliation
Pricing updates and quotes
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Where is messaging used
Financial Services
Utilities & Telecom
Logistics
Insurance
Healthcare
Government
Customer information systems
Billing integration
Smart meter & grid
Command and control

6. Where is messaging used
Financial Services
Utilities & Telecom
Logistics
Insurance
Healthcare
Government
Order processing
eCommerce
Delivery notifications
Inventory management
Routing

7. Where is messaging used
Financial Services
Utilities & Telecom
Logistics
Insurance
Healthcare
Government
Enrollment
Claims processing
Risk modeling
Regulatory filing

8. Where is messaging used
Financial Services
Utilities & Telecom
Logistics
Insurance
Healthcare
Government
Medical records
Care quality metrics
Patient monitoring & diagnostics

9. Where is messaging used
Financial Services
Utilities & Telecom
Logistics
Insurance
Healthcare
Government
You really don’t want to know
How do you think they collect all that data?
Ever use Global Entry?

10. Who is using all this messaging?
Agriculture, Forestry & Fishing
Airline
Automotive
Banking & Capital Markets
Education
Food and Beverage
Gaming
Government
Healthcare
Hospitality & Travel
Insurance
Logistics
Manufacturing
Media & Cable
Power & Utilities
Professional Services
Public Safety
National Security
Retail and Consumer Goods
Telecommunications

11. Service Bus (Queues & Topics)

12. Messaging as a Service (MaaS)
Queues & Topics
Reliable asynchronous communication
Rich features for temporal control
Routing and filtering
Convoys & Sessions (related messages with state)

13. Where does Service Bus fit in?
Decoupling and providing durability are primary drivers for using Service Bus
Queue
Front End / UX
Back End / Processing

14. Common messaging patterns
Load Leveling
Load Balancing
Filtering
Partitioning
Scheduled Delivery
Fan Out
Routing
Claim Checks
Workflow
Taps
Logging
Dead Letter
Sequence
Correlation
Expiration

15. Features of Service Bus
Scheduled delivery
Poison message handling
ForwardTo
Defer
Sessions
Batching
Ordering
Auto-delete on idle
OnMessage
Duplicate detection
Filters
Actions
Transactions

16. The basics of a queue Sender sends message to queue Queue ACKs receipt
Receiver connects to queue & retrieves message
Receiver ACKs complete (or other action)
Sender
Receiver

17. Topics & Subscriptions
Sender only knows about Topic
Receivers only know about Subscriptions
Filters and Actions exist on Subscriptions
type=order
Filter
Subscription 1
“type” = “order”
Topic
type=quote
“type” = “quote”
Subscription 2
Filter

18. Enterprise Messaging is a mature space
Microsoft Ignite 2016
12/7/2017 1:53 AM
Enterprise Messaging is a mature space
Most of “Enterprise” customers will have some
Enterprise Messaging is moving to the cloud…
Azure Service Bus is driving this move!
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. Service Bus’ Three Tiers
Microsoft Ignite 2016
12/7/2017 1:53 AM
Service Bus’ Three Tiers
Three flavors of platform
More features as you go up
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

20. Premium is about predictability
Predictable performance
Predictable price
Larger messages
Higher scale
More innovative features that require more resources

21. Premium also means performance
Microsoft Ignite 2016
12/7/2017 1:53 AM
Premium also means performance
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Event Hubs

23. Distributed data streaming
Event Hubs
A streaming service designed to do low latency distributed stream ingress
A partitioned consumer scale model
A time retention buffer
An elastic component in the middle of your chain

24. Where does Event Hubs fit in?
Ingress
Collection
Presentation / action
Event producers
Stream Processing
Long-term storage
Event Hubs
Stream processing
Cloud gateways (web APIs)
Field gateways
Applications
Legacy IOT (custom protocols)
Devices
IP-capable devices (Windows/Linux)
Low-power devices (RTOS)
Search and query
Cortana Analytics
PowerBI Dashboards
Service Bus
Azure DBs
HDInsight
Azure Storage
Fast Data
Devices to take action
Slow Data
Azure Data Lake

25. Common Event Hubs patterns
Logging / telemetry
Application logging
Device / user / performance telemetry
Dashboarding
Transaction processing (e.g. customer orders/e-commerce)
Anomaly detection (for ex. fraud/outliers)
Data archival
Batch processing

26. Event Hubs features Archive
Proximity (related data is grouped together)
Order
Consistent playback
Tremendous scale

27. How Event Hubs is different from queues
Records a stream
Recording moves forward only
You can play the tape over and over again
FastFwd or Rewind then Play
Tape loops when its done

28. How Event Hubs is different from queues
Records a stream
Recording moves forward only
You can play the tape over and over again
FastFwd or Rewind then Play
Tape loops when its done O
>>
>
<<

29. Event Hubs partitions
A cassette tape actually has Left and Right channels
When you press record, they both record
But the data on each channel is different
The left and right speakers each play one channel
Event Hubs calls these partitions

30. Event Hubs and more partitions
>>
>
<<
Speaker = reader
More partitions = more speakers
Eventually this gets unwieldly O
>>
>
<< O
>>
>
<<

31. Event Hubs conceptual architecture
Azure Event Hub
Consumer Group
Partition 1
HTTP
AMQP
Partition 2
Event Receivers
Partition 3
Consumer Group 2
Event Producers
Partition 4
AMQP

32. Scaling Event Hubs Throughput Units
Reserved capacity that you purchase
1 MB/second or 1,000 events /send ingress
2MB/second or 2000 events per second egress
Namespace wide – across all Event Hubs in that namespace
Overages are throttled
Partition and Throughput Units are separate

33. Event Hubs pricing

34. Event Hubs pricing sample 1
1 TU = $0.72 per day (up to1000/s)
1000 x 60 x 60 x 24= 86,400,000 events per day
86,400,000 / 1,000,000 x $0.028 = $2.42 per day
Total $ $2.42 = $3.14 for ~84GB of data

35. Event Hubs pricing sample 2
1 billion events per day: 12,000 events per second
$0.03 x 12 x 24 = $8.64
1,036,800,000 / 1,000,000 x $0.028 = $29.03
$37.67

36. The majority of Event Hubs customers write their stream to a persistent store…

37. Where does Event Hubs fit in?
Event producers
Collection
Ingress
Stream Processing
Long-term storage
Presentation / action
Applications
Fast Data
Service Bus
Azure DBs
HDInsight
Azure Storage
Search and query
Cortana Analytics
PowerBI Dashboards
Cloud gateways (web APIs)
Event Hubs
Legacy IOT (custom protocols)
Stream processing
Devices
Slow Data
IP-capable devices (Windows/Linux)
Event Hubs Archive
Azure Data Lake
Field gateways
Low-power devices (RTOS)
Devices to take action

38. Event Hubs Archive lets you go fast and slow
Automatically send Event Hub data to your storage
To better serve batching and archival scenarios
To minimize your overhead: no code or other services to configure

39. How Archive works Azure Event Hub
Partition 1
Partition 2
Size: 20MB
Partition 3
Time: 5 min
Partition 4
<Namespace>/<EventHub>/<Partition>/<yyyy/MM/dd/hh/mm/ss>  

40. Naming files the way you want
Microsoft Ignite 2016
12/7/2017 1:53 AM
Naming files the way you want
Complete flexibility, but must have the following:
{namespace name}
{event hub name}
{partition id}
{date} – you could choose day, month and year separately or as a format
{time} – you could choose Hour minute and second separately or together as a format
There is no order restriction on the components
Choose your own delimiter – or none
Examples
${dd}/${yyyy}/${MM}/${HHmmss}/${namespace}/${ehname}/{partitionId}/test
"25/2016/11/173021/Mynamespace1/myeventhub1/1/test"
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

41. Event Hubs Archive Automatically send events to your own storage
No code to run or compute to manage
Enable batch & real time on the same stream
Use existing tools and processes for batch processing
Try it today:

42. Event Hubs Archive is the easiest way to stream data into Azure

43. Daily moment of Zen

44. Relay & Hybrid Connections

45. Cloud relayed communication
Hybrid Connections & Relay
Securely connect to on premises data and services from anywhere
Load balance multiple sites behind a single endpoint
Protect internal services by projecting a cloud endpoint
Does not require any network changes

46. Where does Relay fit in?
When you have scenarios that are too complex or costly for VPN
When you have to keep your data where it is
United States
European Union

47. Common Relay patterns
Frictionless connectivity (no need for VPN/Firewall/Proxy changes)
Live request-response and streaming services
Hybrid cloud + on premises

48. Relay & Hybrid Connections features
WCF or WebSocket transport
One-way, request-response, peer-to-peer
Overlayed endpoint security
Load balancing
Geo-distribution of traffic
Endpoint discovery

49. Relay: Real-time communication proxy
The basis of Hybrid Cloud
Used to decouple
Location
Security
VPN and Firewall constraints
Client
Service

50. How Relay works Service registers cloud endpoint
Client connects to cloud endpoint
Network magic happens
Client
Service

51. A network rendezvous A B C D
Control Channel
Client
Service

52. How much messaging do people really use?

53. Azure Messaging by the numbers
19.36 Trillion
Requests last month
in Event Hubs
7,228,195
Requests per second
average 24/7 %
Success Rate
50ms
Average Event Hubs
send latency
>28 PB
Monthly data volume
2.2 Million
Message Queues and Topics in production
>200,000
Daily active Service Bus Namespaces
580 Billion
Message operations on Azure Service Bus Messaging per month

54. In the approximately 50 min I’ve been up here
12/7/2017 1:53 AM
21,684,585,000
In the approximately 50 min I’ve been up here
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

56. Cloud is a journey… where will you go?

57. Q&A

58. Continue your Ignite learning path
12/7/2017 1:53 AM
Continue your Ignite learning path
Visit Channel 9 to access a wide range of Microsoft training and event recordings
Head to the TechNet Eval Centre to download trials of the latest Microsoft products
Visit Microsoft Virtual Academy for free online training visit
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

59. Thank you Chat with me in the Speaker Lounge Find me @DanRosanova
12/7/2017 1:53 AM
Thank you Chat with me in the Speaker Lounge Find
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.