AMQP, Message Broker Babu Ram Dawadi

overview Why MOM architecture? Messaging broker like RabbitMQ in brief RabbitMQ AMQP – What is it ?

Message Oriented Middleware Architecture Method of communication between different applications or software components between distributed systems. The predominant approaches RMI, CORBA, SOAP web services Calls remote methods with parameters …. And waits for response Used for Scalability and load balancing Decoupling Reliability Availability Asynchronous hand off Queuing and buffering Monitoring and management

AMQP 1.0 Scope AMQP is Message Oriented Middleware (MOM) Transfers application data units from senders to receivers – layer 7 An expectation that the message transfer is via trusted intermediaries – An expectation that messages will be delivered unchanged – An expectation of security – Applications can be separated by (large amounts) of space and time Abstract from the underlying technology – Physical network limits should be hidden (message size, node location) – Technology concerns should be hidden (platform, language, OS) The intermediaries offer various delivery options, as defined by either the sender or the receiver (s) The intermediaries provided various defined qualities of service for the sender and the receiver (s)

AMQP (Advanced Messaging Queue Protocol) Originated in 2003 by John O’Hara at JPMorgan Chase. A protocol to communicate between clients and messaging middleware servers (brokers) Open internet protocols like TCP, SMTP, HTTP. Open wire protocol standard for message-queuing communication. Enables conforming client applications to communicate with conforming messaging middleware services (aka message brokers) Defines Networking protocol Semantics of broker services Use cases Real time feeds for constantly updating information Encrypted assured transaction

AMQP Design Requirement To accommodate existing messaging API standards (for example, Sun's JMS) To allow complete configuration of server wiring via the protocol To handle messages of any size without significant limit To carry multiple channels across a single connection To allow asynchronous command pipelining

To get there … Queues Messages are consumed from queues Stores messages in memory or disk Deliver in sequence Exchanges Messages are routed to queues through exchanges through pre-arranged criteria Matching and routing engines, rather can do more than routing messages. How is the message routed ?? Messages are routed to queues by matching routing keys against binding pattern. Routing key Binding key

AMQP Communication Model Provides a “Shared Queue Space” that is accessible to all interested applications. Message are sent to an Exchange Each message has an associated Routing Key Brokers forward messages to one or more Queues based on the Routing Key Subscriber get messages from named Queues Only one subscriber can get a given message from each Queue

AMQL Object Model Exchange – Receives messages and routes to a set of message queues Queue – Stores messages until they can be processed by the application(s) Binding – Routes messages between Exchange and Queue. Configured externally to the application. Default binding maps routing-key to Queue name Routing Key – label used by the exchange to route Content to the queues Content – Encapsulates application data and provides the methods to send receive, acknowledge, etc.

Messages Carry content (header + body) Content body is opaque block of binary data. Broker never modifies the content body. Have client specified properties Content-type, Encoding, Timestamp, App-Id, User- Id, Headers Delivery mode Non-Persistent Persistent

Producers send messages to exchanges with a routing key e.g. “abc”, or ordered set of keys e.g. “abc.cloud.aws” Exchanges route messages to queues whose binding pattern matches the message routing key or keys. Consumers create queues; these buffer messages for push to consumers Queues are stateful, ordered, and can be persistent, transient, private, shared. Exchanges are stateless routing tables. Consumers tell queues to bind to named exchanges; each binding has a pattern e.g. “abc” or “*.cloud.*” Example of pub-sub message flow Messages

Traditional Topologies Built from Parts Queues are used both for Persistent stores and transient buffers. Link model unifies point-to-point and publish/subscribe – example shows client messages being routed to various Queues – Example mixes traditional Store & Forward and Transient Pub/Sub

AMQP Architectural Model Centralized Multi Broker Each Queue is materialized in a Broker

AMQP1.0 Typical Usage Patterns

Client Producer Client Producer AMQP Broker Client Consumer Entry 1 Entry 2 Entry 3 Session Link Session Link Queue (source) -Persistent Head Tail Highlights Highlights: Only “Source” queue is required and can be read directly by consumer over Link (i.e. dedicated consumer Worker queue and bridging between Source and Worker unnecessary). Highlights Highlights: Only “Source” queue is required and can be read directly by consumer over Link (i.e. dedicated consumer Worker queue and bridging between Source and Worker unnecessary). Point-to-point Queue Delivery

Client Producer Client Producer AMQP Broker Entry 1 Entry 2 Entry 3 Session Link Queue (Source) -Persistent Head Tail Entry 1 Entry 2 Head Link Tail Queue (worker) -Persistent Abstracted Point-to-point Queue Highlights Highlights: One Queue performs the role of holding the “Well Known” name for the outside world. All messages are automatically forward on to the real worker queue. Allows internal topology to change without the outside world seeing (this PO Box) Highlights Highlights: One Queue performs the role of holding the “Well Known” name for the outside world. All messages are automatically forward on to the real worker queue. Allows internal topology to change without the outside world seeing (this PO Box)

Client Producer Client Producer AMQP Broker Client Consumer Entry 1 Entry 2 Entry 3 Session Link Session Link Queue (source) -Persistent 1 Head or 2 ? Tail Client Consumer Session Link Load-Balanced Point-to-point Queue Delivery

Client Publisher Client Publisher AMQP Broker Client Subscriber Entry 1 Entry 2 Entry 3 Session Link Session Link Queue (Source) -Non-persistent Head Tail Client Subscriber Session Link Client Subscriber Session Link Head Dynamic (non-persistent) Pub/Sub Delivery Highlights Highlights: Messages are “garbage collected” in an implementation specific manner after delivery. AMQP makes some guarantees about how long messages are valid for. Highlights Highlights: Messages are “garbage collected” in an implementation specific manner after delivery. AMQP makes some guarantees about how long messages are valid for.

Client Publisher Client Publisher AMQP Broker Entry 1 Entry 2 Entry 3 Session Link Queue (Source) - persistent Head Tail Client Subscriber Session Link Client Subscriber Session Link Head Entry 1 Entry 2 Head Entry 1 Tail Link Queue (Worker) - persistent Durable (persistent) Pub/Sub Delivery