1. VOLTTRON-RabbitMQ Message Bus Overview
Shwetha Niddodi
Chandrika Sivaramakrishnan
Kyle Monson
Craig Allwardt
Jereme Haack

2. Outline Background Message Bus Plugin Framework RabbitMQ Overview
RabbitMQ VOLTTRON
Conclusion

3. Background
VOLTTRON’s ZeroMQ based message bus has been key for meeting the security and interoperability goals of the platform
At the same time, RabbitMQ has become more mature as it has seen major investment by commercial companies.
Rabbit Technologies, now part of Pivotal Technologies (VMWARE spin-out). $105 million investment by GE in 2013.
Used by: Instagram, Indeed.com, Google Cloud Platform, Tesla …
Goals of the Refactor:
Maintain essential features of current message bus and minimize transition cost
Leverage an existing and growing community dedicated to the further development of RabbitMQ
Move services provided currently by VOLTTRON agents to services natively provided by RabbitMQ
Decrease VOLTTRON development time spent on supporting message bus which is now a commodity technology.
Address concerns from community about ZeroMQ
View this effort as essential to the long-term future of the platform
Working with heavy users in the community to get feedback
Reduce long term costs of platform by moving message bus development out of core
Maintain support for ZMQ short term (3 – 5 years) as funding allows

4. Why RabbitMQ?
Supports different messaging patterns, routing topologies
Easy to use, has most of the features already built-in
Well developed security feature
Large scale deployment
Flexibility in deployment
Recommended by VOLTTRON community
Other competitive message buses that were considered.
Kafka – Uses “dumb broker, smart consumer approach” which is opposite of our requirement.
MQTT – Limitation in security feature.
The payload can be secured but not the header

5. Message Bus Plugin Framework
Decouple VOLTTRON from message bus implementation
Similar concept to Historian and Driver frameworks
Provide hooks to allow new solutions to be leveraged
“Proxy Agent” that handles VOLTTRON specific code needed to interact with different types of message buses
Must prevent fracturing of ecosystem
Ensure VOLTTRONs using different messaging systems can still communicate with each other

6. Upgrading to RabbitMQ Based Platform
Minimize changes needed to agent code
No effect unless agent has ZMQ specific code (ForwardHistorian)
No changes required to driver configuration
Data collection unaffected
Backward compatibility
Allow ZMQ and RabbitMQ instances to communicate
Will require new code to connect RabbitMQ based systems together

7. Message Bus Plugin Framework
Consists of five components
New connection class per message bus
Extensions to router module functionality
Extensions to core agent functionality
Adding a proxy agent for each message bus
Authentication related changes

8. Platform Level Changes
On startup, checks the type of message bus used.
Creates appropriate router module
ZMQ Router – Handles actual routing since ZMQ is broker less protocol. Functionality is unchanged.
RMQ Router – Light weight.
Maintains connection to RMQ message bus
Actual routing is handled by RMQ broker
Handles router specific subsystem messages
Handles unrouteable messages

9. Agent Core Changes Application agent code remain unchanged
On startup, checks the type of message bus used.
Maintains connection to appropriate message bus.
All subsystem messages are encapsulated inside a message bus agnostic VIP message object.

10. Compatibility between VOLTTRON instances running different message buses
Proxy Agent – Acts like a bridge between local message bus and remote message bus
Different proxy agent for each type of remote message bus connection.
Maintains connections to internal and external message bus
Route messages from internal to external
Route messages from external to internal

11. Proxy Router Agent acting as bridge`
V2 runs on RMQ
message bus
V1 runs on ZMQ
message bus
RMQ
Message Translation layer
ZMQ
ZMQ Router
RMQ
Message Translation layer
XYZ
ZMQ
Message Translation layer
XYZ
V3 runs on XYZ message bus

12. RabbitMQ Overview (contd.)
RabbitMQ uses AMQP (Advanced Message Queuing Protocol)
Exchanges – Responsible for routing of messages to Queues.
They look at the routing key in the message when deciding how to route messages to queues.

13. RabbitMQ Overview
Queues - Buffer that stores the messages until consumed by consumer.
Bindings – Queues bind to the exchange with binding keys or routing pattern.
Messages are routed to one or many queues based on a matching between a message routing key and binding key.
Consumer 1
Queue
Binding key:“green”
Consumer 2
Producer
Exchange
Queue
Binding key:“red”
Routing Key: “green”
Consumer 3
Queue
Binding key:“green”
Consumer 4
Queue
Binding key:“yellow”

14. RabbitMQ VOLTTRON Uses Pika library – Python library for RabbitMQ
Each VOLTTRON instance creates a single topic exchange – “volttron”
Each Agent Core
Connects to RabbitMQ broker
Creates VIP queue and binds to exchange with binding key
“<instance name>.<agent identity>”
Send/Receive VIP messages using Pika methods
Routes incoming messages to appropriate subsystem
Platform Router
“<instance name>.<router>”
Handles messages intended to the router
Handles unrouteable messages

15. Binding Key: “volttron1.agent_a” agent_b VIPqueue
RPC Subsystem
Agent_A
VIP ID: “agent_a”
Agent_B
VIP ID: “agent_b”
Topic Exchange
agent_a VIP queue
Binding Key: “volttron1.agent_a”
agent_b VIPqueue
Binding Key: “volttron1.agent_b”
Message:
Destination Routing Key: “volttron1.agent_b”
Pika properties:
Type:”rpc”
“user_id”: “volttron1.agent_a”
Body: message arguments
Message:
Destination Routing Key: “volttron1.agent_a”
Pika properties:
Type:”rpc”
app_id: ”volttron1.agent_b”
“user_id”: “volttron1.agent_a”
Message_id: message id
Body: message return result
Alternate Exchange
(Fan out)
Router/Bad Message Handler

16. RabbitMQ PUBSUB Agent functionality remains unchanged.
Agents continue to use same PubSub interfaces
User can continue to use topics delimited by “/”.
RabbitMQ PUBSUB converts it to “.” internally.
Agent topics are internally prefixed “__pubsub__.<platform_id>” to differentiate from main Agent binding.
If agent wants to subscribe to topic from remote instances, it uses
agent.vip.subscribe(“pubsub”, “devices.hvac1”, all_platforms=True”)
It is internally set to “__pubsub__.*.<remainder of topic>”

17. Routing Key: “__pubsub__.volttron1.devices.hvac1” Pika properties:
PubSub Subsystem
Agent_A
VIP ID: “agent_a”
Agent_B
VIP ID: “agent_b”
User
PUBSUB message handler
Publish Message:
Routing Key: “__pubsub__.volttron1.devices.hvac1”
Pika properties:
Type:”pubsub”
user_id: ”volttron1.agent_a”
Message_id: result.ident
Body: message arguments
Subscribe:
Pubsub queue: “volttron1.*”
Routing Key:
“__pubsub__.volttron1.devices.#”
Message_id: message id
Callback handler: user pubsub message handler
VIP queue
Binding Key: “volttron1.agent_a”
Topic Exchange

18. Multi-Platform Communication
With ZMQ based VOLTTRON
Write an agent that would connect to remote instance directly.
Use special agents such as forwarder/data puller agents to forward/receive messages to/from remote instances.
Shovel Plugin can be used in RabbitMQ VOLTTRON
Configure VIP address of all remote instances in $VOLTTRON_HOME/external_discovery.json. Let the router module in each instance manage the connection and message routing for us.
Federation plugin can be used in RabbitMQ VOLTTRON

19. Federation Plugin Used for connecting multiple brokers
Loose coupling of nodes
WAN friendly – Tolerates network intermittency
Specificity – Not everything needs to be federated.
Scalability - Does not require O(n2) connections between n brokers
Allows you to make exchanges and queues federated.
A federated exchange can route messages published upstream to a local queue.
A federated queue lets a local consumer receive messages from an upstream queue
Reference:

20. Shovel Plugin Used to move messages from one broker to another broker
Loose coupling
WAN friendly - Tolerates network intermittency
Dynamic shovels do not need restart of broker
Acts as well-written client application that
Connects to source and destination broker
Consumes messages from the queue
Re-publishes messages to the destination maintaining the same message format (and routing key if needed)
Messages forwarded based on routing key or pattern match
Useful if one of the instance is behind NAT
Shovel setup requires foreknowledge of topic subscriptions.
Limit unneeded traffic across the wire
Does not adapt to subscriptions automatically like a Federation link.
Reference:

21. RPC Multiplatform VOLTTRON - 1 VOLTTRON - 2 Agent_A
VIP ID: “agent_a”
RMQ User ID:
“volttron1.agent_a”
Agent_B
VIP ID: “agent_b”
RMQ User ID:
“volttron1.agent_b”
Agent_B
VIP ID: “agent_b”
RMQ User ID:
“volttron2.agent_b”
Agent_B input queue
Binding Key: “volttron1.agent_a”
Agent_B input queue
Binding Key: “volttron1.agent_b”
Agent_B input queue
Binding Key: “volttron2.agent_b”
Message:
Routing Key: “volttron2.agent_b”
AMPQ:
Type:”rpc”
user_id: ”volttron1.agent_a”
Message:
Routing Key: “volttron1.agent_a”
AMPQ:
Type:”rpc”
user_id: “volttron2.agent_b”
Topic Exchange
Topic Exchange
Federation Link
Federation Link
8/21/2018

22. RPC over NAT Multiplatform
VOLTTRON – 1 (behind NAT)
RPC over NAT Multiplatform
VOLTTRON - 2
Agent_A
VIP ID: “agent_a”
Agent_B
VIP ID: “agent_b”
Agent_B
VIP ID: “agent_b”
Agent_B input queue
Binding Key: “volttron1.agent_a”
Agent_B input queue
Binding Key: “volttron1.agent_b”
Agent_B input queue
Binding Key: “volttron2.agent_b”
Message:
Key: “volttron2.agent_b”
AMPQ:
Type:”rpc”
user_id:”
volttron1.agent_a”
Message:
Key: “volttron1.agent_a”
AMPQ:
Type:”rpc_reply”
Topic Exchange
Topic Exchange
Shovel Key: “volttron1.*” or Federation
Shovel Key: “volttron2.*”

23. Binding Key: “__pubsub__.*.devices.#”
VOLTTRON - 1
PUBSUB Multiplatform
VOLTTRON - 2
Agent_A
VIP ID: “agent_a”
Agent_B
VIP ID: “agent_b”
Agent_B
VIP ID: “agent_b”
Agent_B pubsub queue
Binding Key: “__pubsub__.*.devices.#”
Agent_B VIP queue
Binding Key: “volttron1.agent_a”
Agent_B VIP queue
Binding Key: “volttron1.agent_b”
Agent_B VIP queue
Binding Key: “volttron2.agent_b”
Message:
Key: “__pubsub__.volttron1.devices.pnl.isb1.hvac1”
Topic Exchange
Topic Exchange
Federation Link
Federation Link

24. Authentication with RabbitMQ message bus
RabbitMQ supports multiple authentication mechanisms
For VOLTTRON we use SSL peer verification using with x509 certificates
SSL certificates of interest
Root CA
Server certificate signed by root
Client certificate signed by root

25. RabbitMQ Server SSL certificates
1. Root CA File - CA1 public key
2. Server cert signed by Root CA
3. Server key
Presents agent1.crt and agent1.pem
Presents agent2.crt and agent2.pem
Presents agent3.crt and agent3.pem
Agent 1
agent1.crt – public cert signed by Root CA
agent1.pem – private key
Agent 2
agent2.crt – signed by Root CA
agent2.pem – private key
Agent 3
agent3.crt – signed by Root CA
agent3.pem – private key
Every rabbitmq has a single self signed root ca and server certificate signed by the root CA. This is created at the time of volttron setup and rabbitMQ server is configured and started with these two certificates. Every time an agent is started, the platform automatically creates a agent certificate that signed by the same root ca. When an agent communicates with the rabbitmq message bus it presents it agent certificate and private key to the server and the server validates if it is signed by a ROOT CA it trusts – ie. the root certificate it was started with. Since there is only a single root ca for one volttron instance, all the agents in this instance can communicate with the message bus over ssl.

26. Multi-Platform Connection with SSL certificates
VOLTTRON 1
VOLTTRON 2
Rabbitmq Server - 1
1. CA cert file – contains CA1 public key and CA2 public key
2. Server cert signed by CA1
3. Server private key
Rabbitmq Server - 2
1. CA cert file – contains public key of CA2 public key and CA1 public key
2. Server cert signed by CA2
3. Server private key
Shovel
Presents volttron2 user cert, key
Presents agent1.crt and agent1.pem
Presents agent2.crt and agent2.pem
Federation
Agent 1
agent1.crt – signed by Root CA1
agent1.pem – private key
Agent 2
agent2.crt – signed by Root CA2
agent2.pem – private key
Suppose we have two VOLTTRON instances – VOLTTRON1 and VOLTTRON 2 each with their own root certificates and server and agent certificates singed by their root CA. For VOLTTRON1 to communicate with VOLTTRON2 or vice-versa, each instance’s RabbitMQ server must trust the other instance’s ROOT CA. This can be achieved by combining volttron1 root ca certificate (i.e. the public key of the root certificate) and volttron2’s root ca certificate into a single file and configuring rabbitmq to use this as it trusted root CA. When agent 1 presents it agent1.crt to volttron2, volttron 2 will accept it since agent1.crt was signed by volttron1’s root ca which is now part of its trusted CA.
Presents volttron2 user cert, key

27. Authorization in RabbitMQ message bus
For protected topics, topic permissions are set using RabbitMQ’s management interface based on agent’s user id
Allow publish rights – set read + write permission on the topic for the agent
Allow subscribe rights – Restrict read permission on the topic for the agent
Capabilities on RPC methods – Remains unchanged
Specify required capabilities inside agent’s code
@RPC.allow(‘SET_TEMP’)
@RPC.export
def set_temperature(): ….
Authorization entries (specified via volttron-ctl auth commands)

28. VOLTTRON RabbitMQ Management Utility
Extend “volttron-ctl” utility to include RabbitMQ management commands.
Uses RabbitMQ management http plugin
Some of the functions
Create/Delete vhosts for each platform
Create/Delete unique user, password for each agent
Set permissions on the user
Create/Delete exchanges and queues
Create/Delete federation and shovel setup for multi-platform deployment.
Set topic permissions for protected topics
List the status of
Open Connections
Exchanges
Queues

29. Scalability Tests Single instance can support close to 1000 agents
Tested with 100 instances forwarding messages to single root node connected together
Latest Code in experiment branch:

30. Next Steps
Integrate Volttron Central agent to use RabbitMQ message bus.
Further improvements to installation steps and SSL based authentication.
Scalability tests with large scale deployments.
Complete testing of RabbitMQ VOLTTRON.

31. Conclusion Questions? Suggestions?
Let us know if you would like to be in the message bus refactor working group:

32. Backup

33. Proxy Router Agent acting as bridge
Core
Subsystems
Platform
Router
ZMQ
RMQ
XYZ
ZMQ
RMQ
XYZ
RMQ/XYZ
Message Translation layer
ZMQ
ZMQ/XYZ
Message Translation layer
RMQ
ZMQ/RMQ
Message Translation layer
XYZ
Proxy Agents
Instance V1 runs on RMQ
Agent
Core
Subsystems
Platform
Router
Instance V2 runs on ZMQ
ZMQ
RMQ
XYZ
ZMQ
RMQ
XYZ