1. Messaging Services and Client Software
Lionel Cons – Massimo Paladin
EGI Technical Forum - Prague, 18th September2012

2. Outline Recommendations for messaging services
From use cases to client software
Recommended libraries and software
01/12/2018
CERN

3. Dedicated messaging services of independent brokers
Dedicated services
Application isolation
Application specific broker tuning
Compatible applications can be grouped
Independent brokers
Horizontal scalability
Easier management and operations
Heterogeneous products (when needed)
Goes well with load-balanced DNS
01/12/2018
CERN

4. Dedicated messaging services of independent brokers
produce to any
consume from all
01/12/2018
CERN

5. WLCG messaging services
01/12/2018
CERN

6. Messaging software Many protocol level client libraries available
Many messaging brokers available
Recommendations available in our wiki
Many protocol level client libraries available
Different protocols
Several alternatives per language
STOMP
AMQP
OpenWire …
C / C++ X
Java
Perl
Python
Ruby
01/12/2018
CERN

7. Developing messaging clients
Easy to get something working
Using different protocols and programming languages leads to code duplication
Error handling is not trivial at all (messaging is mostly asynchronous)
Hard to get something working reliably
01/12/2018
CERN

8. How to solve this? What about LEGO bricks? What are our bricks?
Small reusable components
Flexible when combined
What are our bricks?
Message Queue
File based message queue
Simple and robust API
Messaging Transfer Agent
Transfer messages between a broker and a message queue (all combinations)
01/12/2018
CERN

9. Simplifying the producer
01/12/2018
CERN

10. Simplifying the consumer
01/12/2018
CERN

11. Scaling the consumer side
01/12/2018
CERN

12. In practice 1/2
Producers and consumers only need to interact with the Message Queue
Perl: perl-Messaging-Message + perl-Directory-Queue
Python: python-messaging + python-dirq
simple algorithm, easy to port to other languages
01/12/2018
CERN

13. In practice 2/2 Messaging Transfer Agents
STOMP protocol: stompclt (production ready)
AMQP protocol: amqpclt (being tested)
stompclt sender example:
stompclt receiver example:
01/12/2018
CERN

14. How can we handle an elastic service?
01/12/2018
CERN

15. Reliable services through supervision
Proven concept (Erlang/OTP)
Workers do their work
Supervisors monitor workers
All are defined in a supervision tree
Flexible implementation available (simplevisor)
Non intrusive
Handle service evolution
01/12/2018
CERN

16. More about software The EMI Messaging Product Team
Identified the reusable components
Improved the existing ones
Developed the missing ones
All the components are available
Most are production ready
The others are being finalized
All are available in EPEL
01/12/2018
CERN

17. Main messaging brokers
Qpid
MRG
HornetQ
ActiveMQ
Apollo
RabbitMQ
Language
Java
C++
Scala
Erlang
Main Protocols
AMQP
proprietary
STOMP
OpenWire
Owner (*)
Red Hat
Fuse Source
(Progress)
VMware
Red Hat
7 September 2012 : Red Hat completed its acquisition of FuseSource
“Two Gorillas in the Open Source Market Join Forces”
01/12/2018
CERN

18. For more information
CHEP 2012 paper & poster
Our wiki (short:
If you are interested in using messaging or want to provide feedback, contact us
01/12/2018
CERN

19. Thank you!