Felix Ehm CERN BE-CO

Content  Introduction  JMS in the Controls System  Deployment and Operation  Conclusion

Introduction to JMS – The API  Java Messaging Service Messaging API defined in 2002 Decouples Source and Destination Topic / Queue Concepts for named Destinations

Introduction to JMS – A Broker  Technology originated from the financial sector “Few” sources and large set of readers  A Broker is the central entity Accepts messages from 1..* publisher and re-distributes to 0..* subscribers Provides Quality of Service for Messages ○ Persistency, Transactions, Buffering, Expiration Policies, Slow Client Handling Design to distribute data reliably and scalable Load balancing / fail-over / embedded deployment scenarios possible Broker

Introduction to JMS - A Broker  Many broker implementations available Proprietary IBM WebsphereMQ, SonicMQ, TIBCO, RedHat MRG, … OpenSource Apache ActiveMQ, Apache Apollo, RabbitMQ, HornetMQ, …  Broker may be implemented in another language than Java (e.g. Qpid & RabbitMQ)  Communication protocol is non-standard Brokers of different vendors are not interoperable AMQP & STOMP address this issue

Introduction to JMS - Request/Reply  “Direct communication” via Broker using Request – Reply mechanism  Due to intermediate Broker two hops are required RequestorReplier Broker Request MSG Reply MSG Request Queue Temporary Reply Queue creates sends

Introduction to JMS - Selectors  Filter messages from one Destination using Messages Selectors Each message can be enriched with Attributes Subscriber receives filtered messages via defined selector Broker Subscriber selector: BUILDING ≠ A HEADERS CYCLE=A TIME=9845 BODY JMS Message BUILDING=A BUILDING=B BUILDING=C Topic/ Queue

Introduction to JMS - Persistency  JMS defines persistent and non-persistent messages  Broker stores message to disk before acknowledging to sender  Avoids loss of data when broker crashes  Drawbacks Higher disk load Slower message processing Sender 1. send 3. ack 2. store

CERN’s Accelerator Complex

JMS in the Controls System  Purpose Reliable transport of data between (Java) Processes High level controls applications + middle tier servers ○ Alarm System ○ Data Rendering Services ○ (Control) GUIs ○ Logging Services ○ Beam Security System(s) ○ Access Systems, ….  Today vital part of Beam Operations  Product choice: No JMS - No Beam !

JMS in the Controls System - History  History SonicMQ since 2001 for CERN’s Alarm and CO Monitoring system OpenSource ActiveMQ since 2005 ○ Middle tier servers update GUIs ○ Middle tier to Middle tier Phased out in 2013

JMS for Accelerator Controls  Early days only Java clients via JAPC japc-ext-remote contains JMS communication  Today multi-language clients JAPC clients remain Java using ActiveMQ client libraries directly C++/Python using STOMP

 29 production brokers  Single or broker cluster  Large spectrum of usage patterns from few topics to 40K topics from few subscribers to >23 K subscribers from couple msg / minute to 10K msg / sec from few bytes/msg to 3 Mbyte/msg Deployment and Operation

Some numbers :  300 Applications  4’400 Connections  40’000 Subscriptions  85’000 Topics  8 Million msg/h IN, 3.5M msg/h OUT Archived Uptime in 2012: % Not all data which is produced is also consumed

Deployment and Operation  Distribute Load Interconnect two Brokers Consumers/Producer can choose any Broker Sharing of work  Smooth Upgrades possible Client library version != Broker version OK  Drawbacks Consumer might not get all messages in case of network error Routing overhead

The STOMProtocol

 Simple Text Oriented Message Protocol  Defines messaging API + wire format  Many open source clients libraries available Python, C/C++, Java, Perl, Ruby, PHP, Erlang, …  Targets the messaging interoperability among language, platforms and brokers  Supported by many brokers  Very easy to use

Example to send STOMP in Python mysocket = socket.socket(socket.AF_INET,socket.SOCK_STREAM) mysocket.connect( ("jms-co-dev", 61680) ) text = '''SEND\n\ destination:/queue/CERN.DEPLOYMENT\n\ HOST:%s\n\ INSTALLPATH:%s\n\ PRODUCT:%s\n\ TIME:%s\n\ persistent:false\n\n\x00''‘ mysocket.sendall(text) mysocket.sendall("DISCONNECT\x00") mysocket.close()

The AMQProtocol

 Advanced Message Queue Protocol  Defines messaging API + wire format  Next evolution after JMS Consortium of RedHat, Credit Suisse, Microsoft, JPMorgen, Barclays, VMWare, Cisco, …  Released as Version 1.0 More and more brokers implement it  Direction for the future ?

Conclusion  JMS is vital part of Accelerator Controls System  Performs very well for publish - subscribe scenarios  ActiveMQ was a good choice Running reliably Low maintenance and flexible deployments Provides many metrics for diagnostic and monitoring  Multi-language applications can use service  Central (redundant) service

Result Example  Messages received/sec over time time AVG Messages / sec

Result Example  Histogram of delayed message distribution Delay time [msec] # Messages

Result Example  Number of “late” (>300ms) messages for test duration time Delayed Messages

Message Delay Summary  Measuring delayed Message ~ 99.32% of messages delivered < 300ms Rest < 1 sec  Doubling numbers of messages (~9500 msg/sec) 100% between 5 and 10 seconds Further studies for improvement needed

40% less CPU load thanks to optimization Result Example  CPU load time % CPU load Consumer connects Consumer disconnects

Broker Memory Consumption  Initial Memory is proportional to Connections Subscriptions 1000 Subscription / Connection500 Subscriptions / Connection