1. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t MSG Status update Daniel Rodrigues

2. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Presentation Summary MSG overview Timeline ActiveMQ Msg-publish-simple Msg-consumer2oracle Current deployments –OSG –Gridview Next steps

3. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services MSG overview

4. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Features of an Messaging System: –Flexible architecture: Deliver messages, either in point to point (queue)… … or multicast mode (topics) Support Synchronous or Asynchronous communication. –Reliable delivery of messages: Provide reliability to the senders if required Configurable persistency / Master-Slave. –Highly Scalable: Network of Brokers ActiveMQ is an Open Source message broker providing these and many other features. MSG overview

5. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services A real life working example: MSG overview

6. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Timeline 1 st Trimester (Oct/Dez): –Testing of ActiveMQ 4.1, using netcat/bash scripts to test throughput; –Integrated testing with twisted stomp, python framework; 2 nd Trimester (Jan/Mar): –Simple publishers and python consumers developed by Piotr Nyczyk on place for OSG; –Msg-publish-simple and msg-consume2oracle (bugfixes + improvements + testing); –Moving to ActiveMQ 5.0, retesting throughput; –Definition of message-classes for gridview-gridftp parsing records; –Testing different broker configurations: Master-Slave, JDBC persistence, file persistence; –Testing of JMS Java clients (producers and consumers) together with python. 3 rd Trimester(Apr/): –Network of Brokers testing; –GWT (Google Web Toolkit) Client prototype. –Setting up publishers/consumers for gridview-gridftp First 4 machines, now extended for additional 27 machines; –Definition of message-classes for downtimes with OSG and initial protyping. –Moving to ActiveMQ 5.1

7. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services ActiveMQ updates Move from 4.1 => 5.0 –Producer Flow Control! –AMQ Message Store, faster persistence –Command Agent –(Blob Messages) Now moving from 5.0 => 5.1 –More stable –Default AMQ Message Store + bugfixes Issues with recovering from failure: very long time to replay all kept logs! No viable Master/Slave with fast persistence otherwise

8. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services MSG publish simple Simple python publisher script, no other dependency besides… python! Publishes messages over either stomp or http, depending on network availability. Currently version 0.9.3-1: –A few bug fixes since initial version –Priority to stomp over http. –Configurable logging

9. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services MSG consume2oracle Python consumer script – again no other dependency besides… python (and cx_oracle)! How does it work: –Creates a durable subscription to the topics –Receives the messages –Retrieves the records sent along with a message –inserts them into an oracle view Currently version 0.9.1-1: –Bug fixes –Configurable logging Snapshot: –Implementing Status summaries topic (publishes summary information back into MSG)

10. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Current Deployments OSG: –Broker at gridmsg001 activemq 5.0.0 Persistence with journaledJDBC (derby) –msgConsume2Oracle at gridmsg001 Publishing to SAM metricOutput/metricDescription –msgPublishSimple Several publishing from indiana.edu. Gridview: –Broker at gridmsg002 activemq 5.0.0 Persistence with journaledJDBC (derby) –msgPublishSimple Called from gridview-publisher Deployed on 27 gridftp nodes for cms t1transfers. –msgConsume2Oracle gridmsg002: publishing to testDatabase Gridview machine publishing to gridview DB.

11. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Current Deployments Testing : –Brokers: lxb6118 / 6117 -> 5.1 Snapshot Many different configurations along the time (Master/Slave Network of Brokers, file persistence, etc) –Publishers: Java JMS producer (pcitgd24) lxplus based msg-publish-simple –Consumers: msg-consume2oracle (lxb6118/6117) GWT test (pcitgd24)

12. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Next Steps OSG scheduled downtimes –Message Class defined –Some test messages into a test DB done –Waiting Arvind’s prototype (very soon) JobDetails for Dirac3 detailed logging –Interested in getting information such as cputime, and considering using MSG to do it. –Message class under definition MSG System Management: –How to operate a messaging system in production? –We will need to know what’s going on!

13. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Thank you for your attention.

14. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Support Slides

15. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Some recent test results

16. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Some recent test results

17. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Some recent test results

18. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Some recent test results

19. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Console, GridMSG001 Broker restarts: 10 Mar. 18 Apr, too many open connections -> increase of file descriptors. Since 18 April, 220000 messages.

20. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Console, GridMSG002 420087 messages forwarded to each consumer (test + GV).