1. Tim De Borger Principal Solution Consultant May 18 th, 2007 Tuning the ESB How to make the Bus drive faster

2. © 2007 Progress Software Corporation2 Agenda  Quick overview  Current tools  What are we measuring  Message Influences (what is Flow Control?)  Simulation environment  Scaling in different ways  Know what you are doing  Hands on testing  The Simple Calculation  Conclusion

3. © 2007 Progress Software Corporation3 Quick overview  Performance Tuning Art of tuning –Change only 1 parameter at a time –Take your time for it –Always check the same things (even reboot/restart after each run) –Know what you are measuring Check resources –Disk –Memory –CPU –Network Have a plan –What are the requirements –Multiply requirements by factor X (Stress Situation) –When to stop

4. © 2007 Progress Software Corporation4 Current tools  Testing SonicMQ Sonic Test Harness –Sends in a lot of messages –Receives a lot of messages –Difficult to control –Focused on message throughput (Volume)  Testing the ESB No tool is provided (yet) A lot of external influences No Controlled Environment

5. © 2007 Progress Software Corporation5 What are we measuring (or trying to)  We need to be able to understand the time a message is spending in our system  External influences are very important for performance Longer runs for external DB calls WebServices  We need to define the latency of the message on the System (not on A Single Queue or Topic)

6. © 2007 Progress Software Corporation6 Messaging Influences  Message Scenario 1 Running 1 SonicMQ Broker Using 1 Topic Running 1 Publisher (Fast) Running 1 Subscriber (Fast)  Everything is working just fine  Message Scenario 2 Adding another Subscriber !!! Unknown behavior of Subscriber => DANGER!!!!!

7. © 2007 Progress Software Corporation7 Flow Control Demo Point to PointSenderSender PotentialReceiverPotentialReceiver PotentialReceiverPotentialReceiver QueueQueue  Flow Control in PtP: If the Maximum Size of a Queue is reached, Flow Control is kicking in Sender seems to be hanging but is only waiting for space in the Queue Flow Control can be switched off and replaced by an Exception in your code.

8. © 2007 Progress Software Corporation8 Flow Control Demo  In Pub/Sub, the working of Flow Control is more difficult to predict Each Subscriber gets a Buffer (Outgoing Buffer Size) Sonic will try to only have 1 copy of the message in memory When the buffer is reaching a threshold, Flow Control is issued. Threshold depends on message priority Publish and SubscribePublisherPublisherTopicTopic SubscriberSubscriber SubscriberSubscriber

9. © 2007 Progress Software Corporation9 Outgoing Buffer Flow Control Demo Publish and Subscribe PublisherPublisherTopicTopic SubscriberSubscriber SubscriberSubscriber Pub/Sub TEXT Outgoing Buffer Only a pointer is inserted in the Buffer but the calculated size is the message size Ack Danger !!! Slow Subscriber may result in the following situation TEXT FLOW CONTROL

10. © 2007 Progress Software Corporation10 Conclusion of the test  We can have a fully working system that runs without any problems  Adding 1 Client can have dramatic impact 1 Slow SubScriber will make the Publisher Slow Badly written Client(s) can bring any message Broker to its knees  Where to look and debug if this is in an ESB Service? More important: HOW TO SOLVE THIS?

11. © 2007 Progress Software Corporation11 Using the ESB to Simulate the ESB  We developed the Simulation Tool in order to: Have a controlled environment to investigate the impact of introducing an ESB. Have an environment to understand the working and concepts of the ESB –Difference between a Process and loosely linked services –What is the impact of Intra-Container messaging –How to develop a deployment scheme and calculate the number of listeners. (slow 1 st Service, Slow last Service, …) Have the possibility to trace the message through the system –Time-Stamp –Endpoints report in the message body Easy deployable model to test ESB over multiple containers –Impact on Quality of Service –Testing with a DRA Model  We developed it with the ESB itself.

12. © 2007 Progress Software Corporation12 Concept  How to Simulate Test Client that sends an incoming message Test Client that receives the outcome of the ESB (REPLY- TO) Sender puts a time-stamp Receiver gets the timestamp and calculates the message latency in the ESB System. ESB ServiceType for Simulation  What to Simulate CPU External influences Message growth

13. © 2007 Progress Software Corporation13 Simulation environment External Influences Processing Power Message Growth

14. © 2007 Progress Software Corporation14 Scaling in different ways  When looking at the ESB, we can scale in 3 different ways  Horizontal: Moving from 1 listener to X listeners Differences between Process and Service Intra-container messaging  Vertical: Moving from 1 JVM to Y JVM’s Quality of Service might forbid this!  Diagonal: Moving over Z multiple machines

15. © 2007 Progress Software Corporation15 XYZ Scaling X-scaling: Multiple Listeners Y-scaling: Multiple JVMs Z-scaling: Multiple Machines

16. © 2007 Progress Software Corporation16 Know what you are doing  There are limitations: The Obvious: –Disk traffic –CPU –Memory The not so Obvious: –Flow Control –Your JVM:  Thread handling  Memory size  Resources given by the O/S  SonicMQ (move to a cluster etc)

17. © 2007 Progress Software Corporation17 The Files  Files are available: Java code for the Service Type esbstyp definition for the Service type Properties definition for the Service Type Java Client/Receiver  Installing the Service Type Put the jar file in the Classpath of the ESB Container Import the esbstyp file Copy the properties file to the right place

18. © 2007 Progress Software Corporation18 Setup  Create a new ESB container  Create some test endpoints if you want or use Sample Queues/Endpoints that exist  Create a couple of the Benchmark Services  Deploy them into a container  Add the container to an MF Container  Start the container(s) so the environment is running.

19. © 2007 Progress Software Corporation19 Configuration  Create 3 Services from the ESB Benchmarking Type and define the needed Endpoints on them.  Create a Process that uses the 3 services Make the waiting times 500, 1000, 2000 ms for the 3 entries  Set REPLY-TO on the process

20. © 2007 Progress Software Corporation20 Hands on testing: Running Test  In the Benchmarking\bin directory find the bench.bat file.  Set the path right for your SonicMQ installation as it calls SonicMQ.bat from the samples  It is a modified Talk application so make changes to the parameters for connection  Added 3 parameters to give on the Bench.bat command Number of messages to send Sleep time in ms between those messages Size of the initial message to be sent in Bench.bat 20 2000 2000

21. © 2007 Progress Software Corporation21 Hands on testing  First go: 1 ESB Container Make the 2 services each sleep for 500 ms. Run Bench 100 1000 1024 Results? Run Bench 100 100 1024 Results? Why? Solutions?  Play further with different setups. Bench 100 250 1024 Bench 100 100 1024 Bench 100 50 1024 Bench 100 1 1024

22. © 2007 Progress Software Corporation22 Results 1: 1 ESB Container with 1 listener Messages are handled without Delay by the running ESB Service. This results in a flat latency line Messages are handled with a very small Delay by the running ESB Service. This results in a very slow increasing latency line In all other cases, the ESB cannot cope with the publishing rate of the client and the message latency will grow. The faster the publishing rate, the steeper the curve.

23. © 2007 Progress Software Corporation23 Performance Behavior Messages are handled without Delay by the running ESB Service. This results in a flat latency line Messages are handled with a very small Delay by the running ESB Service. This results in a very slow increasing latency line In all other cases, the ESB cannot cope with the publishing rate of the client and the message latency will grow. The faster the publishing rate, the steeper the curve.

24. © 2007 Progress Software Corporation24 Results: Sending 1 message every 2 seconds

25. © 2007 Progress Software Corporation25 Results: Sending 1 message every second

26. © 2007 Progress Software Corporation26 Results: Sending 1 message every 500 ms

27. © 2007 Progress Software Corporation27 Results: Sending 1 message every 250 ms

28. © 2007 Progress Software Corporation28 Results: Sending 1 message every 100 ms

29. © 2007 Progress Software Corporation29 Results: Sending 1 message every 10 ms

30. © 2007 Progress Software Corporation30 Results 2: 1 ESB Container with 2 listeners Messages are handled without Delay by the running ESB Service. This results in a flat latency line In all other cases, the ESB cannot cope with the publishing rate of the client and the message latency will grow. The faster the publishing rate, the steeper the curve.

31. © 2007 Progress Software Corporation31 Results 3: 1 ESB Container with 5 listeners Mission Accomplished

32. © 2007 Progress Software Corporation32 Results 4: 2 ESB Container with 5 listeners Faster messaging results again in increasing the message latency. At one point, the JVM will not allow more Listeners (Threads) and you Need to start a Second ESB Container.

33. © 2007 Progress Software Corporation33 The Simple Calculation  There is a calculation that can be made in order to find out how many Listeners your Service Needs (This calculation is needed for every Service in the Process): Maximum time for the message to go through the Service (Latency) = TSL (2 sec) Time to handle the Load = THL(300 sec) Number of messages in the Load = NM(1000 msg) Formula to give the number of handlers: –((NM / THL) * TSL) (6.67)  For this setting, you should make sure that the number of messages that can be treated in parallel should be 7 or 8.

34. © 2007 Progress Software Corporation34 Conclusion  This tool can be used for testing/simulating: DRA environments Deployments of ESB Components over multiple containers Investigate and test the differences between an ESB Process or linking the ESB Services together  This tool should not be used for Stress- Testing the SonicMQ environment.

35. © 2007 Progress Software Corporation35 Questions

36. © 2007 Progress Software Corporation36 CONNECT EVERYTHING. WORKING ON ACHIEVE ANYTHING. www.sonicsoftware.com