1. SOA-36: Tuning and Scalability for Your Sonic ™ Enterprise Messaging Andreas Gies Principal Architect

2. © 2008 Progress Software Corporation2 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging D I S C L A I M E R Setting Performance Expectations  System performance is highly dependent on machine, application and product version. Performance levels described here may or may not be achievable in a specific deployment.  Tuning techniques often interact with specific features, operating environment characteristics and load factors. You should test every option carefully and make your own decision regarding the relative advantages of each approach. D I S C L A I M E R

3. © 2008 Progress Software Corporation3 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Agenda  Overview – Typical performance projects  Discovery – Defining performance goals  Benchmark – Testing performance  Tuning – Improving results

4. © 2008 Progress Software Corporation4 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Performance Engineering Terms “System Under Test” “Test Harness” “Load” = “Sessions” * “Delivery Rate” “Latency” = ReceiveTime – SendTime “Test Components” “External Components” “Platform” “System Metric” R V “Variable” = client param, app param, system param V V Tip: Limit scope to those test components that are critical to performance and under your control

5. © 2008 Progress Software Corporation5 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Performance project documents  Proposal Goals and Scope Architecture Scenario definition  Project plan Details on architecture and scenarios Test cases Schedule & logistics  Performance report Executive summary Test results Conclusions

6. © 2008 Progress Software Corporation6 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Common Pitfalls  Testing functionality, not performance  No model for scalability  Tuning decisions not applicable to production  Insufficient time to test and tune  Lack of Performance Engineering skills  Inflexibility in exploring high-performance architecture options Rule of Thumb: Schedule a two hour discovery call to clarify customer’s understanding of performance issues.

7. © 2008 Progress Software Corporation7 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging The Performance Engineering Project The Project is Goal Driven Analyze Test Tune For each iteration 1. performance vs goal 2. identify likeliest area for gain Startup tasks 1. success criteria 2. resources & skills 3. tool selection Tip: Schedule daily meetings to share results and reprioritize test plan.

8. © 2008 Progress Software Corporation8 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Performance Project Skills SOLUTION Integration Expert Testing Expert Requirements Expert Cost/Benefit Load/Distribution Design Options

9. © 2008 Progress Software Corporation9 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Tools for a Messaging Benchmark Platform Component System Under Test Test Harness Test Configurator Test Analyzer

10. © 2008 Progress Software Corporation10 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Agenda  Overview – Typical performance projects  Discovery – Defining performance goals  Benchmark – Testing performance  Tuning – Improving results

11. © 2008 Progress Software Corporation11 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Performance Project Specification Rule of Thumb: Focus on message loads over 10 msg/sec, and process loads over 10,000 per hour. Project spec outline:  Project scope  Goals and procedures  Architecture  Business scenarios  Test cases Business view Value Relevance Risk ESB goals Throughput Latency Scalability Architecture Messaging ESB Process SOA Services

12. © 2008 Progress Software Corporation12 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Specification: Platform Architecture Back OfficeFieldDMZ Platform architecture spec  For each Machine: CPU number, type and speed Disk / Memory  Network topology & specs

13. © 2008 Progress Software Corporation13 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Specification: SOA architecture PartnerFieldHQ Finance SFA CRM ESB Tracking PoS Partner ESB SOA architecture spec  Brokers and service containers  Geographical distribution  Scaling model

14. © 2008 Progress Software Corporation14 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Specification: Data Architecture Data Schemas 1…nData Schemas 2…m ? Tip: Transform tools vary in efficiency: XSLT – slowest (but most standard) DataXtend ® SI much faster and most flexible Custom service – fastest, but less flexible Data architecture spec  Key transform events  Schema complexity  Opportunities for caching  Hi-load databases

15. © 2008 Progress Software Corporation15 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Business Scenario patterns Async Process Realtime Feed Request/Reply Tip: The easiest way to manage decoupled SOA processes is the Sonic ESB Itinerary.

16. © 2008 Progress Software Corporation16 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Scenario Specification Scenario spec details:  Business process description  QoS, availability, dependencies  Goals and expected benefits Tip: The User’s language, The User’s diagram, The User’s needs

17. © 2008 Progress Software Corporation17 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Specifying Test Cases Test case spec details:  Scenario being simulated  Load characteristics  Basic ESB process model  Services to deploy or simulate  Variables manipulated  Data measured Inputs Message Protocol QoS ESB process Service impl Distribution Params Results Throughput Latency Validation Tip: Avoid application details; focus on performance.

18. © 2008 Progress Software Corporation18 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Test Variables Tip: External JMS client variables are easily managed with the Test Harness. Sonic ESB  Service config  Endpoints / Connections  Process design  XML operations SonicMQ ®  Connection / session  Quality of Service  Interaction mode  Message def Load  Client sessions  Messages / second  Message size Application  Interaction pattern  Service overhead  Database interaction

19. © 2008 Progress Software Corporation19 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Agenda  Overview – Typical performance projects  Discovery – Defining performance goals  Benchmark – Testing performance  Tuning – Improving results

20. © 2008 Progress Software Corporation20 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Configuring the ESB  Upload services prototype services simulation services  Upload processes test cases simulated external services  Other resources external web services databases

21. © 2008 Progress Software Corporation21 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Provisioning Brokers and Containers  Message brokers  CAA  Cluster / DRA  ESB Containers & Connections Tip: For each client connection scenario define a named JNDI Connection Factory and document its characteristics.

22. © 2008 Progress Software Corporation22 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Client Session Configuration Rule of Thumb: Create one connection for every 10 to 20 sessions Test Harness QoS Msg gen reports ESB Reply

23. © 2008 Progress Software Corporation23 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Configuring Client Quality of Service (QoS)  HTTP and Web service clients  JMS Client Send mode Ack mode Flow control  ESB service  JMS options  Broker d isk overhead Tip: Best compromise is at least once QoS based on CAA, Persistent, Async disk and DUPS_OK.

24. © 2008 Progress Software Corporation24 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Generating Message Content Status Request priority=2 org=1 Cust Info gen random int gen sample xml Addr svc lookup transform rule Tip: Collect sample messages early on, to ensure that routing rules and lookups run as expected.

25. © 2008 Progress Software Corporation25 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Simulating Clients with Test Harness System Under Test Test Harness Connection Broker JNDI Msg Pool Reply Request Message Generation

26. © 2008 Progress Software Corporation26 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Running Performance Test Iterations  Logistics multi-client test intervals Warm-up / cool-down  Data collection  Repeatability efficiency relevance accuracy Rule of Thumb: Start with 10-20 intervals of 30 secs; if steady state is soon reached, reduce to 10 intervals of 10 seconds, plus warmup.

27. © 2008 Progress Software Corporation27 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging The Zen of Benchmarks  Spiral Up  Childish curiosity  Focus on goals  Ensure repeatability  Deal with now  Big picture  Use every resource  Prioritize

28. © 2008 Progress Software Corporation28 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Agenda  Overview – Typical performance projects  Discovery – Defining performance goals  Benchmark – Testing performance  Tuning – Improving results

29. © 2008 Progress Software Corporation29 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Performance Tuning with Sonic ESB ®  Platform factors  Top Ten Tuning Tips  Tuning options Broker ESB

30. © 2008 Progress Software Corporation30 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging ESB System Usage: CPU  Sources of CPU cost: Application code XML Parsing CPU cost of i/o –Network sockets –Log/Data disks Web service protocol –Web Service security Security –Authorization –Message or channel encryption

31. © 2008 Progress Software Corporation31 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging ESB System Usage: Disk  Sources of Disk overhead: Database services Large File / Batch services Message recovery log Message data store Flow to disk overflow Tip: To estimate best-case message log performance, use the DiskPerf utility bundled with Test Harness.

32. © 2008 Progress Software Corporation32 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging ESB System Usage: Network  Sources of Network overhead: Client requests ESB Service invocations ESB Web service callouts CAA broker replication messages Admin messages  Network performance limit: Network bandwidth: –Network Interface Card (NIC) –Switches, routers and firewalls Network load Tip: To estimate best-case network performance, use the DiskPerf utility bundled with Test Harness.

33. © 2008 Progress Software Corporation33 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Tip #1: Increase sender and listener threads to make service & client scale  Increase # Listeners for key entry endpoints  Add more Service/Process instances  Warning: Intra-Container ignores endpoint split scalable services into separate containers turn intra-container messaging off note: sub-Process is always intra-container Container1 ServiceX … Container2 ServiceX … … Tip: Stop increasing Listeners when CPU usage nears maximum acceptable.

34. © 2008 Progress Software Corporation34 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Tip #2: Implement optimal QoS to balance speed versus precision ESB QoS MQ Setting Message Loss Events Duplicate Msg Events N/ADiscardable Buffer overflow, Any crash Missed Ack, Crash of UoW Best Effort NonPersistent, DupsOK ack Broker crash, Client crash Missed Ack, Crash of UoW At Least Once Persistent, DupsOK ack Never Missed Ack, Crash of UoW Exactly OnceTransactedNever Tip: With CAA configured, Best Effort service is equivalent to At Least Once, with substantially lower overhead. (Based on CAA brokers and fault-tolerant connections)

35. © 2008 Progress Software Corporation35 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Tip #3: Re-use JMS objects to reduce setup costs  Objects with client and broker footprint: Connection Session Sender/Receiver/Publisher/Subscriber Temporary destination  Tuning strategies: Reuse JMS objects in client code Share each Connection across sessions Share Sessions across Producers and Consumers –but not across JVM Threads For low-load topics/queues –Use Anonymous Producer –Use wildcard or multi-topic subscription Rule of Thumb: Up to 500 queues per Broker and 10,000 topics per broker.

36. © 2008 Progress Software Corporation36 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Tip #4: Use intra-container service calls to avoid broker hops Broker Dispatch Outbox Marshall Msg Send Msg … … Dispatch Outbox Unmarshall Msg Marshall Msg Call onMessage Send Msg … Receive Msg Unmarshall Msg Call onMessage Instantiate Proc … Inter-Container Messaging Intra-Container Messaging Tip: If you save service resources in sonicfs, the ESB container will dynamically load and cache them.

37. © 2008 Progress Software Corporation37 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Tip #5: Use NonPersistentReplicated mode to reduce disk overhead  Normal broker mechanisms require disk sync Contributes to latency across the board Interferes with batching of packets Limited bandwidth  Disabling disk sync eliminates this overhead Send mode NonPersistentReplicated Optional broker params to disable entirely WARNING: Log-based recovery will lose recent messages BUT: CAA failover will not Rule of Thumb: Network overhead for Replication channel is about ½ the Persistent Msg load of the broker.

38. © 2008 Progress Software Corporation38 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Tip #6: Use XCBR instead of CBR to eliminate Javascript overhead  CBR rules implemented via Javascript Dynamic change with complex rules Very high overhead for runtime engine  XCBR rules extract data fields for comparison Only simple comparisons supported No script engine overhead Use message property data key for best effect Rule of Thumb: Invocation of the Rhino JavaScript engine requires about 100 milliseconds CPU time on a typical platform.

39. © 2008 Progress Software Corporation39 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Tip #7: Use message batching to accelerate message streams  Message transfer overhead is generally fixed  Hidden ack messages amenable to tuning: AsyncNonPersistent mode decouples ack latency Transaction Commit allows 1 ack per N messages DupsOK ack mode allows ‘lazy’ ack from consumer Pre-Fetch Count allows batched transmit to consumer  ESB Design option: send one multi-part message instead of N individual messages  XML transforms and other services handle multi-record data efficiently Producer Broker Consumer Msg … Ack Msg Ack …

40. © 2008 Progress Software Corporation40 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Tip #8: Minimize XML/SOAP operations to avoid parsing overhead  Bypass SOAP and Web services processing Use HTTP Direct Basic instead of SOAP or WS Risk of invalid XML if source is unreliable  Combine multiple XML parsing steps into one Save target XPath results as Message props Also relevant for BPEL correlation ID’s Input Message XML Transform Custom JAXB Input Message propY=9 propX=1 Input Message JAXB Obj Msg part XCBR JAXB Service

41. © 2008 Progress Software Corporation41 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Tip #9: Use high-speed encryption to reduce security overhead  Default SSL encryption uses old RCA stack At least 2X slower than more modern options  Change to any JSSE compliant stack: Modify client DSSL_PROVIDER_CLASS to: progress.message.net.ssl.jsse.jsseSSLImpl Change broker SSL provider from RSA to JSSE  Use more efficient cipher suites: RSA_With_Null_MD5 is the smallest and fastest  Reduce broker memory overhead by deleting any unused ciphers

42. © 2008 Progress Software Corporation42 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Tip #10: Use stream API's to improve large message performance  SonicMQ Recoverable File Channels Uses JMS layer to manage large file transfer Queue-based initiation of transfer High-speed JMS pipeline for blocks of data Recovery continues at point interrupted  Sonic ESB open-source Large Message Service Provides dynamic provisioning Interacts with ESB processes  SonicStream API (version 7.5 or later) Topic-based, pipeline into Java Stream api No recovery

43. © 2008 Progress Software Corporation43 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Broker Tuning Parameters  Core Resources: JVM heap size JVM thread, stack limits DRA, HTTP and Transaction threads TCP settings  Message storage: Queue size and save threshold Pub/sub buffers Message log and store  Message management Encryption Flow control and flow- to-disk Dead message queue management  Connections Mapping to NIC’s Timeouts Load balancing Rule of Thumb: For non-trivial queues, multiply default settings by 10 to 100.

44. © 2008 Progress Software Corporation44 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging ESB Tuning Options  Load balancing and scalability of services: Number of distributed service instances Number of service listener threads  Container Java VM heap size  Intra-Container messaging  Endpoint and connection parameters Same principles as JMS client Tip: Start with small Java heap and only increase -Xms size if it improves performance.

45. © 2008 Progress Software Corporation45 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging  Progress Software Developers Network: http://www.psdn.com Search: “performance”, “sonic test harness”, “tuning”  SonicMQ Performance Tuning Guide  Benchmarking Enterprise Messaging Systems white paper  Sonic Test Harness documentation  Progress Professional Services Developer training courses Sonic Performance Analysis package Other Performance Engineering Resources No “Magic Bullet”, but plenty of places you can go for info:

46. © 2008 Progress Software Corporation46 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Questions ?

47. © 2008 Progress Software Corporation47 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging Thank You

48. © 2008 Progress Software Corporation48 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging