Presentation is loading. Please wait.

Presentation is loading. Please wait.

SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Analyzing, testing and tuning ESB/JMS performance David Hentchel Principal Solution Engineer.

Similar presentations


Presentation on theme: "SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Analyzing, testing and tuning ESB/JMS performance David Hentchel Principal Solution Engineer."— Presentation transcript:

1 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Analyzing, testing and tuning ESB/JMS performance David Hentchel Principal Solution Engineer

2 © 2007 Progress Software Corporation 2 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Agenda Methodology : Review the recommended approach to project and procedures Analysis: Understand how to characterize performance requirements and platform capacity Testing: Learn how to simulate performance scenarios using the Sonic Test Harness Tuning: Know the top ten tuning techniques for the Sonic Enterprise Messaging backbone Analyzing, testing and tuning ESB/JMS performance

3 © 2007 Progress Software Corporation 3 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging D I S C L A I M E R Setting Performance Expectations System performance is highly dependent on machine, application and product version. Performance levels described here may or may not be achievable in a specific deployment. Tuning techniques often interact with specific features, operating environment characteristics and load factors. You should test every option carefully and make your own decision regarding the relative advantages of each approach. D I S C L A I M E R

4 © 2007 Progress Software Corporation 4 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Agenda Methodology : Review the recommended approach to project and procedures Analysis: Understand how to characterize performance requirements and platform capacity Testing: Learn how to simulate performance scenarios using the Sonic Test Harness Tuning: Know the top ten tuning techniques for the Sonic Enterprise Messaging backbone Analyzing, testing and tuning ESB/JMS performance

5 © 2007 Progress Software Corporation 5 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Performance Concepts and Methodology Terms and definitions Performance engineering concepts Managing a performance analysis project Skills needed for the project Performance Tools Project timeline

6 © 2007 Progress Software Corporation 6 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Performance Engineering Terms System Under Test Test Harness Load = Sessions * Delivery Rate Latency = ReceiveTime – SendTime Test Components External Components Platform System Metric R V Variable = client param, app param, system param V V Expert Tip: Limit scope to those test components that are critical to performance and under your control

7 © 2007 Progress Software Corporation 7 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Concepts: Partitioning Resource Usage % CPU Partitionable resources can be broken down as the sum of the contributions of each test component on the system Total resource usage is limited by system capacity Becomes the bottleneck as it nears 100% Goal is linear scalability as additional resource is added Vertical versus Horizontal scalability Total latency is the sum across all resource latencies, i.e.: Latency = CPU_time + Disk_time + Socket_time + wait_sleep ( OverheadX Message rate ) = Load (Writes/sec)(msg/sec)(writes/msg)svcs Bottom-Up Rule: Test and tune each component before you test and tune the aggregate.

8 © 2007 Progress Software Corporation 8 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Concepts: Computer Platform Resources Favorite tools: task mgr, perfmon, top, ping –s, traceroute Disk I/O (read/write) Network I/O (send/receive) CPU time Memory (in use, swap) # Threads Use level of detail appropriate to the question being asked Machine resource (such as %CPU) artifacts: side effects from uncontrolled applications timing of refresh interval correlation with test intervals Scaling across different platforms and resource types

9 © 2007 Progress Software Corporation 9 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging The Performance Engineering Project The Project is Goal Driven Analyze Test Tune For each iteration 1. Test performance vs goal 2. Identify likeliest area for gain Startup tasks 1. Define project completion goals 2. Staff benchmarking skills 3. Acquire test harness Expert Tip: Schedule daily meetings to share results and reprioritize test plan.

10 © 2007 Progress Software Corporation 10 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Performance Project Skills Requirements Expert SLA/QoS levels – minimal & optimal Predicted load – current & future Distribution topology Integration Expert Allowable design options Cost to deploy Cost to maintain Testing Expert Load generation tool Bottleneck diagnosis Tuning and optimization SOLUTION I.E. T.E. R.E. Cost/Benefit Load/Distribution Design Options

11 © 2007 Progress Software Corporation 11 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Tools for a Messaging Benchmark Configurator – creates conditions to bring system under test into known state Harness – the platforms and components whose performance response is being measured Analyzer – tools and procedures to make meaningful conclusions based on result data. Platform Component System Under Test Test Harness Test Configurator Test Analyzer

12 © 2007 Progress Software Corporation 12 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Performance Project Timeline Development Project Week Process Dev Service DevSizing System Test Deployment Plan Launch Performance Project Perf Prototype DesignApplication Tuning

13 © 2007 Progress Software Corporation 13 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Agenda Methodology : Review the recommended approach to project and procedures Analysis : Understand how to characterize performance requirements and platform capacity Testing: Learn how to simulate performance scenarios using the Sonic Test Harness Tuning: Know the top ten tuning techniques for the Sonic Enterprise Messaging backbone Analyzing, testing and tuning ESB/JMS performance

14 © 2007 Progress Software Corporation 14 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Performance Analysis Performance scenarios: requirements and goals Some generic performance scenarios System characterization: platforms and architecture Test cases: specification for benchmark Utilization (% total) Capacity (units/sec) Efficiency (units/msg) X Load (msg/sec) =

15 © 2007 Progress Software Corporation 15 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Performance Scenario Specification First, triage performance-sensitive processes substantial messaging load and latency requirements impact of resource-intensive services Document only process messaging & services leave out Application specific logic – this is a prototype Set specific messaging performance goals Message rate and size Minimum and average latency required Try to quantify actual value and risk Why this use case matters Rule of Thumb: Focus on broker loads over 10 msg/sec or 1 MByte/sec, and service loads over 10,000 per hour.

16 © 2007 Progress Software Corporation 16 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Generic Scenario:Decoupled process Asynchronous, loosely coupled, distributed services. Assumptions: Services allow concurrent, parallel distribution Messaging is lightweight, pub sub End-to-end process completes in real time May be part of a Batch To Real Time pattern Factors to analyze: Speed and scalability of invoked services Distributed topology Quality of Service Aggregate Latency over time across batched invocations Expert Tip: The easiest way to manage decoupled SOA processes is the ESB Itinerary.

17 © 2007 Progress Software Corporation 17 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Generic Scenario:Real time data feed High speed distribution of real time events Assumptions: Read-only data pushed dynamically to users Messages are small Service mediation is simple and fast Latency is very important, but QoS needs are modest Factors to analyze: Quality of Service, esp. worst case for outage and message loss Message rate and fanout (pub sub) Scalability of consumers

18 © 2007 Progress Software Corporation 18 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Generic Scenario: Simple request reply Typical web service call that waits for response Assumptions: client is blocked, pending response small request message, response is larger latency is critical Factors to analyze: Latency of each component service Load balancing of key services Recovery strategy if loop is interrupted Client network, protocol and security specs

19 © 2007 Progress Software Corporation 19 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Example: Performance scenario specification Overall project scope: Project goals and process Deployment environment System architecture For each Scenario: Description of business process Operational constraints (QoS, recovery, availability) Performance goals, including business benefits

20 © 2007 Progress Software Corporation 20 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Characterizing platforms and architecture Scope current and future hardware options available to the project Identify geographical locations, firewalls and predefined service hosting restrictions Define predefined Endpoints and Services Define data models and identify required conversions and transformations.

21 © 2007 Progress Software Corporation 21 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging DMZFieldDMZ Platform configuration specification CPU number type speed Network bandwidth latency speed Disk type speed Firewall cryptos latency Memory size speed Rule of Thumb: LAN 15 – 150 MBytes / second Disk: 2 – 10 MBytes / second XSLT: 200 – 300 KBytes / second

22 © 2007 Progress Software Corporation 22 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Platform Profile: Real-time messaging % capacity System resource limitations 90% 70% 20% 5% Rule of Thumb: Real-time, 1 KB messages, broker performance is about 1,000 to 10,000 msg/sec for each 1 gHz cpu power.

23 © 2007 Progress Software Corporation 23 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Platform Profile: Queued requests % capacity System resource limitations 50% 20% 40% 85% Rule of Thumb: Queued msgs, 1 KB messages, broker performance is about 100 to 1,000 msg/sec for each 1 gHz cpu power.

24 © 2007 Progress Software Corporation 24 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Architecture Spec: Service distribution Identify services performance characteristics Identify high-bandwidth message channels Decide which components can be modified Annotate with message load estimates Back OfficePartnerFieldFront Office CRM Finance SFA CRM SCM ESB Tracking Service ESB PoS ERP Adapter SCM Adapter Partner ESB Integration Broker

25 © 2007 Progress Software Corporation 25 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Architecture Spec: Data Integration Approximate the complexity of data schemas Identify performance critical transformation events Estimate message size Identify acceptable potential services Data Schemas 1…nData Schemas 2…m ? Expert Tip: Transform tools vary in efficiency: XSLT – slowest (but most standard) Semantic modeler – generally faster (e.g. Sonic DXSI) Custom text service – fastest, but not as flexible

26 © 2007 Progress Software Corporation 26 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging DEMO: Test lab setup Test hardware guidelines for lab computers setting up the lab network Test architecture location of test components installation of brokers configuration of service containers Test design assets sample service definitions & wsdls sample test documents

27 © 2007 Progress Software Corporation 27 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Specifying Test Cases Factors to include: Load, sizes, complexity of messaging Range of scalability to try (e.g. min/max msg size) Basic ESB Process model Basic distribution architecture Details to avoid: Application code (unless readily available) Detailed transformation maps Define relevant variables: Fixed factors Test Variables Dependent measures

28 © 2007 Progress Software Corporation 28 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Typical test variables JMS Client variables: Connection / session usage Quality of Service Interaction mode: Message size and shape ESB container variables Service provisioning and parameters Endpoint / Connection parameters Process implementation and design Routing branch or key field for lookup Expert Tip: External JMS client variables are easily managed with the Test Harness.

29 © 2007 Progress Software Corporation 29 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Example: Test Case Specification For each identified Test Case there is a section specifying the following: Overview of test How this use case relates to the scenario Key decision points being examined Functional definition How to simulate the performance impact Description of ESB processes and services Samples messages Design alternatives that will be compared Test definition Variables manipulated Variables measured Completion criteria Throughput and latency goals Issues and options that may merit investigation

30 © 2007 Progress Software Corporation 30 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Agenda Methodology : Review the recommended approach to project and procedures Analysis: Understand how to characterize performance requirements and platform capacity Testing : Learn how to simulate performance scenarios using the Sonic Test Harness Tuning: Know the top ten tuning techniques for the Sonic Enterprise Messaging backbone Analyzing, testing and tuning ESB/JMS performance

31 © 2007 Progress Software Corporation 31 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Testing Performance Staging test services in the test bed Staging brokers and containers Configuring the Sonic Test Harness Running performance tests and gathering data Evaluating results for each test case

32 © 2007 Progress Software Corporation 32 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Staging Test Services Deploying existing services Appropriate to use actual implementation of a service IF robust implementation exists minimal effort to set up in test environment no side effects with other test components Production ready services merit special treatment: perform unit load tests to get baseline document possible tuning / scaling options

33 © 2007 Progress Software Corporation 33 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Staging Test Services Prototyping proposed new services Prototype should include: Correct routing logic for Use Case process Approximately correct resource usage Generic data Prototype does not need: Detailed business logic Exception handling code Invocation of non-critical library calls Its a prototype. Just keep it simple

34 © 2007 Progress Software Corporation 34 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Staging Test Services Simulating non-essential services Use stub service as placeholder for service step that are not performance-sensitive Can return generic data Ensures ESB process for target use case will run correctly Useful stub services: Transform service GetFile service PassThrough service Enrichment service Prototype service (version or later)

35 © 2007 Progress Software Corporation 35 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Demo: Provisioning test services Status Info WSI: Address Svc Status Request Status Query XForm: Build query DBSvr: Query Test Harness Web Portal Adapter: M/F Callout Addr Info Query Result Enriched Result WSI: Address Svc Status Query Status Query PassThru: DBSvr: Query PassThru: Sleep Status Query Result Query Result Business Use CasePerformance Test Case

36 © 2007 Progress Software Corporation 36 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Provisioning test brokers Test broker must be similar to production Correct Sonic release and JVM Expected deployment heap size and settings CAA configuration Optimize network for replication channels Locate on separate host to avoid bottlenecks If failover testing is part of plan: –define fault tolerant (JNDI) connections DRA configuration Set up subset of clusters and WAN simulations Measure local broker configs first, then expand Expert Tip: For each client connection scenario define a named JNDI Connection Factory and document its characteristics.

37 © 2007 Progress Software Corporation 37 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Provisioning ESB containers Use ESB Container to manage service groups name accordingly to service group role plan to reshuffle services during tuning phase provision jar files out of sonicfs Use MF Container to control distribution name according to domain/host location configure Java heap appropriately –for IBM jdk make -Xms = -Xmx –for caching services (e.g. Transform, XML Server), add extra memory for locally-cached data Rule of Thumb: Default heap size is fine for most ESB containers; if memory is limited, reduce size, but no smaller than: 8MB + (service jar size) + (#Listeners) * (200KB)

38 © 2007 Progress Software Corporation 38 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Demo: Setting up containers for test Workbench view of containers Coding and debugging the prototype Runtime view of containers managing the distributed environment reinitializing back to a known state

39 © 2007 Progress Software Corporation 39 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Simulating endpoint producers and consumers Endpoint protocols and performance Test Driver options for various protocols Simulating process/thread configuration Implementing endpoint interaction modes Configuring client Quality of Service (QoS) Generating message content Demo of client/endpoint simulation System Under Test Test Harness

40 © 2007 Progress Software Corporation 40 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Endpoint protocols and performance JMS fastest client protocol strongest range of QoS and Failover HTTP moderate performance and QoS rigid connection model (requires client or router reconnect logic) Web Services slowest performance QoS and recovery depend on WS-* extensions File-based flat file pickup / dropoff, FTP limited to disk speeds (i.e. 1 to 5 MB / sec) appropriate for batch processing scenarios JCA appropriate for EJB server scenarios limited to EJB transaction speeds (i.e. 100 to 1000 msg / sec) Rule of Thumb: HTTP acceptors typically achieve ½ to ¼ the performance of comparable JMS/TCP acceptors.

41 © 2007 Progress Software Corporation 41 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Client session configuration Broker performance depends on scalability of connections and sessions JMS best-practice is one thread per session JMS sessions can efficiently share a connection Use session pool for clients and app servers For test simulation: determine allowable range of client threads test connection/session numbers up to max # threads distribute client processes / drivers across multiple machines, if needed, to avoid client-side bottleneck Rule of Thumb: Create one connection for every 10 to 20 sessions

42 © 2007 Progress Software Corporation 42 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Configuring client Quality of Service (QoS) HTTP and Web Services clients best possible QoS is at least once even with WS-Reliability JMS Client: CAA w NonPersistentReplicated exactly once Many shared subs versus one durable sub NonPersistent Request/Reply at least once Discardable send to avoid queue backup Flow to disk to prevent blocked senders ESB service: Exactly once uses JMS transaction At least once uses client ack Best effort uses dups_ok ack Broker: Sync (default) versus Async disk i/o Expert Tip: Best compromise is at least once QoS based on CAA, Persistent, Async disk and DUPS_OK.

43 © 2007 Progress Software Corporation 43 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Generating message content Simulate message size / distribution for accurate results Message content may trigger ESB routing rules Some services depend on message content key values must match existing data / rules duplicate key value could cause error services that cache content require accurate key distribution Simulating content in the client / driver file-based message pool message template generation Java / object message generator Message properties Status Request priority=2 org=1 Cust Info Sonic Test Harness supports all these gen random int gen sample xml Addr svc lookup transform rule

44 © 2007 Progress Software Corporation 44 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Demo: Simulating clients with Test Harness JNDI connection configuration Producer / Consumer parameters Message generation System Under Test Test Harness Connection Broker JNDI Msg Pool Reply Request Message Generation

45 © 2007 Progress Software Corporation 45 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Running performance test iterations Logistics of test orchestration managing multiple Test Harness clients configuring test intervals test warm-up and cool-down Data collection correlation Ensuring repeatability of results Demo of Test Harness iterations

46 © 2007 Progress Software Corporation 46 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Logistics of test orchestration Managing multiple Test Harness clients Simplest option is multiple command windows –use telnet sessions for remote hosts –initiate test and warmup –hit key in each window Advanced environments can use distributed driver: –Grinder, SLAMD, JMeter, LoadRunner, … Configuring test intervals long enough to detect trend effects short enough to allow fast iteration across tests Test warm-up and cool-down helps eliminate first-time test artifacts ensures steady-state performance numbers Rule of Thumb: Start with intervals of 30 secs; if steady state is soon reached, reduce to 10 intervals of 10 seconds, plus warmup.

47 © 2007 Progress Software Corporation 47 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Data collection correlation Test Harness output: Throughput (msg/sec) Latency (msecs per round trip) Message size (bytes) System measures: CPU usage (%usr, %sys) Disk usage (writes/sec) Broker metrics: Messaging rate (bytes/second) Peak queue size (bytes)

48 © 2007 Progress Software Corporation 48 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Ensuring repeatability of results Experimental method requirement critical in measuring impact of change validate by rerunning identical test Most common artifacts impacting repeatability messages left in queue duplicate files dropped in file system growing database size / duplicate keys disconnected Durable subscribers cached Service artifacts (ESB default) Expert Tip: Run initial tests twice in succession; if results differ by more than 3% or so, investigate why.

49 © 2007 Progress Software Corporation 49 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Demo of Test Harness iterations Baseline test Change test harness properties Rerun test Show spreadsheet across tests

50 © 2007 Progress Software Corporation 50 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Evaluating performance Measurement against goal Short of goal Perform bottleneck analysis / attempt tuning Review option of scaling up resources Review design change options Give up and re-think goal Meet or exceed goal Continue scaling up and tuning til it breaks Give up and declare success

51 © 2007 Progress Software Corporation 51 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Evaluating performance Bottleneck analysis Review of resource consumption Determine cpu-bound, disk-bound, net-bound Identify components using the resource Possibility of offloading to other hosts Examine trends in scalability tests Possibility of improving throughput by adding more client sessions, ESB listeners, clustered brokers, etc. Option of rebalancing (# threads, java heap, priority Go through Top Ten tuning tips and others… Last resort: recode or redesign to save cycles

52 © 2007 Progress Software Corporation 52 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Evaluating performance Compare across test runs Carefully planned test runs yield fertile comparisons: estimate cost/benefit of a feature or option estimate incremental overhead of a tunable parameter narrow the field of concerns and alternatives Advice in collating and analyzing test runs collect test summary results in spreadsheet save raw data and logs in a separate place save test config, so you can replicate later schedule ad hoc review after each test sequence Expert Tip: Verify key conclusions by replicating key test runs that led to that conclusion

53 © 2007 Progress Software Corporation 53 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Update Scenario Demo: Example test result matrix Query Scenario ORX1 KB ORX10 KB ORX100 KB XSVR1 KB12168 XSVR10 KB XSVR100 KB Msg Size DB Svc Thruput * Latency ** Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 * kbytes / second ** milliseconds

54 © 2007 Progress Software Corporation 54 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Agenda Methodology : Review the recommended approach to project and procedures Analysis: Understand how to characterize performance requirements and platform capacity Testing: Learn how to simulate performance scenarios using the Sonic Test Harness Tuning : Know the top ten tuning techniques for the Sonic Enterprise Messaging backbone Analyzing, testing and tuning ESB/JMS performance

55 © 2007 Progress Software Corporation 55 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Performance Tuning with Sonic ESB ® Diagnostics Review of ESB architecture Factors influencing message throughput Factors influencing message latency Factors influencing scalability Top Ten Tuning Tips Other tuning issues Broker parameters Java tuning ESB tuning Specialized ESB services

56 © 2007 Progress Software Corporation 56 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Review ESB Architecture SOA view SERVICES COMMUNICATION BACKBONE NETWORK SERVICE CONTAINER ESB CONTROL System view Message Log DBMS Cache / Swap Threads VM Threads VM Threads VM Threads VM I/O Controller Memory Network Interface Router/Switch

57 © 2007 Progress Software Corporation 57 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging ESB System Usage: CPU Sources of CPU cost: Application code XML Parsing CPU cost of i/o Network sockets Log/Data disks Web Service protocol Web Service security Security Authorization Message or channel encryption

58 © 2007 Progress Software Corporation 58 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging ESB System Usage: Disk Sources of Disk overhead: Database services Large File / Batch services Message recovery log Might not be used if other guarantee mechanisms work Message data store Disconnected consumer Queue save threshold overflow Flow to disk overflow Message storage overhead depends on disk sync behavior Explicit file synchronization ensures data retained on crash Tuning options at disk, filesystem, JVM and Broker levels Expert Tip: To estimate best-case message log performance, use the DiskPerf utility bundled with Test Harness.

59 © 2007 Progress Software Corporation 59 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging ESB System Usage: Network Sources of Network overhead: Client application messages and replies Service to service steps within the ESB (except intra-container) ESB Web Service callouts CAA broker replication messages Metadata (JMX, cluster, DRA) messages (normally < 1%) Computing network bandwidth: Network card: 100 mbit ~= 12 MB/sec, 1 gbit ~= 120 MB/sec Network switches are individually rated Computing network load: message rate X message size include response messages and intermediate steps add ack packets (very small) for each message send Expert Tip: To estimate best-case network performance, use the DiskPerf utility bundled with Test Harness.

60 © 2007 Progress Software Corporation 60 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Tip #1: Increase sender and listener threads to make service & client scale Increase # Listeners for key entry Endpoints Add more Service/Process instances Warning: Intra-Container ignores endpoint split scalable service into separate container turn intra-container messaging off note: sub-Process is always intra-container Container1 ServiceX … Container2 ServiceX … … Expert Tip: Stop increasing Listeners when CPU usage nears maximum acceptable.

61 © 2007 Progress Software Corporation 61 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Tip #2: Implement optimal QoS to balance speed versus precision ESB QoS MQ Setting Message Loss Events Duplicate Msg Events N/ADiscardable Buffer overflow, Any crash Never Best Effort NonPersistent, DupsOK ack Broker crash, Client crash Never At Least Once Persistent, DupsOK ack Never Exactly OnceTransactedNever Expert Tip: With CAA configured, Best Effort service is equivalent to At Least Once, with substantially lower overhead. (Based on CAA brokers and fault-tolerant connections)

62 © 2007 Progress Software Corporation 62 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Tip #3: Re-use JMS objects to reduce setup costs Objects with client and broker footprint: Connection Session Sender/Receiver/Publisher/Subscriber Temporary destination Tuning strategies: Reuse JMS objects in client code Share each Connection across sessions Share Sessions across Producers and Consumers –but not across JVM Threads For low-load topics/queues –Use Anonymous Producer –Use wildcard or multi-topic subscription Rule of Thumb: Up to 500 queues per Broker and 10,000 topics per broker.

63 © 2007 Progress Software Corporation 63 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Tip #4: Use intra-container service calls to avoid broker hops Broker Dispatch Outbox Marshall Msg Send Msg … … Dispatch Outbox Unmarshall Msg Marshall Msg Call onMessage Send Msg … Receive Msg Unmarshall Msg Call onMessage Instantiate Proc … Inter-Container Messaging Intra-Container Messaging v7.5: better! faster!

64 © 2007 Progress Software Corporation 64 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Tip #5: Use NonPersistentReplicated mode to reduce disk overhead Normal broker mechanisms require disk sync contributes to latency across the board interferes with batching of packets limited bandwidth Disabling disk sync eliminates this overhead Send mode NonPersistentReplicated Optional broker params to disable entirely WARNING: Log-based recovery will lose recent messages BUT: CAA failover will not Rule of Thumb: Network overhead for Replication channel is about ½ the Persistent Msg load of the broker.

65 © 2007 Progress Software Corporation 65 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Tip #6: Use XCBR instead of CBR to eliminate Javascript overhead CBR rules implemented via Javascript dynamic change with complex rules very high overhead for runtime engine XCBR rules extract data fields for comparison only simple comparisons supported no script engine overhead use message property data key for best effect Rule of Thumb: Invocation of the Rhino javascript engine requires about 100 milliseconds cpu time on a typical platform.

66 © 2007 Progress Software Corporation 66 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Tip #7: Use message batching to accelerate message streams Message transfer overhead is generally fixed Hidden ack messages amenable to tuning: AsyncNonPersistent mode decouples ack latency Transaction Commit allows 1 ack per N messages DupsOK ack mode allows lazy ack from consumer Pre-Fetch Count allows batched transmit to consumer ESB Design option: send one multi-part message instead of N individual messages XML transforms and other services handle multi-record data efficiently Producer Broker Consumer Msg … Ack Msg Ack …

67 © 2007 Progress Software Corporation 67 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Tip #8: Minimize XML/SOAP operations to avoid parsing overhead Bypass SOAP and Web Services processing Use HTTP Direct Basic instead of SOAP or WS Risk of invalid XML if source is unreliable Combine multiple XML parsing steps into one Save target XPath results as Message props Also relevant for BPEL correlation IDs Input Message XML Transform Custom JAXB Input Message propY=9 propX=1 Input Message JAXB Obj Msg part XCBR JAXB Service

68 © 2007 Progress Software Corporation 68 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Tip #9: Use high-speed encryption to reduce security overhead Default SSL encryption uses old RCA stack At least 2X slower than more modern options Change to any JSSE compliant stack: modify client DSSL_PROVIDER_CLASS to: progress.message.net.ssl.jsse.jsseSSLImpl change broker SSL provider from RSA to JSSE Use more efficient cipher suites: RSA_With_Null_MD5 is the smallest and fastest Reduce broker memory overhead by deleting any unused ciphers

69 © 2007 Progress Software Corporation 69 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Tip #10: Use stream API's to improve large message performance SonicMQ Recoverable File Channels Uses JMS layer to manage large file xfer Queue-based initiation of transfer High-speed JMS pipeline for blocks of data Recovery continues at point interrupted Sonic ESB open-source Large Message Service Provides dynamic provisioning Interacts with ESB processes SonicStream API (version 7.5 or later) Topic-based, pipeline into Java Stream api No recovery

70 © 2007 Progress Software Corporation 70 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Broker Tuning Parameters Core Resources: JVM heap size JVM thread, stack limits DRA, HTTP and Transaction threads TCP settings Message storage: Queue size and save threshold Pub/sub buffers Message log and store Message management Encryption Flow control and flow-to-disk Dead message queue management Connections Mapping to NICs Timeouts Load balancing Rule of Thumb: For non-trivial queues, multiply default settings by 10 to 100.

71 © 2007 Progress Software Corporation 71 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Java Tuning Options Fastest JVM depends a little on the application and a lot on the platform VM heap needs to be large enough to process load, but small enough to avoid system swapping Garbage Collection: default settings good for optimal throughput use advanced (jdk4 or later) GC options to optimize worst-case latency Rule of Thumb: On windows platforms, the Sun JVM is 10% to 50% slower than the default IBM JVM.

72 © 2007 Progress Software Corporation 72 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging ESB Tuning Options Load balancing and scalability of services: number of distributed service instances number of service listener threads Container Java VM heap size Intra-Container messaging Endpoint and connection parameters same principles as JMS client Expert Tip: Start with small Java heap and only increase -Xms size if it improves performance.

73 © 2007 Progress Software Corporation 73 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Discussion of Service tuning Transformations XML Server BPEL Server Database Service DXSI Service

74 © 2007 Progress Software Corporation 74 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Other fun things you can tune Database: indexing, query optimization SOA patterns: federated query, temporal aggregation, split/join, caching XML: DOM, SAX, XStream Disk: Device balancing, RAID, mount params Network: Nagle algorithm, timeouts Expert Tip: If you save service resources in sonicfs, the ESB container will dynamically load and cache them.

75 © 2007 Progress Software Corporation 75 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging SonicMQ Performance Tuning Guide Benchmarking Enterprise Messaging Systems white paper Sonic Test Harness User Guide Progress Professional Services Developer training courses Sonic Performance Analysis package Other Performance Engineering Resources No Magic Bullet, but plenty of places you can go for info:

76 © 2007 Progress Software Corporation 76 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Questions?

77 © 2007 Progress Software Corporation 77 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Thank you for your time

78 © 2007 Progress Software Corporation 78 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging


Download ppt "SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Analyzing, testing and tuning ESB/JMS performance David Hentchel Principal Solution Engineer."

Similar presentations


Ads by Google