Presentation is loading. Please wait.

Presentation is loading. Please wait.

Designing a Workload Scenario for Benchmarking Message-Oriented Middleware Kai Sachs*, Samuel Kounev*, Marc Carter, Alejandro Buchmann* * Databases and.

Similar presentations


Presentation on theme: "Designing a Workload Scenario for Benchmarking Message-Oriented Middleware Kai Sachs*, Samuel Kounev*, Marc Carter, Alejandro Buchmann* * Databases and."— Presentation transcript:

1 Designing a Workload Scenario for Benchmarking Message-Oriented Middleware Kai Sachs*, Samuel Kounev*, Marc Carter, Alejandro Buchmann* * Databases and Distributed Systems Group, TU Darmstadt / Germany Computer Laboratory, University of Cambridge / UK Computer Laboratory, University of Cambridge / UK IBM Hursley Labs, Hursley Park, Winchester / UK IBM Hursley Labs, Hursley Park, Winchester / UK

2 2 Overview I. Introduction II. Workload Requirements and goals of the SPECjms benchmark III. Application Scenario for SPECjms IV. Implementation Details V. Summary

3 3 Message Oriented Middleware (MOM) Used in many business domains Financial services and enterprise applications Health care Supply chain... And in many technologies Enterprise Service Bus (ESB) Service Oriented Architecture (SOA) Enterprise Application Integration (EAI)... Increasing importance Need for benchmark

4 4 Requirements of a benchmark Scenario representative of real-world applications. Exercise all critical services provided by platforms. Not optimized for a specific product. Reproducible results. No inherent scalability limitations.

5 5 Current State of MOM Benchmarking Many proprietary benchmarks for MOM servers Used for performance testing and product comparisons However: These benchmarks do not meet all of the defined requirements Typically they... concentrate on stressing individual MOM features, and do not provide a comprehensive and representative workload for evaluating the overall MOM performance Currently no industry-standard benchmark for MOM Benchmarking SPECjms 2007

6 6 What is SPECjms 2007? Worlds first industry standard benchmark for MOM products supporting Java Message Service (JMS) Developed by the SPEC OSG-Java subcommittee with the participation of: IBM TU Darmstadt Sun Sybase BEA Apache Oracle JBoss

7 7 Goals of SPECjms 2007 I.Provide a standard workload and metrics for measuring and evaluating JMS-based platforms II.Provide a flexible framework for JMS performance analysis

8 8 Overview I. Introduction II. Workload Requirements and goals of the SPECjms benchmark III. Application Scenario for SPECjms IV. Implementation Details V. Summary

9 9 Categories of Workload Requirements Representativeness Comprehensiveness Focus Scalability Configurability

10 10 Categories of Workload Requirements Representativeness Comprehensiveness Focus Scalability Configurability

11 11 Representativeness The goal: Allow users to relate the observed behavior to their own applications and environments. Should simulate the way platform services are exercised in real-life systems. Therefore: It should be based on a representative workload scenario: Communication style and the types of messages should represent a typical transaction mix.

12 12 Comprehensiveness I The goal: Exercise all platform features typically used in the major classes of JMS applications. Point-to-point and publish/subscribe messaging domains should be covered. The features and services stressed should be weighted

13 13 Comprehensiveness II Therefore: A workload transaction mix has to consider many different dimensions: Transactional vs. non-transactional messages. Persistent vs. non-persistent messages. Usage of different message types, e.g. TextMessages, ObjectMessages,... Usage of messages of different sizes (small, medium, large). Publish/subscribe vs. point-to-point messages (topics vs. queues)....

14 14 Focus Focus of the workload / benchmark: performance and scalability of the JMS server's software and hardware components. Therefore: Minimize the impact of other components and services. Especially JMS servers are typically less loaded than e.g. database or application servers. Example: Is a database was used to store business data and manage the application state it could easily become the bottleneck Minimize the impact of client-side operations (e.g. XML parsing).

15 15 Scalability Dimensions of scaling the workload : Horizontal scaling: De/Increase the number of destinations (queues and topics) Keep the traffic per destination constant Vertical scaling: De/Increase traffic per destination Keep the number of destinations fixed Preserve real-life relationships in modeled scenario Additionally: Support for freeform scaling, e.g. user defined traffic per destination and number of destinations

16 16 Configurability I Provide a flexible performance analysis tool: Allows users to configure and customize the workload, e.g. for research purposes Produce and publish standard results for marketing purposes Therefore: Need for a framework which supports tuning, analyzing and optimizing performance of certain features / platforms

17 17 Configurability II A benchmark framework should allow: precise configuration of workload and transaction mix to switch off business interactions (implies that interactions should be decoupled) Providing such a configurability is a great challenge: Freeform mode: Design and implement interactions so that they can be run in different combinations depending on the desired transaction mix Standard mode: It has to be ensured, that the interactions always behave like defined in the application scenario

18 18 Overview I. Introduction II. Workload Requirements and goals of the SPECjms benchmark III. Application Scenario for SPECjms IV. Implementation Framework V. Summary

19 19 The Application Scenario Represents a supply chain of a supermarket company. Participants: Headquarters (HQ) Supermarkets (SM) Distribution Centers (DC) Suppliers (SP). Based on the previously discussed requirements.

20 20 The Application Scenario Why again a Supply Chain Scenario? Excellent basis for defining different interactions: Many destinations, use cases,... Typical real word application Importance of performance (RFID!) Allows scaling the workload in a natural way: Horizontal: e.g. scale the number of SMs Vertical: e.g. scale amount of products sold per SM

21 21 Participants Company HQ Super- markets Distribution Centers =goods and information flow = only information flow Supplier

22 22 Participants - Supermarkets Company HQ Super- markets Distribution Centers =goods and information flow = only information flow Supplier Supermarket (SM) sells goods to end customers. manages its inventory. every supermarket offers different products. every supermarket is supplied by exactly one of the distribution centers.

23 23 Participants - Distribution Center Company HQ Super- markets Distribution Centers =goods and information flow = only information flow Supplier Distribution Center (DC) supplies the supermarket stores which sell goods to end customers. responsible for a set of stores in a given area. is supplied by external suppliers.

24 24 Participants - Suppliers Company HQ Super- markets Distribution Centers =goods and information flow = only information flow Supplier Supplier (SP) deliver goods to distribution centers (based on an offer of the supplier). not every supplier offers the same products. offers either all products of a given product family or none of them.

25 25 Participants - HQ Company HQ Super- markets Distribution Centers =goods and information flow = only information flow Supplier Company HQ manages the accounting of the company. manages information about the goods and products. manages selling prices. monitors the flow of goods and money in the supply chain.

26 26 Interactions The following interactions are part of the scenario: 1.Order / Shipment Handling (SM / DC) 2.(Purchase) Order / Shipment Handling (DC / SP) 3.Price Updates 4.Inventory Management 5.Sales Statistics Collection 6.Product Announcements 7.Credit Card Hotlists

27 27 Interactions The following interactions are part of the scenario: 1.Order / Shipment Handling (SM / DC) 2.(Purchase) Order / Shipment Handling (DC / SP) 3.Price Updates 4.Inventory Management 5.Sales Statistics Collection 6.Product Announcements 7.Credit Card Hotlists

28 28 Interaction 1 Order / Shipment Handling (SM & DC) Includes two major steps: SM sends an order to DC DC ships goods to SM Point-to-Point based communication / Intra-Company.

29 29 Interaction 1 - Order / Shipment Handling Company HQ Super- markets SuppliersSupermarket Company Distribution Centers =goods & info. flow = only info. flow 1 1. A SM sends an order to its DC.

30 30 Interaction 1 - Order / Shipment Handling Company HQ Super- markets SuppliersSupermarket Company Distribution Centers =goods & info. flow =only info. flow The DC sends a confirmation to the SM and dispatches shipment.

31 31 Interaction 1 - Order / Shipment Handling Company HQ Super- markets SuppliersSupermarket Company Distribution Centers =goods & info. flow = only info. flow The DC sends a (non- transactional, non- persistent) message to the HQ (transaction statistics).

32 32 Interaction 1 - Order / Shipment Handling Company HQ Super- markets SuppliersSupermarket Company Distribution Centers =goods & info. flow = only info. flow The shipment arrives at the SM and confirmation is sent to the DC.

33 33 Example: Interaction 2 Purchase Order / Shipment Handling (DC & SPs) Point-to-Point and Publish/Subscribe communication. Inter company communication. Includes six steps:

34 34 Interaction 2 Purchase Order / Shipment Handling Company HQ Super- markets SuppliersSupermarket Company Distribution Centers =goods and info flow = only info flow DC sends a call for offers.

35 35 Interaction 2 Purchase Order / Shipment Handling Company HQ Super- markets SuppliersSupermarket Company Distribution Centers =goods and info flow = only info flow All SPs offering the requested products send an offer.

36 36 Interaction 2 Purchase Order / Shipment Handling Company HQ Super- markets SuppliersSupermarket Company Distribution Centers =goods and info flow = only info flow 3 3. Based on the offers, the DC selects a SP and sends a purchase order to it.

37 37 Interaction 2 Purchase Order / Shipment Handling Company HQ Super- markets SuppliersSupermarket Company Distribution Centers =goods and info flow = only info flow a SP sends an order confirmation to the DC 4.b SP sends an invoice to the HQ 4.c The SP dispatches a shipment to the DC.

38 38 Interaction 2 Purchase Order / Shipment Handling Company HQ Super- markets SuppliersSupermarket Company Distribution Centers =goods and info flow = only info flow 5 5. The shipment arrives at the DC and confirmation is sent to the SP.

39 39 Interaction 2 Purchase Order / Shipment Handling Company HQ Super- markets SuppliersSupermarket Company Distribution Centers =goods and info flow = only info flow 6 6. The DC sends a message to the HQ (transaction statistics).

40 40 Overview I. Introduction II. Workload Requirements and goals of the SPECjms benchmark III. Application Scenario for SPECjms IV. Implementation Details V. Summary

41 41 Message Types and Destinations Message types All messages types supported by the JMS Specification excepted ByteMessages 19 different messages are defined: Three different sizes per message (small, medium, large) with a certain probability Acknowledgment mode: Standard: AUTO_ACKNOWLEDGMENT (can be changed in several interactions) (Non-)Persistent, (Non-)Transactional, Durable,...

42 42 Message Types and Destinations Number of queues per location instance: Number of topics: 3 + one for every product family LocationNo. of queues SM3 SP2 HQ4 DC6

43 43 Event Handler Every Location has a set of different Event Handler (Message Consumer) for different incoming messages. Example: A DC has an Event Handler for: Incoming Orders of the SMs Shipping Confirmations Incoming Offers,.... Event Handlers... contain the business logic of the scenario. are stateless from a business point of view.

44 44 Driver Framework Many locations represented by many event handlers (message consumers) Event handlers may be distributed across many physical machines. Reusable driver framework addresses this issues without any inherent scalability limitations. Plain Java Maximum choice in laying out workload to achieve maximum performance.

45 45 Driver Framework

46 46 A Flexible Framework for Performance Analysis Allows to configure and customize the workload / transaction mixes Provides three different topologies corresponding to three different modes in which the benchmark can be run: Vertical Horizontal Freeform Many features

47 47 A Flexible Framework for Performance Analysis Some features: Number of physical locations (HQ, SM, DC, SP) emulated. Number of agents representing a single physical location. Number of event handlers in an agent of each type. Number of driver instances for each interaction. Total number of invocations of each interaction (as an alternative to specifying a rate). Message size distributions for each interaction. The driver nodes on which agents are run. Number of JVMs run on each node and the way agents are distributed among them. Number of javax.jms.Connection objects shared amongst event handler classes within a single agent.....

48 48 Overview I. Introduction II. Workload Requirements and goals of the SPECjms benchmark III. Application Scenario for SPECjms IV. Implementation Details V. Summary

49 49 Summary The presented scenario models a set of interactions in the supply chain of a supermarket company. These interactions are used as a basis in SPEC's new SPECjms benchmark. SPECjms will be the world's first industry-standard benchmark for MOM products. SPECjms can be used to stress and evaluate the different aspects of JMS performance. SPECjms is more than a benchmark: Offers also a performance analysis tool for JMS-based infrastructures.

50 50 Summary II SPECjms provides a complete level playing field for performance comparisons of competitive MOM products. Producing and publishing standard results (e.g. for marketing purposes) to tune and optimize a platform or to analyze the performance of certain specific features. It provides a flexible performance analysis framework which allows the user to precisely configure the workload and transaction mix to be generated.

51 51 SPEC(Standard Performance Evaluation Corporation): Goal: JMS Benchmark based on real world application. SPECjms Benchmark Company HQ Super- markets Distribution Centers =goods and information flow = only information flow

52 52 SPECjms Scenario Supply chain of a supermarket company Four roles: Supplier, Headquarter, Distribution Center, Supermarktes Selected 7 typical interactions: (Purchase) Order Management Price Updates Product Announcements Inventory Movements Statistical Data Workload verified by multiple vendors Company HQ Super- markets Distribution Centers =goods and information flow = information flow

53 53 Thanks for your attention


Download ppt "Designing a Workload Scenario for Benchmarking Message-Oriented Middleware Kai Sachs*, Samuel Kounev*, Marc Carter, Alejandro Buchmann* * Databases and."

Similar presentations


Ads by Google