Presentation on theme: "Model-driven Performance Analysis Methodology for Scalable Performance Analysis of Distributed Systems Aniruddha Gokhale"— Presentation transcript:
Model-driven Performance Analysis Methodology for Scalable Performance Analysis of Distributed Systems Aniruddha Gokhale firstname.lastname@example.org email@example.com Asst. Professor of EECS, Vanderbilt University, Nashville, TN Swapna Gokhale firstname.lastname@example.org Asst. Professor of CSE, University of Connecticut, Storrs, CT email@example.com Jeff Gray firstname.lastname@example.org Asst. Professor of CIS Univ. of Alabama at Birmingham Birmingham, AL Presented at NSF NGS/CSR PI Meeting Rhodes, Greece, April 25-26, 2006 CSR CNS-0406376, CNS-0509271, CNS-0509296, CNS-0509342
3 Trends in DPSS Development Historically developed using low- level APIs Increasing use of middleware technologies Standards-based COTS middleware helps to: – Control end-to-end resources & QoS – Leverage hardware & software technology advances – Evolve to new environments & requirements Middleware helps capture & codify commonalities across applications in different domains by providing reusable & configurable patterns-based building blocks Examples: CORBA,.Net, J2EE, ICE, MQSeries Developers must decide at design- time which blocks to use to obtain desired functionality and performance
4 The “What If” Performance Analysis Process Model building (basic characteristics) Model validation (simulation/measurements) Model decomposition Model replicationCode generation Generalization of the model
5 Guiding Example: The Reactor Pattern The Reactor architectural pattern allows event-driven applications to demultiplex & dispatch service requests that are delivered to an application from one or more clients. Many networked applications are developed as event-driven programs Common sources of events in these applications include activity on an IPC stream for I/O operations, POSIX signals, Windows handle signaling, & timer expirations Reactor pattern decouples the detection, demultiplexing, & dispatching of events from the handling of events Participants include the Reactor, Event handle, Event demultiplexer, abstract and concrete event handlers Handle owns dispatches * notifies * * handle set Reactor handle_events() register_handler() remove_handler() Event Handler handle_event () get_handle() Concrete Event Handler A handle_event () get_handle() Concrete Event Handler B handle_event () get_handle() Synchronous Event Demuxer select () >
6 Reactor Dynamics Registration Phase Event handlers register themselves with the Reactor for an event type (e.g., input, output, timeout, exception event types) Reactor returns a handle it maintains, which it uses to associate an event type with the registered handler Snapshot Phase Main program delegates thread of control to Reactor, which in turn takes a snapshot of the system to determine which events are enabled in that snapshot For each enabled event, the corresponding event handler is invoked, which services the event When all events in a snapshot are handled, the Reactor proceeds to the next snapshot
7 Characteristics of Base Reactor Single-threaded, select-based Reactor implementation Reactor accepts two types of input events, with one event handler registered for each event type with the Reactor Each event type has a separate queue to hold the incoming events. Buffer capacity for events of type one is and of type two is . Event arrivals are Poisson for type one and type two events with rates and . Event service time is exponential for type one and type two events with rates and 2. In a snapshot, events are serviced non- deterministically (in no particular order). -- Base model of the prioritized reactor presented in NGS 2005. -- Non-deterministic handling makes it interesting/complicated.
8 Performance Metrics Throughput: -Number of events that can be processed -Applications such as telecommunications call processing. Queue length: -Queuing for the event handler queues. -Appropriate scheduling policies for applications with real-time requirements. Total number of events: -Total number of events in the system. -Scheduling decisions. -Resource provisioning required to sustain system demands. Probability of event loss: -Events discarded due to lack of buffer space. -Safety-critical systems. -Levels of resource provisioning. Response time: -Time taken to service the incoming event. - Important from user’s perspective, real-time systems.
9 SRN Model of the Base Reactor Stochastic Reward Nets (SRNs) – Extension of PNs/GSPNs. Part A: Models arrivals, queuing, and service of events. Transitions A1 and A2: Event arrivals. Places B1 and B2: Buffer/queues. Places S1 and S2: Service of the events. Transitions Sr1 and Sr2: Service completions. Inhibitor arcs: Place B1and transition A1 with multiplicity N1 (B2, A2, N2) - Prevents firing of transition A1 when there are N1 tokens in place B1. N1 A1 B1 Sn1 S1 Sr1 N2 A2 B2 Sn2 S2 Sr2 (a) StSnpSht T_StSnp1 T_StSnp2 T_ProcSnp2 T_ProcSnp1 T_EnSnp1 T_EnSnp2 SnpInProg1 SnpInProg2 (b)
10 SRN Model of the Base Reactor Part B: Process of taking successive snapshots Non-deterministic service of events. T_StSnp(i) enabled: Token in StSnpSht && Tokens in B(i) & no Token in S(i). T_EnSnp(i) enabled: No token in S(i). T_ProcSnp(i) enabled: Token in place S(i) and no token in other S(i)s. N1 A1 B1 Sn1 S1 Sr1 N2 A2 B2 Sn2 S2 Sr2 (a) StSnpSht T_StSnp1 T_StSnp2 T_ProcSnp2 T_ProcSnp1 T_EnSnp1 T_EnSnp2 SnpInProg1 SnpInProg2 (b)
11 SRN Model of the Base Reactor N1 A1 B1 Sn1 S1 Sr1 N2 A2 B2 Sn2 S2 Sr2 (a) StSnpSht T_StSnp1 T_StSnp2 T_ProcSnp2 T_ProcSnp1 T_EnSnp1 T_EnSnp2 SnpInProg1 SnpInProg2 (b) Reward rates: Loss probability (Pr(#B(i) == N(i))) Total number of events ((#B(i) + #S(i))) Throughput (rate(“Sr(i)”)) Queue length ((#B(i)) Optimistic and pessimistic bounds on the response times using the tagged-customer approach.
12 SRN Model of the Base Reactor Validate the performance measures Simulation implemented in CSIM Arrival rates, 1 = 2 = 0.4/sec Service rates, 1 = 2 = 2.0/sec Measure Buffer Space N 1 = N 2 = 1N 1 = N 2 = 5 SRNCSIMSRNCSIM T1T1 0.37/sec.0.365/sec.0.399/sec.0.395/sec. Q1Q1 0.0640.05960.120.115 L1L1 0.0640.0024 R 1,l 0.63 sec. 0.676 sec. (avg.) 0.79 sec. 0.830 sec. (avg.) R 1,u 0.86 sec.1.08 sec. Average response time estimate obtained from simulation lies between pessimistic and optimistic response time estimates obtained from SRN.
13 SRN Model of the Base Reactor Response time of event #1 Response time of event #2 Vary the arrival rate of type #1 events. -- Remaining parameters same as before. Response time of type #1, #2 events: -- Approaches pessimistic response time as arrival rate becomes higher. -- Actual response time of type #1 events is higher than type #2 events. Loss probabilities of type #1, #2 events: -- Increases with arrival rate. Sensitivity analysis
14 SRN Model of the Generalized Reactor AmAm N2N2 A2A2 B2B2 Sn 2 S2S2 Sr 2 N1N1 A1A1 B1B1 Sn 1 S1S1 Sr 1 StSnpSht TEnSnp m TEnSnp 1 SnpInProg 1 SnpInProg m NmNm BmBm Sn m SmSm Sr m TStSnp 1 TStSnp m TEnSnp 2 SnpInProg 2 TStSnp 2 TProcSnp 1,2 TProcSnp 1,3 TProcSnp 1,m TProcSnp m,1 TProcSnp m,m-1 m event types handled by the reactor.
15 SRN Model of the Generalized Reactor State-space explosion CTMC for m events, buffer space = 1. Number of states given by: m=2, 17 states m=3, 97 states m=4, 517 states Dramatic superlinear growth, for larger buffer sizes. CTMC for m=2, buffer space = 1. Number of states given by:
16 Model Decomposition Tagged customer approach Tagged event of type #i, b i events in the queue of type #i. Other queues may have more or less events. Current snapshot may be in progress Incoming event will be served after b i +1 snapshots. In each b i +1 snapshots, other events may or may not be handled. Pessimistic case: each b i +1 snapshots handles every event type Service time of each of the b i +1 snapshots: #1 #2 #i #m bibi Tagged event
17 Model Decomposition (b i +2) snapshot, tagged event is serviced. Non-deterministic order of servicing enabled event handles: -- Optimistic case: Tagged type #i event is the first to be serviced. -- Pessimistic case: Tagged type #i event is the last to be serviced. For the pessimistic case, service time of (b i +2) snapshot is the same Arrival, service, queuing of event type #i can be approximately represented by M/G/1/K queuing model. Impact of other event types on a given event type is obtained by inflating the service time the event. Very large buffer space, loss probability is negligible, M/G/1 model. Provides pessimistic or worse case bounds on the performance estimates. Actual performance estimates (obtained from SRN) are expected to be lower/better than the ones provided by model decomposition. Kleinrock: “We are willing to accept the raw facts of life, which state that our models are not perfect pictures of the systems we wish to analyze so we should be willing to accept approximations and bounds in our problem solution….” vo1. 2, page 319
18 Model Decomposition Event type #i, M/G/1 queue Variance of the service time: Expected service time: Traffic intensity: Coefficient of variation: Expected number of events: Response time (Little’s law): Throughput: Closed-form solutions for worse-case bounds on the performance estimates
19 Model Decomposition Met. m = 2m = 3m = 4 SRNDec.SRNDec.SRNDec. ρiρi -0.40-0.60-0.80 TiTi 0.40 EiEi 0.340.470.500.700.820.98 RiRi 0.85s1.18s1.25s1.74s2.05s2.45s Verification of performance estimates obtained from model decomposition Parameters: i = 0.4/sec, i = 2.0/sec, N i = 15 Performance estimates obtained from SRN lower than model decomposition. Exact solution for m=4 took over 12 hours.
20 Addressing Middleware Variability Challenges Per-Block Configuration Variability – Incurred due to variations in implementations & configurations for a patterns-based building block – E.g., single threaded versus thread-pool based reactor implementation dimension that crosscuts the event demultiplexing strategy (e.g., select, poll, WaitForMultipleObjects Manual design time performance modeling and analysis is complex and tedious since middleware building blocks and their compositions incur variabilities that impact performance Compositional Variability – Incurred due to variations in the compositions of these building blocks – Need to address compatibility in the compositions and individual configurations – Dictated by needs of the domain – E.g., Leader-Follower makes no sense in a single threaded Reactor
21 Automated DPSS Design-time Analysis Build and validate performance models for invariant parts of middleware building blocks Weaving of variability concerns manifested in a building block into the performance models Compose and validate performance models of building blocks mirroring the anticipated software design of DPSS systems Estimate end-to-end performance of composed system Iterate until design meets performance requirements MIDDLEWARE Automated design-time performance analysis techniques to estimate the impact of variability in middleware-based DPSS systems
22 Composed System Automation Goals for “What if” Analysis Build and validate performance models for invariant parts of middleware building blocks Weaving of variability concerns manifested in a building block into the performance models Compose and validate performance models of building blocks mirroring the anticipated software design of DRE systems Estimate end-to-end performance of composed system Iterate until design meets performance requirements Automating design-time performance analysis techniques to estimate the impact of variability in middleware-based DPSS systems Invariant model of a pattern Refined model of a pattern variability weave Refined model of a pattern workload system
23 MDE Tool Developer (Metamodeler) Application Developers (Modelers) Technology Enabler: Generic Modeling Environment www.isis.vanderbilt.edu/Projects/gme/default.htm “Write Code That Writes Code That Writes Code!” Decorator GModel GMeta CORE Metamodel XML Paradigm Definition Storage Options … DB #n DB #1 XML … UML / OCL COM XML ODBC Constraint Manager Browser Translator(s) Add-On(s) GME Editor GME Architecture Goal: Correct-by-construction DPSS systems
24 Leveraging Our Existing Solution: CoSMIC CoSMIC tools e.g., PICML used to model application components Captures the data model of the OMG D&C specification Synthesis of static deployment plans for DPSS applications New capabilities being added for QoS provisioning (real-time, fault tolerance) CoSMIC can be downloaded at www.dre.vanderbilt.edu/cosmic
25 POSAML: Modeling DPSS Composition Process POSAML – GME-based modeling language for middleware composition Provides a structural composition model Captures variability in blocks Generative programming capabilities to synthesize different artifacts e.g., benchmarking, configuration, perf modeling.
26 SRNML: Modeling the “What if” Process SRNML – GME-based modeling language for modeling performance models of building blocks Provides behavioral models for building blocks Developed Reactor and Proactor models to date Need to apply to other patterns Demonstrate model compositions and model solving Need to address scalability challenges for models
27 Implemented as a GME plug-in to assist in the rapid adaptation and evolution of models by weaving crosscutting changes into models. ECL Interpreter ECL Parser Defines MetaModel Source Model M o d e l i n g A P I s Defines ECL Transformation Specifications Defines CopyAtom strategy CopyAtom M o d e l i n g A P I s Aspect Weaving Target Model Leveraging Our Solution: C-SAW Model Transformation & Replication Engine
29 Project Status and Work in Progress Ongoing Integration of SRNML (behavioral) and POSAML (structural) Incorporate SRNML & POSAML in CoSMIC and release the software in open source public domain Integrate with C-SAW scalability engine Performance analysis of different building blocks (patterns): – Non-deterministic reactor (all steps). – Prioritized reactor, Active Object, Proactor (Steps #1, #2: Basic model, Model validation) Compose DPSS systems and perform performance analysis (analytical and simulation) of composed systems Validate via automated empirical benchmarking Demonstrate on real systems
30 Selected Publications 1.U. Praphamontripong, S. Gokhale, A. Gokhale, and J. Gray, “Performance Analysis of an Asynchronous Web Server” Proc. of 30 th COMPSAC, To Appear. 2.J. Gray, Y. Lin, and J. Zhang, "Automating Change Evolution in Model-Driven Engineering," IEEE Computer (Special Issue on Model-Driven Engineering), vol. 39, no. 2, February 2006, pp. 51-58 3.Invited (Under Review): J. Gray, Y. Lin, J. Zhang, S. Nordstrom, A. Gokhale, S. Neema, and S. Gokhale, "Replicators: Transformations to Address Model Scalability," voted one of the Best Papers of MoDELS 2005 and invited for an extended submission to the Journal of Software and Systems Modeling. 4.S. Gokhale, A. Gokhale, and J. Gray, "Response Time Analysis of an Event Demultiplexing Pattern in Middleware for Network Services," IEEE GlobeCom, St. Louis, MO, December 2005. 5.Best Paper Award: J. Gray, Y. Lin, J. Zhang, S. Nordstrom, A. Gokhale, S. Neema, and S. Gokhale, "Replicators: Transformations to Address Model Scalability," Model Driven Engineering Languages and Systems (MoDELS) (formerly the UML series of conferences), Springer-Verlag LNCS 3713, Montego Bay, Jamaica, October 2005, pp. 295-308. - Voted one of the Best Papers of MoDELS 2005 and invited for an extended submission to the Journal of Software and Systems Modeling. 6.P. Vandal, U. Praphamontripong, S. Gokhale, A. Gokhale, and J. Gray, "Performance Analysis of the Reactor Pattern in Network Services," 5th International Workshop on Performance Modeling, Evaluation, and Optimization of Parallel and Distributed Systems (PMEO-PDS), held at IPDPS, Rhodes Island, Greece, April 2006. 7.A. Kogekar, D. Kaul, A. Gokhale, P. Vandal, U. Praphamontripong, S. Gokhale, J. Zhang, Y. Lin, J. Gray, "Model-driven Generative Techniques for Scalable Performability Analysis of Distributed Systems," Next Generation Software Workshop, held atIPDPS, Rhodes Island, Greece, April 2006. 8.S. Gokhale, A. Gokhale, and J. Gray, "A Model-Driven Performance Analysis Framework for Distributed Performance-Sensitive Software Systems," Next Generation Software Workshop, held at IPDPS, Denver, CO, April 2005.
31 Concluding Remarks DPSS development incurs significant challenges Model-driven design-time performance analysis is a promising approach Performance models of basic building blocks built Scalability demonstrated Generative programming is key for automation GME is available from www.isis.vanderbilt.edu/Projects/gme/default.htmwww.isis.vanderbilt.edu/Projects/gme/default.htm CoSMIC is available from www.dre.vanderbilt.edu/cosmic C-SAW is available from www.gray-area.org/Research/C-SAW Many hard R&D problems with model-driven engineering remain unresolved!! Model building (basic characteristics) Model validation (simulation/measurements) Generalization of the model Model decomposition Model replication Automatic generation