A Framework for Modeling and Execution of Infrastructure Contention Experiments Carmelo Ragusa, Philip Robinson and Sergej Svorobej MERMAT 2013, FIA, 7 May, Dublin

©2013 SAP AG or an SAP affiliate company. All rights reserved.2 Agenda Introduction –Problem analysis of conducting experiments about resource sharing in multi-tenancy systems Proposed solution –COCOMA framework: objectives, design, benefits and stakeholders Use case –How resource contention affect users for different physical resources partitioning configurations Summary and future work

©2013 SAP AG or an SAP affiliate company. All rights reserved.3 Motivation Resource (memory, disk, network, cpu) contention occurs in shared multy- tenancy environments such as clouds Cloud Infrastructure SuT Unknown More detailed cloud experiments can be interested to investigate how the System under Test (SuT) performs under various contention patterns that may occur in practice. Problem: How can we study these issues? How can we emulate multi- tenancy behaviour? How can we reproduce the same experiments?

©2013 SAP AG or an SAP affiliate company. All rights reserved.4 Requirements for supporting rigorous software testing in contentious, multi-tenant environments Scalability Reproducibility Portability Extensibility Self-containment Controllability

©2013 SAP AG or an SAP affiliate company. All rights reserved.5 Current approaches to reproduce operational conditions such as contention Manual: use of a number of resource-specific tools and command-line operations in order to manipulate operational conditions. Client-Based: simulation of usage, using multithreaded client-side request generators in order to load target environment and raise required operational conditions. Ad-Hoc Scripting: use of custom-built scripts for higher-level coordination of load generation and test execution. Each one fails in same way to meet the previous listed requirements

©2013 SAP AG or an SAP affiliate company. All rights reserved.6 Proposed solution A framework for COntrolled COntentious and Malicious (COCOMA) patterns: deliberately make the platform misbehave – contention, faults and attacks. Cloud Infrastructure SuT Unknown COCOMA Experimenters will be able to: study their system under real world effects conditions control those conditions reproduce exact conditions correlate conditions and results of their system under test use those findings to discover weaknesses and tune/enhance their system

©2013 SAP AG or an SAP affiliate company. All rights reserved.7 Design principles Separation of concerns Provide unified and coherent interface for staging experiments Allowing advanced workload patterns via distribution algorithms Abstraction from low level tools Easy composition of complex patterns Easy extensibility Virtual Machine COCOMA Distribution Algorithms Stressapptest Lookbusy Iperf etc. Emulators Resource contention Experiment XML

©2013 SAP AG or an SAP affiliate company. All rights reserved.8 Concepts and terminology An emulation is composed of pairs of distribution-emulator An emulator in a distribution is bound to a specific resource type A distribution is a workload trend Distributions are broken down into multiple runs to create the desired trend Runs are single instantiation of low level tools For complex scenarios users can specify multiple distribution- emulator pairs EmulationEmulator DistributionResource-Type e.g. CPU, RAM, Net, I/O 1 * 1 * type - duration - granularity - parameters[] e.g. linear, trapezoidal, exp, trace Emulation time Distribution-1 (CPU) Distribution-2 (RAM) Distribution-3 (NET) Distribution-4 (I/O) t1 tn

©2013 SAP AG or an SAP affiliate company. All rights reserved.9 Use case: How resource contention affect users for different physical resources partitioning configurations Test environment: single machine with 4 CPU cores and 16 GB of RAM CPU (cores)RAM (GB range) L (Low) ML (Medium-Low) MH (Medium-High) H (High) COCOMASuTInactive VMFree resources ConfCPURAMNum VMsCPURAMCPURAMCPURAM 1MHH1LLn/a 2ML 1 n/a 3LL1MHHn/a 4MHH3LLn/a 5MHL3LLn/a 6LH1HL 7ML 1LLn/a LML 8 1LLL n/a Each configuration represents VMs with different resources assigned, providing a specific physical resources partitioning We abstract resources values with ranges from Low to High

©2013 SAP AG or an SAP affiliate company. All rights reserved.10 Validation use case sysbench CPU benchmark calculating the first 100K prime numbers memspeed RAM benchmark tiobench IO benchmark

©2013 SAP AG or an SAP affiliate company. All rights reserved.11 Stakeholders Performance testers/engineers Cloud Service Providers Cloud Application Administrators Application Developers and Testers Benchmarks and Standards Groups

©2013 SAP AG or an SAP affiliate company. All rights reserved.12 Summary Experimenters need means that recreate conditions in cloud-like multy-tenancy shared systems to enable testing their solutions Current approaches are limited as they fail to meet requirements for such environments We proposed a solution that abstracts from low level tools enables from simple to complex repeatable scenarios, reducing experimenters effort allows investigating the system behaviour and correlate it to the specific conditions created A simple validation use case was presented

©2013 SAP AG or an SAP affiliate company. All rights reserved.13 Future work Implementing malicious and faultiness modules, and relative distributions Real system workload traces parsing and replay capability Distributed COCOMA solution to enable large complex distributed scenarios Web UI to manage the framework Contention at CPU cache level Tool to record system traces in a COCOMA compatible format for reply purpose Extend emulation analyser for distributed version to get warnings for large scenarios' emulations

©2013 SAP AG or an SAP affiliate company. All rights reserved.14 Acknowledgements This work has being carried out within the BonFIRE project, which has received research funding from the ECs Seventh Framework Programs (EU ICT IP under the Information and Communication Technologies Program)

Thank you Contact information: Carmelo Ragusa SAP HANA Cloud Infrastructure, Belfast COCOMA is released as Open Source under Apache v2 license:

©2013 SAP AG or an SAP affiliate company. All rights reserved.16 © 2013 SAP AG or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Please see for additional trademark information and notices.

©2013 SAP AG or an SAP affiliate company. All rights reserved.17 © 2013 SAP AG oder ein SAP-Konzernunternehmen. Alle Rechte vorbehalten. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden. Einige der von der SAP AG und ihren Distributoren vermarkteten Softwareprodukte enthalten proprietäre Softwarekomponenten anderer Softwareanbieter. Produkte können länderspezifische Unterschiede aufweisen. Die vorliegenden Unterlagen werden von der SAP AG und ihren Konzernunternehmen (SAP-Konzern) bereitgestellt und dienen ausschließlich zu Informationszwecken. Der SAP-Konzern übernimmt keinerlei Haftung oder Gewährleistung für Fehler oder Unvollständigkeiten in dieser Publikation. Der SAP-Konzern steht lediglich für Produkte und Dienstleistungen nach der Maßgabe ein, die in der Vereinbarung über die jeweiligen Produkte und Dienstleistungen ausdrücklich geregelt ist. Keine der hierin enthaltenen Informationen ist als zusätzliche Garantie zu interpretieren. SAP und andere in diesem Dokument erwähnte Produkte und Dienstleistungen von SAP sowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und verschiedenen anderen Ländern weltweit. Weitere Hinweise und Informationen zum Markenrecht finden Sie unter

©2013 SAP AG or an SAP affiliate company. All rights reserved.18 Headline area Drawing area White space The Grid