Software Testing in the Cloud Leah Riungu-Kalliosaari

Objective and Background  Objective: To understand how organizations can successfully use the cloud for testing through empirical observations  Cloud computing: A model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, servers, storage, application and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction (NIST definition)  STaaS: A model for software testing whereby testing of an application is provided as a service across the internet.  Cloud-based testing is provided on-demand and billed on pay- per-use basis, so that the user pays only for the resources they have used

Work distribution during projects MASTO/ESPA - 3 interview rounds (Fall 2009, Spring 2010 & Fall 2010) - 4 publications (2 conference + 1 conference + 1 journal) - Researcher exchange (04-06/2011, Limerick, Ireland) - About half the dissertation work completed STX - About half of the dissertation work to be completed - 1 interview round (Fall 2011) - 1 Journal article; 1 book chapter in progress - Data analysis  formulating results of the thesis - More publications and thesis finalization

What has been done so far?  Conditions that influence software testing as an online service  Research Issues for software testing in the Cloud  The effect of testing in the cloud (most recent)

Effect of Testing in the Cloud (1)  On actual testing work  More efficient performance testing - e.g. Running 40 Amazon instances for generating about realistic user experiences  Quicker testing – e.g. Being able to test run a developed feature in say 10 different environments and getting (almost) immediate results  More realistic test results Riungu-Kalliosaari, L., Taipale, O., Smolander, K., ”Testing in the Cloud: Exploring the Practice”, Special Issue on Software Engineering for Cloud Computing, IEEE Software, March/April 2012

Effect of Testing in the Cloud (2)  On delivery and support of testing services  Better availability of testing tools and options – e.g. being able to carry out technical feasibility studies when choosing the tools to use.  Improved developer-tester communication – due to equal access to the system  Enhanced service delivery for vendors – quicker time to market and meeting of customer demands. Riungu-Kalliosaari, L., Taipale, O., Smolander, K., ”Testing in the Cloud: Exploring the Practice”, Special Issue on Software Engineering for Cloud Computing, IEEE Software, March/April 2012

Roadmap towards testing in the Cloud  Develop an understanding of cloud computing  Consider the benefits and risks  Carry out pilot projects  Safe way to explore potential benefits  Come up with elaborate strategies  Consider what to test; select a cloud provider  Enhance team interaction and prepare for complexities  Developer/tester interaction while working with cloud-based environments for development and testing  Enhance co-operation between research and industry  Aim at addressing cloud-related issues faced by the software industry Riungu-Kalliosaari, L., Taipale, O., Smolander, K., ”Testing in the Cloud: Exploring the Practice”, Special Issue on Software Engineering for Cloud Computing, IEEE Software, March/April 2012

Work in Progress (1)  18 interviews from 11 organizations (Autumn 2011)  Data analysis – an iterative process  Data from previous project also in use  Evaluation of quality attributes  Development of cloud-based testing strategy that can be used while considering adoption of cloud- based testing

WiP (2)… Quality Attributes  Cloud based testing improves testing of certain quality characteristics, for example, performance and scalability  Justification: Cloud testing offers, for example, computing resources in performance testing, human resources in crowd testing, resources for testing scalability. Further cloud testing reduces testing costs e.g. pay per use pricing model.  Cloud-based testing improves certain areas of testing, but it introduces also new issues e.g. security issues, and testing associated with huge data sets.

WiP (3)…Quality Attributes Peformance Cloud-based Testing Scalability Maintainability Elasticity improves controls supports is part of

Assess The need Consider/Evaluate The line of business Choose / Select Critical vs. non- critical applications Size Critical vs. non- critical applications Size Acquire/Re alize/ implement Manag e/Re- evaluat e What would you like to achieve? Security risks/threats Trade-offs What would you like to achieve? Security risks/threats Trade-offs The delivery approach The cloud provider SLAs, TOS, etc The delivery approach The cloud provider SLAs, TOS, etc The cloud service Pay attention to SLAs, TOS, Security The cloud service Pay attention to SLAs, TOS, Security Achieved vs. non- achieved needs, security, trade-offs Maintainability Change management e.g. skills development Achieved vs. non- achieved needs, security, trade-offs Maintainability Change management e.g. skills development WiP (4)…Cloud-based testing strategy

Pleriminary observations from most recent data (1)  There seems to be a correlation between quality and requirements management  Improved requirements management leads to improvement in quality  Global software development  Managing globally distributed teams efficiently  Change management is unique within each organization  Different types of changes  To introduce change, identify the weakest link

Pleriminary observations from most recent data (2)  Cloud awareness is rising  There is need for the benefits of cloud computing to be clearly understood and appreciated before cloud computing can be taken into use  4 out of 11 organizations are using the cloud  1 has a customer that is using the cloud  1 has a customer that is considering to use salesForce.com in future  Very small companies are more willing to use the cloud  No upfront costs  Getting up small functionality up and running quickly  In general, open source tools can be used if deemed necessary