1. DEEPAK BHIMARAJU; EDWARD ALLEN TEST CHALLENGES IN THE CLOUD

2. Speakers Edward Allen – 18 years QA experience – QA Director - Oracle Service Cloud – Co-author of a chapter in the book “Experiences of Test Automation”

3. Speakers Deepak Bhimaraju – Worked as a QA Engineer at Oracle India – Recently moved to a development role

4. The Cloud – Our Partly Sunny Version Multi-tenant Multi-version Enterprise

5. The Cloud – Our Partly Sunny Version Email Chat Collaboration Case Management Software as a Service Custom Data Application Development Middleware Managed Frameworks Platform as a Service Coming Soon? Infrastructure as a Service ConsumeBuildHost

6. On Premise Deployment: Benefits Known hardware Tried and Tested Configurations Physical proximity Image Credits: SilverStartalk, http://en.wikipedia.org/wiki/Computer_network_diagram#/media/File:Sample-network-diagram.png

7. On Premise Deployment: Benefits Legacy software providing stability over time Test approach or process is a known quantity – Primary test focus is functional – Compatibility, Capacity, Scalability, Security testing Image Credits: SilverStartalk, http://en.wikipedia.org/wiki/Computer_network_diagram#/media/File:Sample-network-diagram.png

8. On Premise Deployment: Challenges Scalability Maintenance Setup Costs Latency* Image Credits: Erik Pitti, https://www.flickr.com/photos/epitti/2370848133/

9. Promises of The Cloud Scalable Maintainable Low Latency* Secure Image Credits: Christi Gain, https://www.flickr.com/photos/christigain/5636888777/

10. Challenges of The Cloud Reliance on the Cloud Operations team Pinning responsibility can take time (environment or code or both) Finding bugs early as fixing takes more time.

11. Test Challenges: Multi-Tenant Saas Security Transaction Volumes – Shared services – Spillover from other tenants Scale and Capacity – Tenant growth – Data volume per tenant

12. Test Challenges: Multi-Version SaaS Rapid Scheduled Deliveries – ZERO down-time – No customer patience for break-fixes or post- upgrade work (promise of the Cloud) Customers choose when to uptake features Single-version Shared Services

13. Test Challenges: PaaS Managed Frameworks App Development Custom DataReporting Workflow and Business Logic

14. Test Challenges: IaaS Functional Testing is minimal Must be reliable, secure, compatible Image Credits: Torkild Retvedt, https://www.flickr.com/photos/torkildr/3462607995/

15. Test Challenges: Architecture Reference: Li, et al. (2011). Characteristics of multiple component defects and architectural hotspots: A large system case study. Empirical Software Engineering, 16 (5), 667‐702. Multiple Component Defects (MCDs): Architectural defects which typically span more than one component encompassing multiple, interacting components. Cost of a single MCD can be amplified by the number of tenants.

16. Cost of MCDs is as much as 20 times that of a Functional Defect Test Challenges: Architecture 92% 8% 52% 48% Regular Defect MCD Reference: Li, et al. (2011). Characteristics of multiple component defects and architectural hotspots: A large system case study. Empirical Software Engineering, 16 (5), 667‐702. % of Total App Defects % of Total Repair Efforts

17. Solutions: General Promote and Value Testing as a Profession Greater decision making authority – Technical Level – Product Level Popularize in Universities – Find new test perspectives and approaches – Finding the weakest links in the cloud

18. Solutions: General Utilize the “inter-regnum” Work for other teams on the cloud Create a common Knowledge Base Offer incentives to move developers to QA Equal Pay

19. Solutions: Functional Testing Rapid Release – Automate quickly and in parallel – Manual scripted tests minimized – Exploratory testing

20. Solutions: Cloud Testing Gap Functional Analysis (Integration and System Test) Structural Analysis (Reliability, Security, Scalability, Capacity, Compatibility) Static Analysis Unit Test Tester Developer Attribution to Dr. Bill Curtis SVP and Chief Scientist, CAST Research Labs

21. Solutions: Structural Testing Testing cannot be “after the fact” Much of the work isn’t submitting bug reports Testers have greater product expertise than developers

22. Solutions: Structural Testing Testers lead on capacity, scalability, performance, security Finding best solution among many good options – Or least worse among many bad options – Continual, rapid, iterative development

23. Solutions: Structural Testing Need to be experts in – Customer usage – Availability – System administration – Cloud app configuration – Security – Scalability – Capacity – Performance – Networking – APIs

24. Solutions: Structural Testing Learn craft and discipline of teaching and leading – Testers must lead – Teach usage patterns, data volume modeling, exploits – Unique position to drive discussion Testers train developers – Prevent and Remove – Fewer defects created in the first place Secure coding App configuration Performance

25. Solutions: Structural Testing Expertise demanded of Cloud Testing is ratcheted up We must be up to the challenge – Cannot be satisfied to test boundary conditions and alpha-numeric fields – Cannot continue the exploratory/traditional manual testing debate

26. DEEPAK BHIMARAJU; EDWARD ALLEN TEST CHALLENGES IN THE CLOUD SPECIAL THANKS: BRAD PASCOE, Oracle Corp. PRASAD NAGARAJU, Oracle India NILOY GUPTA, CMU, Pittsburgh