Digital Performance Testing Preeti Kambli, Manager Venkata Goday, Director Capgemini

Abstract 89% of companies will primarily compete on the basis of customer and Omni channel experience CIOs are shifting their investment pattern in response to digital business, with average already spending 18% of their budget for digitalization and set to increase to 28% by end of 2018 Key focus areas for IT organizations - Digitization and customer experience. Digital ecosystem involves enterprises, partners, customers and other stakeholders who experience the Information systems through multiple mediums and access points. Comparisons are quickly drawn based on their interactions, hence its outmost important for IT organizations to provide highest user experience levels. The single source for validating the systems responsiveness is by thoroughly performance testing. This paper talks about the customer user expectations and our solution for digital performance testing

IT Transformation silo Introduction Digital disruption in the today’s age is accelerating Application performance is all about end user experience in the digital era, irrespective of the technology or complexity. With the evolution of Mobiles, Tablets and handheld devices it has become imperative to stay ahead in terms of understanding user experience even before users actually experience it. Business silo behavioural analysis social sentiment analysis heat maps voice of customer data IT Transformation silo system health load time MTTR, TTFB, SDLC Latency and release cycles Need to break the silo and have a more connected customer demand; end user expectation

Challenges in digital performance testing Traditional performance testing practice is more server focused and doesn’t consider end user responsiveness A complex mix of devices, operating systems, network, ever emerging technologies and most importantly – customer expectation makes it difficult to “fix” issues as soon as the customer experience them Often end to end performance tests are bypassed owing to shorter release cycles & quicker time to market – ultimately resulting in unknown performance issues & bad user experience in Production Infrastructure availability for load simulation remains challenge given the large Omni channel spread Lack of standard approaches, frameworks and benchmarks for validation purposes

Factors affecting performance Virtual users & monitoring

Client side application performance Native mobile application check on how much of the application and its data resides on the local device and communicate readily with a server application Device’s own hardware and software configuration come into play Mobile browser-based application performance is usually heavily dependent on network and server application performance Server application needs to be able to recognize the device/browser combination in order to render properly

Server side performance Examining the server performance is similar to measuring website or web app performance Need to decompose the analysis into the components of the server that are providing the services including the database, application server, and associated hardware Architecture has many variables that can result in numerous interaction between its components which could significantly impact performance

Network performance Network protocols impact throughput and delays Test on different networks and in different countries Carriers place overhead on data transmission network latency Components to be measured Traffic b/w Client (desktop, device) & Web/App App Server & DB Attributes Bites Transferred Connection Failures Bandwidth Utilization Transmission Errors

Tool & Techniques No single tool can solve all parts of the problem statement for digital performance as most tools are specialized towards a specific platform and testing aspect Many commercial and open source performance testing tools available for desktop browser based apps Limited options for native mobile apps Omni channel apps end to end performance testing involves multiple devices, OS versions, app versions, and different servers for native and web applications. Functional mobile testing solutions simulate user actions and executing running on different devices. Test tool automation framework help in validating the functional aspect of the application on device. How to integrate functional and performance tools to report on a real world end user digital experience?

APM tools for Omni channel performance monitoring Real user monitoring (RUM) capabilities for Mobile and Browser capabilities providing business insights, user behavior analytics, root case analysis, error rates User Journey Page breakdown metrics Mobile/PC/Browser Number of Sessions User response times Network providers Conversion/Bounce rates Problem detection Alerting Tools – Dynatrace, New Relic, AppDynamics Web page diagnostics tools Identify and diagnose web page and URL performance issues; elements that cause sites to hang and slow down. These issues give users the perceived notion of SPA and website slowness and unresponsiveness. Tools - Google Developers Chrome DevTools, Web Page Analyzer, HPE Network Virtualization (NV) Analytics

Performance tools for server side metrics Measure key KPIs like response times, throughput, transaction/second, hits/second and provide a first glance of the server behaviour under load. Tools - HPE LoadRunner, SOASTA, NeoLoad, Jmeter, BlazeMeter   Device side performance solution/tools Improve the quality of your iOS, Android, and web applications by testing against browsers and real mobile devices hosted at the data center or on the cloud. Optimize performance by simulating real world network conditions and monitoring device vitals consumption (CPU, memory, and battery) Tools - HPE Mobile Center, Perfecto Mobile, Experitest SaaS digital assurance labs, Mobile Labs deviceConnect Network performance tools Capture and emulate real-world network conditions to detect and remediate issues before app deployment. Analyze results and gain insight into the root cause of network performance bottlenecks Tools - HPE Network Virtualization (HPE NV), OpManager

Case StudyDigital Performance Testing for a Leading Media & Entertainment client Business drivers Measure response time experienced by its end users through the recently launched digital delivery platform for marketing of entertainment and news to global audience Define simulation strategy to include device type/configuration, operating system, user base across geography, bandwidth/network and latency Track real user and synthetic test interactions to benchmark performance from different geographies Test environment included setup of dedicated Mobile lab with multiple device types and configurations to simulate browser and native app real time simulations Business Transaction Management

Unified solution for server side, device side, APM and functional test Device side Perf Monitoring Mobile Labs deviceConnect Network performance HPE NV Server side Performance HP LoadRunner Mobile Apps Performance Test Results + Devise stats Data Center - Application Under Test Omni channel Emulated Devices / Operating Sys Test Console M/C (Controller) Hybrid Mobile Requests Generating Concurrent User Load Network Emulation – Varied Bandwidth Automated performance + functional Script Monitoring Engine 3G 4G Wifi Application Scenario Recording & Scripting Application Profiling Metrics & Reports Analysis Load Generators Application, Middle Tier and Database Profiling Tools Tool Captures Application Performance Metrics Performance Test Reports &Tuning Recommendations APM for Monitoring / Diagnostics Appdynamics Performance tuning was carried out for the top time consuming interactions Identified “OutOfMemory” exception due to continuous usage of large object heap memory Identified repetitive calls for same elements/images on same page & were optimized

Summary Need to understand the customer demands and emotions Focus on simulating a real world scenario for performance Assess the various simulation tools and choose the right tool Cohesive digital ecosystem combining the perfect blend of server side, device side, network, APM with a pinch of functional testing touchpoint is equally important. Adoption of functional and device side scenarios will play a critical role in making business decision and having an accurate end user experience measured out of your digital performance testing.

References & Appendix Abbreviation Full form IT Information Technology CIO Chief Information Officer API Application Programming Interface UI User Interface SDLC Software development Life Cycle MTTR Mean Time To Repair TTFB Time To First Byte SPA Single Page Application HTML Hypertext Markup Language APM Application Performance Management HPE Hewlett Packard Enterprise (now Microfocus) UFT Unified Functional Testing CPU Central Processing Unit CoE Center of Excellence https://www.slideshare.net/RyanBateman4/what-is-digital-performance-management http://www.methodsandtools.com/archive/mobiletest1.html https://experitest.com/mobile-cloud-testing/seetestcloud-online/?am_force_theme_layout=desktop https://engineering.linkedin.com/blog/2017/02/measuring-and-optimizing-performance-of-single-page-applications https://en.wikipedia.org/wiki/Single-page_application http://www.techrepublic.com/blog/web-designer/free-diagnostic-tools-for-website-response-and-performance-issues/

Author Biography Preeti Kambli Venkata Goday Performance Testing & Engineering CoE Lead for Capgemini Mumbai 15 year experience in business development support; architecting solutions for clients, competency building for various performance tools and enabling delivery projects in crisis situation Ideated and managed the development of new accelerators for Performance testing and drive the asset industrialization campaign. Venkata Goday Performance Testing & Engineering CoE Head for Capgemini global Leading this practice from the last decade well known thought leader in performance engineering Rich experience in setting up Performance CoE’s for customers and has partnered with several clients for strategic consulting and transformation initiatives Architected and developed many assets in Non-Functional testing space.

Thank You!!!