1. Digital Performance Testing
Preeti Kambli, Manager
Venkata Goday, Director
Capgemini

2. Abstract
89% of companies will primarily compete on the basis of customer and Omni channel experience
CIOs are shifting their investment pattern in response to digital business, with average already spending 18% of their budget for digitalization and set to increase to 28% by end of 2018
Key focus areas for IT organizations - Digitization and customer experience.
Digital ecosystem involves enterprises, partners, customers and other stakeholders who experience the Information systems through multiple mediums and access points.
Comparisons are quickly drawn based on their interactions, hence its outmost important for IT organizations to provide highest user experience levels. The single source for validating the systems responsiveness is by thoroughly performance testing.
This paper talks about the customer user expectations and our solution for digital performance testing

3. IT Transformation silo
Introduction
Digital disruption in the today’s age is accelerating
Application performance is all about end user experience in the digital era, irrespective of the technology or complexity.
With the evolution of Mobiles, Tablets and handheld devices it has become imperative to stay ahead in terms of understanding user experience even before users actually experience it.
Business silo
behavioural analysis
social sentiment analysis
heat maps
voice of customer data
IT Transformation silo
system health
load time
MTTR, TTFB, SDLC
Latency and release cycles
Need to break the silo and have a more connected customer demand; end user expectation

4. Challenges in digital performance testing
Traditional performance testing practice is more server focused and doesn’t consider end user responsiveness
A complex mix of devices, operating systems, network, ever emerging technologies and most importantly – customer expectation makes it difficult to “fix” issues as soon as the customer experience them
Often end to end performance tests are bypassed owing to shorter release cycles & quicker time to market – ultimately resulting in unknown performance issues & bad user experience in Production
Infrastructure availability for load simulation remains challenge given the large Omni channel spread
Lack of standard approaches, frameworks and benchmarks for validation purposes

5. Factors affecting performance
Virtual users & monitoring

6. Client side application performance
Native mobile application
check on how much of the application and its data resides on the local device and communicate readily with a server application
Device’s own hardware and software configuration come into play
Mobile browser-based application performance is usually heavily dependent on network and server application performance
Server application needs to be able to recognize the device/browser combination in order to render properly

7. Server side performance
Examining the server performance is similar to measuring website or web app performance
Need to decompose the analysis into the components of the server that are providing the services including the database, application server, and associated hardware
Architecture has many variables that can result in numerous interaction between its components which could significantly impact performance

8. Network performance Network protocols impact throughput and delays
Test on different networks and in different countries
Carriers place overhead on
data transmission
network latency
Components to be measured
Traffic b/w Client (desktop, device) & Web/App
App Server & DB
Attributes
Bites Transferred
Connection Failures
Bandwidth Utilization
Transmission Errors

9. Tool & Techniques
No single tool can solve all parts of the problem statement for digital performance as most tools are specialized towards a specific platform and testing aspect
Many commercial and open source performance testing tools available for desktop browser based apps
Limited options for native mobile apps
Omni channel apps end to end performance testing involves multiple devices, OS versions, app versions, and different servers for native and web applications.
Functional mobile testing solutions simulate user actions and executing running on different devices. Test tool automation framework help in validating the functional aspect of the application on device.
How to integrate functional and performance tools to report on a real world end user digital experience?

10. APM tools for Omni channel performance monitoring
Real user monitoring (RUM) capabilities for Mobile and Browser capabilities providing business insights, user behavior analytics, root case analysis, error rates
User Journey
Page breakdown metrics
Mobile/PC/Browser
Number of Sessions
User response times
Network providers
Conversion/Bounce rates
Problem detection
Alerting
Tools – Dynatrace, New Relic, AppDynamics
Web page diagnostics tools
Identify and diagnose web page and URL performance issues; elements that cause sites to hang and slow down. These issues give users the perceived notion of SPA and website slowness and unresponsiveness.
Tools - Google Developers Chrome DevTools, Web Page Analyzer, HPE Network Virtualization (NV) Analytics

11. Performance tools for server side metrics
Measure key KPIs like response times, throughput, transaction/second, hits/second and provide a first glance of the server behaviour under load.
Tools - HPE LoadRunner, SOASTA, NeoLoad, Jmeter, BlazeMeter
Device side performance solution/tools
Improve the quality of your iOS, Android, and web applications by testing against browsers and real mobile devices hosted at the data center or on the cloud.
Optimize performance by simulating real world network conditions and monitoring device vitals consumption (CPU, memory, and battery)
Tools - HPE Mobile Center, Perfecto Mobile, Experitest SaaS digital assurance labs, Mobile Labs deviceConnect
Network performance tools
Capture and emulate real-world network conditions to detect and remediate issues before app deployment.
Analyze results and gain insight into the root cause of network performance bottlenecks
Tools - HPE Network Virtualization (HPE NV), OpManager

12. Case StudyDigital Performance Testing for a Leading Media & Entertainment client
Business drivers
Measure response time experienced by its end users through the recently launched digital delivery platform for marketing of entertainment and news to global audience
Define simulation strategy to include device type/configuration, operating system, user base across geography, bandwidth/network and latency
Track real user and synthetic test interactions to benchmark performance from different geographies
Test environment included setup of dedicated Mobile lab with multiple device types and configurations to simulate browser and native app real time simulations
Business Transaction Management

13. Unified solution for server side, device side, APM and functional test
Device side Perf Monitoring Mobile Labs deviceConnect
Network performance
HPE NV
Server side Performance HP LoadRunner
Mobile Apps Performance Test Results + Devise stats
Data Center - Application Under Test
Omni channel Emulated Devices / Operating Sys
Test
Console M/C (Controller)
Hybrid Mobile Requests Generating Concurrent User Load
Network Emulation – Varied Bandwidth
Automated performance + functional Script
Monitoring Engine 3G 4G
Wifi
Application Scenario Recording & Scripting
Application Profiling Metrics & Reports
Analysis
Load Generators
Application, Middle Tier and Database Profiling Tools
Tool Captures Application Performance Metrics
Performance Test Reports &Tuning Recommendations
APM for Monitoring / Diagnostics Appdynamics
Performance tuning was carried out for the top time consuming interactions
Identified “OutOfMemory” exception due to continuous usage of large object heap memory
Identified repetitive calls for same elements/images on same page & were optimized

14. Summary Need to understand the customer demands and emotions
Focus on simulating a real world scenario for performance
Assess the various simulation tools and choose the right tool
Cohesive digital ecosystem combining the perfect blend of server side, device side, network, APM with a pinch of functional testing touchpoint is equally important.
Adoption of functional and device side scenarios will play a critical role in making business decision and having an accurate end user experience measured out of your digital performance testing.

15. References & Appendix
Abbreviation
Full form IT
Information Technology
CIO
Chief Information Officer
API
Application Programming Interface UI
User Interface
SDLC
Software development Life Cycle
MTTR
Mean Time To Repair
TTFB
Time To First Byte
SPA
Single Page Application
HTML
Hypertext Markup Language
APM
Application Performance Management
HPE
Hewlett Packard Enterprise (now Microfocus)
UFT
Unified Functional Testing
CPU
Central Processing Unit
CoE
Center of Excellence

16. Author Biography Preeti Kambli Venkata Goday
Performance Testing & Engineering CoE Lead for Capgemini Mumbai
15 year experience in business development support; architecting solutions for clients, competency building for various performance tools and enabling delivery projects in crisis situation
Ideated and managed the development of new accelerators for Performance testing and drive the asset industrialization campaign.
Venkata Goday
Performance Testing & Engineering CoE Head for Capgemini global
Leading this practice from the last decade
well known thought leader in performance engineering
Rich experience in setting up Performance CoE’s for customers and has partnered with several clients for strategic consulting and transformation initiatives
Architected and developed many assets in Non-Functional testing space.

17. Thank You!!!