1. Digital Engineering & Mobile Solutions Performance Engineering Transformation
Date: 03/11/2015
©2013 Walgreen Co. All rights reserved.

2. Agenda Strategy Solution set-up Diagnostics & Analysis
Key accomplishments summary
Opportunity areas ahead
Superman vs. Arthur production performance
©2013 Walgreen Co. All rights reserved.

3. Strategy for Front-end Performance
Iterative cycles of
Analysis
Implementation
Validation
Feedback
Use data and HAR files for analysis and further
Improvements.
Use Gomez UEM data for comparison with
Production
Analysis
Implementation
Validation
Feedback
Prioritize above-the-fold content.
Reduce number of assets and round trip time.
Defer non-critical assets post Onload event.
Optimize JavaScript execution and rendering
time.
Deliver contents specific to breakpoints and
pages.
Exhaustively test using Cavisson's RBU for
validation.
Use Cavisson's RBU network simulation to assess
user experience on low bandwidth connections. 
Use data and HAR files for analysis and further
improvements.
Use Real Devices for measuring performance
Reduced Page weight
Loading 3rd party calls and Footer after
Onload. Loading images ‘On Demand’
Reduce number of http calls
©2013 Walgreen Co. All rights reserved.

4. Solution set-up for front end performance
Real Browser User
Real Device Testing
Tools used
Cavisson Netstorm & Gomez UEM
Selenium on Real Devices – iOS and Android & Gomez UEM
Breakpoints
Desktop, Mobile
Mobile
Network
3G, 4G & Wi-Fi using Network simulation
3G, 4G & Wi-Fi using Charles Proxy
Browsers
Chrome & Firefox
Safari & Chrome
User Experience views
First and Repeat view
Data Collection
90th %ile & Averages
©2013 Walgreen Co. All rights reserved.

5. Real Device Testing Setup
Network throttling
3G - Down: 750 Kbps, Up: 2Kbps, Lat: 200 ms
4G - Down: 2 Mbps, Up: 20Kbps, Lat: 100 ms
WiFi - Down: 20 Kbps, Up: 1Mbps, Lat: 3 ms
Selenium Scripts
run on Real Devices
Gomez
Monitoring
W3C
Navigation Timing API
Test Reports
for comparison

6. Diagnostics & Analysis
Information used for analysis:
Timing information
Page Timings - Dom Content load, Onload and Page Load
Resource timings - Connection, DNS Lookup, Sending, Waiting , Receiving, Blocking.
Content Types Stats – Number of js, css, images etc. on a page
Request Stats - Third party, Origin or Akamai hit
Host Stats​ - Number of assets served from specific host (adobedtm, img.walgreens.com,
bazaarvoice etc.)
Cache stats – Number of assets served from Cache on repeat view
Headers information inclusive of Akamai and Strangeloop headers
Har files for analysis and one-glance view of the waterfall
©2013 Walgreen Co. All rights reserved.

7. Comparison – Arthur vs. Superman (Adaptive Mobile)
Pages
12/1/ :00 AM - 12/31/ :00 AM
1/26/ :00 AM - 1/27/ :00 AM
Difference Mobile (sec)
Difference Mobile (%)
Superman
Arthur
Home
10.3
5.27
-5.03
-48.83%
Tier1
7.15
4.7
-2.45
-34.27%
Tier2
14.72
5.39
-9.33
-63.38%
Tier3
12.62
8.65
-3.97
-31.46%
VPD
10.37
7.1
-3.27
-31.53%
SearchResults
15.76
8.82
-6.94
-44.04%
Login
12.69
7.56
-5.13
-40.43%
Youraccount
10.99
5.16
-5.83
-53.05%
Cart
13.1
7.74
-5.36
-40.92%
StoreLocator
8.31
6.34
-1.97
-23.71%

8. Strategy for back-end Performance
Iterative cycles of
Validation
Feedback
Analysis
Implementation
Changes are applied in Performance
environment based on Analysis and Feedback
Analysis
Implementation
Validation
Feedback
Generating load from Netstorm to generate
traffic on system under test(SUT)
Assess the impact of changes on SUT which
are part of latest builds
Analysis of system under test during Load test
utilizing diagnostics features of Netstorm.
Analysis of thread dumps and heap dumps using
Memory Analyzer
Compare the performance of current build
with the baseline version
Check system health using various
monitors (memory, Processor, network,
Response times(client and server side))
configured in Netstorm
Using AppDynamics for response time
from App Layer
©2013 Walgreen Co. All rights reserved.

9. Solution set-up for back-end performance | load test
Tool used - Cavisson Netstorm & App Dynamics integration – correlation.
Load Profile – 20% of transactions which represent 80% of our prod volume usage.
Volume/Load profile is about 12% of production volumes.
Data Collection –
Client Side Metrics
Transaction Response Time
Throughput
Hits/second
Number of Passed/Failed transaction
Server Side Metrics
Response time of services and pages from App Layer
Memory, Process, Processor and Disk usage stats
Network Usage stats
Thread Pool stats
Apache server stats
©2013 Walgreen Co. All rights reserved.

10. Catching back end performance issues early | Cruella multi site
Symptoms
High response time for all the transactions (from avg 0.1 sec in Arthur to 7.8 sec in Cruella)
Large number of Young GCs (from 120 in Arthur to 250 in Cruella).
Large number of Tenured GCs (from 4 in Arthur to 60 in Cruella).
Load Averages on App server increased from 7 to 30
CPU utilization on App server increased from 10% to 45%.
'Number of request in process' on web server increased from 200 to 700. 
Resolution
Circular Reference in profile components were identified and were removed
Profile Realm manager was disabled
Proxy Factory was disabled
©2013 Walgreen Co. All rights reserved.

11. Comparison Cruella vs Arthur (with memory issue)
©2013 Walgreen Co. All rights reserved.

12. Comparison Cruella vs Arthur (post implementation of changes)
©2013 Walgreen Co. All rights reserved.

13. Transformation How we used to operate How we operate now
HP Performance center usage for backend performance. Difficulty with analyzing the root cause of performance issue, application vs. environment configuration. Correlating data from HP vs. App Dynamics.
Using Netstorm for backend performance testing. Better and quick analysis by utilizing Netstorm’s monitoring and diagnostics features.
Front-end performance : single browser desktop Fire Fox with limited number of transactions and data collection
Front end performance – Using Netstorm RBU and real devices-selenium scripts to test for Chrome, Firefox and Safari. Further, tests were run for various bandwidth using Netstorm's network simulation and RDT’s Charles proxy
Historically, heavy bandwidth utilization from architecture team to deal with operational performance engineering activities.
 Relatively low bandwidth utilization from Architecture team on operational performance engineering activities
Multiple releases with false alarm
Improved ability to zero in root cause
©2013 Walgreen Co. All rights reserved.

14. Key accomplishments summary
From generating a lot of false alarms, to catch real problems! 3 critical issues caught as a matter of fact in Cruella alone! We would have gone down in production multiple times!
From struggling to identify root-causes and were challenged in find needles in a haystack, now we can collect data, correlate and diagnose efficiently!
From have zero visibility from end-user experience perspective. Now we not only know how fast pages are loading for customers , but also analyze why it is fast or slow with 100% data collection in production.
From hardly predict production performance for just desktop browsers, now we call out production performance for all devices with confidence.
©2013 Walgreen Co. All rights reserved.

15. Opportunity areas ahead
IE Browsers: Nature of problem to solve for ?
IE behaves differently than Chrome, Firefox and Safari. Hence, requires a different strategy.
UEM calls and certain calls (e.g. GlobalHeaderNavigationElements_ab.jsp) cause issues in rendering of pages.
In house web page test – Automated solution for data collection and analysis – In Progress
Old Android devices and browsers:
To mimic the processing power of old phones. Limited processing power affects the behavior of pages on such phones.
3 pages – Offers, Rx Landing & Steps
- consistent slowness across browsers IE
DOM handling – Objects and event handling.
Object handling – JSON
Old android
still under investigation . Blue browser. 90%ile messed up.
3 pages
Offers findings in
Other two pages..
©2013 Walgreen Co. All rights reserved.

16. Q & A ?
©2013 Walgreen Co. All rights reserved.

17. Thank You