Apdex Implementation at AOL CMG International Conference San Diego, California December 5, 2007 Eric Goldsmith Operations Architect Session 45A

Slide 2 Our Environment Operations organization Measuring Web site performance from customer- centric view Full page load measured from outside datacenter Multiple geographic locations Goals Short-term: Identify product issues/outages Long-term: Achieve uniform geographic performance, in parity with competitors

Slide 3 Current Metrics & Shortcomings Response Time & Availability Often dont tell whole user-experience story Reported as averages Hides variance, and is skewed by outliers Reported in absolute numbers No context of a target (goal) value

Slide 4 Goals of Apdex use Inclusive view of performance, availability, and data distribution Building in of a target, and data normalization around it Performance is evaluated qualitatively against a target

Slide 5 Data Source and Collection Using commercial 3 rd -party tool to gather measurements from multiple geographic locations Data of interest for our Apdex calculations 1.Date/Time 2.Measurement Value 3.Success/Error (Error = Frustrated) 4.Test Location Data collection is batched (daily)

Slide 6 Calculation and Graphing in Excel Calculate sub-score for each row (data point) If (error) score = 0 else if (measurement <= T) score = 1 else if (measurement <= F) score = 0.5 else score = 0 Define interval over which to calculate Adpex score –Hourly, daily, weekly, etc. –Segregate by location, if desired –Apdex spec recommends >100 data points per interval Then calculate overall Apdex score for interval =sum(sub-scores) / count(measurements) Get fancy with DSUM() and DCOUNT() Database lookups simplify segregation by date, location, etc.

Slide 7 Target T Determination We chose our targets based on competitor performance For a given Web site, identify its target competitor (may be self) The T marker method we chose initially was based on Best Time Multiple Measure average response time from a good location, then add 50% to build in tolerance for other locations Instead, we averaged data from all locations Our thinking was that the 50% inflation wasnt necessary because of the natural diversity of the data from multiple geographic locations

Slide 8 Example Results Presentation

Slide 9 Example Results Presentation contd

Slide 10 Problems with our initial T Initial results were promising…but as we examined data over time, the Apdex results didnt always correlate well with observations Target competitor never achieves Excellent level Significant performance change not reflected (see next slide)

Slide 11 Example of Initial T Problem 44% reduction in average load time But Apdex score didnt change

Slide 12 Plan B We experimented with various T determination techniques, and eventually settled on the Empirical Data method Find T that results in the proper Apdex for a well studied group In our environment… For a given Web site, identify its target competitor (may be self) –The performance of this competitor is defined as Excellent Determine the smallest T such that the competitors Apdex score remains Excellent for a period of time (at least 1 month)

Slide 13 New T With the new T, the Apdex results correlate better with observations Target competitor now achieves Excellent level Performance change now reflected

Slide 14 Changing T Define technique for reevaluating T on an ongoing basis But dont want to change T too often Suggestions for reevaluating T: Quarterly, looking at prior 3 months of data When a significant product change occurs When requested (from business)

Slide 15 Example - T Change

Slide 16 Apdex vs. Other Metrics

Slide 17 Apdex vs. Performance & Availability Deep Dive 1 Virtually no change in Apdex for B, despite large change in performance and availability. Deep Dive 2 Apdex shows B performing better than A. Perf/Avail charts show opposite.

Slide 18 Deep Dive 1 S377 (0.53) T314 (0.22) F 22 A0.75 S422 (0.60) T195 (0.14) F 88 A0.74 S419 (0.59) T219 (0.15) F 73 A0.74 Virtually no change in Apdex for B, despite large change in performance and availability.

Slide 19 Deep Dive (0.59)390 (0.55) 552 (0.20)215 (0.15) (0.24)436 (0.61) 1074 (0.38)213 (0.15) Apdex shows B performing better than A. Perf/Avail charts show opposite. 506 (0.36)553 (0.79) 919 (0.32)146 (0.10) (0.66)607 (0.84) 469 (0.17)111 (0.08)

Slide 20 Closing Thoughts Were still exploring the application of Apdex in an Operations organization Can Apdex be used to identify the day to day "issues" traditionally identified through analysis of performance and availability metrics? Or is it better suited as a method of performance representation for the business side of the house? Interesting to calc: what would it take for a product to achieve the next "band" of performance What performance level do I need to move from Poor to Fair Help in establishing interim targets

Thank You Questions?