Presentation on theme: "Seth Eliot Senior Knowledge Engineer, Test Excellence."— Presentation transcript:
Seth Eliot Senior Knowledge Engineer, Test Excellence
Services and Cloud A/B Testing of Services Petabytes Processed About Seth
Measuremen t A quantitatively expressed reduction of uncertainty …based on one or more observations Testing …about quality of a system under test…
Data Driven Decision Making Data Driven Validation TiP Real Users Production Environments
Production is Truth
It’s still a lot….
“This process works for Facebook partly because Facebook does not, by and large, need to produce particularly high-quality software” Really?
…maybe less of this
Who's Got the Monkey? Who's Got the Monkey Now? Monkey
Provides insight into real usage Reproducible and well understood scenarios Covers a vast variety of environments Requires proper handling of Personally Identifiable Information (PII) May adversely alter production and production data
“To have a great idea, have a lot of them” -Thomas Edison
“…dice and slice in any way you can possibly fathom”
1B API requests per day Canary Deployment
How many years have you worked in software?
50 % 25 % 12.5 % 6.25 % 3.125 % Chance of neither nor 5 x 100% - 3.125% - 3.125%
These data have exactly the same summary statistics! X Mean = 9.0 y Mean = 7.5 X SD = 3.32 y SD = 2.03 R 2 = 0.67
On tens of thousands of computers Stores hundreds of petabyte
“We know we can't anticipate the 101 things that will go wrong, The only thing we can control is ensuring our team responds appropriately to those situations.“ – Jerry Hook, Executive Producer Halo …Hundreds of thousands of requests per second
Availability (y) over time (x) Predict 75% of dips 24 hours ahead of timeData Machine LearningCosmos
The best observations are often in production A set of observations to reduce uncertainty about quality of a system under test Testing