Presentation is loading. Please wait.

Presentation is loading. Please wait.

Seth Eliot Senior Knowledge Engineer, Test Excellence.

Similar presentations

Presentation on theme: "Seth Eliot Senior Knowledge Engineer, Test Excellence."— Presentation transcript:

1 Seth Eliot Senior Knowledge Engineer, Test Excellence

2 Services and Cloud A/B Testing of Services Petabytes Processed About Seth

3 Measuremen t A quantitatively expressed reduction of uncertainty …based on one or more observations Testing …about quality of a system under test…

4 Data Driven Decision Making Data Driven Validation TiP Real Users Production Environments


6 500 Million measurements per month JSI JavaScript Instrumentation



9 Production is Truth



12 It’s still a lot….

13 “This process works for Facebook partly because Facebook does not, by and large, need to produce particularly high-quality software” Really?

14 …maybe less of this




18 Who's Got the Monkey? Who's Got the Monkey Now? Monkey


20 Provides insight into real usage Reproducible and well understood scenarios Covers a vast variety of environments Requires proper handling of Personally Identifiable Information (PII) May adversely alter production and production data

21 “To have a great idea, have a lot of them” -Thomas Edison

22 “…dice and slice in any way you can possibly fathom”

23 1B API requests per day Canary Deployment

24 Science!

25 How many years have you worked in software?

26 50 % 25 % 12.5 % 6.25 % 3.125 % Chance of neither nor 5 x 100% - 3.125% - 3.125%



29 These data have exactly the same summary statistics! X Mean = 9.0 y Mean = 7.5 X SD = 3.32 y SD = 2.03 R 2 = 0.67

30 

31 On tens of thousands of computers Stores hundreds of petabyte

32 “We know we can't anticipate the 101 things that will go wrong, The only thing we can control is ensuring our team responds appropriately to those situations.“ – Jerry Hook, Executive Producer Halo …Hundreds of thousands of requests per second

33 Availability (y) over time (x) Predict 75% of dips 24 hours ahead of timeData Machine LearningCosmos

34 The best observations are often in production A set of observations to reduce uncertainty about quality of a system under test Testing

35 Do It In Production Testing Where it Counts ?

Download ppt "Seth Eliot Senior Knowledge Engineer, Test Excellence."

Similar presentations

Ads by Google