A/B Testing: An Introduction 50/50 randomized split between two experiences Used in ▫Web development ▫Internet marketing ▫SaaS Products
Why A/B Testing? Web development and internet marketing ▫What metrics do you try to improve? Conversion Sales ROI (on ad spend) Engagement SaaS (cloud-based software) ▫Improve user experience (UX) ▫Increase engagement ▫Drive long-term profitability
When Should You A/B Test? Before introducing a new feature Small day-to-day improvement When you want to improve the customer exp. When you want to increase sales/subscriptions All the time!
How to A/B Test Define goal / question ▫Why are you running the test? Identify metrics ▫How do you identify a successful test? Design test experience Set up data collection ▫Google analytics (very limited), Adobe Marketing Cloud, custom, etc. Analyze data
Let’s Analyze Data Test and Control have different # of users! Google Analytics ▫ ▫Pw: ProductCamp Provo (with the space) ▫Shortcut: A/B Test on Voting Excel Data ▫http://bit.ly/1mdQYJz Limited data pushed into website database User, Test Group, Post (at vote level)
Some Ideas of What to Look For Difference between Test & Control on: ▫Votes per Visitor ▫Visit duration ▫Sessions voted for (somewhat time consuming)
What Insights Have You Found? How do the test and control groups differ? ▫Number of votes per visitor Test > Control ▫Visit duration Test < Control ▫Different sessions voted for?
Statistical Significance Provides confidence in result ▫Insignificant: diff caused by random variation ▫Significant: most likely caused by something Analytics and statistics only reveal correlation Is that a bad thing? Can require more data than we can get Requires more skill to calculate ▫R▫R