Presentation is loading. Please wait.

Presentation is loading. Please wait.

Experimentation Platforms: Build or Buy?

Similar presentations

Presentation on theme: "Experimentation Platforms: Build or Buy?"— Presentation transcript:

1 Experimentation Platforms: Build or Buy?
RecSys 2014 Mike Lo, PayPal

2 Hello! Mike Lo, Sr. Manager - Experimentation, PayPal PayPal
Expectations for today’s talk Manage PayPal’s Experimentation Program Formerly part of eBay’s Program PayPal’s ‘Testing Story’ Some comparison with eBay program. Experimentation Program Focus on Experimentation Platform Product Test Ops Merchant Testing Knowledge Mgt and Insights Experimentation Analytics Build or Buy? We did both at PayPal

3 ANALYTICS Big DATA Beta Ramp Mobile Agile Insights Random Alpha
Words words words Beta Statistics Ramp Mobile ANALYTICS Agile Big DATA Random Alpha Insights

4 Becoming Data-Driven is difficult!
It doesn’t happen overnight. It took us years. Product Process People

5 We bought.

6 PayPal = eBay right? What about the existing platform?
Rationale why we didn’t implement the eBay Experimentation Platform Technology Adoption eBay-focused PayPal’s tech stack Fix Tracking Can’t wait Quick wins Endorsement Program, not the Product. Built ground up for eBay Tightly coupled with eBay systems Need to decouple features

7 Result: Couldn’t Scale Outcome? Had to cut the cord
Ran experiments successfully for 12 months… then … Had to cut the cord Couldn’t Scale Implemented eBay Platform Result:

8 So are you advocating to build rather than buy?
No. It’s all about the TESTING NEEDS of your organization.

9 What happened? No Product Tests Keep it separate? PayPal’s Needs
Like many companies, the need started from Marketing but didn’t end there. Marketing Pages Product Pages Product Pages Testing different content variants Testing different feature variants No Product Tests Keep it separate? PayPal’s Needs Technical Limitations Is there a clean distinction? 20 experiments is nice. But how do we get to 2,000?

10 Why it made sense for PayPal to build instead of buy
3 main reasons PayPal is investing to build Analytical 1 Technical 2 Scale 3

11 1 - High Fidelity Analytics
Our site is an ecosystem Other sites have natural ‘fences’ Single Enterprise Testing Solution Scientific rigor in the results

12 Agile Product Lifecycle
Biz Req Product Req Eng Req QA Release Test Plan Design Code Validate Smoke Test 2 - Technical agility It’s all about going Agile! Alpha Beta AB Test Collect Analyze Ramp Agile Product Lifecycle Who does your dev? Look beyond Coupled with PDLC Code once, push in minutes Allows MVT PayPal’s tech stack Master of your domain Data Site speed performance Rolling Full-Bright

13 3- Scaling to the next level
Utilizing the platform to its fullest potential Volume Sophistication Deeper Insights Thousands of experiments Patented mechanism Utilizing PayPal’s platforms Entire data pipeline Automated design Real-time analytics Data modeling

14 Beware 2 potential unforeseen effects of building a robust platform, if left un-checked… Platform is reduced to a CMS tool Ramping / Throttling / Release tool

15 If you build it, will they come?
Again, it is not just about the Product… follow through with Adoption! Product Process People

16 Questions? Thanks for listening!

Download ppt "Experimentation Platforms: Build or Buy?"

Similar presentations

Ads by Google