Presentation on theme: "Experimentation Platforms: Build or Buy?"— Presentation transcript:
1 Experimentation Platforms: Build or Buy? RecSys 2014Mike Lo, PayPal
2 Hello! Mike Lo, Sr. Manager - Experimentation, PayPal PayPal Expectations for today’s talkManage PayPal’s Experimentation ProgramFormerly part of eBay’s ProgramPayPal’s ‘Testing Story’Some comparison with eBay program.Experimentation ProgramFocus on Experimentation PlatformProductTest OpsMerchant TestingKnowledge Mgt and InsightsExperimentation AnalyticsBuild or Buy?We did both at PayPal
3 ANALYTICS Big DATA Beta Ramp Mobile Agile Insights Random Alpha Words words wordsBetaStatisticsRampMobileANALYTICSAgileBig DATARandomAlphaInsights
4 Becoming Data-Driven is difficult! It doesn’t happen overnight. It took us years.ProductProcessPeople
6 PayPal = eBay right? What about the existing platform? Rationale why we didn’t implement the eBay Experimentation PlatformTechnologyAdoptioneBay-focusedPayPal’s tech stackFix TrackingCan’t waitQuick winsEndorsementProgram, not the Product.Built ground up for eBayTightly coupled with eBay systemsNeed to decouple features
7 Result: Couldn’t Scale Outcome? Had to cut the cord Ran experiments successfully for 12 months… then …Had to cut the cordCouldn’t ScaleImplemented eBay PlatformResult:
8 So are you advocating to build rather than buy? No.It’s all about the TESTING NEEDS of your organization.
9 What happened? No Product Tests Keep it separate? PayPal’s Needs Like many companies, the need started from Marketing but didn’t end there.Marketing PagesProduct PagesProduct PagesTesting different content variantsTesting different feature variantsNo Product TestsKeep it separate?PayPal’s NeedsTechnical LimitationsIs there a clean distinction?20 experiments is nice. But how do we get to 2,000?
10 Why it made sense for PayPal to build instead of buy 3 main reasons PayPal is investing to buildAnalytical1Technical2Scale3
11 1 - High Fidelity Analytics Our site is an ecosystemOther sites have natural ‘fences’Single Enterprise Testing SolutionScientific rigor in the results
12 Agile Product Lifecycle Biz ReqProduct ReqEng ReqQAReleaseTest PlanDesignCodeValidateSmoke Test2 - Technical agilityIt’s all about going Agile!AlphaBetaAB TestCollectAnalyzeRampAgile Product LifecycleWho does your dev?Look beyondCoupled with PDLCCode once, push in minutesAllows MVTPayPal’s tech stackMaster of your domainDataSite speed performanceRolling Full-Bright
13 3- Scaling to the next level Utilizing the platform to its fullest potentialVolumeSophisticationDeeper InsightsThousands of experimentsPatented mechanismUtilizing PayPal’s platformsEntire data pipelineAutomated designReal-time analyticsData modeling
14 Beware2 potential unforeseen effects of building a robust platform, if left un-checked…Platform is reduced to a CMS toolRamping / Throttling / Release tool
15 If you build it, will they come? Again, it is not just about the Product… follow through with Adoption!ProductProcessPeople