Presentation on theme: "Experimentation Platforms: Build or Buy?"— Presentation transcript:
1Experimentation Platforms: Build or Buy? RecSys 2014Mike Lo, PayPal
2Hello! Mike Lo, Sr. Manager - Experimentation, PayPal PayPal Expectations for today’s talkManage PayPal’s Experimentation ProgramFormerly part of eBay’s ProgramPayPal’s ‘Testing Story’Some comparison with eBay program.Experimentation ProgramFocus on Experimentation PlatformProductTest OpsMerchant TestingKnowledge Mgt and InsightsExperimentation AnalyticsBuild or Buy?We did both at PayPal
3ANALYTICS Big DATA Beta Ramp Mobile Agile Insights Random Alpha Words words wordsBetaStatisticsRampMobileANALYTICSAgileBig DATARandomAlphaInsights
4Becoming Data-Driven is difficult! It doesn’t happen overnight. It took us years.ProductProcessPeople
6PayPal = eBay right? What about the existing platform? Rationale why we didn’t implement the eBay Experimentation PlatformTechnologyAdoptioneBay-focusedPayPal’s tech stackFix TrackingCan’t waitQuick winsEndorsementProgram, not the Product.Built ground up for eBayTightly coupled with eBay systemsNeed to decouple features
7Result: Couldn’t Scale Outcome? Had to cut the cord Ran experiments successfully for 12 months… then …Had to cut the cordCouldn’t ScaleImplemented eBay PlatformResult:
8So are you advocating to build rather than buy? No.It’s all about the TESTING NEEDS of your organization.
9What happened? No Product Tests Keep it separate? PayPal’s Needs Like many companies, the need started from Marketing but didn’t end there.Marketing PagesProduct PagesProduct PagesTesting different content variantsTesting different feature variantsNo Product TestsKeep it separate?PayPal’s NeedsTechnical LimitationsIs there a clean distinction?20 experiments is nice. But how do we get to 2,000?
10Why it made sense for PayPal to build instead of buy 3 main reasons PayPal is investing to buildAnalytical1Technical2Scale3
111 - High Fidelity Analytics Our site is an ecosystemOther sites have natural ‘fences’Single Enterprise Testing SolutionScientific rigor in the results
12Agile Product Lifecycle Biz ReqProduct ReqEng ReqQAReleaseTest PlanDesignCodeValidateSmoke Test2 - Technical agilityIt’s all about going Agile!AlphaBetaAB TestCollectAnalyzeRampAgile Product LifecycleWho does your dev?Look beyondCoupled with PDLCCode once, push in minutesAllows MVTPayPal’s tech stackMaster of your domainDataSite speed performanceRolling Full-Bright
133- Scaling to the next level Utilizing the platform to its fullest potentialVolumeSophisticationDeeper InsightsThousands of experimentsPatented mechanismUtilizing PayPal’s platformsEntire data pipelineAutomated designReal-time analyticsData modeling
14Beware2 potential unforeseen effects of building a robust platform, if left un-checked…Platform is reduced to a CMS toolRamping / Throttling / Release tool
15If you build it, will they come? Again, it is not just about the Product… follow through with Adoption!ProductProcessPeople