Presentation is loading. Please wait.

Presentation is loading. Please wait.

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary. Experimentation Platforms: Build or Buy? RecSys 2014 Mike Lo, PayPal.

Similar presentations


Presentation on theme: "© 2014 PayPal Inc. All rights reserved. Confidential and proprietary. Experimentation Platforms: Build or Buy? RecSys 2014 Mike Lo, PayPal."— Presentation transcript:

1 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. Experimentation Platforms: Build or Buy? RecSys 2014 Mike Lo, PayPal

2 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. Product Test Ops Merchant Testing Knowledge Mgt and Insights Experimentation Analytics 2 Hello! Mike Lo, Sr. Manager - Experimentation, PayPal PayPal Experimentation Program Manage PayPal’s Experimentation Program Formerly part of eBay’s Program Build or Buy? We did both at PayPal Expectations for today’s talk Focus on Experimentation Platform PayPal’s ‘Testing Story’ Some comparison with eBay program.

3 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 3 Words words words

4 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 4 Becoming Data-Driven is difficult! It doesn’t happen overnight. It took us years.

5 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 5 We bought.

6 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 6 TechnologyAdoptioneBay-focused PayPal = eBay right? What about the existing platform? Rationale why we didn’t implement the eBay Experimentation Platform PayPal’s tech stack Fix Tracking Can’t wait Quick wins Endorsement Program, not the Product. Built ground up for eBay Tightly coupled with eBay systems Need to decouple features

7 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. Outcome? Ran experiments successfully for 12 months… then … Couldn’t Scale Had to cut the cord 7 Result: Implemented eBay Platform

8 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 8 No. It’s all about the TESTING NEEDS of your organization. So are you advocating to build rather than buy?

9 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 9 No Product TestsKeep it separate?PayPal’s Needs What happened? Like many companies, the need started from Marketing but didn’t end there. Product Pages Marketing Pages Product Pages Testing different content variants Testing different feature variants Technical Limitations Is there a clean distinction? 20 experiments is nice. But how do we get to 2,000?

10 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. Why it made sense for PayPal to build instead of buy 3 main reasons PayPal is investing to build 1 Analytical 10 2 Technical 3 Scale

11 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary High Fidelity Analytics Our site is an ecosystem Single Enterprise Testing Solution Scientific rigor in the results Other sites have natural ‘fences’

12 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 12 Agile Product Lifecycle Who does your dev?Look beyond 2 - Technical agility It’s all about going Agile! PayPal’s tech stack Master of your domain Data Site speed performance Rolling Full-Bright Test PlanDesignCodeValidate Smoke Test Coupled with PDLC Code once, push in minutes Allows MVT Biz Req Product Req Eng Req QARelease AlphaBetaAB TestCollectAnalyzeRamp

13 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 13 VolumeSophisticationDeeper Insights 3- Scaling to the next level Utilizing the platform to its fullest potential Thousands of experiments Patented mechanism Utilizing PayPal’s platforms Entire data pipeline Automated design Real-time analytics Data modeling

14 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 14 Beware 2 potential unforeseen effects of building a robust platform, if left un-checked… Platform is reduced to a CMS tool Ramping / Throttling / Release tool

15 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 15 If you build it, will they come? Again, it is not just about the Product… follow through with Adoption!

16 © 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 16 Questions? Thanks for listening!


Download ppt "© 2014 PayPal Inc. All rights reserved. Confidential and proprietary. Experimentation Platforms: Build or Buy? RecSys 2014 Mike Lo, PayPal."

Similar presentations


Ads by Google