1. Experimentation Platforms: Build or Buy?
RecSys 2014
Mike Lo, PayPal

2. Hello! Mike Lo, Sr. Manager - Experimentation, PayPal PayPal
Expectations for today’s talk
Manage PayPal’s Experimentation Program
Formerly part of eBay’s Program
PayPal’s ‘Testing Story’
Some comparison with eBay program.
Experimentation Program
Focus on Experimentation Platform
Product
Test Ops
Merchant Testing
Knowledge Mgt and Insights
Experimentation Analytics
Build or Buy?
We did both at PayPal

3. ANALYTICS Big DATA Beta Ramp Mobile Agile Insights Random Alpha
Words words words
Beta
Statistics
Ramp
Mobile
ANALYTICS
Agile
Big DATA
Random
Alpha
Insights

4. Becoming Data-Driven is difficult!
It doesn’t happen overnight. It took us years.
Product
Process
People

5. We bought.

6. PayPal = eBay right? What about the existing platform?
Rationale why we didn’t implement the eBay Experimentation Platform
Technology
Adoption
eBay-focused
PayPal’s tech stack
Fix Tracking
Can’t wait
Quick wins
Endorsement
Program, not the Product.
Built ground up for eBay
Tightly coupled with eBay systems
Need to decouple features

7. Result: Couldn’t Scale Outcome? Had to cut the cord
Ran experiments successfully for 12 months… then …
Had to cut the cord
Couldn’t Scale
Implemented eBay Platform
Result:

8. So are you advocating to build rather than buy?
No.
It’s all about the TESTING NEEDS of your organization.

9. What happened? No Product Tests Keep it separate? PayPal’s Needs
Like many companies, the need started from Marketing but didn’t end there.
Marketing Pages
Product Pages
Product Pages
Testing different content variants
Testing different feature variants
No Product Tests
Keep it separate?
PayPal’s Needs
Technical Limitations
Is there a clean distinction?
20 experiments is nice. But how do we get to 2,000?

10. Why it made sense for PayPal to build instead of buy
3 main reasons PayPal is investing to build
Analytical 1
Technical 2
Scale 3

11. 1 - High Fidelity Analytics
Our site is an ecosystem
Other sites have natural ‘fences’
Single Enterprise Testing Solution
Scientific rigor in the results

12. Agile Product Lifecycle
Biz Req
Product Req
Eng Req QA
Release
Test Plan
Design
Code
Validate
Smoke Test
2 - Technical agility
It’s all about going Agile!
Alpha
Beta
AB Test
Collect
Analyze
Ramp
Agile Product Lifecycle
Who does your dev?
Look beyond
Coupled with PDLC
Code once, push in minutes
Allows MVT
PayPal’s tech stack
Master of your domain
Data
Site speed performance
Rolling Full-Bright

13. 3- Scaling to the next level
Utilizing the platform to its fullest potential
Volume
Sophistication
Deeper Insights
Thousands of experiments
Patented mechanism
Utilizing PayPal’s platforms
Entire data pipeline
Automated design
Real-time analytics
Data modeling

14. Beware
2 potential unforeseen effects of building a robust platform, if left un-checked…
Platform is reduced to a CMS tool
Ramping / Throttling / Release tool

15. If you build it, will they come?
Again, it is not just about the Product… follow through with Adoption!
Product
Process
People

16. Questions?
Thanks for listening!