1. A/B Testing for UWP Apps: Experiment for Success
Microsoft Build 2016
12/3/2018 8:16 PM
A/B Testing for UWP Apps: Experiment for Success
Jason Watson
Sr Program Manager, Windows Dev Center
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. 12/3/2018 8:16 PM
A/B Testing Scenarios
Test variations of your app to discover optimizations that boost conversions and engagement:
In-app offers
Ad placements
User flow
Sign-up experience
Game level configuration
Make data-driven decisions about app changes
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. Example Can I boost conversions by changing in-app offer text?
12/3/2018 8:16 PM
Example
Can I boost conversions by changing in-app offer text?
Would “Disable Ads Now!” encourage more clicks?
My conversion rate is 9% today....
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. Winning Variation 12/3/2018 8:16 PM 4514 users (Variation A)
4512 users (Variation B)
11% conversion rate
45% conversion rate
Winning Variation
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Winning Variation 12/3/2018 8:16 PM 4514 users (Variation A)
4512 users (Variation B)
11% conversion rate
45% conversion rate
Winning Variation
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Winning Variation 12/3/2018 8:16 PM 9026 users (Variation B)
45% conversion rate
Winning Variation
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. 12/3/2018 8:16 PM
How does it work?
Update code in your app once to accommodate future experiments
Add reference to the Microsoft Store Engagement and Monetization SDK
Build app logic around A/B testable settings
Create logging events for user views and conversions
Use Windows Dev Center to run experiments
Run experiments to identify winning app variations
Switch your audience to winning app variations
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Demo #1: Update App Code for Experimentation
Microsoft Build 2016
12/3/2018 8:16 PM
Demo #1: Update App Code for Experimentation
Jason Watson
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Demo #2: Act on A/B Test Results
Microsoft Build 2016
12/3/2018 8:16 PM
Demo #2: Act on A/B Test Results
Jason Watson
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Advantages Experiment with your audience using one code base
12/3/2018 8:16 PM
Advantages
Experiment with your audience using one code base
Identify app variations that boost conversions
Switch your audience to a variation in two clicks
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Best Practices Maximize your test sample sizes
12/3/2018 8:16 PM
Best Practices
Maximize your test sample sizes
Use split distributions for variation assignments
Limit your experiment to two variations, including the control group
Run experiments for at least 2 weeks
Do peek at data after 1 day, but don’t peek afterwards
Allow for time to collect statistically significant data
Data from specific time of day can skew results (ex. Saturday mornings)
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Microsoft Build 2016
12/3/2018 8:16 PM
Call to Action
Click on Experimentation in Windows Dev Center to get started.
Download the Microsoft Store Engagement and Monetization SDK:
Re-visit Build on Channel 9.
Continue your education at Microsoft Virtual Academy online.
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. 12/3/2018 8:16 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.