Marketing Experiments I The Marketing Experiments modules explain how to design, implement, and analyze marketing experiments to improve the performance of the marketing function. This module describes advertising before-after experimental design, A/B web testing, and full factorial web experiment design. Marketing Experiments II then provides guidance for how one might extrapolate the results from the experiment to the full context for the product/service. Authors: Raj Venkatesan and Stu James © 2018 Raj Venkatesan, Stu James, and Management by the Numbers, Inc.

Introduction Introduction “Half the money I spend on advertising is wasted; the trouble is I don’t know which half.” - John Wanamaker, Father of Modern Advertising Experimentation is a key component of resource allocation methodology. It allows us to evaluate the consequence of different marketing actions. In this module, we will demonstrate how one can evaluate the following types of questions using marketing experiments: Should advertising spend be increased or decreased? Should price be increased or decreased? What is the most effective message to use? MBTN | Management by the Numbers

Difficult to Evaluate Marketing Options Since marketing decisions are not made in isolation, many other factors are likely in play: Changes in the environment (economy, technology, etc.) Changes in customers’ tastes, preferences, habits Changes in competitors’ actions Changes in distribution or partner activities (change in scope of distribution, pricing policies, etc.) Basic Issue: Would you have achieved the same sales increase without the increased advertising spend? MBTN | Management by the Numbers

Four Factors to Establish Causality Change in Marketing Mix Produces the Change in Sales Increasing Advertising $ Increased Sales No Sales Increase When No Change in Marketing Mix No Increase in Advertising $ Same Sales Time Sequence Increased advertising $ today leads to higher sales tomorrow No Other External Factor For example, a competitor left the market or there was an economic downturn, etc. MBTN | Management by the Numbers

Experiments Experiments One or more independent variable(s) [Advertising $] are manipulated to observe changes in the dependent variable [Sales or Brand Awareness] Choose 1,000 Customers Insight Randomization can match test and control groups on all dimensions simultaneously, given a sufficient sample size. Test Group (500) Exposed to new advertisement for one month Control Group (500) Exposed to old advertisement for one month MBTN | Management by the Numbers

Basic Experiment for Lift Choose 1,000 Customers Test Group (500) Exposed to advertisement highlighting new packaging for one month Control Group (500) Exposed to old advertisement for one month Test Group Total Sales = 2,200 units Control Group Total Sales = 2,000 units Sales Lift Test-Control = 200 units MBTN | Management by the Numbers

Lift Definitions Definitions: Basic Experiment Design Lift (Units) = Test Group Sales (Units) – Control Group Sales (Units) Lift (%) = Test Group Sales (Units) / Control Group Sales (Units) Net Lift (%) = Lift (Units) / Control Group Sales (Units) Question: Based on the example experiment, what is the net lift (%) if test group sales are 2,200 units and control group sales are 2,000 units? Answer: Net Lift % = Lift (Units) / Control Group Sales (Units) Lift (Units) = Test Group Sales – Control Group Sales = 2,200 – 2,000 = 200 Net Lift (%) = 200 / 2000 = 10% MBTN | Management by the Numbers

Before-After Experiment for Lift Test Group (500) Choose 1,000 Customers Old Advertisement Total Sales = 2,100 units Exposed to new advertisement for one month Total Sales = 2,200 units Total Sales = 2,000 units Total Sales = 1,900 Control Group (500) Before – After design controls for underlying differences between the test group and the control group Sales Lift Test-Control = (2,200 – 1,900) – (2,100 – 2,000) = 200 units MBTN | Management by the Numbers

Lift Definitions Definitions: Before/After Experiment Design Lift (Units) = (Test After – Control After) – (Test Before – Control Before) Net Lift (%) = Lift (Units) / Control Group Sales After (Units) Lift (%) = 1 + Net Lift % Question: Based on the example experiment, what is the net lift (%) if test group sales are 2,100 units before 2,200 units after and control group sales are 2,000 units before and 1,900 units after? Answer: Net Lift % = Lift (Units) / Control Group Sales (Units) Lift (Units) = (2,200 - 1,900) – (2,100 – 2,000) = 300 – 100 = 200 Net Lift (%) = 200 / 1,900 = 10.5% MBTN | Management by the Numbers

Web Experiments Web Experiments With the advent of the internet and on-line retailing, marketers now have a new tool available – Web Experiments. Web Experiments offer several advantages over the basic and before/after experiments previously described: Faster Results - from design to conclusion Less Expensive - to develop and execute Multiple Variables may be tested in combination - known as Full Factorial Design. Let’s walk through two types of web experiments, first using A/B testing (which is very similar the basic experiment described earlier but in a web context) and then using full factorial design. MBTN | Management by the Numbers

A/B Web Testing A/B Web Testing Let’s consider a simple A/B test. A company currently is pricing a product at $1.89 and designed an experiment with two additional price points (technically an A/B/C test) that generated the following unit sales: Price Units $1.59 585 $1.89* 434 $2.15 358 * Current price Question: Calculate the lift and sales revenue generated for each price point and increase (or decrease) in revenues from the current price of $1.89. MBTN | Management by the Numbers

A/B Testing Example Answer (Control Group): Answer @$1.59: Revenues at $1.89 = 1.89 * 434 = $820 Answer @$1.59: Lift (Units) = Test Group Sales – Control Group Sales = 585 – 434 = 151 Net Lift (%) = 151 / 434 = 35% Change ($) = Test Group Revenues – Control Group Revenues Change ($) = (585 * $1.59) – 820 = 930 – 820 = $110 Answer @$2.15: Lift (Units) = Test Group Sales – Control Group Sales = 358 – 434 = -76 Net Lift (%) = -76 / 434 = - 18% Change ($) = (358 * $2.15) – $820 = 770 – 820 = -$50 For this experiment, if objective is maximizing revenues, then pricing at $1.59 is the best choice. However, this would not take into consideration margins, competitive dynamics, etc. MBTN | Management by the Numbers

A/B Web Experiments - Concerns Insight While A/B web experiments offer many potential benefits, one must also be careful interpreting results. Here are a few concerns to consider when using A/B web experiments: Depending on how the experiment is designed, it is not always possible to choose your target audience. If the business has a broad customer base, that may not matter, but if the focus is on a narrower market, design the experiment such that it reaches the target audience, not every random visitor. Using results from an experiment with insufficient or skewed data. This might be due to small sample size or seasonal effects (holidays, days of week, etc.) Failing to account for first time visitors vs. frequent visitors. A frequent visitor may have a different reaction to a perceived change than a first time visitor who has no pre-existing perception or expectation. The result from the experiment may have a small, but statistically valid result; but then deemed not worthwhile to implement. It is important not to underestimate the cumulative effect of many small gains in lift. For example, if a gain of 5% is achieved each month due to testing improvements, this results in a cumulative gain of 80% for the year! MBTN | Management by the Numbers

Full Factorial Web Experiments Now let’s demonstrate the benefits of full factorial design. A company decides it would like to test the impact of changes to price and advertising copy on sales and has designed two traditional (A/B/C) experiments with the following results: Price Net Impact $1.59 $930 $1.89* $820 $2.15 $770 Price Net Impact $1.59 $930 $1.89* $820 $2.15 $770 Ad Copy Net Impact “Lasts Longer” $1,112 “Tastes Better” $1,030 “Good for You”* $820 Ad Copy Net Impact “Lasts Longer” $1,112 “Tastes Better” $1,030 “Good for You”* $820 * Current price and ad copy. Note that price tests are all run with the current ad copy of “Good for You” and ad copy tests are all run at the current price of $1.89. Simple test results suggest a change in price to $1.59 and a change in ad copy to “Lasts Longer”. MBTN | Management by the Numbers

Full Factorial Web Experiments Now let’s consider a web experiment using full factorial design where each price and copy combination are tested in combination. Note that we now have four additional price/ad combinations and recall A/B test results suggesting $1.59/“Lasts Longer” as optimal. Advertisement Copy Price $1.59 $1.89* $2.15 “Lasts Longer” $1,315 $1,112 $1,206 “Tastes Better” $957 $1,030 $1,500 “Good for You”* $930 $820 $770 Full factorial design would require a larger sample size to be valid, but it also takes into consideration the interaction of multiple marketing variables. MBTN | Management by the Numbers

Full Factorial Web Experiments Advertisement Copy Price $1.59 $1.89* $2.15 “Lasts Longer” $1,315 $1,112 $1,206 “Tastes Better” $957 $1,030 $1,500 “Good for You”* $930 $820 $770 Full factorial design results suggest a different choice than a traditional experiment. While “Lasts Longer” and $1.59 is an improvement, an increase in price to $2.15 in combination with a change in ad copy to “Tastes Better” appears to be the best alternative. In marketing, analyzing decision variables in isolation doesn’t always provide the optimal choice. If this is difficult to imagine, consider the option of new product introduction with and without advertising. MBTN | Management by the Numbers

Further Reference Further Reference Designing and analyzing marketing experiments is only the first half of the managerial decision process. Next is learning about how to apply test/experiment results to larger contexts and how to estimate the economic impact of those choices. These topics are addressed in Marketing Experiments II. For Further Reference: Marketing Metrics by Farris, Bendle, Pfeifer and Reibstein, 2nd edition, chapter 8. MBTN Modules: Marketing Experiments II MBTN | Management by the Numbers