1. Marketing Experiments I
The Marketing Experiments modules explain how to design, implement, and analyze marketing experiments to improve the performance of the marketing function. This module describes advertising before-after experimental design, A/B web testing, and full factorial web experiment design. Marketing Experiments II then provides guidance for how one might extrapolate the results from the experiment to the full context for the product/service.
Authors: Raj Venkatesan and Stu James
© 2018 Raj Venkatesan, Stu James, and Management by the Numbers, Inc.

2. Introduction
Introduction
“Half the money I spend on advertising is wasted;
the trouble is I don’t know which half.”
- John Wanamaker, Father of Modern Advertising
Experimentation is a key component of resource allocation methodology. It allows us to evaluate the consequence of different marketing actions.
In this module, we will demonstrate how one can evaluate the following types of questions using marketing experiments:
Should advertising spend be increased or decreased?
Should price be increased or decreased?
What is the most effective message to use?
MBTN | Management by the Numbers

3. Difficult to Evaluate Marketing Options
Since marketing decisions are not made in isolation, many other factors are likely in play:
Changes in the environment (economy, technology, etc.)
Changes in customers’ tastes, preferences, habits
Changes in competitors’ actions
Changes in distribution or partner activities (change in scope of distribution, pricing policies, etc.)
Basic Issue: Would you have achieved the same sales increase without the increased advertising spend?
MBTN | Management by the Numbers

4. Four Factors to Establish Causality
Change in Marketing Mix Produces the Change in Sales
Increasing Advertising $ Increased Sales
No Sales Increase When No Change in Marketing Mix
No Increase in Advertising $ Same Sales
Time Sequence
Increased advertising $ today leads to higher sales tomorrow
No Other External Factor
For example, a competitor left the market or there was an economic downturn, etc.
MBTN | Management by the Numbers

5. Experiments
Experiments
One or more independent variable(s) [Advertising $] are manipulated to observe changes in the dependent variable [Sales or Brand Awareness]
Choose 1,000 Customers
Insight
Randomization can match test and control groups on all dimensions simultaneously, given a sufficient sample size.
Test Group (500) Exposed to new advertisement for one month
Control Group (500) Exposed to old advertisement for one month
MBTN | Management by the Numbers

6. Basic Experiment for Lift
Choose 1,000 Customers
Test Group (500) Exposed to advertisement highlighting new packaging for one month
Control Group (500) Exposed to old advertisement for one month
Test Group Total Sales = 2,200 units
Control Group Total Sales = 2,000 units
Sales Lift Test-Control = 200 units
MBTN | Management by the Numbers

7. Lift Definitions Definitions: Basic Experiment Design
Lift (Units) = Test Group Sales (Units) – Control Group Sales (Units)
Lift (%) = Test Group Sales (Units) / Control Group Sales (Units)
Net Lift (%) = Lift (Units) / Control Group Sales (Units)
Question: Based on the example experiment, what is the net lift (%) if test group sales are 2,200 units and control group sales are 2,000 units?
Answer:
Net Lift % = Lift (Units) / Control Group Sales (Units)
Lift (Units) = Test Group Sales – Control Group Sales = 2,200 – 2,000 = 200
Net Lift (%) = 200 / 2000 = 10%
MBTN | Management by the Numbers

8. Before-After Experiment for Lift
Test Group (500)
Choose 1,000 Customers
Old Advertisement
Total Sales = 2,100 units
Exposed to new advertisement for one month
Total Sales = 2,200 units
Total Sales = 2,000 units
Total Sales = 1,900
Control Group (500)
Before – After design controls for underlying differences between the test group and the control group
Sales Lift Test-Control = (2,200 – 1,900) – (2,100 – 2,000) = 200 units
MBTN | Management by the Numbers

9. Lift Definitions Definitions: Before/After Experiment Design
Lift (Units) = (Test After – Control After) – (Test Before – Control Before)
Net Lift (%) = Lift (Units) / Control Group Sales After (Units)
Lift (%) = 1 + Net Lift %
Question: Based on the example experiment, what is the net lift (%) if test group sales are 2,100 units before 2,200 units after and control group sales are 2,000 units before and 1,900 units after?
Answer:
Net Lift % = Lift (Units) / Control Group Sales (Units)
Lift (Units) = (2, ,900) – (2,100 – 2,000) = 300 – 100 = 200
Net Lift (%) = 200 / 1,900 = 10.5%
MBTN | Management by the Numbers

10. Web Experiments
Web Experiments
With the advent of the internet and on-line retailing, marketers now have a new tool available – Web Experiments.
Web Experiments offer several advantages over the basic and before/after experiments previously described:
Faster Results - from design to conclusion
Less Expensive - to develop and execute
Multiple Variables may be tested in combination - known as Full Factorial Design.
Let’s walk through two types of web experiments, first using A/B testing (which is very similar the basic experiment described earlier but in a web context) and then using full factorial design.
MBTN | Management by the Numbers

11. A/B Web Testing
A/B Web Testing
Let’s consider a simple A/B test. A company currently is pricing a product at $1.89 and designed an experiment with two additional price points (technically an A/B/C test) that generated the following unit sales:
Price
Units
$1.59
585
$1.89*
434
$2.15
358
* Current price
Question: Calculate the lift and sales revenue generated for each price point and increase (or decrease) in revenues from the current price of $1.89.
MBTN | Management by the Numbers

12. A/B Testing Example Answer (Control Group): Answer @$1.59:
Revenues at $1.89 = 1.89 * 434 = $820
Lift (Units) = Test Group Sales – Control Group Sales = 585 – 434 = 151
Net Lift (%) = 151 / 434 = 35%
Change ($) = Test Group Revenues – Control Group Revenues
Change ($) = (585 * $1.59) – 820 = 930 – 820 = $110
Lift (Units) = Test Group Sales – Control Group Sales = 358 – 434 = -76
Net Lift (%) = -76 / 434 = - 18%
Change ($) = (358 * $2.15) – $820 = 770 – 820 = -$50
For this experiment, if objective is maximizing revenues, then pricing at $1.59 is the best choice. However, this would not take into consideration margins, competitive dynamics, etc.
MBTN | Management by the Numbers

13. A/B Web Experiments - Concerns
Insight
While A/B web experiments offer many potential benefits, one must also be careful interpreting results. Here are a few concerns to consider when using A/B web experiments:
Depending on how the experiment is designed, it is not always possible to choose your target audience. If the business has a broad customer base, that may not matter, but if the focus is on a narrower market, design the experiment such that it reaches the target audience, not every random visitor.
Using results from an experiment with insufficient or skewed data. This might be due to small sample size or seasonal effects (holidays, days of week, etc.)
Failing to account for first time visitors vs. frequent visitors. A frequent visitor may have a different reaction to a perceived change than a first time visitor who has no pre-existing perception or expectation.
The result from the experiment may have a small, but statistically valid result; but then deemed not worthwhile to implement. It is important not to underestimate the cumulative effect of many small gains in lift. For example, if a gain of 5% is achieved each month due to testing improvements, this results in a cumulative gain of 80% for the year!
MBTN | Management by the Numbers

14. Full Factorial Web Experiments
Now let’s demonstrate the benefits of full factorial design. A company decides it would like to test the impact of changes to price and advertising copy on sales and has designed two traditional (A/B/C) experiments with the following results:
Price
Net Impact
$1.59
$930
$1.89*
$820
$2.15
$770
Price
Net Impact
$1.59
$930
$1.89*
$820
$2.15
$770
Ad Copy
Net Impact
“Lasts Longer”
$1,112
“Tastes Better”
$1,030
“Good for You”*
$820
Ad Copy
Net Impact
“Lasts Longer”
$1,112
“Tastes Better”
$1,030
“Good for You”*
$820
* Current price and ad copy. Note that price tests are all run with the current ad copy of “Good for You” and ad copy tests are all run at the current price of $1.89.
Simple test results suggest a change in price to $1.59 and a change in ad copy to “Lasts Longer”.
MBTN | Management by the Numbers

15. Full Factorial Web Experiments
Now let’s consider a web experiment using full factorial design where each price and copy combination are tested in combination. Note that we now have four additional price/ad combinations and recall A/B test results suggesting $1.59/“Lasts Longer” as optimal.
Advertisement Copy
Price
$1.59
$1.89*
$2.15
“Lasts Longer”
$1,315
$1,112
$1,206
“Tastes Better”
$957
$1,030
$1,500
“Good for You”*
$930
$820
$770
Full factorial design would require a larger sample size to be valid, but it also takes into consideration the interaction of multiple marketing variables.
MBTN | Management by the Numbers

16. Full Factorial Web Experiments
Advertisement Copy
Price
$1.59
$1.89*
$2.15
“Lasts Longer”
$1,315
$1,112
$1,206
“Tastes Better”
$957
$1,030
$1,500
“Good for You”*
$930
$820
$770
Full factorial design results suggest a different choice than a traditional experiment. While “Lasts Longer” and $1.59 is an improvement, an increase in price to $2.15 in combination with a change in ad copy to “Tastes Better” appears to be the best alternative.
In marketing, analyzing decision variables in isolation doesn’t always provide the optimal choice. If this is difficult to imagine, consider the option of new product introduction with and without advertising.
MBTN | Management by the Numbers

17. Further Reference
Further Reference
Designing and analyzing marketing experiments is only the first half of the managerial decision process. Next is learning about how to apply test/experiment results to larger contexts and how to estimate the economic impact of those choices. These topics are addressed in Marketing Experiments II.
For Further Reference:
Marketing Metrics by Farris, Bendle, Pfeifer and Reibstein, 2nd edition, chapter 8.
MBTN Modules: Marketing Experiments II
MBTN | Management by the Numbers