Presentation on theme: "1 Email Test Case Study. Email Test Scenario Chase Bank’s email acquisition program with AOL was delivering CPAs consistently well above target. We."— Presentation transcript:
Email Test Scenario Chase Bank’s email acquisition program with AOL was delivering CPAs consistently well above target. We structured a testing program to immediately introduce new email creative designs, measure response, and optimize email designs accordingly. The program, thus far, has included 3 individual, sequential tests: #1 - An all-new email creative execution #2 - A single variable email iteration #3 - Subject line testing Testing was scheduled for May – July, 2007, so learnings could be leveraged in the peak back-to-school & holiday seasons.
Email Test Structure #1 These 3 email versions ran head-to-head Control EmailAll New CreativeSingle Variable Test
Email Test Success #1 Both the all-new creative & the single variable iteration decreased CPA. 12% decrease in CPA Control EmailAll New CreativeSingle Variable Test
5 Email Test Success #1 CPAs from both test emails were equally improved, indicating that the new email design is strong, and repositioning the calls-to-action does not adversely affect response. 12% decrease in CPA All New CreativeSingle Variable Test 12% decrease in CPA
Email Test Structure #2 Next, a multi-variable test with 2 additional CTA buttons was developed. The new control and the test version ran head-to-head in June. New Control EmailSingle Variable Test
Email Test Success #2 This simple multi-variable test drove an incremental 6% decrease in CPA. 6% decrease in CPA New Control EmailSingle Variable Test
Email Test Success #3 Then we tested subject lines head-to-head for each email version: Subject Line 1: “Special Offer for AOL Members” Subject Line2 : “Special Rewards for AOL Members” Subject Line 1, with a single word difference from Subject Line 2, delivered a 15% lower CPA. “Special Offer for AOL Members” Decreased CPA by 15%
9 Email Test Insights & Actions The all-new email design drove double-digit decreases in CPA. Iterations of the all-new design, with repositioned calls-to-action, performed as well as the all-new design in reducing CPA. Subtle changes to the new control, with additional calls-to-action, decreased CPA still further. And a single-word difference in subject lines drove additional performance gains. Continual testing – of both all-new designs as well as small changes to existing designs – is an extremely efficient and highly effective way to significantly improve campaign performance.