1. 1 Stats are Sexy! Metrics and Testing That’s HOT!

2. 2 2 “The only truth is what actually happens.” The number of people who did the thing you wanted them to do. Give. Act. Volunteer. Come to Get Together.

3. 3 3 The data Oh NO! Let’s take a closer look...

4. 4 4 What are all these metrics? It’s easy to get lost in the result metrics, some common calculations are: Delivery rate Open rate Click-through rate Conversion rate Tell-a-Friend rates

5. 5 5 Delivery Delivered Emails, not yet opened Delivery Rate = Number of emails Received/ Total emails Sent

6. 6 6 Opening the email Opened email Number of emailed members who opened at least 1 email/ Total Emails Sent Open Rate =

7. 7 7 Beware, Preview Pane Affects Open Rates

8. 8 8 Opened email

9. 9 9 Click-Through Click-through Rate = Number of emailed members who clicked on at least one link/ Total Emails Sent

10. 10 The Response Response Rate = Number of emailed members who participated in the campaign / Total emails Sent

11. 11 Tell-a-Friend Usage Number of emailed members who told at least 1 friend/ Total emails Sent Tell-a-Friend Usage Rate =

12. 12 Tell-a-friend Recruitment Number of Friends who joined a center/ Total Emails Sent Tell-a-Friend Recruitment Rate =

13. 13 Which Rates are the most important? Conversion Rates are the best measure of a campaign’s immediate success Click-through rates can be good indicators of how members are interacting with your messages Open rates are deceiving for short term data, but if measured over several years can become interesting. They also are good for normalizing other data.

14. 14 So, what can you do with all these statistics?

15. 15 This is a numbers game … … do the math If you want 10 people to TAKE ACTION, then: (at least) 20 people need to click to the advocacy page (at least) 30 people need to open your e-mail message (at least) 100 people need to receive your message

16. 16 This is a numbers game … … do the math If you want 10 people to DONATE, then: (at least) 50 people need to click to the donation page (at least) 250 people need to open your e- mail message (at least) 1,000 people need to receive your message Yes, your results may vary…dramatically!

17. 17 It all starts with the list Good direct marketing is all about the list It starts with the list It ends with the list It is all about the list But building a good list requires you to be good at everything else: your work, copy, layout, timing, etc.

18. 18 Typical response rates Nothing is typical, really Big lists usually = lower response… …and smaller lists usually ROCK But if you make us…

19. 19 Where can I find my industry standards? Marketingsherpa.com Emarketer.com Opt-In News Your own campaigns!

20. 20 Making the most of the numbers to get the most out of your online program… Segmenting and Testing

21. 21 Segmenting is… …dividing your e-mail list when running campaigns to learn more, personalize, and generate higher results.

22. 22 Segmentation example Donors People on direct mail list with e-mail E-mail subscribers RecipientsAmt of Gifts # of Gifts Avg gift Resp. rate 3,500$2,29040$57.751.13% RecipientsAmt of Gifts # of Gifts Avg gift Resp. rate 7,250$3,38592$36.791.26% RecipientsAmt of Gifts # of Gifts Avg gift Resp. rate 7,000$77526$29.810.37%

23. 23 Before segmenting…

24. 24 After segmenting…

25. 25 Other segmentation strategies Giving level Campaign-based Donors who have taken action Donors who have not taken action Non-donors who have taken action Non-donor, non-action-takers Recency Source One-time and lapsed online donors Active and inactive non-donors

26. 26 Testing … …Once you’ve determined how best to segment your particular list, you can begin to test what works within each of these segments.

27. 27 Testing Open rates Time of day, day of week Subject lines Straightforward subject lines vs suggestive ones. Your action will help a child eat today vs. How can you help the orphans? To Mention The Ask Or Not? Looking ahead, more animals need your help in 2007 vs. Last chance for your tax-deductible gift to animals Subject lines that in different ways ask for a renewal of commitment. Renew your commitment for 2007 vs. Your membership is expiring vs. Renew today—and save even more puppies

28. 28 Testing, continued Click through rates Premiums Offer Signer Short or long copy Link placement Conversion rates Short vs. long page Images and insets First things first! Giving levels

29. 29 How to create a good test... Pick one variable you want to test Keep everything else the same Randomly 50/50 split the population of the campaign Compare the results from each test group using a “significance calculator” “Significant” results mean the results can be repeated

30. 30 Testing Do’s and Don’t’s DO Ask a question and design a test to answer it Select a RANDOM sample of your list Test ONE thing at a time Keep everything the same except for the thing you are testing Check for statistical significance Make sure your list segments are large enough to produce meaningful results Understand that some test results are only applicable to the message you were testing

31. 31 Testing Do’s and Don’t’s DON’T Assume that test results apply across lists or list segments Bite off more than you can chew Read too much into the numbers Assume that every test will yield statistically significant results

32. 32 Testing Gone Wrong

33. 33 What Happened? Too many variables Tried to answer every question at once The result: Hours of work to set up 10 segments Data was messy and difficult to decipher We didn’t learn much about our list

34. 34 What We Should Have Done Picked one variable Had a clear question to answer Split the list in half or thirds instead of in tenths

35. 35 And the winner is…. Subject Line Test What metric should be used to evaluate a subject line? Which is the winner?

36. 36 And the winner is… Open Rate Two-way tie MessageEmails Sent Open Rate Click Through RateActions Response Rate Bush's Budget: Guns Not Butter1,00028.50%13.00%535.30% What's At Stake99931.90%15.50%525.20% Action Alert: Defend 100 Hours Agenda99927.70%14.80%494.90% Action Alert: Organize agaist Bush's budget1,00031.90%18.20%515.10% Organize Against Bush's Budget1,00028.80%16.70%545.40% Total4,99830.60%16.90%2595.18%

37. 37 And the winner is… Link placement test Which is the most important metric? Which segment is the winner?

38. 38 And the winner is… Click Through Rate Below the first paragraph

39. 39 And the winner is…. Short Email vs. Long Email What metric(s) should you look at? Which message is the winner?

40. 40 And the winner is… Click through rates and response rates Trick question – the results are statistically insignificant. You can’t always eyeball test results – be sure to test for statistical significance

41. 41 And the winner is… Which campaign is more successful? Why? 41

42. 42 And the winner is… Arctic campaign most likely to raise money (fewer donors, but higher average gift) Public lands campaign most likely to get engage donors (more donors, lower average gift). Depending on the goals of the campaign (raise a lot of $, or get more people to give), both are a success. 42

43. 43 And the winner is…. From line testing – by list segment What should the organization do moving forward? 43

44. 44 And the winner is… Even thought the inactive segment had a lower open rate with the from line “Kathy, Organization” all of the other segments saw improved open rates. You could segment out the inactive folks and continue to send them messages with the old from line, but it’s probably not worth the effort. 44

45. 45 Digging Deeper: 5 Questions you could ask

46. 46 Question 1 Which group of people on my email list is the most likely to take action, tell a friend, or donate?

47. 47 Question 1, Sample answer Source 1Source 2 Members in Group 2,4034,014 Activists* 59.84%29.45% Donors 5.45%2.72% TAF 35.41%9.12% Super activists 4.99%2.99% Members who opened an email in 2005 57.26%55.23% Members who clicked 70.41%54.53%

48. 48 Answering Question 1 Two words: Source Codes Suffix on imports or on any link someone may click to come on your list, starts with ‘?source=’ Advocacy campaign (?source=advo1) Viral Flash game or movie (?source=funny_movie) Banner ads (?source=yahoo_banner) Paid source (?source=care2) Donation campaign (?source=moneybags) Source codes are attached to a member so you always know where they originally came from

49. 49 Answering Question 1 Each cell in the table is a member select Query members who have the source code you want AND have donated, taken action, etc. The results can help you compare how valuable names from various sources are to your organization

50. 50 Question 2 How do I know how fast my list is growing?

51. 51 Answering Question 2 List Growth Rate: # new sign-ups/# of deliverable addresses Shows % list growth over a period of time (month, 3-months, year) List Hurdle Rate: # removals/# of deliverable addresses Shows % of list that leaves over a period of time # removals tells you how many new names you need to get before you achieve net growth

52. 52 Question 3 How come my advocacy response rates are so low?

53. 53 Question 4 How do I find out if I’m sending too much email?

54. 54 Question 5 How valuable is an email address?