Einstein on Evaluation: Spencer on Stats 2 nd November 2011

“Not everything that can be counted counts, and not everything that counts can be counted”

“Not everything that can be counted counts...”

What can be counted? Clip/story counts Cost per impression Audience impressions Weighted Media Costs (WMC) Advertising Value Equivalent (AVE) Tone / sentiment Message Photo Quote Placement

What can be counted? Who thinks what? How many think x? How many do x? How many have seen x? Opinions?

6 So, what counts? What counts is success

What is success? Success Pronunciation: / sək’s ɛ s / noun [mass noun] 1 the accomplishment of an aim or purpose: the mission had some success in restoring confidence in the country 2 archaic the good or bad outcome of an undertaking: the good or ill success of their peacekeeping mission Origin: mid 16th century: from Latin successus, from the verb succedere 'come close after' (see SUCCEED )

By using smart objectives S pecific M easurable A chievable R ealistic T ime And how can we count success?

The ladder of success Did the message resonate and stick? Did the target audience have the opportunity to see it? Did the message appear? Did an attitude or behaviour change? Did the audience actually see it? Was the message placed in the right medium? Was the messages changed or distorted? Was the message compelling to the medium or intermediary?

The laddering approach to measurement Measures of effectiveness (MoE) Measures of performance (MoP)

Measures, metrics & methods Metrics Methods Measures Attitudes & behaviours Message pick- up/take out Outcome Audience penetration (direct) Reach of audience (indirect) Message placement Media uptake Pre/post surveys Pre-testing/ evaluation Events/ ‘contacts’ Media analysis (readership/listenership/ viewership) Media evaluation (traditional & new social media) Media coverage/ interviews Qual. & Quant. research Out- take Output

Methods – Primary Research

Quantitative approaches Face to Face research Telephone research Online research Postal research (rare) Data collection methods:

Face to Face research Perfect for:- Subject sensitive studies Where other methods (phone, s) impractical Pen & paper or CAPI* where available Street (max 5 mins.) or in home (20 to 40 mins.) Benefits: Very thorough, can interview people in situ and comfort of their own home, rich information Drawbacks: Expensive fieldwork, time consuming, some areas are “no go” areas *Note -Computer Aided Personal Interviewing

Telephone research Ideal for geographically dispersed populations Pre-provided telephone numbers or “direct dial” Trained interviewers calling from a telephone facility mins is the norm Benefits: Potential to reach broad population Drawbacks: Expensive NB mobile phone interviewing.

Online research Fastest growing data collection tool But only suitable where internet penetration high DIY option Using survey software Outsource data processing to DP House Write own questionnaire, script and web-host Online Panel option Benefits: inexpensive, quick & easy & good for showing stimulus material Drawbacks: not suitable for hard to reach or low literacy populations

Designing sample Samples Consider the population and the different subgroups you will need to analyse dictates overall sample size and design. Random sampling techniques produce representative results Census data is a challenge Why is it important to consider base sizes? The larger the base size the more confidence we have in the results being a true reflection of the study results We cannot act on the results if the sample size is not robust and the results therefore reliable Sample Size Answers accurate + or - %* (using a statistical measure of 95% confidence limit) % % % % % % * For example if your base size is 1600, and the results show that 50% of the sample have voted in the recent election, you can be confident that 47.7%- 52.3% of the total population have voted.

Principles of good research practice (applies to all methods) Pilot where you can – test out the survey if at all possible with the appropriate population Check the flow, the language, the layout, do not ask for feats of memory from the respondent Make the questionnaire as inviting to complete and as interesting as possible But do not make it so complex it confuses them Include some open ended questions if possible so people can express their own views in their own way Give time for analysis at the end – it is the output that is important not the research process in itself

Recommendations Face-to-Face methodologies should be considered first Do not rely upon government statistics; NGOs often have good reports on social movements and situations Contact NGOs who have completed research in the area. Find out about the work they have done and with who Be aware of social mores when assigning interviewers Timelines must be flexible, and projects on a best efforts basis Monitor political and social situation closely.

Qualitative approaches Data collection methods: Focus Groups o M ini groups of 4 to 5 (or full groups of 6 to 9 attending) In-depth interviews o Face to face or telephone

Qualitative Approaches – some rules of thumb In depth interviews work when: People cannot travel easily and it is easier for you to go to them (disability, ill health, lack of mobility, suspicious, too busy..) People you want to speak to are geographically dispersed The subject is highly sensitive and people would not want to discuss openly with others there There is time for the executives to do individual phoning or to travel to interviews Advantages: Much richness in the encounters; puts most onus on the researcher; reaches “difficult to reach” groups Disadvantages: Time consuming, costly, more difficult to cover a wider sample Focus Groups work when: People can travel to the groups and are prepared to do so The subject is one that people are happy to discuss in the open and share their thoughts with others There is enough sample to cluster people so that you can run a group (ie not too geographically dispersed) Advantages: Richness in the exchange and thrust and parry of the interchange; takes people out of their usual environment; more effective in terms of researcher time, more cost effective to reach a wider sample (than depths) Disadvantages: Logistics (viewing facilities etc) can be costly; people do not always turn up; analysis more intense & what people says more difficult to disaggregate

Methods – Secondary Research

Media evaluation What counts? The principles of effective communication count... Reach of audience: did you reach them sufficient times for message to stick? Delivery of message: did the message even appear, and how did it appear (credibly, negatively?)? AVE/WMC/OTS/GIs – what purpose do they serve?

Considerations:- It provides a deep dive into online discourse that is taking place but is not intended to provide a representative study of people living in a certain community Supplements, rather than replaces, other sources of information that are available to the organisation, whether gathered via primary research studies or informally from stakeholders and ‘allies’ It is a channel for communications and like any comms. needs to be measured! Monitoring the online environment Uses & Benefits:- Proxy for public opinions Will tell you how people feel about issues and how online discussion is enhancing or undermining a position Identify how current events shape content and discourse about organisations or social issues in online spaces.

What counts is the message Around half of the content analysed had a key strategic message embedded in it – primarily support for pro-democracy initiatives Base: 820 items of Online Content (Jan 2008 to Oct 2009)

Content analysed sourced from: Websites and online medias Blogs and microblogging sites Forums/ Discussion groups Social Network sites File sharing sites (e.g. YouTube and Flickr) Wikis/ Answer sites/ Podcasts Software tools used to uncover content posted: Alterian SM2 Free search tools e.g. Google Methodology

Deciding what counts Rear view Forward looking How did the campaign perform in terms of:- -Media -Return How effective was the campaign in terms of:- -Messaging -Attitude/behavioural shift Stop doing Start doing Keep doing

Decisions that count Shoot the messenger? Shoot the message?

“Not everything that counts is counted...”

Accounting for flat-lining of measures

And accounting for casuality issues

How they already act & feel about voting Behaviou r Attitude

Combined effect of communication - vote at some elections Q14: Please indicate how you feel about voting in terms of the following aspects.. Setting an example for younger people (very important, important, not very important, not at all important)

Quantifying campaign effect

Calculating cost benefit

When you can’t count, important to triangulate the data Primary survey data Ethnographics Secondary (media) data

Thank you and questions Claire Spencer FCIPR, i to i research limited DD: +44 (0) Mob: +44 (0)