Presentation on theme: "How to interpret results of public opinion surveys? How to respond? EU Accession Communications Network Ohrid – 21-22 September 2006 Leendert de Voogd."— Presentation transcript:
How to interpret results of public opinion surveys? How to respond? EU Accession Communications Network Ohrid – 21-22 September 2006 Leendert de Voogd
2 How to interpret results of public opinion surveys? How to respond? CONTENT Back to basics How to interpret the results? Some hints Case studies What strategy?
Back to basics
4 A survey, what for? It is not necessary to interview everyone or we simply cannot. There could be many reasons for conducting a survey. Some examples: To identify attitudes, needs, concerns and/or expectations about specific topics To measure opinions (agreement or opposition) to certain ideas To test specific messages and communication plans A survey will never: Replace a census Replace an election It is a picture at a given time, in a given context, of the diversity of opinions
5 Types of research Two types of research: quantitative and qualitative Quantitative: the objective is to quantify opinions related to a specific issue. This can be done for different targets (general public, specific target groups, opinion leaders…) Qualitative: to explore in depth motivations, opinions, attitudes and behaviours of individuals facing a given problematic. This is can be done through focus groups or in-depth interviews. Both types are/could be complementary
6 Data collection modes PAPI, CAPI, CATI,…CAWI CATI stands for Computer Assisted Telephone Interviewing Face-to-face, telephone or Internet? FACE TO FACE (PAPI/CAPI) TELEPHONE (CATI) INTERNET (CAWI) For long questionnaire Material to show For difficult questions For fast results Targets difficult to reach For difficult areas to get in Self administration Specific targets Cheaper
7 Representativeness of a sample First step: definition of the universe to represent Second step: definition of a sample The first quality of a sample is to be representative of the surveyed universe. This means without BIAS. - Sampling methods mostly used: random or quota sampling The law of large numbers: The law says that with a large enough sample many odd coincidences are likely to happen. The weight of the universe to survey has almost no incidence on the size of the sample
8 Margin of error Survey results remain approximations, the accuracy of which, everything being equal, rests upon the sample size and upon the observed percentage. Depending on sample sizes, the real percentages vary within precise confidence limits*. The formula: ± 2 (p*q)/n For 1000 interviews Observed %10% or 90%20% or 80%30% or 70%40% or 60%50% Confidence limits ± 1.9 points± 2.5 points± 2.7 points± 3.0 points± 3.1 points For 10 000 interviews Observed %10% or 90%20% or 80%30% or 70%40% or 60%50% Confidence limits ± 0.6 point± 0.8 point± 0.9 point± 1 point * at 95% level of confidence
9 Margin of error (contd) An illustration: As a conclusion: if we had interviewed the whole population (i.e. universe) the result for the Yes option would have lied between 51.85% and 58.15% In other words, voting intentions of 51% vs. 49% do not mean much, except that it is too close to call… Survey results (n=1000) Hyp.: Whole population Yes55%51.85 < X < 58.15 No45%41.85 < Y < 48.15
How to interpret results?
11 Interpretation of the results Interpretation of the results is the step following the tabulation of the responses. Rely on own basic knowledge: Theoretical knowledge in sociology, economy, psychology and political science Marketing models Analyse the influence of social factors: Gender: could refer to own perceptions of role differences between men & women Age: could indicate an evolution in time but could also reveal generations effect Education: cultural indicator but also social status indicator Profession: social status indicator but also economical indicator Constructed variables: opinion leaders (e.g. people who talk politics on a regular basis at home and who try to convince other people)
12 Interpretation of the results (contd) While interpreting results we must keep extra attention to the following: At an individual level: Age or ageing effect : public opinion vary with age, whatever the generation considered is Generation effect : public opinion is typical from one generation; the youngest, while ageing, do not have the same response pattern as the oldest Period effect: public opinion evolve in time, whatever the age. At a collective level: Structural changes models: major social or economic evolutions, technical innovations… Political or economical cycles
13 Interpretation of the results (contd) Behaviours models Behaviours Social and demographic variables Attitudes
15 Context of the study In an information society, public opinion is reacting almost immediately to particular events. This has a clear impact on survey results. Some examples: Bombings of Sarajevos markets obliged us to conduct new fieldwork in some countries Shifts in American public opinion after 9/11: support for Bush increased Better image of NGOs after Tsunami in South-East Asia Level of information on the Euro after the changeover took place (+20 points between November 2001 and January 2002)
16 Basis of respondents Before drawing any conclusion on one particular result, one must look at the basis of respondents (i.e. how many people answer the question?) A general rule that could be followed: A minimum of 50 respondents per variable considered seems necessary for a sample of 1000 interviews A minimum of 100 respondents per region considered seems necessary to speak of some regional trends A minimum of 1000 respondents per variable considered seems necessary for a sample of 25000 interviews (e.g. in the EU)
17 Going beyond the obvious Cross tabulations could be very helpful but sometimes are not really necessary. Vertical % (DK not shown) Total EU 25 Has benefited from EU membership Has not benefited from EU membership EU membership: a good thing 56%82%24% EU membership: a bad thing 13%2%32% EU membership: neither a good nor a bad thing 28%14%42%
18 Going beyond the obvious (contd) E.G.: Level of information with the support to further enlargement Vertical % Total EU 25 Feel well informed about enlargement Feel badly informed about enlargement FOR further enlargement 45%53%41% AGAINST further enlargement 42%41%43% DK13%6%16% 2 hyp.: either good information produces support to enlargement; or people who already support enlargement are more interested in this topic and tend to seek more information.
19 Going beyond the obvious (contd) Be careful to redundancy. The same persons are present in other sub- groups. Examples given: - Managers have similar answers to those who finished their education after the age of 20. Of course… they are often the same persons. - Less educated persons have similar patterns of responses than those aged 65+. It does not bring added value
20 Significant differences A difference between two percentages has to be significant if one wants to comment it.
21 Correlation does not imply causality There is a positive correlation between two variables if they are moving in the same direction. This does not mean that one variable is the cause (or the consequence) of the other. Example given: - People have a good image of the EU and feel happy. On the contrary : - People acknowledge to have received better information on one specific issue and objectively know more things about the topic Assumptions can still be made
22 Scales Commentators tend to forget what is behind a sub-total. It is essential to have a closer look at the distribution of responses. 65% is not equal to 65%
23 Multi-answers vs. single answer possible The same written percentage can have a different weight depending on how many answers the respondent could give. When analysing multi-answers questions, it is possible to process the data on the basis of responses and not respondents.
25 EU Constitution Surveys prior to referenda in France and the Netherlands showed that: Reasons for supporting the NO were clearly more concrete than those mentioned by supporters of the YES. Other public opinion indicators (like public opinion mood, support for government in place…) were not helping the YES Historical indicators should have been taken into account: - Citizens in the former Member States were not consulted for the various waves of enlargement. Especially not for the last one that took place in 2004. - Very close win of the YES in France in 1992 Clear signs of fragility
26 Threats to peace in the world One of our survey was clearly misinterpreted by the media. Original question: For each of the following countries, tell me if in your opinion, it represents or not a threat to peace in the world Interpretation made: EU poll: Israel 'biggest threat' to world peace Over half of Europeans think Israel poses the "biggest threat to world peace," according to a controversial poll commissioned by the European Commission. Source: WorldNetDaily.com YESNODK Israel59%37%4% Iran53%41%6% North Korea53%40%7%
28 Changing communication plans Surveys can identified new themes, new issues, new messages and new targets to work on. Results showed recently that European citizens made a difference between information sources legitimate to talk about a certain topic and sources they are using to get informed. Potential actions: Mobilize these legitimate sources and make them intervene in traditional media. Focus on some particular publics, geographical areas. Use new messages, simplify them
29 Going forward EU Constitution: Recent surveys about the future of Europe showed a significant support for a EU Constitution (even in France and the Netherlands). Results also clearly indicate that amongst supporters of a Constitution, there is a need to renegotiate and adapt the content of the current draft to include more political and social aspects. Further enlargements: EU citizens call for more information and communication about benefits and challenges of the process (especially in economical and social terms) National Governments and media have a crucial role to play
grazie mille! thank you! vielen Dank! tack! merci! bedankt! muchas gracias! kiitos!