Applied Opinion Research Training Workshop Day 3

Preparing to Conduct Research Sharon Felzer

Conducting Research Going into the Field  Prepare Materials Translate and back-translate questionnaires and guidelines Provide sufficient copies, return mail envelopes, etc.  Determine Schedule Do not field right before or during holidays Allow enough time to recruit sample, conduct research, analyze data, and prepare reports

Conducting Research Going into the Field Gather Your Resources  Translators  Local contacts to recruit samples  Local interviewers/moderators  Transcribers/Data entry  Data analysts  Contractors (ESOMAR Website)

Conducting Research Instrument Quality Control  Vet among core research team Be sure research objectives will be met  Pilot test Be sure participants will understand instructions, questions, and skip patterns Be sure length is appropriate Be sure guideline encourages discussion  Back-translate Be sure questions ask what they were intended to ask

Conducting Research Avoid Common Pitfalls  Poorly designed instruments Research is only as useful as the instruments used  Poorly recruited sample Findings are highly dependent on the sample used  Unskilled interviewers/moderators  Generalizing beyond research/sample  No buy-in  Poor timing of research  Poorly designed Terms of Reference  Cost overrun

Conducting Research When to Bring in Contractors/Experts  Language barrier  Cultural barrier  Large or multiple samples  Local contacts needed to encourage participation  Preventing bias  Complicated research objectives

Analyzing and Reporting Results Mary McIntosh

Analyzing & Reporting Results Essential Components of Successful Research:  Appropriate sampling  Valid instruments  Accurate translations  Skilled interviewers/moderators  Effective timing  Accurate data entry/transcriptions  Appropriate analyses and reporting  Reasonable response rate

Analyzing & Reporting Results Analyzing Data Use statistics appropriate to responses  Rating scale: means and standard deviations or ranges  Dichotomous and multiple choice:frequencies of respondents ’ responses  Ranking: counts of #1 ratings, etc.  Open-ended/Qualitative: no statistics appropriate unless transcripts are coded

Analyzing & Reporting Results Analyzing Quantitative Data Pay attention to :  Sample base  Number of respondents –The smaller the n, the less stable the parameter estimates  Mean or frequency of response  Standard deviation –The larger the standard deviation, the less reliable is your estimate of the mean  Test of significance –If the test is not significant, you cannot say it is a significant difference

Analyzing & Reporting Results Analyzing Quantitative Data Tests of significance Tells you whether differences in numbers are meaningful, or significant  Most commonly conducted on frequency and mean data  A significance test of p <.05 tells you that there is less than a 5% chance that this difference in mean responses is due to chance  A test of significance must be conducted before you can say that a difference exists  However, if a test is significant, it is not necessarily meaningful

Analyzing & Reporting Results Group 1 –Mean = 3.00 –Std. Dev. = 1.00 –95% of respondents ’ ratings between 1 and 5 Group 3 –Mean = 3.00 –Std. Dev. = 2.00 –95% of respondents ’ ratings between 1 and 7 Group 2 –Mean = 7.00 –Std. Dev. = 1.00 –95% of respondents ’ ratings between 5 and 9 Group 4 –Mean = 7.00 –Std. Dev. = 2.00 –95% of respondents ’ ratings between 3 and 10 Analyzing Quantitative Data Tests of significance Not different Significantly different

Analyzing & Reporting Results Group 1 –45% agreed –N = 200 Group 3 –45% agreed –N = 200 Group 2 –55% agreed –N = 300 Group 4 –55% agreed –N = 100 Analyzing Quantitative Data Tests of significance Not different Significantly different

Analyzing & Reporting Results Analyzing Qualitative Data Pay attention to:  Sample base  Trends in responses  Potential group differences in responses  Any potential bias in responding

Analyzing & Reporting Results Reporting Results Keep in Mind:  Your data is from a sample of the population only –Be very careful about generalizing your results to the entire population  Only a subset of your intended sample actually took part in your research –Low response rates suggest potential response bias  You may want to consider weighting your data to correct for bias

Analyzing & Reporting Results Reporting Results Keep in Mind:  Your questionnaire or guideline asked a finite number of questions –It is quite possible that you did not ask about an important factor  Despite your best efforts, there may still have been biases –Researcher biases (more likely in qualitative) –Sample biases –Cultural biases

Analyzing & Reporting Results Reporting Results Keep in Mind:  Although the data from quantitative data seems quite scientific, it is still subject to interpretation  Reporting attitudes and opinions is not the same as reporting facts

Analyzing & Reporting Results Reporting Results Reporting the Sample:  The population  Sampling frame  Sample design  Rationale for sample design  Sample size  Response rate  You may want to report the number of respondents who answered each question if there are high numbers of “ Don ’ t Know ” or “ Refused ” responses

Analyzing & Reporting Results Reporting Results Reporting the Sample:  What if your response rate is lower than expected or what is traditional for that type of research in that country?  Report that this may be a potential limitation. For instance, some respondents may have been discouraged to participate by superiors. Therefore, the sample of respondents that did take part in the research may not be representative of the population  If possible, check the characteristics of those who participated versus those who did not

Analyzing & Reporting Results Reporting Results Reporting the Sample Because your data is only from a sample of the population, you must be very careful about generalizing your results to the entire population WRONG: NGOs believe the Bank is not effective RIGHT: NGOs who were interviewed believe that the Bank was not effective

Analyzing & Reporting Results Reporting Results Quantitative Data:  Although the data from quantitative data seems quite scientific, it is still subject to interpretation –A mean of 7 on a 10-point scale may be a somewhat positive response in one context (e.g., culture) or a very positive response in another context (e.g., culture) –Don ’ t assume scales are uniform and consistent across topics and cultures

Analyzing & Reporting Results Reporting Results Quantitative Data:  Part of your interpretation involves deciding how to report the data –Mean scores »Respondents from NGOs gave Bank effectiveness a mean rating of 7.5 –Aggregate frequencies (e.g., High/Medium/Low) »35% of respondents from NGOs gave a high rating for the Bank ’ s effectiveness –Response frequencies »25% of respondents from NGOs gave a rating of 6 for the Bank ’ s effectiveness

Analyzing & Reporting Results Reporting Results Quantitative Data :  Report the data in the way that your readers will be most likely to understand  Include appropriate charts, graphs, or tables to represent the data pictorially to assist your readers

Analyzing & Reporting Results Reporting Results Qualitative Data:  Because this data is highly subjective and completely dependent on the particular sample you have drawn, you must be very careful in reporting qualitative findings WRONG: The Bank program failed because it was poorly designed RIGHT: Respondents in a beneficiaries focus group reported that one potential reason the Bank program was not that effective was that the design was not optimal given the situation  Always include the caveat that your findings are not necessarily representative of the population (or the +/-)

Analyzing & Reporting Results Reporting Results: Right & Wrong –Finding: When asked to choose what the Bank ’ s greatest value was, 25% of respondents (the largest %) chose financial resources WRONG: The Bank ’ s value is only in its financial resources RIGHT: A quarter of respondents chose financial resources as the Bank ’ s greatest value. This was followed by …

Analyzing & Reporting Results Reporting Results Right & Wrong –Finding: When asked about the Bank ’ s overall effectiveness, NGO ’ s mean was 6.5 and private sector ’ s mean was 7.8, a significant difference WRONG: NGOs do not think that the Bank is an effective organization. In contrast, people in the private sector think that the Bank is highly effective RIGHT: Respondents from private sector organizations rated the Bank ’ s overall effectiveness significantly higher (7.8) than respondents from NGOs (6.5).

Analyzing & Reporting Results Reporting Results Right & Wrong –Finding: When asked about the Bank ’ s overall effectiveness, the 3 media respondents gave a mean of 9.0 WRONG: The media think that the Bank is very effective RIGHT: Although respondents from the media rated the Bank ’ s overall effectiveness quite high (9.0), there were only three media respondents, thus, these results are suggestive at best »Another alternative is to not report results from samples sizes that you judge too small to be reliable

Analyzing & Reporting Results Reporting Results Right & Wrong –Finding: In a focus group of medical professionals, several said that the Bank needs to initiate a vaccine program WRONG: The Bank needs to initiate a vaccine program RIGHT: In a focus group of medical professionals, it was recommended that the Bank initiate a vaccine program

Hands On Work: Analyzing and Reporting Results