Analyzing your Survey Data: The Impact of the Campaign

KAP Data analysis Part 2: BR, BC, TR and CR

By the end of this lesson you will be able to: Analyze your data according to your analytical plan (BR, BC, TR, CR) Analyze your data based on your reporting needs Complete Sections 3.4 B – D of your Campaign Learning Report that identify the impact of your campaign along the Theory of Change

Structure for the Day Check in9:30 – 10:00 AM Individual Work time10:00 – 12:00 Check in1:30 – 2:00 PM Individual Work time2:00 – 3:00 Check in/Assignment3:00 – 3:30 4

Assessing Impact Provides an overview of the results of the campaign along the: – BR SMART Objectives – BC SMART Objectives and – TR SMART Objectives – CR SMART Objectives

Flow/Guidelines 1.Review of your KAP Monitoring Plan (BR, BC, TR, CR) and Hypothesis 2.Complete the tables in your section 3.4 per Target Audience 3.Write a short narrative after each table describing how well your campaign achieved its SMART Objectives 4.Assess the impact of your campaign along TOC

Analyze Data: Statistics clarify truth Show critical data in body of report (SMART Objectives) –Tables –Charts Show all your data in tables in Appendix: –Honest thing to do –May show you where your campaign failed so you can fix it in future!

Analyze Data: Comparability of Surveys Best: –All Chi-Square tests < 95% OK: –A few Chi-Square tests ≥ 95% –Differences in frequencies small (5 to 10 percentage points) So-so: –Several Chi-Square tests ≥ 95% –Differences in frequencies large (10 to 20 percentage points) Unacceptable: –Many Chi-Square tests ≥ 95% –Differences in frequencies large (> 20 percentage points)

Worse Case Scenario Assume you have large differences in gender: –Baseline = 35% male –Post-campaign = 56% male –Difference of 21 percentage points Problem: If a dependent variable increases by 15 pp, it could be due to 2 things: –Pride campaign impact –More men in 2 nd sample Solution is to “control” for gender using filters: –Filter for men, run analysis –Filter for women, run analysis

Reminder # 1 For BR, BC, TR and CR, KAP SMART objectives are based on self-reported data. Triangulate results with non-KAP metrics

Analyzing BR Example: BR KAP Question (36) I am going to read you a number of statements about the management of the local no-take area. For each statement, I would like you to tell me if you strongly agree, agree, disagree, or strongly disagree with it. (E) There is enough money and other resources to fully manage and enforce the rules of the no-take area [ ] SA [ ] A [ ] D [ ] SD [ ] NS/DK (G) The rules of the no-take area are unclear and local fishers don't understand them [ ] SA [ ] A [ ] D [ ] SD [ ] NS/DK BR: Non-KAP Measure?

Example: BC Question (43)During the past 6 months, would you say that you have been regularly involved, occasionally involved, or not involved with the creation and/or the management of a no- take fishing area in your local area (A)[ ] Regularly involved [ ] Occasionally involved [ ] Never involved [ ] Don't know / not applicable Analyzing BC BC: Non-KAP Measure?

Example: TR Question (41)I am going to read you a list of different types of fishers, and for each one, I would like you to tell me whether you remember seeing someone like that fishing in this area in the past 6 months (show the NTZ on a map of the area but don't mention whether it is NTZ or not) (A) Subsistence fishers from your village [ ] Seen [ ] Not seen [ ] Not sure / Don't remember (B) Subsistence fishers from nearby villages [ ] Seen [ ] Not seen [ ] Not sure / Don't remember Analyzing TR TR: Non-KAP Measure?

Example: CR Question Has your catch increased, decreased or stayed the same as a result of the Lola Marine Sanctuary? (If the person does not fish or glean mark as NA) [ ] Decreased [ ] Increased [ ] Stayed the Same [ ] N/A Analyzing CR CR: Non-KAP Measure?

Structured Time: KAP monitoring Plan data survey template Proceed to section 3.4A in your campaign learning report SMART Objective ToC Category Metric (KAP or non- KAP) Pre- campai gn result Pre- campaign frequency error (if KAP) Tar get Post- campaig n result Post- campaig n frequenc y error (if KAP) Chi- squared significan ce (if KAP) Change (in pp if applicabl e) Knowledge Attitude Interpersonal Communicatio n

Reminder # 2 ToCSMART ObjectivesMetricMethod Baseline (Pre- campaign) Result (Post- campaign ) Change (in percent age points) Chi- Square (X 2 ) Significan ce SMART Objective Attainme nt Interpersonal communication GENERAL COMMUNITY By September 2010, 25 % of Southern Residents who do not hunt will have spoken with someone about wildland fires in the past 6 months (a 10 percentage point increase from 15%; N=237). Q28: In the past 6 months, have you talked to anyone about wildland fires? KAP Survey Analysis >95 significant 190% 1.GENERAL COMMUNITY (NON-HUNTERS)

Interpersonal Communication Our goal for interpersonal communication was simple. We wanted people to talk to each other about wildland fires, about the causes, the effects, the billboards, etc. We achieved 190% of the objective for the general community, though the data for the hunters was not statistically significant and cannot be assessed. This may be attributed to a very small sample size of this audience in the pre campaign survey. Respondents were also asked if they had heard that there were wildland fires in Guam’s watersheds. While cannot be linked directly to interpersonal communication, it is an interesting question to look at to see if there is an increase in people hearing about fires since the pre campaign survey. The general results of all of the respondents showed an increase from 22% to 45% of people who had heard about wildland fires, indicating that there is indeed more information about fires being communicated. While many of the campaign images focused on the impacts of wildland fires, such as the campaign poster and billboards, they did not directly state that fires were caused by people and that Guam had a very low occurrence of natural fires. The campaign display game addressed this as did the community and school presentations, but it may have been advantageous to include this messaging into more campaign materials. It may also be a good idea to do some supplemental surveys with more direct questions about these important knowledge concepts to get a more accurate representation of what people know.

Reminder # 2 Use the exact SMART objectives language Ex. Strongly agree is not the same with agree SMART Objective TA1 - Fishers The number of Burgos/Uba local fishers who says strongly agree that the Burgos Marine Sanctuary regulations need to be followed will increase from 86.9% measured in February, 2011 to 89.9% in July, 2012 ( an increase of 3pp, Q30F in the KAP survey) SurveyPro result

Scenario There is an increase in reported BC but the SMART Objective target are not met. WHY?

Could it be how you set your SMART target? Historic data Baseline - Diffusion of innovation Type of audience  Increase/Decrease  Maintain

The potential for change for different types of objectives across the Theory of Change 22 CriteriaKAICBC Average Percent of Target Audience Changed 22pp13pp28pp14pp Sample Size (n of campaigns) Historic Data Pride campaign average results summary till 2010

Baseline and diffusion of innovation According to Diffusions of Innovation the Rate of Change Depends on the Starting Point Innovators – 2.5% Early Adopters – 13.5% The Late Majority – 34% Laggards – 16% The Early Majority – 34% Source: Everett Rogers, graph from Wikipedia.org

The potential of change for different types of audiences 24 Baseline\Target Audience (data based on median of knowledge) General Public Influencer Resource User <20%20pp24pp1.7pp 20% to 40%23pp37pp25pp 40% to 60%31pp33pp21pp >60%17pp8pp16pp Selective perception(Hassinger) – people who don’t “want” to know don’t seem to learn. The critical mass phenomenon / social norms

3. Using right words for accuracy The SMART objective should use the same words as the question & as the answer option used

Reminder # 3 Check if you use the right filter. The latest appears in the dropdown menu. Take note of the code of the filters you use.

Stages of behavior question Can it be used to identify the current stage of behavior?

Reminder # 5 Try other figures beside using tables When you are done with your analysis, do a publish report

Reminder # 6 When you are done with your analysis, do a publish report