1. Soul City: Institute for Health and Development Communication
Monitoring and evaluation for social and behavioural change communication
Dr Renay Weiner
Soul City: Institute for Health and Development Communication

2. Outline
Soul City: background What do we measure? Selecting indicators Approach to evaluation Issues and challenges

3. Soul City: mission
Mission is to make an improvement in people's health and quality of life by:
Harnessing the power of the mass media
Developing high quality education material through thorough research and evaluation involving the communities we serve
While remaining an organisation not for gain, ensure the sustainability of our  effort through sound business principles and practices
Being strong advocates for health and development
Measuring impact on health and development

4. SC model of SBCC
Effect change at individual, community and societal levels
Methods: MEDIA; SOCIAL MOBILISATION and ADVOCACY
Learning and Reflection; self-efficacy
Draw on different theories of SBCC

5. 5 Pillars of Health Promotion(Ottawa Charter):
Create enabling environments
Develop personal skills
Advocate for healthy public policy
Re-orientate services
Stimulate community action

6. Social Cognitive Theory
A person’s behaviour is influenced by :
Personal factors (cognitive, affective, biological events)
Environmental factors
Social Learning Theory
Modelling
Self efficacy
People learn by observing behaviour of others and the outcomes thereof
People adopt behaviours if they believe they are capable.
Important theory specifically for edutainment because it includes concept of traveling on a journey with characters and seeing the results of risky / negative behaviour and can role model positive behaviours and outcomes)
In many instances, without social change, individual behaviour change, is almost impossible
Soul City aims to impact on individual, community and societal levels to enhance those things that help behaviour change, and decrease those things that hinder.

7. Changing behaviour is a complex endeavour
In many instances, without social change, individual behaviour change, is almost impossible
Soul City aims to impact on individual, community and societal levels to enhance those things that help behaviour change, and decrease those things that hinder.

9. Soul City Institute for Health and Development Communication
, Soul City is tackling two important social and public health issues in South Africa: HIV/AIDS and violence.
Campaign approach
Onelove campaign: SA and regional
PhuzaWize: SA

10. example of regional level OL

11. Soul City: interventions and activities
MEDIA
Soul City TV drama: 13 episodes
Soul Buddyz TV drama
Soul City radio drama: translated into all languages
Other: Kwanda; Love Stories, BOM
Print booklets
SOCIAL MOBILISATION
Soul Buddyz programme
Adult training and community dialogues
ADVOCACY

13. What do we monitor and evaluate?
Need to consider objectives of programme/project
REACH OF Onelove
REACH numbers and % by age and sex geographical location province and geotype

14. Selecting indicators: deciding what to measure
MEDIA
SOCIAL MOBILISATION
ADVOCACY
Social Knowledge
Community---Intervention --- Attitudes Behaviours---- Prevalence
Individual Self-efficacy Incidence
INPUT OUTCOME IMPACT
PROCESS OUTCOME IMPACT
OUTPUT

15. Selecting indicators Example: HIV prevention
Intervention
(output)
Knowledge
Attitudes
Self efficacy
Behaviours
Prevalence/Incidence
% of adults exposed to the TV/radio/print intervention
% of adults with accurate HIV prevention knowledge
% of respondents who used a condom at last sex
HIV prevalence amongst pregnant women aged years
% of respondents who agree that a woman can refuse to have unprotected sex with her husband if he has an STI

16. Approach to evaluation: quantitative and qualitative approaches
Qualitative questions
Relate to `quality’ of
variations in experience or the
meaning of experience for different people
Classify events or phenomena
How?
Why?
Reasons for abstaining---lack of opportunities versus choice to self-protect
Quantitative questions
Relate to `quantities’
Count or measure phenomena
How many?
What proportion?
How often?
eg % of youth that abstained in past year

17. What do we monitor and evaluate?
PROCESS EVALUATION
How was the intervention implemented?
Was the intervention implemented according to plan?
Under what conditions was implementation successful?

18. What do we monitor and evaluate?
IMPACT/OUTCOMES Individual level knowledge attitudes/beliefs self-efficacy behaviours Community level -Social norms -Community cohesion/social capital -Civic action -Sustainability

19. Evaluation: quantitative and qualitative approaches
Aims to quantify research findings using numbers, %, ratios etc
Quantify frequency of disease/health events, risks, measures associations
Standardised procedures decided in advance
Samples used to approximate status in the population
Measures are estimates, with defined levels of certainty eg HIV prevalence: 10% (95% CI:8-12%)
Random sampling allows generalisability to population
Sample sizes large enough to ensure precision of estimates

20. Impact evaluations of SBCC require household surveys

21. Evaluation: quantitative and qualitative approaches
Concepts presented in written words and images
Good for explaining how and why; good for exploring beliefs and behaviours
Methods and Qs can change as data collected; can measure impacts that have not been predicted and generate new ideas
Sampling is purposive---data not statistically generalisable
Sample sizes small
Analysis: extract themes, organise data into coherent findings
“Yes, Desert Soul has made a difference to my life because I used to have many boyfriends. When I read this book I have changed. That means that I have dumped all my boyfriends because I read that it is dangerous to have many partners that increases the risk of being infected by HIV”. (Rural female, 26-35; FG 8)

22. Results: Impact - Behaviour

23. Results: Impact - Behaviour
“I have realised the importance of protected sex and have since refrained from sex without a condom. The booklet also influenced me to go for voluntary counselling and find out my HIV status” – (Lesotho)
“I never used to like condoms but now with my girlfriend we buy and use” (Zambia)

24. Quantitative approaches: RCT and evaluating mass media intervention
Cannot allocate intervention- implemented via national public media
Cannot restrict intervention-aimed for maximum reach of largest possible audience
Intervention tested in development phase through target audience research
Recent systematic review effect of communication programs on HIV/AIDS—no randomised studies for full coverage programmes (Bertrand et al)

25. Evaluation methodologies: quantitative approaches
What study design to use to evaluate impact of health communication interventions?
Observational
Cross-sectional, case-control, cohort
Intervention
Quasi-experimental (pre-post),
RCT (individual/cluster) `GOLD STANDARD’
randomisation

26. Evaluation methodologies: quantitative approaches
What study design to use to evaluate impact of health communication interventions?
Consider: level of implementation of the
intervention
SC intervention
Level of implementation
MEDIA- TV
Population
MEDIA-radio
MEDIA-print
Individual (support other media)
SOCMOB-Soul Buddyz; community training
Individual/family/schools
ADVOCACY

27. Quantitative approaches: alternative study designs
Cross-sectional analytic
exposure to SC measured
impact compared in exposed and non exposed;
dose response relationships measured
reverse causality, need to control confounders
Cohort
follow-up same group over time
impact compared in exposed and non exposed
3. Pre-post/quasi-experimental
baseline for comparison

28. Evaluation: qualitative methods examples
Focus Group Discussions: facilitator guides group; participants have similar characteristics
Interviews in-depth: guided by the respondent
semi-structured: loose set of questions
Ethnographic observation: immersion in another’s culture
Most Significant Change methodology

29. Participatory M and E AIMS
To generate data on activities being carried out at the local community level
To increase beneficiaries commitment and understanding in planning and implementing community based development projects
Examples: Community Dialogues

30. Evaluation: triangulation
Refers to the use of two or more different research methods in combination, principally as a check on validity
SC series 7 and 8
Quantitative data
HIV/AIDS is the result of sinning
Multimedia exposure Agree/strongly agree
None %
Low %
High %
Qualitative data
`To me, to watch SC changed my whole life because I was affected with HIV and AIDS .I was able to teach others to disclose to their partners….’ (female KZN, rural 16-25)

31. Evaluation: triangulation
Routine data:
National media monitoring
Television Audience Measurement Surveys
2. Helpline data
AIDS helpline

33. Community responses Letters and newspaper articles

34. Issues and challenges Multilevel intervention—multiple sources of data
New activities---new indicators:
dynamic M and E plan
Reach
Media monitoring: data limited especially radio; cannot tell if same or new viewers
Survey data better but infrequent

35. Monitoring media: special issues
Need ongoing monitoring data to estimate reach
Two sources of mass media monitoring data:
SAARF media monitoring
Response to advice lines
Print media monitoring: distribution data

36. Media surveys: SAARF South African Advertising Research Foundation
1.AMPS: All Media Products Survey
Annual national household survey; F2F interviews
sample of adults
TV viewing information: past 4 weeks; past 7 days, yesterday
2. TAMS: TV Audience measurement survey
`Peoplemeters’ attached to TV equipment of sampled households to monitor what is being watched
Every household member must log onto the system when viewing
Measures viewership patterns at home only
Daily reports provided

37. Issues and challenges REACH
Print distribution as a proxy for booklet reach—survey data better
Training data: from partners —m and e support/quality audits; feedback
IMPACT
For mass media cannot do RCT
Costs of surveys

38. Conclusion
Monitoring and evaluating social change communication complex
Quantitative and qualitative approaches complementary
Alternative study designs to RCT needed for evaluating media interventions