Utilizing Evidence to Guide Program Implementation The Use of Routine Lot Quality Assurance Sampling (LQAS) Surveys at Household Level Kironde S, Ekochu E, Businge D, Nkwake A, Kiracho G John Snow Inc./Uganda Program for Human and Holistic Development (JSI/UPHOLD) XVII International Conference on AIDS, August 3-8, 2008

Purpose of the Presentation To briefly introduce the Lot Quality Assurance Sampling (LQAS) survey methodology To share other illustrative results from LQAS surveys and how these have been utilized in various technical areas JSI/UPHOLD has supported To share lessons learned, benefits and challenges

LQAS is a simple, low cost random sampling survey methodology… Refers to ‘Lot Quality Assurance Sampling’ Introduced in 1920s as an adaptation for quality control in industry In Public Health, can be used locally at the level of a ‘supervision area’ as is subsequently illustrated Can provide an accurate measure of coverage or health system quality at a more aggregate level (e.g., program catchment area, district or refugee camp) Can enable targeting of interventions within a district

What are the LQAS Principles? County A County D County E County C County B Assume a program covers a whole district Each county is then called a ‘Supervision Area’ and the district a ‘Supervision Unit’ LQAS utilizes a minimum of 19 items (e.g., households, schools, health units) from each ‘supervision area’ in order to assess an indicator

What can LQAS tell us? Good coverage Below desired coverage Indicator: Percent of children under age five who slept under a mosquito net the night before the survey Good coverage County A = 80 County D = 20 County E = 45 County C = 85 County B = 35 Below desired coverage

What can be learned from the findings? Good and above established benchmark Study and learn what is working well Identify what can be applied to other Supervision areas Learn causes of low coverage Focus efforts and resources on these supervision areas Poor or below the established Benchmark Improve coverage of whole program area by improving coverage in these SAs

Normally, a sample size of 19 is utilized from each supervision area… Little is added to the precision of the measure by using a sample larger than 19, notwithstanding the level of coverage to be assessed Sample sizes less than 19 however, see a rapid deterioration in the precision of the measure. This is particularly problematic when coverage benchmarks vary

Why LQAS was chosen in Uganda Decentralization in the 1990s brought a new set of demands and challenges as planning and implementation of social services became a core mandate of districts Existing information systems had certain unique challenges e.g.: The Demographic Health Survey (UDHS) though household based is every five years (long interval for projects) and does not provide district-specific information The Health Management Information System (HMIS) is mainly facility-based, which presents potential problems with representativeness in the general population

LQAS implementation is representative of all regions in Uganda UPHOLD Supported District Key:

JSI/UPHOLD has institutionalized LQAS at district level for evidence-based planning and decision-making District capacity to annually collect data for planning, monitoring and evaluation using LQAS has been enhanced: 303 district staff in 37 districts have been trained to use the methodology Four LQAS surveys have been carried out (2004, 2005, 2006 and 2007) hence making trend analysis possible for each district Annual presentation and discussion of district-specific LQAS results is conducted in each participant district User friendly district and county/sub-county specific data is now available for evidence-based planning and tracking of progress across a wide range of parameters Over time other Development Partners have also participated in the annual LQAS surveys – something which shows buy-in for the methodology

Illustrative Results

Uptake of HIV Counselling & Testing Services can be tracked Percent of adults who tested for and received their HIV test results Trained ~18,400 service providers Increased HIV testing sites from 32 to 683 (including outreaches) Provided grant support to AIC Awarded 29 grants to CSOs to promote HIV/AIDS related activities Awarded HIV/AIDS grants to 28 district local governments Procured and distributed 1,331 bicycles to support community mobilization Used BCC to mitigate against stigma, gender based violence and other factors that hinder uptake of HCT services Supported the formation and functioning of over 1,700 Radio Listening Clubs reaching ~11m people 62% of individuals received HCT from outreaches or their homes Support supervision of HIV/AIDS services with TB, STIs and malaria Increased from; 982,189 in 2004 to 1,759,941 in 2007

Malaria Interventions at Household level can also be tracked Percent of children under-5 years sleeping under a treated net in the previous night UPHOLD has supported the re-treatment of 511,990 nets in 6 original UPHOLD supported districts in 4 rounds of net re-treatment (2004-2007)* UPHOLD has distributed 311,311 insecticide treated nets to under fives since 2004 Increased BCC campaigns on use of ITNs through LG and CSO grants as well as Radio Listening Clubs In 2007, GFATM financed the procurement and distribution of ~1.8m ITNs – some of them n UPHOLD supported districts Increased from; 262,117 in 2004 to 908,852 in 2007 * DFID and GFATM supported the Net re-treatment Campaign of 2005

Fewer Children Under 5 Years Reported Fever (2004-07) Percent of children under 5 years who had fever in past 2 weeks prior to the survey Increased BCC campaigns on use of bed nets and other forms of malaria prevention By 2007, ~54% of households owned a net of any kind in UPHOLD supported districts (c.f. 34% nationally [UDHS, 2006]) UPHOLD supported four net re-treatment campaigns to increase efficacy of bed nets (as per previous slide) Children with fever have decreased from 1,250,095 in 2004 to 724,597 in 2007

Deliveries in Formal Health Sector can be tracked Percent of deliveries in last two years preceding the survey that took place in a health facility Trained ~1,862 health workers on Reproductive Health Awarded grants to civil society (e.g., UPMA) to train private midwives and to provide services Supported five grantees to mobilize communities for Reproductive Health services Procured and provided about 26,000 ‘Mama Kits’ to pregnant women in the ‘conflict’ districts of Northern Uganda

LQAS can been used to estimate Bed Net requirements at district level

Knowledge of population size and ITN coverage can accurately future estimate demand District Estimated Population of <5s (2006) % ITN Coverage by Sept. 2004 (LQAS) % ITN Coverage by Sept. 2005 (LQAS) # of ITNs distributed by UPHOLD Dec 2005 – Mar 2006 Estimated % increase in bed net coverage in <5s Estimated % new ITN coverage after distribution Actual ITN coverage as per 2006 LQAS Survey p Value Bugiri 100,104 4.2% 19.6% 13,000 12.1% 31.7% 32.1% 0.980 (ns) Bushenyi 159,920 9.4% 22,800 14.1% 23.5% 25.8% 0.518 (ns) Mayuge 75,259 2.1% 4.7% 10,000 13.1% 17.8% 18.4% 0.981 (ns) Rukungiri 58,993 3.7% 10.5% 9,931 16.7% 27.2% 23.7% 0.513 (ns)

Graphical Illustration The 2006 LQAS Survey validated the accuracy of the intended coverage estimates

LQAS results compare well with other standard Survey Methodologies SAMPLE INDICATORS LQAS results compare well with other standard Survey Methodologies Comparison Indicator Household Survey HIV/AIDS LQAS 2004 UHSBS* 2004-05 % Adults (15 years and above) who reported being tested for HIV and received their results 16 14 Malaria LQAS 2006 UDHS** 2006 % Children under five with fever in two weeks preceding the survey 43 41 % Children with fever who received timely treatment 77 75 % Households with a bed net of any kind 39 34 * UHSBS = Uganda HIV/AIDS Sero-Behavioural Survey 2004-05 **UDHS = Uganda Demographic and Health Survey, 2006

Benefits of LQAS as a Survey Methodology Low sample size needs (n=19 in most cases) Simple to apply yet has very specific conclusions District level people can be trained to entirely ‘own’ this methodology Provides reliable information at low & affordable cost Fast – ‘supervision areas’ are able to conduct self-evaluation and obtain results immediately after the survey Results are locally relevant and can be utilized in district level annual planning and decision-making

Benefits of LQAS to Districts Change of mindsets towards reliance on data for evidence-based planning Districts have relied on LQAS data to identify priority target areas for implementation and to focus work plans Districts can use LQAS to do own evaluations in different technical areas e.g., education, agriculture etc. Strengthened partnership between districts and NGOs in provision of related services

Challenges Initial costs for training and capacity building may be quite high Being a relatively new methodology, there are concerns about the validity and reliability May require significant district personnel time and hence requires appropriate pre-planning Cannot answer the ‘why?’ Follow up studies are required to establish the reasons for low coverage/poor performance

Acknowledgements JSI/UPHOLD would wish to thank the following for their contribution to this work: USAID for provision of funding District Local Governments for participation in the surveys UNICEF, NUMAT for participation and part funding of some of the LQAS Surveys All respondents at household and facility level