1. Monitoring and Assessing Pharmaceutical Policies
Mr Enrico Cinnella
WHO/EMP, Geneva
18 November 2009

2. Outline of the Presentation
Introduction
Level I
Level II
Facility Survey
Household Survey

3. Why countries measure? As a baseline to inform decisions
Priority Setting
To check how well (or badly) you are performing
Two reasons to measure:
Fist one is at the beginning of the cycle: to decide what to do.
Second one is during the implementation: to see whether results are being achieved.

4. Country A Knowing the situation RATIONAL USE Last update of EML: 2009
Last update of STGs: 2008
Antiobiotics sold over the counter: NO
QUALITY CONTROL
System in place for quality control: NO
Samples tested for post marketing surveillance: NO
System in place for ADR: NO
Collecting information to have an overall view of the situation of the pharmaceutical sector.
Without that it is difficult to decide in which area to work and what to do.

5. See how well you are doing
Monitoring the availability of medicines is a way of measuring whether policies are achieving their expected results.
Linear monitoring across a period of time.
It can be seen how availability in the Public Sector is increasing over time.
Mention sources of data

6. Who can use the results ?
Countries - focus action, prioritize, measure achievement
International agencies
to assess the structure and capability of countries, assess the progress, accomplishment and impact of aid
Professional groups, NGOs and academia
to focus advocacy activities and information campaigns
Health facilities to be aware of institutional problems & improve situations
Countries -Country budget planning % basis to request fund and support using data and evidence.
Donors are now keen to measure progress and output. WHO EC log frame is supported by indicators with target
Presentation, report, discussion , debate, backed by data.

7. Pharmaceutical indicators
Variables that measure situations and change
Numerical ( numbers, percentage, or averages)
Binomials (yes” and “no)”
Useful tools to track the performance of particular aspects or activities of the pharmaceutical system
Linked to an important input, process, or outcome
may be numerical and expressed in terms of numbers, percentage, or averages. They may also be expressed as binomials such as “yes” and “no.” Indicators are useful tools for managers to use to track the performance of particular aspects or activities of the pharmaceutical supply system as well as the performance of the overall system. A well-defined indicator is clearly linked to an important input, process, or outcome. A well-selected indicator will help managers quickly identify potential problems in critical areas. Indicators are extremely helpful to communicate important performance gains and losses to other stakeholders of the pharmaceutical supply system. Indicators can be developed for different levels of the supply system.
Managers should be aware that there are well-established indicators to measure the performance of the different components of the pharmaceutical supply system. However, because systems can be organized in different ways, managers should adapt or modify internationally recognized indicators to reflect the realities of their own system if necessary.
Field test idicatros before final includsion in the set and at countries with those who will collect the data

8. The WHO System Level I Questionnaire/rapid assessment/checklist
Arrays achievement & weaknessess, illustrate sectoral approaches
Level II
Comprehensive monitoring of pharmaceutical strategy outcome and impact
Measures attainment of objectives
Level III
More detailed indicators for monitoring and evaluating specific areas/components
Questionnaire
(Health Officials)
Level I
Core structure
& process indicators
Level I
Core structure
& process indicators
Systematic
survey
Level II
Core outcome/impact indicators
& household survey
Level II
Core outcome/impact indicators
& household survey
Level I- Global overview of policies, structures and process
Level II- Results and Outcomes through surveys conducted at facility and household level
BOTH COVER the whole spectrum of EMP work
Level III goes into detail of individual topics
Level III
Indicator tools for specific components
of the pharmaceutical sector
Pricing ●Traditional medicine
Human Resources ● Assessing regulatory capacity
Procurement and Supply

9. Core outcome/impact indicators
LEVEL I
Level I
Core structure
& process indicators
Level II
Core outcome/impact indicators
& household survey
Level III
Indicator tools for specific components
of the pharmaceutical sector
Pricing ●Traditional medicine
Human Resources ● Assessing regulatory capacity
Procurement and Supply

10. Level I- A Global Survey
Questionnaires sent to MoH officials every four years (1999, 2003 and 2007). Mostly yes/no question.
Responses collected and compiled into a global database and global report.
Questionnaire developed and send to MoH at regular intervals.
Explain well how the questionnaire is sent through official channels- Regions- Countries- MoH
Simple questionnaire, mostly yes/no question.
Easy analysis.
Global report is produced, but regions can and do produce their own reports.

11. Level I- What does it cover?
Policies and Structures in 6 areas:
Medicines Policies
Regulation
Procurement and Supply
Financing
Production and Trade
Rational Use
The whole Pharmaceutical situation is covered

12. Advantages of the level I
Little Financial and HR Investment.
Number of countries covered (156 in 2007).
Allows
An overview of the global situation
Comparisons across regions
Comparisons across time
Financial investment is close to O
Time investment is limited compared to other surveys and studies.
Most member states are covered.

13. The Global situation in 2007 Official NMP document updated within the last 5 years
It allows to have an overview of the global situation, but also to compare across regions.
Most African countries have a NMP document, but not many of them have updated it recently. This may change soon as many countries are about to deliver (Mali, Uganda, Tanzania).

14. Have we made progress? Standard Treatment Guidelines 2003-7
Level I allow to compare the situation every 4 years.
Progress has been made in all income levels. Low-income countries have performed as well as high income ones. In this case the African Region is doing particularly well.

15. Limitations of WHO Level I
Data quality
Ownership of the data
Does not tell anything about outcomes and impact
Data quality not always great. The level of respondents varies from country to country. Difficult to come back to them due to number of countries.
Ownership of data. Countries give data, but them to access them have to write (and know that we have them). They make an effort to put information from different sources together, but do not benefit fully from this effort.
Outcomes. Having STGs is fine, but what really matters is whether treatment of patients does follow them or not. Level I does not tell us anything about it.

16. The SADC experience
Assessment of Pharmaceutical situation in 15 countries
Tools similar to level I, but including indicators from other surveys
Main changes in the process:
Tool prefilled (more speed, less effort and duplication).
Data quality- official endorsement.
Data ownership- data used to build Profiles to be shared with countries. Countries will have all their information in a single document.
Geographically Limited assessment.
Tools similar to level I, but pulling together info from other surveys to give a more complete picture.
Tool prefilled: less burden on countries, if we have the information we will not ask again for it, only to confirm.
Official endorsement should better guarantee the quality of data.
Information used to build profiles that will be sent back to countries. Countries will then have all their pharmaceutical information compiled in a single document.

17. Core outcome/impact indicators
Level II
Level I
Core structure
& process indicators
Level II
Core outcome/impact indicators
& household survey
Level III
Indicator tools for specific components
of the pharmaceutical sector
Pricing ●Traditional medicine
Human Resources ● Assessing regulatory capacity
Procurement and Supply

18. Level I vs Level II LEVEL I Single Questionnaire Little Investment
Policies and Structures
Based on EMP's strategy and vision
LEVEL II
Survey with sampling
More expensive and time-consuming
Outcomes
Based on EMP's strategy and vision
Single questionnaire- Survey-same questions asked in a number of facilities/setting.
Expensive:60 days (10x6) or 150 days (10x6)
Structures vs Outcomes
Both of them covered the full pharmaceutical situation, from affordability to rational use.

19. From Structures to outcomes
Level I- Is there a EML?
Level II- Is the EML available at facility level? What is the percentage of medicines prescribed that are in the EML?
LEVEL II- Are policies achieving their effect?
Availability- of a list of tracer medicines, and number of stock out days.
Affordability- (number of days of pay to purchase treatment for selected illnesses).
Quality- % Adequately labelled medicines, % expired medicines.
Rational use- INRUD Prescribing indicators.
Structures and Outcomes
INRUD: number of medicines prescribed per patient, % of patients given antibiotics, % of patients given injections; % of medicines prescribed in generic name; % of medicines prescribed that are in the EML,
What is covered by Level II

20. Facility Survey
5 Areas selected- The Capital (or richest), the most rural one (or poorest) and another 3.
In each Areas select 1 Warehouse, 6 Public Health Care Facilities, 6 private drugs outlets. 5
Warehouses 30
Public Health Care Facilities
Private Medicines Outlets
Sampling of facility survey. May be adapted to country situation (e.g. Tonga)

21. Some results Results from Uganda. High antibiotics use.
Most medicines are in the EML
Generic Prescription good.

22. Household Survey

23. The household survey sampling scheme
5 regions in the country
From each region select 6 public health facilities (30 reference public health facilities)
In each of reference facility, select 30 households (900 households) 8

24. Household Survey Household situations
How they access their medicines, where they get them
How much they pay
Identify access and affordability in relation to socio economic indicators, barriers
Examine use of medicines (acute and chronic diseases)
Perceptions on access, use and quality; handling of medicines
Covers all areas of Medicine Policy- Again from Access to Rational Use.
Household perspective.
Allow more than facility survey- How many are accessing medicines? From where are they buying/getting them?
Gives an idea of how people are doing in terms of access to medicines, but also on their opinions and perceptions.

25. Households with medicines at Home
Allow to see whether different socioeconomic groups have differential access to medicines. In this case, richer people are more likely to have medicines at home.

26. Geographical issues COUNTRY A
Do you receive free medicines at public health care facilities?
95%
56%
Allow also to see whether there differences between areas of the same countries.
Here, the further we go from the capital, the less likely people are to get free medicines from public facilities?
96%
97%
77%

27. Medicines Expenditure
Putting countries together is a good way of benchmarking and have an idea of how well you are doing.
Here we can that catastrophic expenditure on medicines is very different in countries that have a different financing system (free for B, completely paying for C).

28. Source of medicines
Source of medicines, may be public facilities or private retailers.
Country A private pharmacy.
Country B Public Dispensary.
Country C: Public Hospitals and Private Pharmacies
This is very useful to see where to aim effort to control quality/promote rational dispensing.

29. Tools and Reports Level I Assessment 2003 Manual Level II Assessment
Manual Level II Assessment
Uganda Facility Survey 2008
Uganda Household Survey 2008

30. Thank you for your attention