2 Manual Evaluation: Are we doing what we think we are doing? Associate Professor Dee ManginDirector Primary Care Research UnitDepartment of Public Health and General PracticeChristchruch School of Medicine,University of OtagoYou will be wondering what a family doctor from New Zealand is doing… and at the endI am here from the other side of the world because the issue we are trying to address here is globalORPharmaceutical promotion is the greatest threat globally to rational use of medicines and its influence is increasingNow more than ever we need to to include in the medical and pharmacy school curricula teaching to assist students to understanding the range of promotional strategies and to develop strategies to minimise inappropriate prescribing as a resultAnd up until now there has been neither curriculum time nor resource material for medical and pharmacy schools20 May 2009WHA
3 Rationale for testing the manual in a pilot study OverviewRationale for testing the manual in a pilot studyProtocol for testingTimelinesWhat I would like to describe for you in the next 15 minutes is outline the planned pilot study testing of this WHO / HAI teaching manual – covering the reasons why it is essential that we do pilot it and the way we plan to do it.We would really welcome interest from any of you who are associated with medical and pharmacy student teaching who are interested in being involved as a pilot study site
4 BackgroundWHO/HAI have developed a comprehensive teaching package comprising a manual and accompanying resource material as well as training for teachers implementing and thisSo we have this very comprehensive manual and teaching resource package that Barbara has spoken about
5 BackgroundIt is planned to evaluate this in a pilot study to look at its effectiveness and at areas for improvement in the final manualWHY DO THIS STUDY?Why not just send it out?Why do we need to do a pilot study?
6 BackgroundMany educational resources and techniques are implemented without ever having any assessment of their effectivenessMany – dare I say most educational ideas and resources are implemented without piloting– without ever really knowing whether they are effectiveAnd alsoWithout knowing whether there is some possibly counterproductive effectCounterproductive? Some of you will thinkHow could providing information be counterproductiveIn this instance we are implementing a manual in an environment where most physicians and pharmacists acknowledge that people are influenced by promotion, but believe that they individually arent.This is called the illusion of unique invulnerability.It is possible that in this environment, an educational strategy might further increase the confidence of individuals in engaging with the pharmaceutical industry, believing they are even less vulnerable to such influence when armed with knowledge.
7 So we don’t know whether our efforts are effective So, like this cohort of roman soldiers marching onOften we don’t know whether our efforts are effective
8 Our wish to think that they are can be misleading…… OverviewOur wish to think that they are can be misleading……Many strategies for medical education have little effect on prescribing when they are tested in studiesAnd our desire to think they are can sometimes be misleadingMany strategies for medical education have little effect on participants when assessed in a research frameworkPrinted informationLectures …. Like this oneGuidelines have limited effect
9 Evidence for prescribing education strategies that have been found effective The effect of dissemination only strategies such as didactic lectures and written information are small at best and of unknown clinical significanceFarmer et al. Cochrane Database of Systematic Reviews 2008;3Facilitated teaching with evidence based resources and audit and feedback have been found to have a moderate effectRichards et al Family Practice 2000A previous WHO educational resource (WHO Guide to Good Prescribing) using facilitated teaching for medical students on pharmacotherapy showed a significant effect on students skills in a randomised controlled trialImpact of a short course in pharmacotherapy for undergraduate medical students: An international randomised controlled study. de Vries T Henning R Hogerzeil H Bapna J et al The Lancet 1995What we do know about some of the effecgtive strategies is that they generally have a small to moderate effectFacilitated teaching with evidence based resources similar to the WHO manual proposal, combined with audit and feedback have a moderate effectA previous WHO resource – the companion =volume for this manual – was implemented within a pilot study and found to have a significant effect on students prescribing skills in a test situation
10 Randomised controlled trial 10 volunteer sites Study DesignRandomised controlled trial10 volunteer sites5 intervention sites 5 control sitesQuantitative and qualitative methods to assess change in students knowledge attitudes and skillsSo for these reasonsto both test the effect and improve the content of this manualwe are planning a randomised controlled trial of implementationWhy an RCCT?There are a number of potential biases in designs other than RCTsBefore and after designs with no comparison group are likely to show effects in the students that are dues to influences other than the manualWhile at medical and pharmacy school students are subject to a wide range of influences and in addition mature a great deal in their thinking in each year.Satisfaction surveys are likely to be subject to Social desirability bias – that is students may choose the answers they think they ‘ought’ toWe are proposing to randomise by site rather then randomised students within a site as many of the changes are attitudinal ones. During medical and pharmacy training one of the most important influences on attitudes are peers and it would be impossible to avoid ‘contamination’ through discussion between students who did and did not receive teaching using the manual.We will use quantitative methods to look at shifts in quantifiable aspects of students knowledge attitudes and skillsBut there are a number of sources of bias in using just this kind of assessment –There is still a risk of Social desirability bias – that is students may choose the answers they think they ‘ought’ to in both armsFor this reason we are planning an complementary qualitative analysis that will use before and after focus groups to understand the kinds of more subtle effects the implementation of the manual has had on student attitudes and behaviour. This part of the evaluation will also involve key informant interviews with the teachers to tease out aspects of the manual that were helpful and unhelpful – what caught students interest and what was counterproductive.
11 What is the effect on students QuestionsWhat is the effect on studentsknowledge, attitudes and skills of students in understanding and responding to pharmaceutical promotionWhat are the strengths and weaknesses of the content of the manual and resources providedassessed across a variety of settingsThese are the two important questions to answerOne about the effect on students of teaching using these resourcesThe other is around the material provided – its strengths and weaknesses and how it might be improved
12 Data collectionBefore and after assessment of knowledge, attitudes and skills of studentsQuestionnaireFocus groupsKey informant interviews and content evaluation by teachers in the intervention groupStudents will have before and after questionnaires covering their knowledge attitudes and skills as well as before and after focus groups to explore more deeply their attitudes and planned engagement style with promotion in particularTeachers will provide feedback on the content of the manual and also participate in interviews to get their views on how well the manual worked overall
13 Two comparisons between these randomised intervention and control groups The difference between the before and after manual assessments at the beginning and end of the year compared to the control sitesThe difference between their assessments at the end of the year and students at the end of the year in the previous year groupAgain compared to the differences at the control sitesWe will compare the difference in before and after assessments in the sites that had the manual and resources with those that carried on with their normal teaching curriculum.We will do a second independent comparison between the students at the end of the previous group, before the manual is used at the site , with the next group of students in at the same stage of training, who have receive d teaching using the manual and resources. These differences between consecutive years will also be compared between intervention and control sites.This will give us 2 independent ways of assessing what the effect of the manual has been on these students
14 Comparison 1Change in scores at intervention sites compared to control sitesintervention sites control sitesmanual teaching as usualBefore Beforecompared toAfter AfterFor those who like diagrams…differencedifference
15 Comparison 2Difference in scores between study groups at the end of the year and students at the end of the previous yearintervention sites control sitesStudents from previous year tested for comparisonStudents from previous year tested for comparisonStudy startStudy startManual implementedTeaching as usualStudents at end of study yearStudents at end of study year
16 Comparison 2Difference in scores between study groups at the end of the year and students at the end of the previous yearintervention sites control sitesStudy startStudy startManual implementedTeaching as usualStudents from previous year tested for comparisonStudents from previous year tested for comparisonStudents at end of study yearStudents at end of study year
17 Medical or pharmacy students Volunteer sitesMedical or pharmacy studentsSite prepared to teach all manual chaptersSite able to teach within one academic year (12 months)Ethics approval will be gained where requiredThe volunteer sites we need are medical or pharmacy students who are taught in spanish english or russian.THe volunteer sites need to be able to teach all or most of the manual chapters and do so within one academic year in order that we can test and get the results in a way that is timely enough to allow revision before release of the final version.We will of course gain ethics approval in all sites that require this.
18 Volunteer sitesTraining, travel and accommodation will be provided for a teacher from each siteThe evaluation assessments and analysis will be organised and co-ordinated by the study team based at HAIThese involve the before and after questionnaires and focus groups for students/interviews for teachers at each siteWhat will the volunteer sites getTeacher will come to Amsterdam to receive 3 days of training in the manual the resources and the implementationThe research team will organise and co-ordinate the before and after assessments at each site . That is the questionnaires and focus groups
19 Volunteer sites: Timeline Plan to implement as academic years begin from later in 2009 through to 2010Training for teachers from sites randomised to intervention in Amsterdam Sept 2009‘Before’ assessments as academic years start‘After’ assessments completed late 2010 as years endTraining for teachers from sites randomised to control in Amsterdam in Sept 2010This is the timelineThe draft manual is ready and we plan to implement it as a pilot later this year and through to early 2010 as academic years start in different countries.As sites volunteer we would complete the training and assessment of students prior to the manual teaching, and then aim to complete the after manual assessments late in 2010.At that point the ‘control ‘ site teachers would then receive the manual and training in its implementation in Amsterdam.
20 It is not newPromotion is not new but its influence is increasing
21 As promotion of pharmaceuticals becomes and increasingly important influence Countering the adverse effects of promotion is vitally important for the rational use of medicines in ensuring that patients receive the best that medical science has to offer.
22 We need to ensure medical and pharmacy graduates are well equipped to understand and deal with the effects of promotion on their prescribing decisionsSo our imperative to provide health professionals with the tools to deal with it also becomes more urgentBecauseIn the end
23 Our common goal is the relief of suffering. Inappropriate medication decisions made in response to misleading promotion can increase the burden of suffering rather than relive it
24 It is important that we not just have good ideas But that we know whether our collective efforts are effectiveWe not to just have good ideas about how we might do this ,We also need to know that our collective efforts are effective
25 In order that disaster does not follow disaster
26 Volunteers! Pilot study Testing the manual in your setting Expression of interest forms at the backof the room ORSo this is my last word about the manual evaluationWE NEED YOU