Presentation on theme: "Pilot Studies Nuts and Bolts Deborah Grady, MD, MPH Professor of Medicine Associate Dean for Clinical and Translational Research UCSF."— Presentation transcript:
Pilot Studies Nuts and Bolts Deborah Grady, MD, MPH Professor of Medicine Associate Dean for Clinical and Translational Research UCSF
What is a Pilot Study? Early study of a new idea designed to obtain preliminary data to support a larger studyEarly study of a new idea designed to obtain preliminary data to support a larger study –Usually small –Goals often focus on feasibility and logistics –Outcomes often surrogates –Typically unfunded or funded intramurally –Require a lot of work - protocol, IRB approval, data forms, recruitment, data entry, analysis… Distinguish from pre-testing of methods, questionnaires, measuresDistinguish from pre-testing of methods, questionnaires, measures
Clear Reasons to Do a Pilot Study Demonstrate feasibilityDemonstrate feasibility –Determine if you can do it Improve logisticsImprove logistics –Determine the fastest, easiest, most cost- effective way to do it Estimate costEstimate cost Demonstrate to fundersDemonstrate to funders –that you can do it within budget!
Possible Reasons to Do a Pilot Study Decide whether to do larger studyDecide whether to do larger study Estimate parameters for designing larger studyEstimate parameters for designing larger study –Rate of outcomes in placebo group –Benefit (effect size) –Variability of outcome measure
Yoga to Prevent Diabetes Evidence suggests yoga decreases stress, improves metabolic risksEvidence suggests yoga decreases stress, improves metabolic risks Possible larger studyPossible larger study –randomized trial of yoga vs. control in adults vs. control in adults with metabolic syndrome with metabolic syndrome –outcome - new diabetes
Yoga Pilot Study Design Randomized trial in persons yo with metabolic syndromeRandomized trial in persons yo with metabolic syndrome Intervention - 10 yoga poses taught in group classes x 10 weeks + home practiceIntervention - 10 yoga poses taught in group classes x 10 weeks + home practice Outcomes - diabetes; fasting glucose, insulin, lipids; insulin sensitivity, heart rate variability, measures of stress, BMI, etc.Outcomes - diabetes; fasting glucose, insulin, lipids; insulin sensitivity, heart rate variability, measures of stress, BMI, etc.
Feasibility RecruitmentRecruitment –Sources of participants –N of participants –% eligible –% willing to enroll –Time and cost of recruitment Responded to Ads/fliers 388 Telephone Screen 255 Eligible for Clinical Screen 89 Eligible 28 Yoga14Control12
Feasibility Interventions and measuresInterventions and measures –Can obese persons perform the postures? –Will there be side effects? –Can participants do insulin clamp, 2-h GTT, other tests and questionnaires? Adherence and follow-upAdherence and follow-up –Will participants stay in the study and practice at home?
Logistics and Cost What staff are needed?What staff are needed? How should the flow of visits be organized?How should the flow of visits be organized? Where will participants park?Where will participants park? How long do visits take?How long do visits take? Can participants answer questionnaires while completing GTT?Can participants answer questionnaires while completing GTT? How much will the trial cost?
Yoga Pilot Study Findings Trial is feasibleTrial is feasible –Able to recruit and randomize –Participants able to do 9 Yoga postures –Adherence excellent 13.7 classes attended (goal 14)13.7 classes attended (goal 14) 100 min practice/week (goal 90)100 min practice/week (goal 90) Logistics improvedLogistics improved –One posture (Downward Dog) dropped –Added video for home practice –Visit time reduced Cost reasonableCost reasonable
Yoga Pilot Study Results Variable Difference P-value Favors Weight, kg Y BMI, kg/m Y Waist, cm SBP, mmHg Y DBP, mmHg Y Perceived Stress Scale Y Energy Level Y Psych Well-being Y Depression (CES-D) Y Self-rated health Y Insomnia (% none) Y
Yoga Pilot Study Results Variable Difference P-value Favors Insulin Sens Index C Fast. Glucose C 2-hour Glucose HDL- cholesterol C Triglycerides Y ALT C YIKES!
Yoga Pilot Bottom Line Full-scale Yoga study funded by NCCAM and in progressFull-scale Yoga study funded by NCCAM and in progress
Should Pilot Studies Include Randomization? Maybe not…Maybe not… –Nonrandomized studies can generally be smaller, faster and cheaper –Obtain more data on intervention if all participants receive it Maybe…Maybe… –If need to know if randomization is possible –If need to know anything about the control group (adherence, placebo effect, etc) –If trying to estimate between group effects
Do Pilots Need Control Groups ? Phase I trials (evaluate short-term safety)Phase I trials (evaluate short-term safety) –generally uncontrolled Phase II trials (evaluate range of doses)Phase II trials (evaluate range of doses) –generally randomized and controlled Feasibility, logistics and costFeasibility, logistics and cost –controls generally not needed Effect and sample size estimatesEffect and sample size estimates –controls helpful, esp. if need to control for placebo effect
How Do You Estimate Sample Size for a Pilot Study Sample size often set by traditionSample size often set by tradition –Lab studies –Many phase I trials start with 3, add 3 –Pilots for feasibility often 5-20 –Pilots to estimate effect size can be large Sample size often dictated by resources and timeSample size often dictated by resources and time
Should You Publish Pilot Results? Restorative yoga in adults with metabolic syndrome: a randomized, controlled pilot trial. Cohen BE, Chang AA, Grady D, Kanaya AM. Metab Syndr Relat Disord. 2008, 3:223-9.Restorative yoga in adults with metabolic syndrome: a randomized, controlled pilot trial. Cohen BE, Chang AA, Grady D, Kanaya AM. Metab Syndr Relat Disord. 2008, 3: PubMed search for “pilot” in title, last 12 months, in EnglishPubMed search for “pilot” in title, last 12 months, in English –2649 publications!
Funding for Pilots UCSFUCSF –Research Allocation Program (RAP) –Training programs, mentors NIH - R03, R21, U34, K awardsNIH - R03, R21, U34, K awards OtherOther –Foundations, philanthropy –Pharmaceutical companies PI discretionary fundsPI discretionary funds
Summary: Pilot Studies Crucial to the development and testing of novel ideas and interventionsCrucial to the development and testing of novel ideas and interventions Necessarily small and under-poweredNecessarily small and under-powered Have different or additional goalsHave different or additional goals Are recognized as a specific study design by NIH, local funders, journalsAre recognized as a specific study design by NIH, local funders, journals But little attention to design, methods and interpretation of pilot studiesBut little attention to design, methods and interpretation of pilot studies
Nuts and Bolts Resources ContractsResources Contracts Budgets Study teamBudgets Study team Space RecruitmentSpace Recruitment Start-up FormsStart-up Forms Adherence and follow-upAdherence and follow-up Money Matters CloseoutMoney Matters Closeout
UCSF Resources Senior colleagues in your disciplineSenior colleagues in your discipline Experienced trialists and trial staffExperienced trialists and trial staff CTSI Consultation ServiceCTSI Consultation Service –design –ethics –statistical issues –database development and management
Contracts Required if trial supported by industryRequired if trial supported by industry Must be approved by UCSF Industry Contracts & company’s legal divisionMust be approved by UCSF Industry Contracts & company’s legal division –lots of puzzling legalese –you should understand and revise, esp. scope of work –best to start with UCSF template, revise and submit to company Between UCSF Regents and companyBetween UCSF Regents and company –signed by PI
Elements of a Contract Scope of workScope of work Timetable for deliverablesTimetable for deliverables Budget and timing of paymentsBudget and timing of payments –get a bolus up front –don’t link payment to deliverables out of your control Ownership of data and publishing rightsOwnership of data and publishing rights Rules for breaking contractRules for breaking contract Confidentiality, indemnificationConfidentiality, indemnification Patents and inventionsPatents and inventions
Budgets NIH-style (NIH, VA, AHRQ, CDC)NIH-style (NIH, VA, AHRQ, CDC) –prepared far in advance –governed by strict rules funds restricted to categoriesfunds restricted to categories permission for carry-over requiredpermission for carry-over required certain expenses not allowedcertain expenses not allowed subject to government auditsubject to government audit Pharmaceutical company budgetsPharmaceutical company budgets –prepared just before trial starts –negotiable –much more flexible
Money Matters Pre-award managerPre-award manager –help prepare budget and “face” pages –help with budget justification –make sure you follow rules/timelines Post-award managerPost-award manager –pay salaries, buy equipment and supplies –monthly report of expenditures –projections over life of trial
Who is the Study Team? Principle investigatorPrinciple investigator Project director/clinic coordinatorProject director/clinic coordinator RecruiterRecruiter Data managerData manager Programmer/analystProgrammer/analyst StatisticianStatistician Quality control supervisorQuality control supervisor Administrative assistantAdministrative assistant Financial/personnel managerFinancial/personnel manager
How to Hire the Study Team Work with your department or unit personnel (HR) managerWork with your department or unit personnel (HR) manager –write a job description –decide on job series, step, salary –post the job at UCSF and advertise –review resumes –interview –select
Where to Find the Study Team Other studiesOther studies UCSF employeesUCSF employees Recent graduates, studentsRecent graduates, students Friends and colleaguesFriends and colleagues Craig’s ListCraig’s List Chronicle, webChronicle, web
Training the Study Team Describe, emphasize and demonstrate the importance of:Describe, emphasize and demonstrate the importance of: –following the protocol/operations manual –meeting recruitment goals –complete adherence and follow-up –full outcome ascertainment –maintaining participant safety and confidentiality –ethical conduct of research Formal training and certification of staffFormal training and certification of staff
Operations Manual and Training Meeting Purpose and design of the trialPurpose and design of the trial Each measurement and visit procedureEach measurement and visit procedure –methods –qualified personnel –calibration and testing Completing and altering study formsCompleting and altering study forms Data entry or submissionData entry or submission Safety and ethical issuesSafety and ethical issues Certification and training of new staffCertification and training of new staff
Team Leadership Management training usefulManagement training useful You are responsible and the bossYou are responsible and the boss Keep your staff happyKeep your staff happy –involvement in the science –salary, working conditions –level of responsibility Set an exampleSet an example Promote good staff interactionsPromote good staff interactions Some staff should be firedSome staff should be fired
Space ConsiderationsConsiderations –accessibility, parking –design and décor –privacy –clinical needs, special tests –safety measures –cost CTSI CRCsCTSI CRCs On-campus or medical spaceOn-campus or medical space Off campus rentalOff campus rental
Recruitment Hire an experienced recruiterHire an experienced recruiter Provide adequate time and moneyProvide adequate time and money Monitor resultsMonitor results Make changesMake changes –different approaches –more centers –longer recruitment –change eligibility
Plan Multiple Approaches to Recruitment Referral from providersReferral from providers Targeted mailing (age, gender, disease)Targeted mailing (age, gender, disease) UCSF Integrated Data RepositoryUCSF Integrated Data Repository Kaiser, DMV, HCFA, registriesKaiser, DMV, HCFA, registries AdvertiseAdvertise –hospital, clinics, special sites, churches –radio, newspaper, TV, internet (Facebook) –celebrities, leaders CTSI Participant Recruitment ServiceCTSI Participant Recruitment Service Follow HIPPA and confidentiality rules
Forms and Database Development Define variables, create data dictionaryDefine variables, create data dictionary Determine method of data entryDetermine method of data entry –Humans –Machine readable forms Design database (CTSI Consultation)Design database (CTSI Consultation) Use prior forms - forms librariesUse prior forms - forms libraries Pretest, validatePretest, validate
Adherence and Follow-up Monitor ratesMonitor rates Obtain contact data at baselineObtain contact data at baseline Pleasant, professional staffPleasant, professional staff Develop personal relationshipsDevelop personal relationships Encourage resumption of intervention and return to follow-upEncourage resumption of intervention and return to follow-up Cards, small gifts, meetings, partiesCards, small gifts, meetings, parties
Quality Control Standard Operating ProceduresStandard Operating Procedures –protocol –operations manual –statistical analysis plan –data and safety monitoring plan Training and certification of staffTraining and certification of staff –ethical conduct of research –measurements and other visit procedures –data entry Performance reviewPerformance review
Quality Control Special procedures for quality controlSpecial procedures for quality control –study drug –laboratory procedures –data quality missing, out of range, illogicalmissing, out of range, illogical variation by site, investigator, staff membervariation by site, investigator, staff member Periodic checks of source dataPeriodic checks of source data Periodic reports (missing forms/data, queries, differences, drift)Periodic reports (missing forms/data, queries, differences, drift)
Closeout Final visit and post-trial plansFinal visit and post-trial plans –make final measurements –say goodbye and thank you –inform of test results –?inform of treatment status –?inform of trial results –?make clinical recommendation
Summary Trails are costly and complicatedTrails are costly and complicated –get advice from experienced colleagues –get materials used in prior trials –use UCSF services financial, legal, IRB/CHRfinancial, legal, IRB/CHR CTSI Clinical Research CenterCTSI Clinical Research Center –Space, phlebotomy, study staff CTSI Consultation ServicesCTSI Consultation Services –Design, ethics, statistics, database management