1. Evaluation Report: April 1, 2015 – March 31, 2016
Implementing a National Evaluation to Demonstrate Reach and Impact of the National Network of STD Clinical Prevention Training Centers
Christopher Voegeli, PhD, MPH, Susan Dreisbach, PhD, Tarus Sandoval, BA,
Helen Burnside, MS, Sharon Devine, PhD, JD
Evaluation Report: April 1, 2015 – March 31, 2016
Background and Purpose
The CDC funds the National Network of Sexually Transmitted Diseases Clinical Prevention Training Centers (NNPTC) to train clinicians to screen, diagnose, and treat STDs.
In 2014 the CDC funded the National Evaluation Center (NEC) to create and implement a national evaluation to measure the reach of training to primary care and other healthcare providers in high STD morbidity areas and to assess the impact of training on clinical practices to screen for, diagnose, and treat STDs.
Methods
Reach to priority training audiences was determined by participant demographics and practice characteristics.
Audience 1 - non-STD expert, general practitioners in high morbidity counties who provided direct care to at-risk populations
Audience 2 - expert STD/HIV clinicians in high morbidity counties who provided direct care to at- risk populations
Audience 3 - administrators in settings that should screen for and treat STDs
Audience 4 - client-facing providers other than clinicians working in high morbidity counties who provide services to at-risk populations
Discussion
The NNPTC implemented an online national registration and evaluation that is capturing standardized data across PTCs.
Post and 90 day response rates are above national trends for surveys (Sheehan, 2001)
We intend to capture data from a greater proportion of training participants to more fully evaluate the impact on clinical practices and compare impact between courses.
Reach to clinicians in states and counties with high morbidity can be improved by targeting more professional associations, CHCs, FQHCs, and health systems.
68% of all trainees who completed an evaluation intend to make at least one change based on the training.
63% of 90-day respondents reported having made at least one change at the time of the 90 day follow-up.
The proportion of 90-day self-reported practice changes exceeds expectations based on literature (Webb and Shehan, 2006).
Participants completing more intensive courses (4 hrs - one day or more )) self-reported statistically significant improvements after 90 days in the frequency of recommended screenings, recommending HPV vaccination s, and taking a sexual history, suggesting a strong impact on clinical practices.
90 day improvements in recommended practices were less for courses lasting less than 4 hours
Many training events were not evaluated.
Some trainings such as grand rounds and conference sessions do not lend themselves to pre-registration which precludes automated evaluation
Innovations are needed to increase capture of registration and evaluation data without additional burden to NNPTC resources
Sheehan KB. survey response rates: A review. J Computer-Mediated Communication; 2001:6,2.
Webb T, Sheehan P. Does Changing Behavioral Intentions Engender Behavior Change? A Meta-Analysis of the Experimental Evidence. Psychological Bulletin, 2006: 132, 2, 249–268
Setting
The NNPTC consists of eight regional prevention training centers (PTCs) that train healthcare providers to screen, diagnose, treat and counsel those with STDs.
The centers are located in Baltimore, Birmingham, Boston, Denver, New York City, Oakland, Seattle and St. Louis; they serve all 50 states, District of Columbia, Puerto Rico, and the US Virgin Islands.
Results
The NNPTC presented 509 trainings to 25,270 participants in the evaluation period.
PTCs collaborated on 28% of trainings.
Registration data were collected for 21% of all training participants.
Of the 4,568 participants that completed an online registration with all data:
1,755 (38%) are clinicians in high morbidity counties who provide direct care to at-risk populations; 12% are primary care providers with little STD care experience.
3129 (68%) are in one of the priority audiences.
4090 (76%) work in high morbidity counties
Response rates were 59% for post-course evaluations and 30% for 90 day follow-up surveys.
85% - 88% of participants were “satisfied” or “very satisfied” with all aspects of the training.
Most courses showed statistically significant gains in self- reported retrospective confidence to demonstrate knowledge and skills needed to screen for, diagnose, and treat chlamydia (CT), gonorrhea (GC), syphilis, and HPV according to the 2015 CDC Treatment Guidelines.
Results
68% of post-training respondents intend to make at least one change based on the training (n=1,579) and 63% of 90 day follow-up respondents report having made at least one change (n=661)
After 90 days, there were statistically significant self- reported retrospective increases in
Frequency of screening females under 25 and MSM annually for CT and GC
Frequency of conducting an HIV test on patients newly diagnosed with syphilis
Frequency of discussing HPV recommendations with patients
Frequency of conducting a sexual history and asking about extra genital sites
Median frequency at 90 days was >91% for all recommended screening practices except HPV vaccination with MSM (76-90%).
3,343 (73%) participants used the 2015 STD Treatment Guidelines before training and 565 (86% of respondents) were using them 90 days after training.
Post course knowledge about treatment for CT, GC, and syphilis ranged from 84% - 90% correct answers. Post- course knowledge about the need for 3 month retesting after CT, GC, or trichomonas ranged from 65%-74% indicating a need for greater training emphasis.
Methods
The NEC developed and implemented an online national registration and evaluation system.
PTCs created courses, invited participants to register online, and selected the appropriate evaluation instruments for the course content. The system sent course participants a link to the online evaluation after the training and again 90 days later.
Online evaluation surveys assessed satisfaction as well as immediate and longer term changes in confidence, knowledge, skills, intention to change clinical practices, and self-reported changes in practice patterns.
Data was collected electronically on courses, registrants, and evaluation indicators between April 1st 2015 and March 31st
A descriptive analysis was performed on:
Knowledge
Reach
Satisfaction
Use of the CDC’s 2015 STD Treatment Guidelines
Intended and self-reported changes in STD clinical practices over time