2. Tamatha Thomas-Haase, MPA Jessie Saul, PhD February 4 and 6, 2009 Maintaining Quality Across Quitlines In North America

3. On today’s call we hope to: Review the recommendations for measuring and reporting quitline reach and quit rate; Highlight potential challenges to implementation of the standards and potential solutions; and Provide members an opportunity to respond to the recommendations, challenges and proposed solutions.

4. Measuring Reach of Quitline Programs Before you begin Define the quitline and timeframe of interest. For example, you might be interested in assessing the reach of a state quitline over fiscal year 2006-2007. Potential Challenge: Data may not be easily available for the specific time frame of interest. Potential Solution: NAQC will provide assistance to best determine how to access data for the time period of interest.

5. Reliably defining and estimating the size of the target population (the denominator) To estimate the size of the target population, select a population survey that is relevant, covers the geographic area and time of interest. Potential Challenge: A quitline may target and serve all tobacco users, but the best population survey only asks about cigarette use (e.g., BRFSS). Potential Solution: Use the survey data for smokers, and restrict the numerator to smokers when calculating reach.

6. Most quitlines provide service to all tobacco users rather than only to smokers. Therefore, the target population should be all tobacco users. Potential Challenge: Some quitlines only serve cigarette users. OR population survey data are only available for cigarette users. Potential Solution: Use population estimates for cigarette users only. OR restrict the numerator to cigarette users only to calculate reach.

7. Treatment reach can be calculated for any age group, but the target population should be adults unless otherwise specified. Potential Challenge: Canada’s population surveys report on 15+ and 25+. Potential Solution: Note where target population and/or population includes those other than 18+.

8. To assess Canadian provincial quitlines, consider using the CTUMS. To assess a U.S. state quitline, consider using the BRFSS once the survey includes questions about all tobacco use in 2009. Potential Challenge: What if you want to compare internationally? Potential Solution: Use the best survey for each country, and note in the reporting how the surveys differed, which may limit comparability of reach rates.

9. Opt for a survey that is likely to provide an accurate estimate because of the sound sampling scheme, large sample size, and high participation rate [for special populations]. If there is no survey estimate of all tobacco users, select a survey that estimates the number of smokers in the population and restrict the numerator to smokers who received evidence-based treatment.

10. Identifying specific evidence-based services Identify the evidence-based services provided by the quitline. Research supports telephone counseling (proactive and reactive, single or multiple) and pharmacotherapy as evidence-based, but ongoing research will likely expand the list. Potential Challenge: How much of an evidence-based service “counts”? Potential Solution: NAQC to work with quitlines to figure out the cutoff for service delivery. Proposed: completed one counseling session, and/or received NRT.

11. Record and report measures that reflect the workload of the quitline staff including the number of calls answered, completed intakes, and types and amount of services provided. If possible, track the number of calls that do not result in an intake and report on the disposition of these calls.

12. Verify that the quitline records the type of service each client receives in a way that differentiates evidence-based treatment from other services. Potential Challenge: Some quitlines may not record this information with enough detail to be useful. Some service providers may not report this information regularly. Potential Solution: NAQC could create a reporting template for reach including all recommended variables associated with the reach calculation. Service providers could use this as a starting point for reporting so all clients would have the same expectations regarding reporting of reach.

13. Although not directly related to treatment reach, consider instituting quality control measures that would ensure treatments are provided with high fidelity to the intended protocol.

14. Who is in the numerator? Record and report quitline call volume and number of callers who complete the quitline intake. Potential Challenge: Lack of clarity about “call volume.” Potential differences in the timing of intake. Potential Solution: NAQC to define who should be included in “call volume” numbers. Pranks, wrong numbers, etc. should not be included. Proxies should.

15. Include fax referral and click to call in the measure of treatment reach if the clients go on to receive one of the evidence-based treatments. Potential Challenge: Quitlines may not distinguish between route of entry into the system (e.g., intakes from fax referral vs. intakes from direct calls) Potential Solution: No need to report mode of entry separately – treatment reach can be reported as a single number.

16. Count each client only once in the measure of treatment reach, regardless of the number of services they receive in the time period. Potential Challenge: Some quitlines may not have good methods of checking for duplicate enrollments, or may not have a need for such methods. Potential Solution: Quitlines should de-duplicate individuals calling the quitline to the best of their ability, and should report on which methods were used to do this. Any caller receiving any service in the 12 month period should be counted as having been served.

17. Calculate treatment reach by dividing the # target population who received quitline evidence- based treatment by the # total target population. Potential Challenge: Need to define cutoff for what counts as evidence- based service. Potential Solution: At least one counseling session completed and/or received NRT or other meds.

18. Measuring promotion or awareness of the quitline Work with those who promote quitlines to demonstrate the response to specific media efforts.

19. Setting a reasonable goal for quitline treatment reach Given that the focus of this whitepaper is to clarify the terms used to discuss quitline reach, the question of reasonable goals is beyond its scope. However, it is clear that efforts should be made to increase quitline treatment reach. Treatment reach can be increased by driving more calls to the quitline through promotional efforts, by supplying sufficient resources to provide treatment, and/or by ensuring that more callers are funneled through to evidence-based treatment.

20. Sample benchmarking chart for treatment reach

21. Ensuring priority populations are reached with quitlines To report on reach to priority populations, assess proportion of quitline participants from a particular subgroup relative to their proportion in the general population. Potential Challenge: Could be difficult to identify accurate proportions of subgroups of smokers from general population surveys. Potential Solution: NAQC could compile a list of possible data sources on its website for smaller populations.

22. Because estimates of subgroups are likely to be based on small sample sizes, it would be important to work with a statistician if you want to compare reach across these populations.

23. Calculating reach in states with more than one quitline To assess treatment reach for several quitlines in a state (or province) collaborate with the various service providers to determine the number of the target population who receive evidence- based treatment through any of the services.

24. Measuring Quit Rates

25. Denominator Include in the denominator all tobacco users who register for services, consent to follow- up, receive some evidence-based treatment, and have not been quit at intake or registration for more than 30 consecutive days. Potential Challenge: Some quitlines may not capture information on length of current quit attempt for those quit at registration. Potential Solution: NAQC could recommend adding a question about length of current quit to the MDS, and could work with quitlines to facilitate implementation of the new question.

26. Individuals who complete registration and intake but do not receive any counseling should not be included in quit rate calculations. Potential Challenge: Some quitlines may not be able to distinguish easily where intake ends and counseling begins. Potential Solution: Use completion of the first counseling session as the indicator for inclusion in the quit rate calculation rather than completion of intake.

27. A reasonable minimum for having received some treatment is the receipt of at least one telephone counseling session. Potential Challenge: What about those who begin, but do not finish the first counseling session? Potential Solution: Anyone who completes 10+ minutes of the first counseling session “counts” as having received evidence-based treatment.

28. The definition of treatment is expected to evolve as quitlines become more involved in provision of a range of tobacco cessation services (e.g. pharmacological therapy, web- assisted tobacco interventions, etc.). Failure to deliver counseling to individuals who register for services is a quitline quality issue that should be addressed in companion papers to this report.

29. Timing of follow-up Conduct follow up 7 months following quitline enrollment. Potential Challenge: For those quitlines who have not adopted the MDS recommended 7-month follow-up, this may be a larger challenge. Potential Solution: Adopting a 7-month follow-up would be accordance with both the quit rate standard and the MDS.

30. Duration of abstinence Measure and report 30-day point prevalence abstinence. Potential Challenge: 22 quitlines do not ask this question. Potential Solution: NAQC could provide talking points or rationale for including the 30-day question in follow-up surveys.

31. Intake and follow-up questions Use Intake items 5a or b and 7 and follow-up item 10 in Table 2 of the paper to measure 30-day point prevalence, the proposed outcome measure in this paper. Potential Challenge: 22 quitlines do not ask follow-up item 10 (30-day point prevalence question) Potential Solution: NAQC could provide talking points or rationale for including the 30-day question in follow-up surveys.

32. Biochemical validation Do not conduct biochemical validation. It is not recommended and the literature shows that self-reported smoking behavior is an adequate means to measure quit rates for tobacco cessation programs. Potential Challenge: none anticipated

33. Remain aware that self-reported quit rates are likely to include some small amount of inflation, and that this inflation is likely to be higher if the special populations you measured are facing an elevated expectancy to quit or feel a greater need to hide smoking behaviors.

34. Reducing missing data Select a combination of strategies appropriate to the quitline’s unique resources and needs in order to obtain a follow up response rate of 50%. Potential Challenge: 50% may be difficult to obtain for certain populations or quitlines. Potential Solution: Report quit rate along with response rate. NAQC can provide assistance for evaluating whether additional strategies to increase response rates would be helpful or worth the cost.

35. Report the follow-up survey response rate along with the quit rate. Potential Challenge: none, as long as both are calculated Potential Solution: NAQC could provide sample reporting templates indicating which items should be reported with a quit rate.

36. When response rates fall below the goal of 50%, utilize advance letters and incentives which literature shows to be cost effective. Consider obtaining consent and additional contact information at intake and carefully attend to policies and protocols regarding survey introductions, conversion of soft refusals, and the number of attempts in efforts to increase response rates. Quit rates should be interpreted with caution.

37. Missing outcome data Use the Responder Rate (RR) Quit Rate (number quit/number of follow- up survey respondents) as the primary measure for reporting quitline outcomes. Potential Challenge: This may be a shift for some quitlines. Potential Solution: NAQC could provide talking points to help explain any changes in how quit rates are reported.

38. DUPLICATE SLIDE? (same as #35?) Always report the response rate alongside the Responder Rate quit rate. Potential Challenge: Potential Solution:

39. Do not use more complex imputation-based methods for estimating quit rates (i.e. imputation) as part of standard quitline evaluation.

40. Specialized programs and dose response Report information about program intensity with quit rates (e.g, duration of programming, availability of pharmacological aids, special counseling strategies and content). Potential Challenge: Difficult to know how much to report or in what format. Potential Solution: NAQC could provide reporting templates for quit rates and all recommended additional information.

41. Calculate numerators and denominators separately for each program. (For example, the callers participating in a multi-call program providing NRT could be grouped in separate numerators and denominators from those participating in a one-call brief counseling intervention.)

42. Those already quit Include all callers who quit 30 or fewer days prior to calling the quitline in the numerator and denominator used in the abstinence rate calculation. Any caller who started their quit 31 or more days prior to calling should be excluded. Potential Challenge: Many quitlines may not ask about length of current quit at intake. Potential Solution: NAQC could recommend adding this question to the MDS for consistency and standardization of how the question is asked.

43. For quitlines that have a program designed specifically to serve those already quit, callers who have already quit should be treated as a special target population and a separate numerator and denominator should be calculated for this group.

44. Subject selection for evaluation Solicit consent to follow-up early on, such as at the close of intake or registration. Consider consulting a Human Subjects Research or Institutional Review Board regarding consent procedures as appropriate. Report consent rate with the quit rate. Conduct follow-up on an ongoing rolling basis, with a random sample of registered callers. If a rolling follow-up is not feasible, use multiple cohorts or time-limited sampling (following up with callers who register during a limited time period) over the course of the year as a viable second choice. Where possible, and as decided based on individual program needs for precision and budgetary constraints, complete at least 400 follow-up surveys as part of an outcome evaluation.

45. Mode Use telephone as the mode of follow-up, as long as the target goal of a 50% response rate can be achieved. Potential Challenge: None anticipated.

46. First bullet is a duplicate of first bullet on slide #36 Depending on available budget, use one or more strategies to help reach a target 50% response rate include mailing pre- notification letters, providing monetary incentives, and making at least 15 attempts to reach each participant. If the minimum response rate cannot be achieved, explore using a mixed-mode survey.

47. The evaluation team Use an evaluator with experience in quitline evaluation. Evaluation may be conducted by an external evaluator or internally by quitline service providers as long as those individuals conducting the evaluation are entirely separate from and independent of the counseling staff. Potential Challenge: None anticipated

48. Select an evaluation team based upon transparency of reporting and demonstrated ability to achieve adequate response rates on follow-up evaluation surveys. Potential Challenge: Transparency of reporting may be difficult to identify. Potential Solution: NAQC can provide reporting templates for quit rates – evaluators would need to agree to use it or something like it even if they have not reported all the information in that format before.

49. Confidence intervals Report confidence intervals with all abstinence rates, and include the number of subjects used in the calculation. Potential Challenge: Difficult to know in what contexts this applies. Potential Solution: In all official reports, confidence intervals and number of subjects should be included with quit rates. Other uses of quit rates are up to the individual quitline’s discretion.

50. Logistic regression and odds ratios Explore the possibility of reporting abstinence rate findings as odds ratios via logistic regression. If conducted, relative risk and differential probabilities should also be reported.

51. Interpreting quit rates Report information about the demographic and clinical characteristics of callers and program characteristics with quit rates. Potential Challenge: Difficult to know exactly which variables to report. Potential Solution: NAQC can provide sample reporting templates indicating exactly which information should be included.

52. Use caution when comparing your quit rate to those of other quitlines. Consider the similarity of the quitline programs, as well as the demographic and tobacco use characteristics of respondents. Potential Challenge: It can be very tempting to look only at quit rates. Potential Solution: NAQC could conduct benchmarking for similar quitlines to ensure appropriate comparisons are made.