1. Is Your Data Management System Flexible for Quality Control Activities? Winny Roshala, CTR Data Standards and Quality Control Unit NAACCR: June 13-19, 2009, San Diego, CA

2. CCR Visual Editing Standards: Accuracy Rates Implemented January 1, 2000 with 100% visual editing on 13 data items Automated software was developed to calculate accuracy rates Accuracy Rate Standard: 97%

3. CCR Visual Editing Standards: Purpose Assure high quality data for analysis Provide consistency in the visual editing process Quantify the accuracy of cancer data from cancer reporting facilities Standardize accuracy rates Standardize format for reporting rates to registrars/facilities

4. CCR Visual Editing Discrepancies Defined as the quality or state of being discrepant, i.e., disagreeing, being at variance A discrepancy arises when a more appropriate code should have been selected for a data item based on submitted documentation

5. CCR Visual Editing Discrepancies CCR Visual Editing Discrepancies Discrepancies are counted prior to cases being linked or consolidated Each data item is considered one potential discrepancy with the following exceptions: ◦ Site/subsite ◦ LN’s Pos/Examined ◦ Site Specific Factor fields

6. CCR Visual Editing Standards: Calculation of Accuracy Rates Percent Discrepant: Number of discrepancies divided by the number of abstracts, multiplied by the number of data items Accuracy Rate: 100% less the percent discrepant

7. CCR Visual Editing Standards: Historical Perspective In December 2005, in order to reduce a backlog, admissions from abstractors with an accuracy rate of 99% were no longer visually edited This “push through” represented approximately 64% of admissions

8. CCR Visual Editing Standards: Historical Perspective Due to budget cuts, this percentage was reduced again by adding abstractors with an accuracy rate of 98% to the admissions no longer visually edited

9. CCR Visual Editing Standards: Historical Perspective In February 2008, due to a further reduction in state funding, the CCR changed it’s approach to reducing the proportion of cancer registry abstracts that are visually edited Instead of focusing on individuals, the CCR went to a random sampling of cases for visual editing, reducing it from 100 percent to 40 percent

10. CCR Visual Editing Standards Quality for the remaining 60% of abstracts will be monitored by targeted visual editing and through recoding and reabstracting audits Hospital registrars continued to receive monthly Discrepancy Reports

11. Visually Edited Data Items County of Residence at Diagnosis Sex Race Spanish/Hispanic Origin Date of Diagnosis Diagnostic Confirmation Site/Subsite* Laterality (Only paired sites listed in Volume I) Histology Grade CS Tumor Size CS Extension CS Lymph Nodes Number of Regional Nodes Positive/Examined* CS Metastasis at Diagnosis CS Site Specific Factors 1-6* Class of Case * Counted as one discrepancy

12. VISUAL EDITING SAMPLING PLAN Run the edits against the admission Set a flag indicating whether any edit errors exist Check the admission as to whether it qualifies for required review and set a flag indicating true/false Check the site to determine whether it is one of the sites that require 100% visual editing and set a flag indicating true/false

13. The following sites had a high discrepancy rate and will continue to undergo 100% visual editing:  Lip  Nasal Cavity & Middle Ear  Accessory Sinuses  Thymus  Heart, Mediastinum, and Pleura  Retroperitoneum and Peritoneum  Adrenal Glands  Other Endocrine Glands  Other Ill-Defined Sites  Unknown Primary Site

14. VISUAL EDITING SAMPLING PLAN If the site is determined not to require 100% VE, using the system function Random, generate a number between 0 and 99 Set a flag indicating VE Required to true if the number generated is 37 or less, else set the flag to false Once all of the checks are complete, and if all flags are set to false, the admission will bypass Visual Editing

15. Issues to Consider Percent of cases randomly selected for visual editing List of sites which require 100% visual editing New review tasks that need to be added to the database

16. Issues to Consider Programming changes may require little time, however the deployment of programming changes may require a full build Deployment may be delayed to comply with a scheduled release

17. EUREKA RECODING AUDIT MODULE (RAM) What About the 60% of Cases Bypassing Visual Editing?

18. RAM Features Ability to select data items and text fields Accessibility Audit current data Automatically sends cases from the primary auditor to the secondary auditor Generates reports Multi-purpose tool

19. Audit Sample Request Request ItemParameter Name Lymphoma Recoding Audit - 2009 Data LevelAdmission Level Sample Size60 cases from each region Case StatusBYPASS Cases Only Sample Specifications 1) Resident within reporting region only 2) Reporting facility within resident region 3) Combine regions 1 and 8 as one sample 4) Combine regions 7 and 10 and 3 and 4 as one sample (Region 4: 10 cases/ Region 3: 20 cases/ Region 7/10: 30 cases) 5) Remove all ACTUR reporting facilities (159990,349990,379991,429990,489990) Type of Cases (Year)Cases Loaded 3/1/2008 to 12/31/2008 Diagnosis Year2007 and 2008 Primary SiteLymph Nodes (C77.0 – C77.9)

20. Audit Sample Request Request ItemParameter GenderAll Class of CaseClass 0, 1, and 2 - ONLY Histology (Parameters, if any) 9540 - 9989 Histology_m3= 2,3 (Behavior) 3 – Invasive - ONLY CA residents only (at DX) Type Reporting Source 1 ( hospital/clinic) Recoding Dates (Start and End) May 26, 2009 to June 12, 2009

21. 21 Primary Auditor Recoding Screen

22. 22 Reconciliation Screen

23. 23 RAM Disposition Report

24. Summary With diminishing resources, changes in the CCR visual editing practices were necessary Due to the flexibility in our data management system, Eureka, we have the ability to quickly refine the sampling plan for visual editing as needed This has resulted in our ability to redirect resources while carefully monitoring quality control on our data

25. Summary The development of Eureka RAM has enabled us to focus on the cases bypassing the visual editing process RAM is instrumental in quickly identifying problem areas in coding and instruction Training efforts can be mobilized earlier Future uses for RAM may include training of new staff, targeted visual editing and special studies

26. Acknowledgements Nancy Schlag, CCR Operations Section Chief Andrew Sutliff, Eureka Programmer Analyst Kyle Ziegler, Audit Coordinator, Quality Control Specialist Vic Belen, Administrative Assistant

27. Contact Information Winny Roshala, BA, CTR California Cancer Registry 1825 Bell St., Suite 102 Sacramento, CA 95825 Phone: (916) 779-0313 Email: wroshala@ccr.ca.gov