1. The Progress of npcr audits What have we done, what have we learned, and where are we going now
Click to edit subtitle
Click to enter your Division Name here

2. History of Npcr audits
Public Law :
“Participate in an independent audit of compliance with NPCR standards, authorized by Public Law , conducted by CDC approved organization/entity ”
Technical Assistance & Audit
Data Completeness & Quality Audit
Data Quality Evaluations – Version 1 & 2
Read first part of slide – then relate that today I will touch base on the last four (4) audits

3. Technical Assistance and Audit (TAA) 2001-2005
Assessment of Completeness and Data Quality of
Female Breast
Colon and Rectum
Lung and Bronchus
Prostate
Data Years
TAA timeframe: October, 2000 – September, 2005 – reviewing data from encompassing 46 CCR’s.
The primary focus of these audit activities were on female breast, colorectal, lung and prostate primaries. The “source” of these audits were an evaluation of hospital resources and processes.
The first being an assessment of reporting completeness through a case-finding exercise. Pathology reports were the primary source reviewed for inclusion in the CCR level – assessing whether the patient and tumor information was reported to the state registry. If time allowed, then a review other sources – MRDI and/or clinic logs was also performed.
Data quality was evaluated via re-abstracting from source/hospital records.
The results of both these activities was then compared to the CCR merged information resulting in
Reconciliation by the CCR to either
a. explain missing cases or differences in data elements based on additional information at the CCR level or
b. or the identification of cases that were not reported to the CCR

4. Completeness
Review of MRDI demonstrated that the most often missed cases were:
Hematopoietic Diseases
Clinical only cases:
Lung, Pancreas, Liver, and Brain/CNS
Complete review of Pathology Cases revealed that:
Breast
Digestive were more often missed
Out-patient clinics and logs had it’s own issues
Completeness rates were between 95.1% and 97.7% over the five year period

5. Data completeness and Quality audit (DCQA) 2006-2011
Assessment of Completeness and Data Quality of
ALL Sites
Summary Stage and Collaborative Stage
ALL Treatment information
Data Years
With the overall completeness in reporting well within NPCR standards, Casefinding was enhanced with a look at all sites. And data quality took a close look at Summary stage and collaborative stage and an evaluation of all treatment information.

6. Year 1 case-finding Results:
Most often missed cases:
Reportable Hematopoietic Diseases
Non-malignant CNS
Digestive
Respiratory
Breast
Year 1 Completeness:
Ranged from: 92% to 99%
Average: 97%
Most Common Missed:
RHD, non-malignant CNS, digestive, respiratory and female breast

7. And we took a snap-shot look at the stage errors related to our most common sites. CS errors were an issue – with an overall error rate between 6-13% for Summary Stage. Not all the Summary Stage issues were related to errors in the data elements. We found that if a change/update had been made to a portion of CS – Extent, Lymph Nodes or Mets – sometimes SS was not recalculated.

8. I presented this information a while back, and you can see from this slide that Surgery information needed some additional attention. Results of our audits revealed various issues that were consistent from 2004 – 2006.

9. NPCR Data Quality Evaluation 2012 - 2016
Focus:
Data within the Central Cancer Registry (CCR) for
Breast,
Colon/Rectum,
Lung,
Corpus Uteri and
Prostate
Purpose:
Data Completeness and Quality
Identify Data Limitations
Identify Training Needs:
Data Reporters
Central Cancer Registry Staff
In 2012, NPCR audits took a new focus. The purpose was to evaluate the completeness and quality from the data within the CCR for the major sites. The results help NPCR understand some of the CCR procedures regarding data completeness and overall data quality, as well as to understand some of the data limitations - either due to the accuracy or the completeness of the information that is recorded at the source level and subsequently the summary information that is ultimately used in research
As well as to identify the training needs – both of the data reporters as well as the central cancer registry staff.

10. NPCR Data quality evaluation - dqe:
Process:
Visual Review – Source Level
Manual Consolidation
Comparison to CCR Merged Data
Evaluation of QC Proceduress
Power Point Presentations
Ascertainment of Treatment Data:
Breast
Colorectal
The process involves a visual review of the abstract information; manual consolidation of multiple abstracts with a comparison to the CCR merged data. WE also did a review of the CCR QC procedures in comparison to CDC-NPCR recommended guidelines and provided a summary of findings to the CCR via a written report and a power point presentation.
We also run clinical checks on breast and colo-rectal primary cases; The primary purpose of the Clinical Check edits is to evaluate reported
prognostic and treatment items for cancer cases with specific tumor characteristics. For example – was the required prognostic/staging information reported for the breast and colon primaries
If the reported treatment does not appear to be consistent with widely recognized standards of care or cases fail to contain known prognostic characteristics, a warning is generated.

11. Common Issues: Required Data Elements: Consolidation: Is there text?
Is it accurate?
Are dates included in text?
Consolidation:
Was it done?
Was the source reviewed?
The NPCR Data Quality Evaluation reviewed the “source information” in the CCR – rather than reviewing the records at the hospital level – through a visual review of required data elements – including TEXT DOCUMENTATION. Text is a critical piece related to quality and validates whether or not each source record or abstract contains the necessary/correct information – to validate the code. And if there is text – does it correspond to the coded data value? Are dates included in the text fields to validate items like the date(s) of surgery, chemotherapy and/or radiation therapy so a researcher could determine if patient received adjuvant tx? Text is evaluated with the coded data values – for accuracy and completeness from the reporting source. Because if the pieces of the puzzle are not correct – it could impact the outcome.
Let me show you some of the common errors related to stage that we encountered – next slide

12. Common issues:
Tumor Size: 999 vs. Specific Value Using the Correct Report Lack of Documentation Size Eval: Applying Correct Rules Update/Consolidation Neoadjuvant therapy
Let me start with Tumor Size: we found that it was coded to unknown or to 990/991 – C50 – when there was a specific size available in the text fields. There were also issues regarding whether the size should come from the imaging, the physical exam or the pathology specimen – when all three were available. And our favorite – there were often times where we could find no text to validate the code that had been entered.
And related to Tumor size – the Size Eval field also had data quality issues. While this may have been a CS data item, and we know that CS is no longer required by NPCR– the correct coding or knowledge related to this data item results in the correct assignment to either a clinical or pathologic category. We found issues related to applying the rules – bx vs. resection; or often this item wouldn’t get updated when additional information was added. And we saw data quality errors when patient received neoadjuvant therapy and the clinical size should have been used instead of the pathologic size.

13. Data Item Accuracy Data Item Accuracy 95% CI Minimum Maximum
Primary Site
99.90%
99.8% %
99.00%
100.00%
Histologic Type
98.30%
97.8%--98.7%
94.30%
Behavior
99.70%
99.5%--99.9%
97.90%
Grade
94.50%
93.8%--95.3%
84.60%
99.20%
Laterality
98.90%
Derived Summary Stage 2000
96.10%
95.5%--96.8%
90.30%
This slide is for all sites combined. Overall, the quality of the tumor-specific data items was very good. However accurate information re: tumor grade had remained an issue

14. Data Quality Evaluation – Version 2 Data sources
Consolidated record level file - Diagnosis years 2008 – 2014
Focus: Select cancer sites: Abstract record level file
Urinary Bladder* Accuracy & Consolidation
Melanoma of skin*
Colon Policy and Procedure Manual
Lung
Female Breast Multiple Primary Rules
Prostate
NPCR is currently in the process of completing the Data Quality Evaluation – Version 2 with our states. The cancer sites that are included in the data item review has changed slightly where you see that we added bladder, melanoma, to the original sites. The overall focus is again a review of the central cancer registry’s source information – the abstracts that make up the summary information – for accuracy of text to coded values and an evaluation of the consolidation process/logic as well as a review of cases related to the application of the multiple primary rules – thus this is why bladder and melanoma of skins were added to the mix.

15. Methodology Reconsolidation process MPH Rules
Assess data agreement and reproducibility
Consistent interpretation and abstracting
Estimate agreement rates
Identify training opportunities
MPH Rules
Determine agreement and consistency in applying rules
During the reconsolidation process, data were assessed for agreement and reproducibility – to determine the consistency in interpretation and abstracting, estimate agreement rates, and identify training opportunities (if needed). The sample size included 438 cases from each state, depending upon data availability. Then, the multiple primary and histology rules were applied to determine agreement and consistency in applying the rules. A sample of records with a sequence number >01, for all sites, was selected and each tumor for that patient was reviewed.

16. Data Quality Factors Use most accurate or specific information
DQE – Version 2 results shows Consistently, each CCR’s identified errors are related to these issues:
Most accurate or specific information wasn’t used to assign the proper code; e.g., date of diagnosis, primary site, histology
An unknown or less specific code was recorded when text provided information for the known or more specific code; e.g., incorrect grading system used or highest grade not assigned; specific or combination histology information was available
Text information was lacking to support the assigned code
Information from a metastatic site rather than the primary site was used to assign codes; e.g., histology, behavior, grade; subsequent primary assigned for metastatic disease
Important to code the full extent of the stage; e.g., LN involvement or metastatic disease was missed, progression of disease was recorded as mets, incorrect behavior assigned which directly affects stage
Subsequent treatment was recorded as first course; when using COC treatment guidelines, failed to record date of death as reason no treatment; no treatment and active surveillance are considered treatment; diagnostic procedures were recorded as definitive treatment or were used to assign dates of first course.
Use most accurate or specific information
Code known over unknown or less specific
Document codes with text
Primary site vs. metastatic site
Stage
Treatment
First course treatment vs. subsequent treatment
No therapy/Active Surveillance
Diagnostic vs. definitive procedures

17. Multiple Primary Factors
Primary site
Synchronous tumors in paired organs
Metastatic disease
Progression of disease
Same histology
Coding error
Non-reportable histology
Ambiguous terminology
Sequence number
Factors associated with the multiple primary issues consistently related to assigning the primary site; e.g., 4th digit for colon cases, lesions in both breasts, metastatic site or progression of disease recorded as new primaries, same histology w/n or beyond established timeframe (colon >1 yr apart, lung w/ same laterality/histology w/n 3 yrs). Coding errors in either the site or histology lead to incorrect disposition of multiple records. Histologies without a definitive diagnosis that includes “malignant” are not reportable but were assigned as a new primary. An ambiguous term was used to assign a new primary; e.g., suspicious cytology for bladder. Sequence number didn’t reflect the tumors over the patient’s life; e.g., h/o previous cancer.

18. This infographic provides a snapshot of the evaluation year for all programs combined. You may have trouble reading this but I wanted to show it again. In the right hand corner you can see the errors by primary site, lung having the most errors, 24%. In the bottom left hand corner you see the errors categorized. Stage and treatment have about equal percent of errors. And in the bottom right, it shows the data elements with lowest accuracy proportions.
Overall a 97% accuracy rate for the ten registries that participated is a job well done. I just want to thank Dylan Holt from Westat for putting this together for us.

19. For year 2 of this DQE, all CCRs combined, the data quality and the multiple primary accuracy rate is 97.9%. Treatment information contributed the most to the errors identified. Ten CCRs were evaluated contributing >4,000 reconsolidated cases with >93,000 data items reviewed and consolidated. The MPH sample contained >8,500 tumors from >3,800 patients. For the sites shown on this slide, other sites had the lowest agreement for the MPH evaluation followed by colon. I’ll show another slide later that breaks this down further. For the data quality portion, melanoma had the highest agreement while bladder had the lowest.

20. NPCR DQE Plan for 2020-2023 Three focus areas:
Completeness of staging, biomarker, treatment, and treatment- related information
Validation of staging, biomarker, treatment, and treatment-related information
Evaluation of duplicate rates
Soon we will be launching our next version of the NPCR Data Quality Evaluation. It will run 2020 – 2023 and will focus on: READ SLIDE

21. Thank you Mary Lewis, CTR
Thank you for this opportunity to share with you the progress of the NPCR Audits.
Click to enter your Division Name here