1. Reasons for Rework in Clinical Data Provisioning: A Root-Cause Analysis
Amanda L. Do, MPH1,2, Ruby Y. Wan, MS1,2, Robert W. Follett1,2, Marianne Zachariah1,2,
and Douglas S. Bell MD, PhD1,2
1Department of Medicine, UCLA David Geffen School of Medicine, Los Angeles, CA 2UCLA Clinical & Translational Science Institute, Los Angeles, CA
Accelerating Discoveries Toward Better Health
Secondary use of electronic health record (EHR) data in clinical research has become more prevalent with the deployment of large-scale EHR systems. Challenges frequently arise in repurposing clinical data for research, and many investigators are unable to relate their needs to the existing clinical data.
To address these challenges, the UCLA Clinical & Translational Science Institute’s (CTSI) Informatics Program (IP) has implemented, and continues to refine, a data provisioning service model. The model includes consulting and programming services to assist investigators in defining EHR data requests that address their research questions. However, investigators sometimes find that a dataset does not meet their needs only after it has been provided, which costs the IP additional time and resources to remedy. The IP seeks to understand the reasons for this kind of rework in an effort to identify and rectify their root causes. IP hopes to use the findings from this analysis to further enhance the service model and minimize future rework.
IP acts as the storefront for provisioning healthcare-related datasets for research projects at UCLA:
Provide initial consultation (orient the investigators to the request process and ask them to fill out the data request form)
Data request form: standardized template with two components 1) definition of patient selection criteria and 2) selection of demographics and other output variables e.g. diagnoses, lab results, etc.
Review, analyze and optimize data request to ensure institutional approvals (IRB and Compliance) and technical feasibility (data requested is attainable from the existing EHR)
Explore data sources, program and test data extraction code, quality check data
Data source: UCLA’s Integrated Clinical and Research Data Repository (xDR), a clinical data warehouse of UCLA’s CareConnect (Epic) EHR data, linked with data from older legacy systems and other sources (e.g. Transplant Registry)
Communicate with the investigators when clarification is needed
Deliver data through secure, compliant methods
Data requests were considered complete once all template variables were queried, exported, and delivered to the investigators.
Study Data
165 data requests completed between March 2014 and February 2017
Reviewing Rework Requests
Two IP data consultants reviewed all completed data requests and associated materials:
Data request form filled out by the investigator and confirmed by IP staff
Consulting and programming notes
Project management system records (REDCap and JIRA)
correspondence with the investigator or amongst IP staff
Institutional Review Board (IRB) applications, approval notices, and other related study materials
Additional files sent by the investigator e.g. study protocols, recruitment materials
Defining Rework and Assigning Rework Reasons
Rework was defined as any instance in which the original query needed to be revisited and modified after the original dataset was extracted and delivered.
Based on previous work and experience, IP consultants classified reasons for rework into 6 categories (Table 1). Each request was then assigned any one or combination of the 6 reasons by IP consultants and programmers.
Calculating Rework Cost
Total hours logged by consultants and programmers for each rework request were obtained from project management system records
Consultants and programmers confirmed, out of the total number of hours, the time spent on each rework reason. Requests could have more than 1 rework reason.
Cost was then calculated for each request based on IP’s current hourly service rates
Introduction
Results
Distribution of Rework
Out of 165 data requests completed between March 2014 and February 2017, 39 requests (23.6%) required rework. 16 out of the 39 rework requests (41.0%) had only 1 cause for rework. The remaining 23 requests (59.0%) had more than 1 rework reason: 14 requests with two reasons, 3 with three or four reasons, 2 requests with five reasons, and 1 request with all six possible rework reasons. By year, there were 6 rework requests in 2014, 22 in 2015 and 11 in 2016.
Hours and Cost Spent on Rework
Total time spent on rework: 795 hours
Total cost of rework (based on current hourly rates): $102,000
Adjustment of patient selection criteria was the greatest cause for rework, accounting for ~446 hours and a cost of ~$57,800 (Figure 1, Figure 2)
Total time spent on original rework requests (prior to any rework): 1,268 hours
Total cost of original rework requests (prior to any rework): ~$149,400
Average cost of original rework request (prior to any rework): ~$3,800
Average cost per any request (with or without rework): ~$2,600
Overall, most investigators were able to use the data provisioned through the UCLA CTSI IP service model without needing to request revisions. Nonetheless, substantial resources ($102K) were still spent on rework, costing the equivalent of 68% of the original data extraction expense.
Adjustment of patient selection criteria and additional data variables have imposed the most significant strains on IP resources, accounting for ~80% of all rework.
The root causes of rework could be insufficient preparation on the part of the investigators in defining the study scope and planning data analysis or lack of careful upfront analysis of the research plan by IP. However, the requests that ended up needing rework may have been inherently more complex, e.g. patient populations harder to define, as evidenced by the higher average cost ($3,800) compared to that ($2,600) of all requests, regardless of rework.
To reduce the occurrence of rework, IP has continually worked to improve the service model to ensure patient selection criteria and output variables will adequately answer the investigators’ research questions. This includes fine-tuning the data request form to make it more comprehensive, e.g. addition of descriptive footnotes. IP has also implemented a more rigorous scientific review during initial consultations.
The project described was supported by the National Center for Advancing Translational Science through UCLA CTSI Grant UL1TR
Data Provisioning Service Model
Methods
Figure 1: Total Hours of Rework, by Reason
Figure 2: Cost of Rework, by Reason
Discussion & Conclusion
Table 1: Rework Reasons and Counts
Acknowledgments