Download presentation
Presentation is loading. Please wait.
1
Patient Matching Algorithm Challenge
Informational Webinar Caitlin Ryan, PMP| IRIS Health Solutions LLC, Contract Support to ONC Adam Culbertson, M.S., M.S. | HIMSS Innovator in Residence, ONC
2
Background on Matching About the Challenge
Agenda ONC Overview Background on Matching About the Challenge Eligibility Requirements Registration Project Submissions Winners and Prizes Calculating Metrics Creating Test Data Challenge Q&A
3
Office of the National Coordinator for Health IT (ONC)
The Office of the National Coordinator for Health Information Technology (ONC) is at the forefront of the administration’s health IT efforts and is a resource to the entire health system to support the adoption of health information technology and the promotion of nationwide health information exchange to improve health care, ONC is organizationally located within the Office of the Secretary for the U.S. Department of Health and Human Services (HHS), ONC is the principal federal entity charged with coordination of nationwide efforts to implement and use the most advanced health information technology and the electronic exchange of health information.
4
ONC Challenges Overview
The statutory authority for Challenges are hosted under Section of the America COMPETES Reauthorization Act of (Public L. No ). ONC Tech Lab - Innovation Spotlight areas of high interest to ONC and HHS Direct attention to new market opportunities Continue work with start-up community and administer challenge contests Increase awareness and uptake of new standards and data
5
ONC Roadmap Connecting Health and Care for the Nation: A Shared Nationwide Interoperability Roadmap Released in 2015 A 10 Year Vision to Achieve An Interoperable Health IT Infrastructure Section L: Accurate Individual Data Matching, states that patient matching is fundamental requirement in achieving interoperability.
6
Patient Matching Definition
Patient matching: Comparing data from multiple sources to identify records that represent the same patient. Also called merge-purge, record linkage and entity resolution in other fields. Entity Resolution, Data Matching, Record Linkage, Object Identification, Merge-Purge, Linked Data, Data Deduplication, Identify Resolution, Schema Matching
7
Source: Culbertson, A. Patient Matching A-Z, Wednesday, March 2ndst HIMSS 2016, Las Vegas, NV
8
Significant Dates in (Patient) Matching
Campbell, K et al A Comparison of Link Plus, The Link King, and a “Basic” Deterministic Algorithm A Framework for Cross-Organizational Patient Identity Management HIMSS Patient Identify Integrity Toolkit, Patient Key Performance Indicators RAND Health Report Identity Crisis: An Examination of the Costs and Benefits of a Unique Patient Identifier for the US Health Care System 2008 2015 Kho, Abel N., et al Design and Implementation of a Privacy Preserving Electronic Health Record Linkage Tool Dunn Record Linkage 1946 Fellegi & Sunter A Theory of Record Linkage 1969 Winkler Matching and Record Linkage 2011 Soundex US Patent 1918 Newcombe, Kennedy, & Axford Automatic Linkage of Vital Records 1959 Grannis, et al Analysis of Identifier Performance Using a Deterministic Linkage Algorithm 2002 Grannis, et al Privacy and Security Solutions for Interoperable Health Information Exchange 2009 Audacious Inquiry and ONC Patient Identification and Matching Final Report 2014 Joffe et al A Benchmark Comparison of Deterministic and Probabilistic Methods for Defining Manual Review Datasets in Duplicate Records Reconciliation HIMSS Patient Identity Integrity Exemplars Intelligence Industry post 9/11 realized that had a problem matching non-Anglo names Problem faced in many verticals such intelligence finance, marketing A Comparison of Link Plus, The Link King, and a “Basic” Deterministic Algorithm Campbell, K et al. (2008) (Notes: Campbell, Kevin M., Dennis Deck, and Antoinette Krupski. "Record linkage software in the public domain: a comparison of Link Plus, The Link King, and a basic' deterministic algorithm." Health Informatics Journal 14.1 (2008): 5-15.) A benchmark comparison of deterministic and probabilistic methods for defining manual review datasets in duplicate records reconciliation Joffe et al. Jan 14. (Notes: Joffe E, Byrne MJ, Reeder P, et al. A benchmark comparison of deterministic and probabilistic methods for defining manual review datasets in duplicate records reconciliation. Journal of the American Medical Informatics Association : JAMIA. 2014;21(1): doi: /amiajnl ) “work of the IIR will lead to establishing metrics of patient matching technology approaches and create a pathway for evaluating solutions.” and " IIR to develop a vision, strategy, and implementation plan for the near-term deployment of consistent patient data matching in health that builds on the body of work from HHS’s Office of the National Coordinator for Health IT (ONC) and healthcare community partners. The IIR will also assess the longer-term applicability of identity management methods, processes and technologies currently in use in healthcare and other sectors.” Design and implementation of a privacy preserving electronic health record linkage tool in Chicago." Journal of the American Medical Informatics Association (2015): ocv038. Dusetzina, Stacie B., et al Linking Data for Health Services Research: A Framework and Instructional Guide HIMSS hires Innovator In Residence (IIR) focused on Patient Matching Source: Culbertson, A. & Miller, K., Patient Matching EHR Ailments: Going from Placebo to Cure, Tuesday, March 1st HIMSS 2016 Las Vegas, NV
9
The 5 Step Data Match Process
Data pre-processing Indexing Comparison Classification Evaluation Pre-processing: characterizes the data and ensures the elements have the same structure and the content follows the same format Indexing: organizing the data to support better pairing (blocking and the use of a blocking key common) Comparison: identifying the similarity between two records seeking comparison vectors Classification: based on comparison results, records are found to be matches, non-matches or potential matches Data pre-processing: 3 to 4 steps: Remove unwanted characters and words Expand abbreviations and correct misspellings Segment attributes into well-defined and consistent output attributes Verify the correctness of attribute values Important to note that this step must not overwrite the original input data – but new attributes can and should be created that contain the cleaned and standardized data. Evaluation: comparing match results with the known ground truth or gold standard. Data Matching; Concepts and techniques for record linkage, entity resolutions, and duplication detection, Peter Christen 2012
10
Data quality issues make matching more complicated
Problem Patient data matching has been noted as one key barriers to achieving interoperability in the Nations Road Map for Health IT Patient Matching causes issues for over 50% of health information managers1 Problem will increases as we increase the volume of health data sharing Data quality issues make matching more complicated Lack of knowledge of patient matching algorithms performance, adoption of metrics 1) MMS - all known milestones ONC Annual Plan – subset of database focused on ONC ownership incremented into quarterly milestones ONC Quarterly Dashboard – subset of annual plan for quarterly monitoring and cascade into NC & SES perforamnce plans ONC Bi-Weekly / Monthly Milestone monitoring report 1)
11
Data entry errors are compound data matching complexity
Data Quality Data Quality is a Key Garbage in and Garbage out Data entry errors are compound data matching complexity Various algorithmic solutions to address these, not perfect Types of errors: Missing or Incomplete Values Inaccurate data Fat finger errors Information is out of date Transposed names Misspelled names
12
"If you can't measure it, you can't improve it.“
Solution "If you can't measure it, you can't improve it.“ -Peter Drucker
13
ONC’s Patient Matching Algorithm Challenge
The goal of this challenge is to: Bring about greater transparency and data on the performance of existing patient matching algorithms, Spur the adoption of performance metrics for patient data matching algorithm vendors, and positively impact other aspects of patient matching such as deduplication and linking to clinical data. Website:
14
Eligibility Requirements
There is no age requirement for this challenge. All members of a team must meet the eligibility requirements. Shall have registered to participate in the Challenge under the challenge requirements by ONC. Shall have complied with all the stated requirements of the Challenge. Businesses must be incorporated in and maintained a primary place of business in the United States; Individuals must be a citizen or permanent resident of the United States. Shall not be an HHS employee.
15
Eligibility Requirements (cont’d)
May not be a federal entity or federal employee acting within the scope of their employment. Federal grantees may not use federal funds to develop COMPETES Act challenge applications unless consistent with the purpose of their grant award. Federal contractors may not use federal funds from a contract to develop COMPETES Act challenge applications or to fund efforts in support of a COMPETES Act challenge submission. Participants must also agree to indemnify the Federal Government against third party claims for damages arising from or related to Challenge activities.
16
Receive performance scores and appear on a Challenge leaderboard.
Challenge Process Register your team Contestants will unlock a test data set provided by ONC to run algorithms Run Algorithm Submit results for evaluation which will be scored against an “answer key” Receive performance scores and appear on a Challenge leaderboard. Repeat submission until satisfied with the result or have hit 100 submissions or end date has passed. Participants will unlock and download the test data set at the time of registration. Participants will then run their algorithms and submit their results to the scoring server on the Challenge Website. A small set of true match pairs (that have been created and verified through manual review) exist within the large data set and will serve as the “answer key” against which participants’ submissions will be scored.
17
Challenge Process Synthetic Data Download Data in CSV File
Set Download Data in CSV File Submit Linked Data Scoring Server (Gold Standard)) Returns a Score Brain image source: Submit Results to Leader Board
18
Create a username and password (1 account per team)
Registration Visit the challenge website and fill in all required fields of the registration form Create a username and password (1 account per team) Enter a team name which will be used on the leader board Can be used to keep team identities private Acknowledge and agree to all terms and rules of the Challenge
19
Known potential duplicate pairs mimic real world scenarios
Challenge Dataset Dataset synthetically generated by Just Associates using a proprietary software algorithm Based on real-world data in an MPI and actual data discrepancies in each field across the fields Known potential duplicate pairs mimic real world scenarios Does not contain PHI Approximately 1M Patient Records Available early June, will send out when data set is made available
20
Fields include: Challenge Dataset
Enterprise ID, LAST NAME, FIRST NAME, MIDDLE NAME, SUFFIX, DOB, GENDER, SSN, ADDRESS1, ADDRESS2, CITY, STATE, ZIP, PHONE, PHONE2, , ALIAS, MOTHERS_MAIDEN_NAME, MRN, SSNs(Most are within the 800 range) Data Format CSV Also available in FHIR bundle
21
1 John Smith 1-1-1990 202-223-9910 Washington, DC
Challenge Dataset 1 John Smith Washington, DC 2 Carol Jones Bethesda, MD 3 Bobby Johnson Arlington, VA 4 Johnny Smith Washington, DC 1 John Smith Washington, DC 4 Johnny Smith Washington, DC Carol Jones Bethesda, MD 3 Bobby Johnson Arlington, VA Scoring Server
22
One dataset will be provided to all participants
Submission Process One dataset will be provided to all participants Participants will submit their matches to the ONC scoring server The answer key, separate from the dataset provided to participants, will be used to score submissions. Submission Data format CSV, Enterprise ID 1, Enterprise ID 4, 0.90 Optionally FHIR bundle Submit Enterprise ID, Enterprise ID linked to, Optional submit confidence score for probabilistic algorithms
23
Project Submissions Teams will submit the results, matched records, of their algorithm tests The submission period* will be open for 3 months 100 submissions from each individual/team Can submit at any time during the “submission period” Challenge will open on June 12th 12:00 p.m. E.S.T. Submissions will be allowed until 11:59pm on the last day of the submission period *Submission period dates have not been determined. Once the test data set is available we will add these dates to the challenge website.
24
Tradeoffs between Precision and Recall
Project Submissions Calculation Precision Recall Tradeoffs between Precision and Recall F-Score is the harmonic mean between precision and recall F score. = Precision If Jon Smith is the record you are looking for and you search for Jon Smith. Precision is the likelihood that the record you have that corresponds to john smith is actually Jon Smith Recall – If Bob Jones has 100 records in a database how many of Bob Jones records to you get? If you get 90 Recall you got 90 of bob Jones Records Different use cases have different requirements, and different balance of tradeoffs. Security vs Healthcare
25
Participants will receive a
Returned Results Participants will receive a F-Score Precision Recall Run ID Month one will include a beta period New matches found will be manually reviewed to determine match status Previous submissions will be rescored with updated answer key and leader boards updated After beta period all future submissions will be scored against updated answer key only
26
Leader Board Example
27
Leader Board Example
28
Precision Recall Best first f-score run
Winners and Prizes The Total Prize Purse for this challenge is $75,000 Judging will be based upon the empirical evaluation of the performance of the algorithms. Highest F-Score 1st- $25,000 2nd- $20,000 3rd- $15,000 Best in Category: ($5,000 ea. Category) Precision Recall Best first f-score run Judging will be based upon the empirical evaluation of the performance of the algorithms. For the purposes of scoring, all scores will be assessed to three places after the decimal (i.e., 0.xxx). F-score judging and award conditions: In general, the top three algorithms with the highest F-scores will be selected as the winners. In the event that a team achieves F-scores that would consecutively occupy 1st, 2nd, and/or 3rd place (or any consecutive combination or otherwise (e.g., (1st and 3rd) or (2nd & 3rd)), the team will be awarded only one prize, which will be the highest available prize. After that award, that teams remaining score(s) will be skipped until a different competitor can be awarded the next prize level. In the event of a tie in F-scores, the winners will be awarded their prizes based on least amount of “runs”/tries that it took to get such score. For example, if two teams tied for first place and one got its high F-score on its 31st try and the other team on its 57th try, the first team would be awarded 1st place and the other team 2nd place. The same tie-breaking approach will be applied for all award positions. If the least amount of tries method subsequently results in a tie, the teams will be awarded that place and the prize will be split evenly to all who won that place. Best in category prizes In general, best in category prizes will be made for precision, recall, and best first run. Unlike the F-score award rules, a team is permitted to win one or all of these “best of” category prizes. In the event that a tie occurs for the precision and recall categories, the same least amount of tries method (as noted in F-score) will be used to break the tie and same even award split method will be used if the least amount of tries method does not break the tie. If the best first F-score run category results in a tie, the award will be split evenly to all who won the category.
29
Precision: Best precision with recall >= 90%
Best in Category Best F-Score 1st Place 2nd Place 3rd Place Best 1st Run F-Score: This prize will be awarded to the contestant/team whose first submission to the scoring server results in the highest f-score. Precision: Best precision with recall >= 90% Recall: Best Recall with precision >= 90%
30
Metrics for Algorithm Performance
31
Ideal outcome of any matching exercise is correctly answering this one question hundreds or thousands of times, Are these two things the same thing? Correctly identifying all the true positives and true negatives while minimizing the number of errors, false positives and false negatives Patient Matching Goal Source: Culbertson, A. Patient Matching A-Z, Wednesday, March 2ndst HIMSS 2016, Las Vegas, NV
32
Patient Matching Terminology
True Positive- The two records represent the same patient True Negative- The two records don't represent the same patient Source: Culbertson, A. Patient Matching A-Z, Wednesday, March 2ndst HIMSS 2016, Las Vegas, NV
33
Patient Matching Terminology
False Negative: The algorithm misses a record that should be matched False Positive: The algorithm creates a link to two records that don’t actually match Source: Culbertson, A. & Miller, K., Patient Matching EHR Ailments: Going from Placebo to Cure, Tuesday, March 1st HIMSS 2016 Las Vegas, NV
34
Evaluation EHR A EHR B Truth (Gold Standard) Algorithm Match Type
Jonathan True Positive Sally Non-Match True Negative False Positive Jon False Negative Good Bad Add color to highligh differences Source: Culbertson, A. Patient Matching A-Z, Wednesday, March 2ndst HIMSS 2016, Las Vegas, NV
35
Evaluation EHR A EHR B Truth (Gold Standard) Algorithm Match Type
Jonathan True Positive Sally Non-Match True Negative False Positive Jon False Negative Bad Add color to highligh differences Source: Culbertson, A. Patient Matching A-Z, Wednesday, March 2ndst HIMSS 2016, Las Vegas, NV
36
Evaluation EHR A EHR B Truth (Gold Standard) Algorithm Match Type
Jonathan True Positive Sally Non-Match True Negative False Positive Jon False Negative Bad Add color to highlight differences Source: Culbertson, A. Patient Matching A-Z, Wednesday, March 2ndst HIMSS 2016, Las Vegas, NV
37
Precision = True Positives / (True Positives + False Positives)
Evaluation Truth Algorithm Positive Negative True Positive False Positive False Negative True Negative Precision Recall = True Positives / (True Positives + False Negatives) Recall Source: Culbertson, A. Patient Matching A-Z, Wednesday, March 2ndst HIMSS 2016, Las Vegas, NV
38
Tradeoffs between Precision and Recall
Evaluation Calculation Precision Recall Tradeoffs between Precision and Recall F-Score is the harmonic mean between precision and recall F score. = Precision If Jon Smith is the record you are looking for and you search for Jon Smith. Precision is the likelihood that the record you have that corresponds to john smith is actually Jon Smith Recall – If Bob Jones has 100 records in a database how many of Bob Jones records to you get? If you get 90 Recall you got 90 of bob Jones Records Different use cases have different requirements, and different balance of tradeoffs. Security vs Healthcare
39
Creating Test Data Sets
40
Development of Test Data Set
Patient Database Select Potential Matches (aka Adjudication Pool) Manual Reviewer 1 Reviewer 2 Reviewer 3 Human-Reviewed Match Decisions (Answer Key == Ground Truth Data Set) MITRE help in graphics Compare Algorithm and Test Data Set Source: Culbertson, A. & Miller, K., Patient Matching EHR Ailments: Going from Placebo to Cure, Tuesday, March 1st HIMSS 2016 Las Vegas, NV
41
Development of Ground Truth Sets
Identify data set that reflects real word use case Develop potential duplicates Human adjudication review and classification Match or Non-Match Estimate truth Pooled methods using multiple matching methods
42
Issues In Establishing Ground Truth
First step in evaluation is to determine why the evaluation is being conducted Different truth for different applications Security Applications vs Patient Health Record What is the cost of missing a match? Security: Lives are lost Health: Patient safety event, missed medications, allergies, etc… death But…this is situation today. What is the cost of wrongly identifying a match? Security : Passenger is inconvenienced / delayed Health: Patient safety event, wrong medication, treatment, liability, death Criteria for truth must be carefully established and well-understood E.g. Question posed to annotators must be carefully phrased Summary for Healthcare Use Case
43
Issues In Establishing Ground Truth (cont’d)
Different truth for different applications Credit check Security applications Customer support De-duplication of mailing lists What is the cost of missing a match? New record entered into database Irritated customer Lives are lost Criteria for truth must be carefully established and well-understood by annotators Question posed to annotators must be carefully phrased
44
Issues In Establishing Ground Truth (cont’d)
How much time / expertise is available to judge (/discount) false positives? Needs to reflect real word test use case Evaluation results are only as good as the truth on which they are based And only as appropriate as the evaluation is to the task that will be performed with the operational system Absolute recall impossible to measure without completely known test set (i.e. “You don’t know what you’re missing.”) Estimate with pooled results
45
B Smith Bill Smythe William Smythe W Smith ??
Examples B Smith Bill Smythe William Smythe W Smith ?? DOB: 10/12/1972 October 11, 1972 December 10, 1972 12/10/72 October 12, 1927 Use all same name example throughout
46
Webinars on how to participate and challenge overview
Get Involved Webinars on how to participate and challenge overview May 24th Kicking off Patient Data Matching Algorithm Challenge in June Participant Discussion Board Website:
47
Acknowledgments Thank you to the following individuals and organizations for their involvement in the planning and development of this challenge : Debbie Bucci and the ONC Team HIMSS North America, Tom Leary, HIMSS Greg Downing, HHS Idea Lab, ONC Jerry and Beth Just and the Just Associates Team Keith Miller and Andy Gregorowicz, MITRE Caitlin Ryan, IRIS Health Solutions Capital Consulting Corporation Team
48
FOR ADDITIONAL QUESTIONS/INFORMATION CONTACT:
Adam Culbertson, Debbie Bucci, (preferred) Phone:
49
Thank you for your interest!
The ONC Team
Similar presentations
© 2025 SlidePlayer.com Inc.
All rights reserved.