S&I Framework LRI Validation Suite Vocabulary Testing Proposal (Lab Results Interface) Robert Snelick National Institute of Standards and Technology October 18 th, 2011 Contact:

2 How do we Test the Required Vocabulary Standards? Limit discussion to LOINC –In particular the use of LOINC for the observation result identifier (OBX.3) How do we interpret the Meaningful Use Requirements? –“Incorporate results. Electronically attribute, associate, or link a laboratory test result to a laboratory order or patient record.” –How do we ensure that the LOINC codes are used meaningfully? How do we ensure this: “The LOINC code is the anchor that discretely and accurately identifies the lab test that was ordered/performed. No matter what a hospital or physician chooses to call the test, the LOINC code is the one constant that tells everyone which test it was.” How do we test the sender (the LIS)? How do we test the receiver (the EHR)? How do we test the universe of possible codes? Considerations: –Meaningful Use Requirements –Labs and EHR use local codes today –EHR likely to translate/map LOINC code to local code and display local text name on user’s screen (what impact does this have on testing?) Two perspectives: –“LOINC codes should not be tested for “correct” values, just that it is a valid LOINC code”—what is the purpose of the standardized code then? –“If local codes are sent and used in conjunction with the standardized codes then we are no closer to achieving terminology interoperability”

3 Meaningful Use Requirements LOINC is a named standard for lab test results in the § g Reportable Lab Results Stage 1 ONC certification criterion. The § (h) Incorporate laboratory test results criterion states: –Receive results. Electronically receive clinical laboratory test results in a structured format and display such results in human readable format. –Display test report information. Electronically display all the information for a test report specified at 42 CFR (c)(1) through (7). –Incorporate results. Electronically attribute, associate, or link a laboratory test result to a laboratory order or patient record. The criteria use the following verbiage when referring to LOINC: –“§ (g) Reportable lab results. Electronically record, modify, retrieve, and submit reportable clinical lab results in accordance with the standard (and applicable implementation specifications) specified in § (c) and, at a minimum, the version of the standard specified in § (c).” –§ (c) states: “Standard. Logical Observation Identifiers Names and Codes (LOINC®) version 2.27, when such codes were received within an electronic transaction from a laboratory (incorporated by reference in § ).” –In the ONC Final Rule Preamble for i Exchange Clinical Information and Patient Summary Record ONC stated the following: “For the purposes of electronically exchanging a patient summary record, we expect the patient summary record to include health information that is coded, where applicable, in accordance with adopted vocabulary standards. Therefore, unless otherwise required in the context of a meaningful use objective and measure, an eligible professional (or eligible hospital) would be permitted to map or crosswalk local/proprietary codes to the adopted vocabulary standards prior to transmitting a patient summary record.” –MU Stage-2 requirements are not finalized (See the last slide for a summary of current discussions)

4 Testing LOINC—LIS (Sending Application) Lab Test Order (options) –Test case will indicate a lab test order on a paper form –Test case may indicate the lab test order in electronic format This is simulated (i.e., it is assumed that the LIS received the order)? A specific LOINC code in OBR.4 may be specified (Lab dependent) Lab Results –Test Case will provide lab test order/results data sheet (indicates all the results of the lab test) –A LIS user enters the information into the LIS system (including the appropriate LOINC code for the order) –The LIS system generates an LRI message Validation –The message is imported in the LRI LIS validation tool –The LIS LRI validation tool validates the message –For LOINC the validation tool will examine the content of OBX.3 for a valid observation identifier (LOINC code)

5 Testing LOINC—LIS (Sending Application) For a given order request a finite number of valid observation identifiers (LOINC codes) are identified and associated with that order For example if the order requested is a Hemoglobin in blood test (LOINC code is 718-7) then the following LOINC codes would be considered valid: LOINC CodeTest Long Name Hemoglobin [Mass/volume] in Blood (Methodless) Hemoglobin [Mass/volume] in Blood (by Calculation) Hemoglobin [Mass/volume] in Arterial blood (Methodless) Hemoglobin [Mass/volume] in Capillary blood (Methodless) Hemoglobin [Mass/volume] in Mixed venous blood (Methodless) Hemoglobin [Mass/volume] in Venous blood (Methodless) The LRI LIS validation will check OBX.3 (OBX.3.1 or OBX.3.4) for one of the listed LOINC codes for this test case The LRI LIS validation tool in-scope tests are limited to the most common lab tests (i.e., the entire LOINC database is not in scope) for testing The list of the in-scope lab tests and their respective LOINC codes can be found in a spreadsheet located on the validation suite WG wiki Current version: Categorized Tests for LRI Testing Compiled ST v11.xlsx 1.Test tool can give guidance— default to expert 2.Verify LOINC mappings with Regenstrief

6 LIS Validation Tool Test Flow (Context Based) LIS LIS Test Tool select test case Test Data Sheet Manual entry of test data  Load  Cut/paste  Send HL7 V2 Lab Results Message Use Case Test Case Validation Report EHR transmits Hemoglobin blood test order to the LIS for John Doe along with pertinent demographic and order request data Lab Results Data Sheet John Doe DOB: 05/23/1959 Gender: M Result: 13.7 g/dL Range 13.2 – 16.2 Status: Final A lab test is ordered for a patient The specimen is collected, and is received and processed in the lab The lab result is produced and stored in the LIS database The lab result is transmitted to an ambulatory EHR The lab result is viewed in the ambulatory EHR 1.Local code 2.No code—text 3.LOINC code No LOINC given for results Test tool inspects OBX.3 for a specific LOINC code Inspect for only or inspect for one of 718-7, , , , , and ? For Discussion: 1.No suggested method given (Should we?) 2.With no method given we should expect one from a set of valid LOINC codes for this order (Lab dependent?) 3.If a specific method is given (is this typical/possible?) then should we look for a specific LOINC code in the sent message? For Discussion: 1.Does the LAB expect a local code for the order in a electronic order request? 2.Use Cases to consider: agreement/no agreement between LIS and EHR for local codes It is assumed Lab has capabilities for manual entry Data Entry DB Message

7 Validation of LOINC for Sender of OBX.3 Requirements from LRI IG (OBX.3) –Usage: R; Datatype: CWE –Unique identifier for the type of observation. This field provides a code for the type of observation. OBX-3 in conjunction with OBX-4 Observation Sub-ID should uniquely identify this OBX from all other OBXs associated with this OBR. –LOINC is used as the coding system for this field. Local codes may also be used in conjunction with LOINC codes –When populating this field with values, this guide does not give preference to the triplet in which the standard (LOINC) code should appear. Example: 718-7^Hemoglobin [Mass/volume] in Blood^LN Validation (For context-based testing) 1.Valid LOINC code for the lab test ordered (one of a pre-determined set) 2.Valid OBX.3.1 and OBX.3.2 pair (or OBX.3.4 and OBX.3.5 pair) 3.Valid LOINC format (subsumed by 1) 4.Valid LOINC code from LOINC database (subsumed by 1)

8 Testing LOINC—EHR (Receiving Application) Lab Result LRI test message is sent to the EHR –Test case will indicate the lab result in a data sheet –A LRI message is generated and sent to the EHR –Message will contain a LOINC code in OBX.3 (observation identifier) How to test that the EHR incorporates the LOINC code? –Option 1: Send only the LOINC code in OBX.3 EHR can translate into internal code and display local text string of lab test name If only the LOINC code is sent then this provides a mechanism to better ascertain that the EHR incorporated the LOINC code (since no local code is provided) EHR can use LOINC code directly –Option 2: Send a LOINC code and local code in OBX.3 Is this a necessary option? Can the actual incorporate be circumvented? Validation (details to follow) –The lab results are inspected by viewing the EHR display (GUI) –Certain elements can be displayed as equivalent concept (e.g., lab test display name) –Certain elements are to be displayed exactly as received (e.g., result and units)

9 LRI EHR Test Harness Test Flow—Testing the EHR (Model 1) EHR Lab Message ACK Validation LRI EHR Test Harness Communication LRI Test Message Validation Report Test Data Test Case Use Case Juror Document ACK (LN) SNameValueREF Range FHemoglobin – 16.2 g/dL For Discussion: 1.What are the requirements on the EHR with regards to LOINC— given the LRI IG, CLIA, and meaningful use? 2.What are the EHR display (GUI) requirements for the received lab results? 3.What are the EHR storage requirements for the received lab results? 4.Given that the LOINC code is likely to be translated into the EHR internal representation, what lab test names need to be displayed in exact form and what lab test names can be displayed in an equivalence text (same concept)? 5.What qualifies as “equivalence—same concept”? How is that determined? Can it be? Expert inspector only? Should displaying of the LOINC text (only) be the requirement—i.e., no mapping? 6.What is the impact of sending only the LOINC code (i.e., no local code) on testing? Translated into internal representation Displayed as internal display text representation

10 Processing of Standardized Vocabulary (Proposed) Receiver requirements for Standardized Vocabulary: –The receiver shall persist (store) the original standardized code and the original standardized code text as received in exact representation –The receiver may perform a translation/mapping to a locally defined representation –If a translation is performed the concept of the original standardized code shall be maintained

11 Assessment of Standardized Vocabulary (Proposed) Procedure for assessing the receiver for incorporation of standardized vocabulary: –Note: discussion is limited to standardized vocabulary (i.e., the LOINC code received and associated components, e.g., the display text). The lab results values are not in scope for this discussion. –Where applicable to meet CLIA and MU requirements the receiver shall display on the EHR GUI the equivalent representation of the received coded lab test name. The receiver shall display at least one of the following: Original standardized code text Original standardized code (probably not very useful, but is it acceptable?) Local code text (local display representation) –The receiver shall be capable of demonstrating the persistent of the original standardized code and the original standardized code text. Acceptable methods for attestation: Administrative access to database Inspector approved method –If applicable the receiver shall be capable of demonstrating the linkage of the original standardized code and the locally translated/mapped code. Acceptable methods for attestation: Administrative access to database Browse capabilities of configuration files Inspector approved method

12 What does Equivalent Representation (Concept) Mean? The exact original code The exact original code text (Make this the only acceptable method?) For translated/mapped local code text an equivalent representation as determined by a clinical terminology expert. The following rules and guidance are given to promote consistency in assessment: –Rule: A terminology shall never be made more specific in the translation/mapping. –Rule: A terminology shall never be made more specific in the display of standardized terminology. –Guidance: A limited number of synonyms are provided to assist the clinical terminology expert. Note: a predefined finite set is not possible—there are too many possible equivalent local representations. –Guidance: The inspector must consider the context in which the code is used as this may impact the translation/mapping.

13 Discussion (Mapping to a more general code) What should be the assessment if a specific terminology code is received that is mapped to a less specific code? –For example, (Hemoglobin [Mass/volume] in Mixed venous blood) to (Hemoglobin [Mass/volume] in Blood) Assuming this is relevant –This mapping will lead to a loss of information if the original code is not persisted. –Is it valid to display a representation that has less specificity? –Is it valid to display a representation that has less specificity as long as the original data is persisted in the system? –Is it acceptable for a loss of information to occur when data is rendered as a report or forwarded on to another system (e.g., public health)? That is, we sent a specific code and then a general code is forwarded on to public health. Is it acceptable for either the general or specific LOINC code to be forwarded? LOINC codes often indicate a specific processing method for a particular lab test; the NLM has recommended use of LOINC codes that do not specify a method (i.e., are “method-less”) –What is the testing impact? –Do we focus on the general codes? –Does method-less mean less specific?

14 Can we Leverage the MU Stage-1 Requirements? LAB (or any sending system or tool) EHR (System under Test) Public Health HL7 ELR IG 2010 Direct Data Entry ORU Message ORU Message Elements 1.Meaningful Use Stage-1 requirements includes the EHRs to send reportable lab results to public health using the HL7 Version Implementation Guide: ELR to Public Health (Release 1: February 2010) 2.NIST test tool already exists and used for Meaningful Use stage-1 certification 3.Identify which test cases fall into the category of reportable lab results 4.Identify the differences in the implementation guides (data elements, etc.) 5.Provide data sheet for additional information that needs to be entered into the EHR system 6.Correlate content of public health message to original LRI test case message 7.This process provides additional automated testing 8.Test EHR workflow ORU Elements Meaningful Use Scope Stage 1 NIST Message Validation Tool Meaningful Use Scope Stage 2 S&I Framework LRI

15 LRI EHR Test Harness Test Flow—Testing the EHR (Model 2) EHR Lab Message ACK Validation LRI EHR Test Harness Communication LRI Test Message Validation Report Test Data Test Case Use Case Juror Document ACK (LN) SNameValueREF Range FHemoglobin – 16.2 g/dL For Discussion: 1.Employ a testing technique that leverages other EHR capabilities and MU requirements 2.In this case, leveraging reportable lab results to public health 3.It is expected that the EHR send exactly the same LOINC code as it received? 4.Can the EHR translate a specific LOINC code to a general LOINC code? 5.In general, what transformation rules are acceptable if any? Translated into internal representation Displayed as internal display text representation MU-1 Public Health Message Public Health Lab Message Validation Tool (LN)

16 Other Issues for Testing Standardized Vocabulary EHR – Options for implementing standardized vocabulary –Use standardized coding internally –Map to local codes –Need to account for options in test procedure How do we test for complete coverage of recommended standardized vocabulary? –Create all messages for all possible (recommended) terms (for important code system, e.g., LOINC)—This is the preferred approach –Alternative approach: Create a subset and inspect/verify configuration tables for coverage and accuracy –The LAB is not restricted to sending the recommended list of LOINC codes and the EHR should not fail when it does not recognize a LOINC code. How can this be tested; should it be tested? –We should test that the EHR recognizes an invalid LOINC code and rejects the message. Given that LOINC changes often what mechanisms will a system use to identify invalid codes (based on a certain format?) We could send messages where code and text don’t match as another negative test –Selection of approach may be made on a case-by-case analysis –It may be necessary to provide coverage for all recommended LOINC codes. However, for other vocabularies it may not be a priority (e.g., state) or may not be feasible (SNOMED). What is the testing approach?

17 NIST’s understanding of the request from the Vocabulary Task Force (For Meaningful Use Stage-2) EHR MU Capabilities Discussed on June 6 th 2011 TF call: Require that an EHR be capable of understanding a subset of 2000 most commonly used LOINC codes for lab results reporting –Subset would be published by Regenstrief (see LOINC Mapper’s Guide to Top Lab Tests v1.0a) EHR would need to be capable of * –R1: performing computable activities on received LOINC codes - storing LOINC codes received in a lab results message in such a way that the EHR can perform computable activities on the received LOINC codes –R2: don’t break if receive unknown LOINC code - displaying the text description of the lab test if the received LOINC code is not recognized by the EHR (text description as provided in the received message) –R3: transmitting LOINC codes - instantiating correct LOINC codes in messages and documents being sent from the EHR * Note that the required capabilities are still being discussed in the Vocabulary TF. The criterion statements have not been written by ONC. This is NIST’s understanding as of 06/06/2011. Source: Ken Gebhart, NIST