IHE-PCD , HL7 HC Dev WG, ISO/IEEE 11073, and NIST Medical Device Communication and IHE-PCD Cycle 4 Test Strategy IHE-PCD, HL7, ISO/IEEE Joint WG Meetings (@ Atlanta, GA) 23 September 2009 NIST Presentation to IHE-PCD, IEEE MDC, HL7 Healthcare Devices SIG in Atlanta George, on September 23, 2009.

Medical Device Test Effort NIST Team Members John Garguilo (john.garguilo@nist.gov, 301-975-5248) Sandra Martinez (sandra.martinez@nist.gov, 301-975-3579) Maria Cherkaoui (maria.cherkaoui@nist.gov Guest Researcher) *Jing Gao (jing.gao@nist.gov Guest Researcher) * Jing joined our project in May ’09 Project Web site: www.nist.gov/medicaldevices

NIST MDC Testing Topics IHE-PCD Test Strategy NIST Framework for Testing Distributed Healthcare Applications IHE-PCD Testing Framework Instance Cycle 4 Pre-connectathon (and virtual-connectathon) Test Plan Support both Conformance & Interoperability RTM Terminology Management System Update NIST ISO/IEEE 11073 Tooling ICSGenerator, ValidatePDU NIST ISO/IEEE 11073 Standards Development Work MDC X73 XML Schema (X73-P10202)

Test Environments Instance Testing Isolated System Testing Conformance (e.g., against HL7 2.x or CDA) Implementation conforms to Spec. on which it is based IHE Model: ~Virtual and Pre-Connectathon Isolated System Testing Includes Instance Testing Activities Protocol Conformance Functional Behavior Conformance Features and Operational behavior correspond to Specs. Peer-to-Peer System Testing Includes Isolated System Testing Activities Interoperability Testing Testing complete application environment May include interacting w/ Database, using Network Communications, or interacting w/ other hardware, apps, or systems if appropriate IHE Model: ~Connectathon In IHE Model: Instance and Isolated testing ~ Virtual and Pre-connectathon Peer-to-Peer System Testing ~ Connectathon

Test Environment Message Validation NIST V2 Testing Tools: IHE-PCD Validation of IHE-PCD message(s) and corresponding HL7 Profile(s) Syntax and Semantic Content Validation Against HL7 V2 message (e.g., PCD-01) Message structure (e.g., MSH,PID,PV1,OBR,NTE,{{OBX},OBX,OBX,OBX,…}) Against HL7 profile (Msg_type^Event_type^ e.g., ORU^R01^…) Against HL7 and/or user provided tables Example of user provided table is RTM for Ref_IDs, Units, etc. Against ‘validation context’, including specific values Defined in XML (e.g., specific test case values) 5

NIST V2 Testing Tools and Services Testing Validation Types Validation against ‘failure types’: VERSION*: The version in the message and in the profile should match. MESSAGE_STRUCTURE_ID*: The message type (MSH.9 element) in the profile and in the message should match. MESSAGE_STRUCTURE: The message should have a valid message structure (correct usage, correct cardinality, and correct element name). USAGE: R elements should be present; X elements should not be present in the message. CARDINALITY: Elements should be present at least the minimum times and at most the maximum times specified in the profile. It should also take into account the usage of the element (X element with a minimum of 4 should not be present in the message). LENGTH: The value of the element should have a length equal or less than the value specified in the profile. DATATYPE: For the datatype NM, DT, DTM, SI and TM, the value of the element should match the regular expression defined in the standard. DATA: The value of the element should match a constant specified in the profile, a value set specified in a table, a value or a regular expression specified in the message validation context. MESSAGE_VALIDATION_CONTEXT*: This is a user input error when the location specified in the message validation context can't be found in the message. TABLE_NOT_FOUND*: This is a user input when a table can't be found in the table files (TableProfileDocument). AMBIGUOUS_PROFILE*: The profile should not be ambiguous. 6

Instance Testing HL7 V2 Message Validation Test Management Services Web Application Client HL7 V2 Message Validation Criteria HL7 V2 Message Validation Report Results HL7 V2 Message Validation Report HL7 V2 Message Test Artifacts Conformance Profiles HL7 Tables Validation Context Files Generation Context Files User Test Harness (Java Code) Test Execution Validating message for compliance to HL7 Conformance Testing: Showing that messages are in compliance w/ HL7 V2.x and constraints specified in IHE-PCD framework documents.

Instance System Testing IHE-PCD RTM Validation Test Management Services HL7 V2 Message RTM Validation Criteria Web Application Client HL7 V2 Message Validation Report Results RTM Message Validation Report HL7 V2 Message RTM Message Validation IHE PCD DOC Test Agent Test Harness (Java Code) User Test Execution Test Artifacts Conformance Profiles HL7 Tables Validation Context Files Generation Context Files Validating the HL7 message for compliance to RTM (e.g., a specified Ref_id had an appropriate unit) Evaluation criteria as defined in IHE-PCD RTM profile The need is to test that units, enumerations, external lead sites, and the parent OBXs are checked. Vendors would provide the messages, perhaps in email. This would assure that units of measure and nomenclature are correct. Verify is this is the type of testing to be done: RTM verification can be triggered whenever an observation identifier is recognized in the message, based on the four-level hierarchy used by PCD-01 and related messages, Get first observation (parameter) (term code) from msg and check against Rosetta - If observation is found in Rosetta => term check ok! - Check the units for that parameter - If the unit matches the unit identified in Rosetta for the term being tested => unit check ok! - If the unit does not match the unit identified in Rosetta for the term being tested => invalid unit for term xxx! - Check enumerations - Check external lead sites (OBX20) - If observation is not in the table => “term” is missing! Need type of errors to identify. Conformance Testing: Showing that messages are in compliance w/ Harmonized-Rosetta terminology Time Security 8

Future: Isolated System Testing IHE-PCD Application Functional Behavior Testing Test Management Services IHE-PCD DEC DOC Test Scenario Web Application Client Vendor HL7 V2 Message Validation HL7 V2 Message Generation Results Message Validation Reports IHE-PCD DOR Test Agent IHE-PCD DOC Test Agent Test Harness (Java Code) Test Execution RTM Message Validation Report Test Artifacts Conformance Profiles HL7 Tables Validation Context Files Generation Context Files Conformance Testing: May include HL7 message syntax validation, RTM, and/or application functional behavior testing This Isolated System Tests for behavior of the SUT (e.g., an invalid message is sent to “Consumer” and acts appropriately – acknowledgement reject - “AR – message has been rejected by receiving application” ) IHE-PCD DOC System Time Security Router 9

Future: Isolated System Testing IHE-PCD RTM Validation Test Management Services HL7 V2 Message HL7 V2 Message RTM Validation Criteria Web Application Client User HL7 V2 Message Validation Report Results RTM Message Validation Report RTM Message Validation IHE PCD DOC Test Agent Test Harness (Java Code) Test Execution HL7 V2 Message IHE-PCD DOR System Test Artifacts Conformance Profiles HL7 Tables Validation Context Files Generation Context Files Validating the HL7 message for compliance to RTM (e.g., a specified Ref_id had an appropriate unit) Evaluation criteria as defined in IHE-PCD RTM profile The need is to test that units, enumerations, external lead sites, and the parent OBXs are checked. Vendors would provide the messages, perhaps in email. This would assure that units of measure and nomenclature are correct. Verify is this is the type of testing to be done: RTM verification can be triggered whenever an observation identifier is recognized in the message, based on the four-level hierarchy used by PCD-01 and related messages, Get first observation (parameter) (term code) from msg and check against Rosetta - If observation is found in Rosetta => term check ok! - Check the units for that parameter - If the unit matches the unit identified in Rosetta for the term being tested => unit check ok! - If the unit does not match the unit identified in Rosetta for the term being tested => invalid unit for term xxx! - Check enumerations - Check external lead sites (OBX20) - If observation is not in the table => “term” is missing! Need type of errors to identify. Conformance Testing: Showing that messages are in compliance w/ Harmonized-Rosetta terminology Time Security 10

Future: Peer-to-Peer System Testing IHE-PCD Application Functional Behavior Testing Test Management Services Vendor IHE-PCD DEC DOC Test Scenario Web Application Client HL7 V2 Message Validation HL7 V2 Message Generation Results Message Validation Reports IHE-PCD DOR Test Agent IHE-PCD DOC Test Agent Test Harness (Java Code) Test Execution RTM Message Validation Report Conformance Testing: May include HL7 message syntax validation, RTM, and/or application functional behavior testing Interoperability Testing: system(s) (functional) behavior Test Artifacts Conformance Profiles HL7 Tables Validation Context Files Generation Context Files This Peer-to-Peer System Tests for behavior of the SUT (e.g., Multiple Patient Monitor systems displaying parametric data” ) IHE-PCD DOC System IHE-PCD DOC System Time Security Router/Logger/Proxy 11

HL7 Message Validation Study Example PIV (PCD-03)

PIV-PCD-03 Test Case 60101 Validation using the IHE TF/Supplements MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L ORC|RE|12345|||||||||||||||||N60101 RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC RXR|IV||IVP OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

PIV-PCD-03 Test Case 60101 Validation Using HL7 Profile Tables MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L ORC|RE|12345|||||||||||||||||N60101 RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC RXR|IV||IVP OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

PIV-PCD-03 Test Case 60101 hRTM Validation MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L ORC|RE|12345|||||||||||||||||N60101 RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC RXR|IV||IVP OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

PIV-PCD-03 Test Case 60101 Use/Test Case Specific Validation MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L ORC|RE|12345|||||||||||||||||N60101 RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC RXR|IV||IVP OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

PIV-PCD-03 Test Case 60101 Overall Validation MSH|^~\&|BBRAUN_TEST_IOP^0012210000000001^EUI-64|IOP|PAT_DEVICE_BBRAUN^0012210000000001^EUI-64|BBRAUN|20090112090938-0600||RGV^O15^RGV_O15|1|P|2.5||||||ASCII|EN^English^ISO659||IHE_PCD_PIV_001 PID|||60101AB^^^AA1^PI||Bains^Allen^^^^^L ORC|RE|12345|||||||||||||||||N60101 RXG|1|||1^Heparin|500||mL^mL^UCUM||||||||1000|[iU]/h^[iU]/h^UCUM|25000|267616^MDC_DIM_X_INTL_UNIT^MDC|||||500|263762^MDC_DIM_MILLI_L^MDC RXR|IV||IVP OBX|1||69986^MDC_DEV_PUMP_INFUS_VMD^MDC|||||||||||||||^^A26672^BBraun

PIV-PCD-03 Test Case 60101 Validation Coverage Coverage % = Number of Values in Message / Number of Values Checked IHE Supplement Validation 43% Validation using Profile Tables 24% Test Case Specific Validation 19% Overall Validation 74% Overall Validation without profile 62% Note: remaining 24% (from overall validation percentage) is validated at the message structure level via HL7; these include message segments that are not of primary interest to the IHE-PCD. If this level of validation is considered, then nearly 100% validation coverage can be expected.

Message Validation Context TF/Supplement Validation DEC-PCD-01-ValidationContext.xml SPD-PCD-02-ValidationContext.xml PIV-PCD-03-ValidationContext.xml ACM-PCD-04-ValidationContext.xml IDCO-PCD-09-ValidationContext.xml

Message Validation Context hRTM Validation DEC-PCD-01-ValidationContext.xml PIV-PCD-03-ValidationContext.xml ACM-PCD-04-ValidationContext.xml Needed: IDCO-PCD-09ValidationContext.xml

Message Validation Context Test Case Specific Validation PIV(for PCD-03) 60101-ValidationContext.xml 60102-ValidationContext.xml 60103-ValidationContext.xml 60104-ValidationContext.xml 60105-ValidationContext.xml 60106-ValidationContext.xml 60107-ValidationContext.xml 60108-ValidationContext.xml 60109-1-ValidationContext.xml 60109-2-ValidationContext.xml 60109-3-ValidationContext.xml 60109-4-ValidationContext.xml 60110-ValidationContext.xml

IHE-PCD ’09-’10 Pre-Connectathon NIST Support (see recommendation document: "NIST Support for IHE-2009v1.docx") Primary Goal: Provide tools to test conformance to HL7 and IHE-PCD Integration Profiles such as DEC, DEC-SPD, ACM, PIV, IDCO, and RTM Instance Testing System Under Test (SUT)  Test Agent (TA) HL7 syntax validation (basic validation) IHE-PCD profile syntax IHE-PCD profile semantic validation RTM validation Test case specific validation Develop Test Plan across IHE-PCD Integration Profiles Test Scenarios Interaction diagrams Pre-conditions Validation criteria – all test-able requirements (assertions) Test cases Validation context file(s)

IHE-PCD ’09-’10 Pre-Connectathon Continued Proposed conformance testing for ’09-’10 Pre-Connectathon All DOR messages (PCD-01) sent during DOR testing. All IOP messages (PCD-03) sent during IOP testing. All IDCO-DOR messages (PCD-09) sent during IDCO testing. All DOF messages (PCD-01) sent during DOF testing. All DOC messages (PCD-02) sent during DOF testing. All AR messages (PCD-04) sent during AR testing.

Show NIST V2 Tools (via Web) Demonstration NIST HL7 Web Upload HL7 IHE-PCD ORU^RO1 profile Upload HL7 IHE-PCD context validation file (XML) Upload HL7 IHE-PCD message (ER7 format) Validate Generate and review resulting ‘Report’ Review errors

RTM Terminology Management System Update

RTM Mgmt System Changes (from last WG mtgs in May) Features Completed Enhanced lookup of REFIDs (description of REFID’s) and provided an interface to propose new terms Implemented an interface for browsing terms in X73 Nomenclature

RTMMS In progress and next steps… Features based on the Rosetta Supplement Incorporate “Enumerations” management capability(ies) Add ranking capabilities of implementation priority of valid terms in the Rosetta table Implement Rosetta validation against hRTM Add ability to edit hRTM table and save changes Implement hRTM version control and management system 

RTMMS Future New Enhancements Features based on the new/recent enhancements to HRTM Assign dimensions to the units Implement Rosetta rules Implement generation of hRTM based on Rosetta rules Generate comment table based on the Recommended Rosetta Comment Submittal Format Implement RCH table User oriented features Regular Expression based column filtering Implement “change trailing” capabilities To identify occurred changes, time they were made, users who made them… Enhance registration process Incorporate enhanced X73 Nomenclature database 

NIST ISO/IEEE 11073 Tooling Update

X73 NIST Tooling… ICSGenerator On horizon: MDC Semantic Database User Interface Work Met w/ Todd Cooper, captured issues, made changes Updated issues list – used as basis for task list Integrating database into ICSGenerator… Lite-weight database identified To include hRTM… Initial discussions X73-compliant Device Containment Generation of partial OBX segments On horizon: MDC Semantic Database Terminology Nomenclature Medical Device Profiles and other Test Artifacts

ISO/IEEE 11073 Standards Work Update

ISO/IEEE 11073 Standards Work MDC X73 XML Schema (X73-P10202) Verification and validation of XML Schema continues via tool usage Support for Personal Health Device Working Group X73-20601 104xx device specializations Continue to validate via tooling prior to ballot Report errors for correction prior to balloting

Thanks for your attention  Please visit us at: Thank YOU! Discussion Propose “ORU^R01^ORU^Rxx” naming convention for IHE-PCD HL7 profiles xx denotes profile specific for Integration profile/device class (use of DEC/PCD-01) Questions? Thanks for your attention  Please visit us at: http://www.nist.gov/medicaldevices 33

Extra Slides

NIST HIT Testing Infrastructure A Framework for Building Test Systems Test Management Services User Test Description Validation Generation Results Test Data Test Agent Test Harness Test Execution Facilitator Evaluation Agent Aggregated Service Test Artifacts Time NIST HIT Testing Infrastructure … This slide is a reminder of what/how the infrastructure is preliminarily defined and looks like… A second SUT could also be shown in the case of peer-to-peer testing… IHE-PCD DOC System Monitor Report Security Other Services Specialized Services Router/Logger/Proxy