1. Part 1: HL7 Testing Tools (Supporting Meaningful Use and IHE) Part 2: Towards a Testing Infrastructure (Lessons Learned and Leveraging Resources)
Robert Snelick
March 21, 2011

2. Outline: Key Theme is Automation and Leveraging Tools
HL7 Testing Toolkit
HL7 is the most widely used messaging standard to exchange administrative and clinical data (>90% of U.S. hospitals)
Meaningful Use Testing
Data Instance Validation
Reportable Lab Results to Public Health
Immunization to Public Health
Patient Identification Management and Query Testing
Isolated System Testing
IHE PIX/PDQ Testing Tool
Towards a Testing Infrastructure
Expand the scope to other protocols
End-to-end
Interoperability Test bed (Peer-to-peer System Testing)
Sustainability/Transfer/Exit Strategy

3. NIST HL7 Testing Toolkit
APIs and components to support tool development
Applications and Services
HL7
Artifacts
Web
Services
Message Generation
Message Validation
Message
Encoding
Profile
Validation
Generation
HL7 Communication
Data Sources
Test Agents
Java APIs
Message
Maker
3rd Party
Applications
and Testing
Environments
Java
Database
Message
Validation
Profile
Validation
Testing Framework
Web
Applications
IHE PIX/PDQ
IHE PCD
MU EHRPH
MU ImmunizationPH
NHIN PIX/PDQ
CCHIT Lab Messages
Message Generation
Message Validation
NIST HL7
Test Data
HL7 Data
Repository MU
Validation
Profile
Registry

4. Meaningful Use HL7 V2 Validation Testing
Reportable Lab Results and Immunization
Robert Snelick
March 21, 2011

5. ARRA Meaningful Use Certification Testing
HL7 V2 Tools used for certifying EHR systems and models (Stage 1)
Process
Test Procedure
Data elements and message structure defined
Test Data Sets
Validation Tool (
EHR Reportable Lab Results to Public Health
HL7 V2.5.1 Implementation Guide and LOINC value set
EHR Immunization to Public Health
HL7 V2.3.1, HL7 V2.5.1, and CVX value set
Leverages the HL7 V2 Toolkit
Project Timeline
Research/Development: March 2010 to October 2010
Maintenance/Support: October 2010  Current
Has general industry applicability
Current Use: Accredited Testing Certification Bodies
Stage 2?

6. Lab Results to Public Health MU Test Procedure
HL7 ELR IG 2010
Meaningful Use Scope
(Stage 2)
LAB
(or any sending system or utility)
EHR
(System under Test)
Public Health
Meaningful Use Scope
(Stage 1)
ORU Message
ORU Message
ORU Elements
Direct Data Entry
Message
Captured
NIST Validation
Tool
Validation
Report
ORU Message Data
The EHR is the system being tested. The EHR system is required to send (create) messages that conform to the referenced standards (HL7 V2.5.1 Lab IG and LOINC).
Test data can be entered into EHR directly via the EHR’s user interface or be imported via an incoming message
The EHR is expected to process the test data to create a message. This message is captured and uploaded into the testing tool for validation.
The test can be conducted using NIST supplied test data or vendor supplied test data. When using NIST data sets, the appropriate validation context needs to be selected. When choosing vendor supplied data an appropriate generic validation context needs to be selected.

7. Immunization Message Validation

8. Validation Report

9. Supporting Pre-connectathon and Connectathon Testing
IHE PIX/PDQ Testing
Supporting Pre-connectathon and Connectathon Testing
Robert Snelick
March 21, 2011

10. IHE PIX/PDQ Testing
Patient Identification Management and Patient Demographics
Critical integration profiles
PIX Source, PIX Manager, PIX Consumer, PDQ Client, PDQ Supplier
Used for North American and European IHE Affiliates
Supports both Connectathon and Pre-connectathon Testing Events
Supports both HL7 V2 and V3 Messages Standards
Testing Tool
Isolated System Testing
Interactive on-line web application tool
Test Event Management Capabilities
Leverages the HL7 Toolkit
Completely automated
Template/Mold for other use cases based on HL7 transactions
Timeline
Research/Development: June 2009  October 2009 (Prototype)
Research/Development: June 2010  October 2010 (Production)
Upgrade/Maintenance: March 2011  October 2011 (Version 3)

11. Patient Identification Matching and Linking
East Side Hospital
System Under Test
ADTA04 – Register Patient
(Master Patient ID - 101)
PIX Source
Simulator
(Patient
Registration) 1
PIX
Manager
Application
XDS 1a
ACKA04 – Acknowledgement
RSPK23– Get
Corresponding
Identifier Response
(Validate for response
with master ID 101) 4
ADTA04 – Register Patient
(Local Patient ID – F432)
QBPQ23 – Get
Corresponding Identifiers
(Query with local ID F-432) 3 2
PIX Source
Simulator
(Patient
Registration)
PIX and
Document
Consumer
Simulator 2a
ACKA04 – Acknowledgement
Retrieve
Document
St. Mary’s Medical Center

12. PIX/PDQ Test Tool: Operational Process
INTERACTION/REPORTS
END-USER
(VENDOR)
MANUAL OR AUTOMATED SUT
Web Application Interface
NIST PIX/PDQ
HL7 v2/v3 TEST TOOL
MESSAGES (TEST OBJECTS)
SYSTEM UNDER TEST (SUT)
STIMULUS OR RESPONSE (MESSAGES)
(via the communication protocol)
SPECIFICATIONS
(test material that defines test assertions)

13. Test Case : Patient Match, Link , and Retrieve
PIX/PDQ Test Tool
System Under Test 1
ADTA04 – Register patient
“ALAN ALPHA” (ID = PIX10501) in domain “HIMSS2005”
End User
(Vendor)
NIST
PIX
Source
Test Agent
Vendor
PIX
Manager
Application 1a
ACKA04 – Send acknowledgement
ADTA04 – Register patient
“ALAN ALPHA” (ID = XYZ10501) in domain “XREF2005” 2 2a
ACKA04 – Send acknowledgement
ADTA04 – Register patient
“CARL SIMPSON” (ID = ABC10501) in domain “HIMSS2005” 3 3a
ACKA04 – Send acknowledgement
Web Application Interface
NIST
PIX
Consumer
Test Agent 4
QBPQ23 – Send query message for patient “ALAN ALPHA” in domain “HIMSS2005” seeking the patient ID in domain “XREF2005” 4a
RSPK23– Send response indicating that the identifier (“XYZ10501”) for patient “ALPHA” was found in domain “XREF2005”
QBPQ23 – Send query message for patient “CARL SIMPSON” in domain “HIMSS2005” seeking the patient ID in domain “XREF2005” 5 5a
RSPK23 – Send response message indicating no
data was found “NF”

14. Select Test

15. View Test Steps and Run

16. View Interactive Log

17. Summary Results

18. North American Pre-Connectathon Results (Fall 2010)

19. Summary of Vendor Test Results

20. Vendor Saved Test Results

21. Community Involvement: PIX/PDQ Google Group
2009/2010 Prototype: ~60 Group Members; ~25 vendors completed tests
2010/2011: > 200 Group Members; ~70 vendors completed tests so far

22. Testing Infrastructure
Robert Snelick
March 21, 2011

23. Motivation Group has built tooling for various healthcare IT domains
HL7 V2, HL7 V3, IHE XDS, CDA, Patient Care Devices, NHIN, IHE PIX/PDQ
Typically built test tools one at a time (independently)
We have had many successes and have positively impacted industry
However, we can’t build tools fast enough one at time
Problem MU reportable lab tool took 4 man months (immunization the same)
In some instances we needed to become domain experts (e.g., lab); probably not the best use of NIST resources
Muddled down in administration of tests/maintenance of tools—instead of innovation
Demand greatly outpacing industry/NIST capabilities to produce tools
Recognized a need to work more efficiently
Need to “build tools to build tools”, we’re heading in that direction…
V2 Tool Kit (leveraged for CCHIT Message Generation, PIX/PDQ, PCD, connectathon, NHIN, Meaningful Use)
External validation services (V2 and CDA); common interface/reporting
More and more to come…
ARRA funding allocated to build TI in 2009 (but not yet awarded)
Group has traditionally worked with and built tooling for various healthcare IT domains
HL7 V2, HL7 V3, XDS, CDA, Patient Care Devices, NHIN
Built test tools one at a time (independently)
At the end of the day you’ve built one test tool (which is great, but…)
In the process we need to gain domain expertise; is this really a NIST function?
Necessary to some extent to test our tooling methodology (and the IT part)
Many shortcomings with specifications and testing process became apparent
Demand greatly outpacing industry capabilities to produce test tools
Recognized a need to work more efficiently
Need to “build tools to build tools”
Don’t duplicate tooling (not only at NIST but in the community), we’re starting…
V2 Tool Kit (leveraged for CCHIT Message Generation, PIX/PDQ, PCD, connectathon)
External validation services (V2 and CDA); common interface/reporting
More and more to come…
Easier said then done—more applicable in some areas then others
TI is not a do over; we’ll build on or adapt existing toolkit design/code base—not everything can be assimilated (cost/benefit)

24. Testing Infrastructure
Heading in this direction with the HL7 Projects
Proceeding with pilot HL7 to better understand requirements
Testing model is widely applicable
HL7 V2 IG, CDC , IHE, PHIN, NHIN, HITSP, ISDS, etc
Flood of implementation guides
For example, MU to add 3 more HL7 IGs and also more elaborate testing scenarios
Important that our tools are leveraged and we need to further engage industry experts
Easier said then done—more applicable in some areas then others
TI is not a do over (migration); we’ll build on or adapt existing toolkit design/code base—not everything can be assimilated (cost/benefit)
Build TI with targeted (production) tools in mine
build out TI components for specific tools (develop abstract design along the way)
End-to-End PIX/PDQ
End-to-End PCD (collaborate with working group from start)
NHIN (Patient Discovery)
ONC MU (Lab, Immunization, Bio-surveillance, Patient Management)
Transfer technologies to SDOs (HL7, IHE, ONC, etc)

25. How can we Address the Increased Demand for Tools?
Implementation Guide Authoring Tools (Create more “testable” specifications!)
Implementation
Guides
(Number is increasing rapidly—no testing tools available)
HL7 v2
MU Bio-Surveillance
NHIN
HL7 V3 Patient
Discovery
HL7 v2 MU
incorporate Lab
Results &
Reportable Lab
Results
MU Immunization
Reporting
IHE HL7 v2
PCD
Technical
Framework
PIX/PDQ
IHE HL7 v3
IHE HL7
v2 and v3
In General
HL7 v2 and v3
Guides in General
(HL7, CDC, PHIN,
ISDS, ONC, etc.)
20+ IHE integration profiles using V2!
Test Case Management Tool (Let domain experts help write the test cases!)
V2 Implementation Guide Authoring Tool
Many HL7 V2 Implementation Guides are being written
IHE, CDC-public health, some specified for meaning use
All with different style, conformance requirement interpretations, etc.
Written in text; hard to extract conformance requirements
V2 is a different beast then newer XML standards
But V2 is here to stay and MU/IHE will continue to use
Tools are lacking—MU test development made that clear
Provide a mechanism to produce consistent HL7 V2 IG
Models V2 message definition and constraints
Explicitly defines conformance requirements that can be processed automatically
Publish Text (English) version of the Implementation Guide
Pre-defined IG Skeleton
Modifications made in tool and republished
Test Case Management Tool
Creates, Maintains, and Manages Test Plans
Embedded Standard Constraint Model or Imports Test Artifacts
Supports Multistep Test Cases (Test Scenarios)
Defines Each Test Case (Test Steps)
Builds Test Environment
Defines transactions, actors, and sequence to execute tests
Defines generation context files and validation context files
Defines configuration parameters
Creates executable test script in XML format
Independent of any execution engine
Generates English Document Test Plan
Describes test cases and expected results
Provides traceability to specification reference
Overall has to incorporate/use results of the other tools
TI Goal: Tool Building Development Environment
Build testing tools more rapidly and with better correctness
IHE has 40/50 integration profiles with 100’s of transactions
30/40 actors; V3, V3, DICOM, XDS, CDA
New specifications being developed each year
New tests being developed each year
NIST can build Testing Infrastructure and Demonstrate Usefulness (PIX/PDQ as one example) then replicate
Industry/Domain Experts can use TI to develop Test Tools
Bulk of work completed with TI
Industry supplies test agent functionality, writes test cases, test data, validation criteria
Show industry “how to fish”
Applies to other industry organizations and SDOs
CDC/Public Health(PHIN), HL7, Meaningful Use, etc.
Test Cases and Testing Artifacts
Test System (Allow NIST/others to build tools quickly)

26. “Reusable” Test System
Phase 3
HL7 Test Tool
Phase 1
SDOs
Implementation Guide Authoring Tool
Test
Flow
Engine
User Interface
Test Agent
Support
Testers
HL7 v2
Test Agent
Core
Test
Cases
Test Case
Management
Tool
Phase 2
End-User
(Vendor)
Test Agent
Support
Message Generation and Validation
Test Engine
Web Application
Test
Management
Test Agent
Core
Communication
Real-
Time
Log
Vendor
System
Under
Test
Test Agent
Support
Current/Planned Projects with Testing Infrastructure
To be developed under the testing infrastructure umbrella
IHE PIX/PDQ V2/V3 End-to-end
Specify the integration using the Implementation Guide Authoring Tool
Build/manage Test Cases using the Test Case Management Tool
Execute Tests using components of the Testing Infrastructure
V2/V3 Message Generation Services
V2/V3 Message Validation Services
Data Service—Test Data Development
Registry/Repository Services
PIX/PDQ Test Agents
SOAP Validation Service
Test Harness
Logging Service
Test Analyzer Service
Reporting Service
Meaningful Use 2011 (Lab Results and Immunization)
Meaningful Use 2013
Test Agent
Core
Reports
HL7 v3
Special Validation
MLLP (v2) or SOAP (v3)
NIST Server/Other Server
Tool Builder Tasks

27. Broader View: Conceptual View of TI System Components
Specification (IG) Authoring Tools
STAKEHOLDERS
NIST TEST INFRASTRUCTURE SYSTEM
SUT
Test System Instance
Resource Repository
System(s) Under Test
HITSP
User Interface
Test Cases, Machine-Readable Test Scripts
Templates, Common File Format Specifications
Test System Control
Certification
Body
Test Environments Supported
Instance Testing
Test Harness
Isolated System Testing
Workflow Management /Execution Engine
Test Analyzer
Peer-to-Peer System Testing
NHIN
Test System Development Components
Services
Test Management Tools and Test Systems
Test Agent
Proxy
Evaluation Agent
Vendors
Test Initiation and Test Identification Management
Test Framework
Test Framework
Test Framework
Test Case Creation
Implementers
Testing Infrastructure Key Objectives
Provide a variety of testing services to health IT stakeholders
Support conformance and interoperability testing for numerous healthcare messaging and document data exchange standards
Provide a set of reusable/flexible components and services
“A Framework for Building Test Systems—an SOA Approach”
Assortment of testing infrastructure uses
Enables user customization
Support different delivery mechanisms including APIs, desktop applications, web applications, and web services
Provide a feedback loop to enhance the healthcare standards
Integrated existing tools when appropriate
Collaborate and harmonize with other tooling efforts
Roll out tools and resources incrementally
Generation
Test
Data
Validation
Report
Remote
Services
Configuration Settings Management
Existing
Tool
Time
Security
Logging
Log
Analyzer
Other Industry
Consortia
User Repository Management
Registry/ Repository
Aggregated
Specialized
Other
Services
External
Service
Real-Time Monitor

28. Tool Building Development Environment
Meaningful Use Web
Application
Validation Tool
PIX/PDQ
Pre-Connectathon Test Tool
NHIN Test Tool
IHE Connectathon Test Tool
General CDA Validation
IHE XDS
V2 Toolkit (includes communication framework)
V3 Tools
General HL7 V2 tools (Message Maker, Validation W/S, app)
IHE CDA (General CDA + more)
IHE PCD (HL7 V2 Toolkit/Validation + more)
IHE PIX/PDQ (HL7 V2 Toolkit + Test Agents + more)
IHE PIX/PDQ (V3 Tools + CDA + XDS + more)
MU CDA (General CDA + more)
MU HL7 V2 Immunization (HL7 V2 Toolkit/Validation + more)
MU HL7 V2 Lab Results (HL7 V2 Toolkit/Validation + more)
NHIN (XDS + V3 PIX/PDQ + more)
Modularized/Flexible/Reusable Components
Able to react quickly to test tool needs, e.g., Meaningful Use
Enable the healthcare community/domain experts to build tools
Provide a low barrier of use; high return

29. Summary What have we done? Lessons Learned/Industry Needs
HL7 Tool Kit (V2 and V3)
Tools
General HL7 Utilities
Message Maker
CCHIT Incorporate Lab Message certification
Meaningful Use (Stage 1: Reportable Lab Results and Immunization)
IHE PIX/PDQ Pre-connectathon and connectathon
IHE PCD Pre-connectathon and connectathon
Lessons Learned/Industry Needs
Demand for tools increasing; need a community effort
Need tools to build tools
Need to react quickly to requests for tools
Need more automated tools
What are doing?
Improving HL7 conformance standards (HL7 V2 conformance co-chair)
Building core components and infrastructure
Building prototypes and tools to work smarter
HL7 V2 End-to-End Test System (PIX/PDQ and PCD)
Implementation Guide Authoring Tool
Test Case Management Tool
Test System
Support NHIN Testing
Building reference test data (messages)
Where are we headed?
Build out broader testing infrastructure components
Support more elaborate meaningful use scenarios
Virtual Interoperability Test bed
Engage industry to adopt tools
Transfer technology such that others can build the tools (e.g., ONC, IHE, etc.)