Certification and Conformance Approaches for Canada’s EHR Elaine Sawatsky, Senior Management Consultant, Privacy & Security, Canada

Conformance: Canada’s EHR Infoway Conformance Services in Canada Supporting the Implementation of pan-Canadian Standards Provincial/Territorial work On the ground experiences

Conformance: Canada’s EHR Sharing information requires a quality assessment of products to ensure interoperability, to reduce risk to patients, purchasers and the ‘system’. Standards support sharing data and bringing it together Improves comparability and harmonisation between products. Certification: “a process of validation of products in a given context against specific validation criteria.” Certification criteria are – even regarding the same functionality – up to a certain degree similar but also different between jurisdictions in: » The way they are formulated & validated » The importance given to each of them Certification should be unambiguous, fair and trustworthy

Conformance Services Vision To support the successful implementation of pan-Canadian Standards within projects through the provision of conformance testing artefacts and tooling. – To improve the consistent implementation of pan- Canadian Standards – To reduce the effort required to implement pan- Canadian Standards – To reduce the cost of implementing pan-Canadian Standards Strategic Intent of Conformance Services: – Ensure that there is a national service to support the vendor conformance and certification of iEHR, public health and EMR solutions.

SK EHR Blueprint and Concepts

The need: Bridging The Gap The challenge is to bridge the gap that exists between the health informatician (modeler) and the clinical integration specialist (solution developer). EHR in Operation EHR in Operation Health Informatics Modeler Integration Specialist Modeling Culture Delivering Specifications Project and Product Culture Delivers Production Solutions Standards Development Standards Implementation, Conformance & Maintenance Need Options Develop SFU Authoring Vocabulary Profiles Assessment Certification ReleaseChange Helpdesk Tooling 6

Conformance: Canada’s EHR

Approach There is recognition that there is a need for conformance testing to ensure alignment The approach is founded on international standards Prototype & Learn – conformance testing is not well understood yet Build Methods & Processes with stakeholder input Build tooling & leverage : Standards – HL7 v3, LOINC, SNOMED, XDSI Tooling Component model – integration is key! Sharing of testing artefacts – huge value to stakeholders

Conformance: Canada’s EHR Key Messages There is recognition that we all need to do end to end and integrated testing We are committed to leveraging work of others We are making good progress in developing processes and tooling to assist our jurisdictions in implementing “Conformant” solutions

Conformance Continuum Conformance includes: – Message content and structure – Vocabulary and terminology – Interoperability profiles and message sequencing – Sending and receiving applications and intermediaries Conformance Testing includes: – Functionality of application – Basic privacy and security services related to the functionality – Workflow (to an optimal level determined by conformance continuum) 10

Our Approach First, develop Profiles for all pan-Canadian standards Develop a process & mechanisms linking the pan-Canadian Profiles through to conformance test Develop, deploy, continuously improve “quick-hit” tools Provide business and technical support for Investment Projects to set up and execute conformance testing Document & refine requirements for long-term solution Harvest artefacts for sharing with other jurisdictions Rationalize jurisdiction artefacts to create pan-Canadian versions Establish a pan-Canadian Working Group of early participant jurisdictions and their vendor partners to provide guidance and assistance in the development of Conformance Testing artefacts

Pan-Canadian Standards Evaluation Adoption Strategies Tools and Templates Standards Knowledge Management Liaison Promotion & Communication Licenses Change Management Standards Selection Strategy Stable for Use Final Approval Scope of Part 4 Our Approach

Conformance: Canada’s EHR Benefits: Financial savings through reduced duplication of efforts in test authoring Provinces leverage other provinces’ Business Rules Financial savings though tools based test message development Provinces use Infoway tools and other provinces’ messages Breadth of testing improved through increased quantity and quality of test cases Provinces are committed to sharing test artefacts.

Conformance: Canada’s EHR Benefits: Depth of testing improved through consistent test rules and tooling Provinces’ use of tooling has improved UAT testing Too early to accrue benefits Quality and consistency of conformance testing improved between Investment Projects Too early to accrue benefits Improved consistency in Investment Project implementations of pan-Canadian standards Too early to accrue benefits

Benefits Providing Implementation support for the development of Test Authoring and Test Execution will enable jurisdictions to leverage and share Conformance Profiles and testing artefacts, leading to: – Improved quality of testing – Financial savings through reduced duplication of efforts – Breadth of testing improved through increased quantity and quality of test cases – Improved quality and consistency of conformance testing – Financial savings though tools based test message development – Depth of testing improved through consistent test rules and tooling – Improved consistency in implementations of pan-Canadian standards

Conformance: Canada’s EHR Conformance Services Overview Main elements influencing content of test criteria: Legal, national, cultural (business and social) Sophistication of the industry area Environment Certification is the process Conformance is the result Must be good enough, not best or everyone will fail Area subject matter experts – what will the medical professional need?

Stakeholder Engagement Governance Collaboration on Test Authoring components Collaboration on supporting tools development and implementation Registry of vendor systems & certification status Registry of implementation projects Ongoing liaison/communication with jurisdictions and vendors Promotion and Education around Conformance Services Overview of Conformance Services Test Authoring Services Business Rule Management Conformance Profile Management Vocabulary Maintenance Test Case Authoring Test Data Management Message Instance Authoring Test Execution Services Conformance self assessment & evaluation Conformance independent verification & validation Conformance certification Technology Test Execution Tools  Conformance Validation  Test Case Validation  Self-test, monitoring Web presence for jurisdictions and vendors  Registration,  artifacts library,  test results Test Authoring Tools  Repository,  remote user access  shared authentication 17

Strategic Direction Analysis Project Project Objective: – Develop a preferred option for the future delivery of Conformance Services. Define and identify existing conformance services models; Develop a shared vision for the delivery of Conformance Services for the Canadian healthcare sector Develop conformance services delivery options for the Canadian healthcare sector; and detail the preferred option. Defined Strategic Intent of Conformance Services: – “Ensure that there is a national service to support the vendor conformance and certification of iEHR, public health and EMR solutions.” RFP ReleaseFebruary 15 ‘08 Project StartMarch 15 ‘08 Draft recommendationsJune 30 ‘08 Final Recommendations AvailableAugust 30 ‘08 18

Investment Project Support Overview Project Objective: – Provide immediate Standards Implementation Support for Test Authoring & Test Execution to key Infoway Investment projects Project Approach: – Develop a process & mechanisms linking the pan-Canadian Profiles through to conformance test – Develop, deploy, continuously improve “quick-hit” tools – Rollout to qualified projects (if proof of concept successful) – Harvest artefacts for sharing with other jurisdictions – Provide lessons learned and input into CS Strategic Direction Project – Provide transition and support to recommended Strategic Direction Project Kick-offNovember 1 ‘07 Proof of concept (2-3 projects)March 31 ’08 Proposed Project Extension: Limited rollout (4-6 projects)September ‘08 19

Investment Project Support Activities The specific services that Infoway is offering to assist projects Assistance in developing test strategies – Sort out overlap between conformance testing vs. UAT Education on development of business rules & test cases Technical education on development of message instances and test data Sharing of testing artefacts Quality review of artefacts developed by investment projects Full set of tools to assist getting to Conformance Testing Assistance to jurisdiction in rationalizing and integrating their testing tools with Infoway conformance tools Harvesting of testing artefacts for sharing with jurisdictions Sharing of Lessons Learned Facilitating inter-jurisdiction collaboration on conformance testing 20

Quality Review Harvest artefacts (2-3 days) 3 Introduction Workshop 0 21 Engagement Process to Conformance Introduction Workshop 0 Jurisdiction Conformance Pilot Testing, Quality Review Harvesting artefacts, Lessons learned (3-5 days) 5 Jurisdiction End-to-End Test Case Development (20-40 days) 4 Quality Review Harvest artefacts (2-3 days) 3 Jurisdiction End-to-End Test Case Development (10%) (15-20 days) 2 Test case Examples Harvested artefacts End to End Test Authoring Examples Jurisdiction Test Artefacts Tools End to End Test Execution Examples Jurisdiction Test Authoring Artefacts Jurisdiction still need to complete, but ready to perform Conformance Testing Ongoing Harvesting End to End Conformance Process Workshop (2-3 days) 1

22 Conformance Constructs – End-to-End Process Jurisdiction Input Best Practice External Application Conformance Profiles Standards Artefacts Test Cases HL7 Test Harness Test Case Authoring Test Data Instance Editor (Messages) Vocabulary Data - “Quick-hit” tools - “Existing” tools - HL7 V3 Business & Technical Rules Implementation Guides

Conformance: Canada’s EHR Major Tooling Components Business Rules Manager- Maintains Business Rules Conformance Profile Manager - browse pan-Canadian Conformance Profiles Implementation Conformance Profile Manager - browse profiles and create/maintain ICPs Static Model Constraint Tool - captures changes to models, and traceability information Instance Editor - creates Conformance Test Scenario Files, Message Fragments and Parameter Files as XML output. Test Case Author - Maintains Test Cases Test Authoring Manager - The main user interface Test Execution Manager - links the Test Authoring Manager to control the test execution Test Authoring Reports Manager - run reports

Conformance: Canada’s EHR Conformance Test Execution Scenarios There are different ways to perform Conformance Test Execution. This process uses the pCCT Test Execution manager and / or the HL7 Test Harness to allow the user to test both POS and iEHR systems as “real” or “simulated” applications under test. 1.Scenario Real POS with Simulated iEHR Application 2.Scenario Simulated POS with Simulated iEHR Application 3.Scenario Simulated POS with Real iEHR Application 4.Scenario Real POS with Real iEHR Application

Lessons Learned – Considerations for AB Testing artefact reuse is seen to be a big win One province has 100 business rules & 300 test cases – target is test cases Another target will be significantly higher (~2,500 test cases) Others will leverage oand likely add more Toolset are not all or nothing One province using blend of Infoway tools and home grown Uses Infoway tools and processes first to perform testing then conformance testing of PoS One province using Infoway tools for go forward & PoS testing Specific initiatives will likely use blend Infoway is investing in tools so jurisdictions don’t have to!

Conformance: Canada’s EHR Tooling Vision Create a suite of integrated Conformance Test tools based on Open Source principles and accepted pan-Canadian standards that allows jurisdictions, and vendors to perform conformance testing Includes: – Well documented processes and procedures – A repository of structured conformance profiles, test cases, test data and business rules that can be reused – A suite of re-usable conformance testing solution components able to easily exchange structured data between each component and projects – An Enhanced Web Based Presence to allow remote administration, self registration, shared resources management, self testing / monitoring and publication of results

Lessons Learned – Considerations for AB Many jurisdiction projects have left UAT & Conformance Testing until very late in their project lifecycle There is not a clear understanding of the continuum of conformance testing Profile  Business rules  Test cases  Messaging  Conformance test Traceability is key to conformance test Profile  Business rules  Test cases  Messaging  Conformance test Using Profiles to help them define user roles What does a physician do vs. project focus Tools can and are providing immediate benefit to jurisdiction projects for publishing Consolidated view of all business rules Publication of specification for vendor 27

Conformance Profiles Pan-Canadian Conformance Profile (pCCP) Implementation Conformance Profile (ICP) pCCP Extension Constraint Example: BC ICP #84 #84 BC Extensions BC Constraint Likely Initial State Desired State #47 #84 Library of Conformance Profiles Sask BC Nfld Sask BC Nfld + +

Conformance: Canada’s EHR Tooling for Test Authoring Components (TAC) – Use prototyping methodology to conduct staged development – Test with jurisdictional review and input – Document processes, requirements and changes – Garner feedback and experience to inform the refinement of detailed specifications for long-term open source tooling development – Address immediate jurisdictional needs & provide tangible benefits All test artefacts exportable Work with jurisdictions to rationalize and integrate tooling – Align test cases, business rules across jurisdictions – Promote early sharing and leveraging of work by jurisdictions

Conformance: Canada’s EHR Tooling for Test Execution – Leverage an existing open source test harness (previous Infoway investment), adding ability for jurisdictions to easily configure to reflect local technology solutions such as transport and security – Leverage Eclipse Instance Editor for message and vocabulary creation – Use data exported from test authoring tools Key is to ensure tools export into a format that can be consumed by HL7 Test Harness Invest in documenting processes, requirements, integration – Rationalize and integrate tools – Develop guidelines for local Test Execution prototyping environments including a reference implementation – Develop processes to ensure data integrity between different groups of users during prototyping phase

Conformance: Canada’s EHR It’s not easy: In order to get to a conformance test, you need to start with the Profile, know what the business rules are and create test data and message instances that will then test a scenario Once you have done this for every business and technical scenario, only then can you think about performing a conformance test By our simple calculations, this will be thousands and thousands of test cases!

0 – Step 0 Engagement 1 – Step 1 Workshop 2 – Step 2 Activities 3 – Step 3 Workshop 4 – Step 4 Activities 5 – Step 5 Workshop PlannedScheduled/UnderwayCompleted 0, ,5 0,1,2,3 Panorama 0 0, 1 0 0, Mar-08 Conformance Services Activity in Canada

Conformance: Canada’s EHR Provinces have done functional Certification for EMRs specifically. Other certification means defining differences in function, scope, perspective (e.g. Social interaction between users, staff and patients), Legal requirements are different between provinces, policies are different: Infoway cannot set policy but provinces must Provinces must work in collaboration with Health Authorities and professional bodies. Conformance criteria will be different from one business environment to another.

Conformance: Canada’s EHR Work is based on existing provincial models – Establish business rules to support policy set by Ministries of Health, College of Physicians and Surgeons and College of Pharmacists – Publish compliance specifications that include all the transactions needed at the point-of-service (e.g. pharmacy claims, pharmacy prescriptions, Client Registry, Provider Registry, Common, Lab, Immunization, etc.) – Provide support to vendors during their development – Conduct compliance testing of vendor software once they pass a pre-test – Support change agents (chain stores, vendors, etc.) during their training and deployment activities

Conformance: Canada’s EHR POS Software Compliance Principles POS systems – Must enhance patient safety & Support clinician workflow – Must be secure and ensure privacy of patient medical records – Must support interoperability standards – Are integral in creating and maintaining quality data for use throughout the health care system POS vendors must be able to differentiate their products Compliance documents – Are referred to in some provinces legislation – Communicate what rules need to be followed, not how the rules are implemented in vendor software – Provide one place to house Ministry policy, College rules and application capabilities – Provide a level playing field for all vendors regardless of size

Conformance: Canada’s EHR POS Conformance Specifications Define Business rules – Based on existing business rules from compliance documents and input from Clinical Working Group – Identify items that need further clarification Develop Compliance specifications – Based on existing compliance documents and published message specifications from Infoway – Builds on Infoway Conformance Profile definitions – Leverage, where possible, work from other jurisdictions Materials include: – POS Software Vendor Compliance specifications; – Vendor registration and development support processes; – Compliance testing process; – Process for approving vendors with compliant software Stakeholders included in joint review of material prior to publishing

Conformance: Canada’s EHR Compliance – Lessons Learned It takes time for the vendors to develop code to the level that warrants compliance testing Vendors may present software that was not stable enough to warrant compliance testing Vendors can require multiple attempts at compliance testing Thorough compliance testing takes time, especially on the very first test of new software or a major release change Good rules are needed on what constitutes an initial test and a follow-up test for each version of the vendor software Good rules are needed on when penalty and charges are applied to the test

Conformance: Canada’s EHR Sample tests were provided for vendors Focus needed to ensure that the vendors could prove they ran the tests and the tests worked before scheduling a compliance testing Need mechanisms to ensure compliance team provided consistent responses to all vendors throughout the compliance support process Good rules are needed around mandatory training that a vendor must provide e.g. training plans, training outcomes, etc. Good test facilities are needed for vendors and for compliance testing Multiple training environments are needed for each vendor Good tools are needed for the compliance testers that assist them in developing the test cases and support them during the testing

Conformance: Canada’s EHR Clinical guidance was critical during vendor development and compliance testing to clarify options and workflow – Requires ongoing clinical involvement Specifications, clinical guidance and support needs to include details Compliance team requires good communication and negotiation skills and need to be able to stand their ground under pressure Compliance documentation needs to be targeted to specific audiences and also clearly identify elements common across audiences A tool that enables easy linking and/or packaging of modules is highly desirable. Documents that are difficult to maintain don’t get maintained Business context for the compliance material is just as important as the rules themselves

Conformance: Canada’s EHR Conformance includes: – Message content and structure – Vocabulary and terminology – Interoperability profiles and message sequencing – Sending and receiving applications and intermediaries Conformance Testing includes: – Functionality of application – Basic privacy and security services related to the functionality – Workflow (to an optimal level determined by conformance continuum)

Contact Standards Collaborative Information Desk Bureau d'information de l'Unité collaborative de normalisation d’Inforoute or or Privacy and Security, SME

Holding slide