1e-Health 2013 Challenges & Opportunities TEC Talk Service Interrupted… AHS Experience with IT Major Incidents &Clinical Involvemente-Health Challenges & Opportunities TEC TalkJill speaking...Welcome to our TEC talk presentation. My name is Jill Robert and I along with Wendy Tagart are AHS employees with the IT department.BiographiesWendy Tegart BSc, BA - Provincial Director, Service ManagementWendy Tegart is a member of the management team within Alberta Health Services with responsibilities for improving IT service delivery by leveraging ITSM. Wendy's career has been focused on service management over the past ten years in various organizations including Alberta Health Services, Calgary Health Region and University Health Services in Toronto.As Director, IT Service Management, Wendy is building an ITSM provincial solution that will centre on delivering results by aligning IT and business goals with a focus on operational and service excellence, governance and best practices.Jill Robert BScN, RN – Provincial Director, IT Strategic PartnershipsJill has approximately 10 years of bedside nursing experience in critical care and another decade in the area of clinical informatics. She’s played a vital role in building, implementing and supporting clinical information systems both in Halifax and Calgary. With her knowledge and experience in clinical education and organizational change management, Jill supports Alberta Health Services to ensure the IT investment strategy in major projects and operations will bring value to clinicians and the organization as a whole.Wendy Tegart, Provincial Director Service ManagementJill Robert, IT Strategic Partner
2Faculty/Presenter Disclosure Faculty: Wendy Tegart & Jill RobertRelationships with commercial interests:Grants/Research Support: Not applicableSpeakers Bureau/Honoraria: Not applicableConsulting Fees: Not applicableOther: Employees of Alberta Health ServicesJill speaking...This slide must be visually presented to the audience AND verbalized by the speaker.As per the disclosure requirement, we have “Nothing to Disclose”.Nothing to Disclose
3Agenda Alberta Health Services Overview Major Incident Process 1Alberta Health Services Overview2Major Incident Process3Major Incident Roles4Communication Approach5Clinical Involvement6Next StepsJill speaking...We’re excited to share with you our evolving work at Alberta Health Services around supporting our organization when there are major IT service interruptions. Unfortunately, we have more experience in this area than we’d like. We’ll share the processes, roles involved and learnings we’ve garnered.But first, I’d like to provide a little background to where we come from and who we are representing...7Questions
4Annual Service Volumes (2011-12) Alberta Health Service OverviewAlberta Health Services (AHS)Responsible for delivering health services to the 3.8 million people living in Alberta, over 661,848 square kilometers servedAnnual Service Volumes ( )Acute Care2,029,191 Emergency Department Visits376,115 Hospital Discharges2,602,384 Total Hospital Days50,099 Births99 Acute care hospitals and 5 stand-alone psychiatric facilitiesJill speaking....Alberta Health Services was formed in In essence, Alberta dissolved nine regional health authorities, the Cancer Board, the Mental Health Board and the Alcohol and Drug Abuse Commission, and rolled them into one entity called Alberta Health Services. A province-wide health delivery system that is charged with delivering equitable services to 3.8 million Albertans no matter where they live in the province.Primary Care104,704 Home Care Clients766,146 Health Link calls393,964 EMS Calls/Events
5AHS Scale of Effort Largest Employer in Alberta, 5th largest in Canada Alberta Health Service OverviewAHS Scale of EffortLargest Employer in Alberta, 5th largest in Canada100,000 employees7,000 physicians120,000 network IDsScope of AHS-IT1,514 production apps (163 critical)34 data centers4,721 servers (physical and virtual)75,000 workstations48,000 tickets generated monthly550 concurrent users in ITSM tool1,300 IT Staff (+ outsourced partners)Jill speaking...The Alberta Health Services merger is the largest in Canadian history. It involves an $8 billion operation with approx 100,000 employees, along with physicians and other providers.From an information technology perspective, the AHS IT department has approx 1300 staff supporting 1500 server-based applications, 34 data centres and troubleshooting 48,000 IT service incidents per month.
6Context to Current Realities Major Incident ProcessContext to Current RealitiesComplexities of Electronic Health Record in AlbertaLocal vs Provincial IT service deliveryGiven the complexities of the AHS IT landscape, aging and varied technical infrastructure and critical service requirements to support patient care...“Downtimes happen...”How do we minimize organizational and clinical impact and provide robust support when the technology fails?Jill Speaking...While healthcare organizations increasingly adopt information systems for point-of-care support, these Electronic Health Record worlds have become progressively complex from a system integration and data integrity perspective. The building components of the Alberta Health Services EHR are extremely complicated from both the upstream and downstream components including: registries, repositories, patient care systems, interfaces, and how information is accessed/viewed across the enterprise through the Alberta Netcare Portal.When AHS formed in 2008, twelve IT departments needed to be consolidated. We have spent a lot of effort in forming, storming and norming as we aim to move away from local IT solutions to provincial service delivery models. There is added complexity with aging and varied technical infrastructure, limited funds and increased project demands while at the same time having to ensure that the critical IT services to support patient care are supportable, sustainable and scalable.While we have made some headway, there is still a lot of work ahead of us.All of us in this room know that “Downtimes happen...” and while the outage may not be for long, in the eyes of our customers, the measure of the IT department can be based on their most recent downtime experience.
7Super Bowl 2013 – infamous power outage Major Incident ProcessSuper Bowl 2013 – infamous power outageWendy speaking...How many of you watched Super Bowl 2013? Over 100 million people worldwide tuned in to watch this years Super Bowl XLVII. It could be argued that it was the most viewed and infamous power outage. There is varied accounts of why it happened from a power line issue, to a software glitch, to a hardware issue. It felt like forever, but the actual electricity outage was for 33 minutes.When outages happen in a healthcare environment, it is also critical as it can impact patient safety and these types of outages put us in the news as well.SHAW Communications OutageIn July 2012, Calgary experienced a huge outage with the Shaw Communications Building electrical fire that impacted radio stations, internet service, 911 service in the core, cable TV service, much of city hall’s phone system and all AHS applications hosted in that site (including and patient care systems mainly for Calgary zone).It was related to a electrical fire caused by a transformer explosion, there was redundancy for power through dual generators but unfortunately the sprinkler system caused those to shut down as well. It took 2.5 days to bring back up the AHS systems as water in the building/data center was an issue. It then took AHS an additional 3 days to enter paper information used during the outage and to manage data reconciliation efforts.In January 2013, Shaw disclosed that it received $5-million from its insurance company to help repair the damage. Moreover, the company said it was facing additional costs of $6-million for ongoing restoration. Total equalling $11M.
8What is a Major Incident (MI)? Major Incident ProcessWhat is a Major Incident (MI)?IT has a provincial Incident Management Process to manage all Incidents. When an Incident is of a certain scale, scope, or impact, a “Major” Incident is launched.The goal of the Incident process is to return an IT Service to operational status.Throughout AHS-IT, we employ this common process to ensure that major IT service issues are quickly identified and appropriately responded to.The purpose of the MI process is to supplement the Incident process with additional resources, escalation, communication and record keeping.Wendy speaking...As most systems were and are still structured geographically, this multi-merger added complexity to the management of both scheduled and unscheduled downtimes.Arriving at a common definition for an IT Major Incident and creating a single process for escalation, communication and documentation in a distributed support environment was critical.A key aspect has been developing close partnerships with all levels of clinical operations to assess and minimize impact to patient care. Leveraging ITIL best practices of Incident, Change, Configuration, Knowledge and Problem Management processes was vital for success.
9Is this a “Critical” Incident? Major Incident ProcessIs this a “Critical” Incident?Urgency and Impact must both be High to create a critical incident. Critical Incidents must be escalated to the IROC immediately. Critical incidents are:a major outage affecting a large number of customersan essential service and/or a business unit where there is no available resolution or work around to provide a return to business operationsMust also consider:Patient safety may be at risk or reduced effectiveness of patient careThe safety of AHS staff and personnelImpact to confidentiality of data, or reliability of dataDegradation of a service including data, applications, or infrastructure.A Senior Admin from the business is requesting a Major Incident be declared (requires immediate escalation to IROC)Wendy speaking...How do we identify IT Major Incidents? All critical priority incidents are Major Incidents. (MI) – and no this is not a Myocardial Infraction (heart attack) but I am surprised that they have not given our CIO one yet.The business, a senior leader, an Incident Response On Call who is a Director or a Service Owner could request the MI process be invoked. In any case the incident should immediately be escalated to the IROC to initiate the Major Incident process. Incident Management activities that drive resolution of the Incident should begin and continue while the MI process is initiated.Critical incidents are:a major outage affecting a large number of customers – like the Shaw Court firean essential service and/or a business unit where there is no available resolution or work around to provide a return to business operations – like a network outage that affects DI PACS which would impact patient careOur IROCs are at the Director Level as it takes operational experience and being well seasoned to make the judgement call of when to launch an MI. Additionally, each Service/Application should have a clear understanding of business expectations that identifies when a Service or Application qualifies for an MI and clearly identifies which support groups need to be contacted in the event that an MI occurs.Like medicine, determining if an incident meets the critical of an MI - is more of an art form than a science.
10Is this a “Critical” Incident? Urgency Major Incident ProcessIs this a “Critical” Incident? UrgencyWendy speaking...How do we establish the priority of an incident – is this a critical incident?Priority is a combination of impact and urgency.The combination of a High Impact and a High Urgency create a Critical Priority Incident.High Urgency is described as …Read description
11Is this a “Critical” Incident? Impact Major Incident ProcessIs this a “Critical” Incident? ImpactWendy speaking...Read descriptionWho is affected etc.
12Priority Major Incident Process Wendy speaking... Once the assessment is completed and determines the high impact plus high urgency equals a critical priority the major incident process must be initiated.When a critical incident is logged, the IT staff must initiate the process.The MI Process must be initiated as soon as the critical incident is identified.
13Major Incidents by Month Major Incident ProcessMajor Incidents by MonthWendy speaking...How many of these do we manage? The data is not encouraging.The large spike at the end of 2011 coincides with the CIO directive to ensure all Major Incidents are reported numbers are consistent in this change in reporting of MI’s.In 2013, all incidents with a critical priority must be immediately escalated as an MI.The MI process is more of a organizational change management effort than an ITIL process - as we are trying to create a culture where it is better to declare an MI and stand it down, than not declare one and delay the communication of the issue or the resolution.
14IT Major Incident Roles An IT service Incident is typically managed by the IT Service Desk and/or a specific IT Service team. When an MI is initiated, some additional resources brought in include:IT Incident Response On Call (IROC)This is a group of IT Directors who share an On Call responsibility for MI’s. Once contacted, the IROC is responsible for managing the MI Process so the Service Desk and Service team can concentrate on resolving the Incident.IT Security & Compliance On CallOn Call IT Security staff to respond to MI’s with a security component.IT Senior Leader On CallThis group of IT senior leaders is available to provide additional guidance and authority if/as required by the particular MI.Problem ManagerChair and facilitate communication bridge meetings. Notify IT staff of updates.Wendy speaking...Now we’re going to expand on some of the additional roles and processes that occur in Major Incidents.
15Major Incident RolesClinical RolesNot all MIs require the engagement of clinical experts, but when required these roles provide context to clinical impact and urgencyClinical InformaticsThis is a group of Physicians and non-physiciansClinical Operations Administrator On-callOn Call AHS leaders including Executive Directors and Site Administrators. May provide front line resources to support in downtime and reconciliation effortsSenior Leadership On-callThis group of AHS Senior leaders include Facility Medical directors and VPSHealth Information ManagementHealth Record Management experts with data and record integrity expertiseZonal Emergency Operations Centres (ZEOCs)Tied into Emergency PreparednessJill speaking...In order to understand what the incident or technical failure means to front line clinicians, the potential risk to patient care and what messages need to be delivered ASAP, we have a number of clinical folks actively participating in the MI process. This ranges from physicians to nurses to health information management experts. We also link into the clinical administrative leadership – accessing their existing on call processes to notify them of MIs and to determine if and when staffing augmentation is required. And finally, when all hell breaks loose, we call on the disaster preparedness processes and leverage the Zone Emergency Operations Centres.
16Bridges Types (conference calls) Communication ApproachBridges Types (conference calls)Technical BridgePart of the Incident Management process, as is initiated independently of MI processOpened when collaboration by several parties is required during incident resolution activitiesCommunications BridgeLaunched by IROC to bring the right stakeholders together to identify the problem and direct its resolution.Problem manager assists by recording chronology, participants, decisions and resultsDirects communications within IT and the user communityClinical BridgeUsually chaired by a Clinical Informatics physicianWendy speaking...Technical Bridge: is important to the resolution of Major Incidents. It is part of the Incident Management process that is initiated independently of the Major Incident process. The technical bridge should be opened as soon as it is required.Communications Bridge: is a call that is lead by the IROC and supported by the Problem Manager between technical specialists, vendors, IT leads, business representatives and other people integral to identifying the problem and directing the resolution.The purpose is to identify the impact, action items to resolution, key players, communication strategy and timelines.The Major Incident Communications bridge will also direct communications with IT, and to the user community as necessary during the resolution process.Finally, it supplements the Incident process with additional resources, escalation, communication and record keeping.Note: Depending on the complexity and impact of the MI, separate bridge calls are kicked off to address technical, clinical, and communication aspects of the MI. The challenge is to ensure that it is clear who is the liaison between the bridges to provide updates.
17MI Heads Up Notification Communication ApproachMI Heads Up NotificationWendy speaking...Sample notification coming to the Major Incident notification mailing list.We ask that all teams who are involved in some aspect of supporting the service in question to call into the bridge at the designated time to get and provide more information.The first bridge call is considered an all hands on deck situation, subsequent bridge calls should involve only those who have an interest in the resolution of the incident.This will also include Technical and Clinical bridge numbers if they are available or required.
18Communications to Customers Communication ApproachCommunications to CustomersIT - Service Issue InformationMessage may be sent to users of IT Services. In relation to unexpected/unplanned service issues. Say who is this information is intended for / pertains to. Speak in terms the customer will understand.Briefly and directly tell users what is happening and what impact they will experience. Note that IT teams are working to resolve the issue and restore Services.Acknowledge the issue/inconvenience and provide contact information for the relevant zone/FHE Service Desk. If appropriate, state that an update will be provided within a specific timeframe.Replace all text in this section with pertinent information. Review the notifications guide if there are questions on when to use this format.Impact SummaryClearly state what, from the users perspective, is not working. Also set out the specific locations affected by this Service Issue.NOTE: Any exclusions or caveats to what you've stated above regarding this Service Issue.Wendy speaking...This is the format designed for communication to customer in the event of a service outage. During the MI call there should be a discussion of if an announcement to customers is needed, who should draft the text and who should send it.In the event of a large scale outage, Corporate communications will take on the roll of communication with customers and they will use their own templates.
19MI Root Cause Code Definitions Communication ApproachMI Root Cause Code DefinitionsCause CodeSummaryApplication/Software BugThe failure is caused by a problem within the packed software itself.CommunicationFailure is caused by a missed communication.DataUnexpected or corrupted data elements caused the failure.EnvironmentThe failure is caused by an uncontrolled element of the physical world where redundancy would not have reasonably mitigated the effect.EquipmentFailure due to age, malfunction or fault in the physical equipment where redundancy would not have reasonably mitigated the effect..IT Third Party VendorRoot cause lies with the vendor providing a service.ProcessMissing or undeveloped process caused the failure. There was an oversight in the process; a branch of the process isn’t properly developed or missed entirely.SecurityAn IT Security failure caused the issue.TrainingThe failure was caused by lack of understanding, incorrect qualification or insufficient training.OtherA mistake was made where existing process, if followed correctly, should have avoided the failure.UnknownRoot cause undetermined.Wendy speaking...Starting in Jan 2013, we have tightened up the root cause definitions to better pin point areas for improvement and see trending issues.After every MI, there is a meeting with the core stakeholders to finalize the details of the MI including root cause, and action items to be completed so that this MI does not reoccur.
20Clinical Support During a MI Clinical InvolvementClinical Support During a MITransparencyCommunicationUnderstanding and translating the clinical impactTimely and frequent “clinical speak” communication about the incident and immediate risk mitigation measuresSupportRobust downtime procedures owned by clinical operationsBedside to boardroom engagement and supportIn his book The Quantum Age of IT, Charles Araujo (A-row-zhu) talks about working for a health care organization that was having many MI’s due to old infrastructure. The organization openly acknowledged that the problem could not be fixed overnight. Speaks to their honesty. But what they did do was assign people to call the senior executives and give them the heads up when issues were brewing. One person, one exec. Instant communication and improved transparency. Ironically, despite the MIs, they did find that their IT satisfaction scores improved.One of our challenges has been the translation of an incident into what it means to clinicians or AHS staff in a language that makes sense. The point is succinct communication which focuses on the perceived area of greatest clinical risk, what staff need to do in the mean time, and don’t speak techie! We have a core group of nurses and physicians that help us craft meaningful communications during an MI.I heard the other day that the rule of 7 (communicate your message 7 times in 7 different ways) has now been replaced by the rule of 14. I think that speaks for itself – we so often rely on to get our messages out. We’ve provided some of the templates that we use, but we also use broadcast faxes, humans (staff rounding to units), phone calls, overhead pages, splash screens, our intranet site and so forth.The last comment I’d like to share is about the ironic challenge we face - many organizations have the uphill battle with CIS adoption, but then the pendulum swings. We now are in a situation where we have clinicians that have never worked with a paper record and those that did, forget what it was like and how to revert back to paper processes in emergency situations. Downtime procedures and clinical staff owning and embedding them into their annual ‘refreshers’ or ‘re-certification’ processes is key to minimize disruption during a major incident. Continual effort is required to train staff on how to operate in the event when a system is not available.
21Clinical InvolvementMore than clinical involvement... it’s about relationships, partnerships and supporting safe patient care!Jill speaking...When Wendy and I were preparing for this talk, we realized how patronizing the use of “involvement” was. When we reflect on the most major incident we’ve experienced, the Shaw Communications outage, the complete shut down of the entire AHS Calgary IT network and along with it, the use of its applications both corporate and clinical, and on top of that, the entire AHS system (from High Level in the north to Lethbridge in the south) – the success factors in supporting that event were much greater than just “clinical involvement” but entire AHS organizational teamwork which was possible because of relationships, partnerships and the sole focus on providing and supporting safe patient care. It was about leveraging the Zone Emergency Operations Centre – corporate, clinical and IT teams organizing and operating out of one central location, working 24/7 and dispatching people, information and support wherever it was needed. For example our IT administrative assistants put on their running shoes and delivered lab and DI reports to patient care units. It was about hourly updates distributed to the front lines and conference calls with the entire Calgary AHS clinical administration and 3 to 4 daily meetings at each facility with patient care managers. It wasn’t about blame, it was about the organization (clinical, corporate and IT) pulling together to mitigate the patient care risk of a major major problem.
22Next Steps Continuous improvement per incident review Develop service improvement plans overall driven by business requirementsExamine different scales of MIs and support requirementsLeveraging the successes of the MI process to other risk areasContinually examine clinical business risk tolerance/value and architecture of information systemsSimplify – application consolidation, migration to a provincial patient care platform and large scale reliability/redundancyWendy speaking...Work is ongoing to improve communications between IT and the clinical business, to better understand the business continuity needs, and to better align with the corporate Emerging Incidents and Disaster Preparedness processes.It’s like that old saying that you need to expect the unexpected. When the unexpected does arrive you have to be prepared to come back from that downtime swiftly and with as little disruption to your business as possible. With the right technology and the right best practices in place, you can minimize the damage and decrease the chance of downtime seriously hampering your ability to do business.There is nothing like a Major Incident to get the circulation flowing... With every MI we’ve learned, and we’ve absorbed and we are committed to continually find ways to improve it... Because unfortunately no matter how redundant the infrastructure or applications are... Downtimes happen and IT will fall. What matters is what we learn from it and how we use this information to improve.1. Good judgment comes from bad experience.2. Experience comes from bad judgment.-- Higdon's LawHigdon’s LawGood judgement comes from bad experience.Experience comes from bad judgement.
23Comments / Questions Questions? Insanity: Doing the same thing over and over again and expecting different results. ~ Albert Einstein