Evaluation of Hospital Drills: Using the Tool Amy Kaji, MD, MPH November 16 th, 2005 Acute Care College Medical Student Seminar

Hospital Disaster Drills Why are drills necessary? Hospitals will be called upon to provide care to the ill, injured, exposed, and concerned Joint Commission on Accreditation of Healthcare Organizations (JCAHO) Requirement May help train employees Tests aspects of hospital response

Hospital Disaster Drills Types Computer simulation Tabletop exercises Operationalized drills involving specific victim scenarios Evaluations Can help maximize the value of the drill Based on accurate observation Benefit of standardization

Johns Hopkins University Evidence- based Practice Center (JHU EPC) Developed an evaluation tool for hospital drills Assess impact of drill as hospital response progresses and develops Presents topics for evaluation in a systematic manner Identify strengths and weaknesses in hospital disaster preparedness Promotes targeted efforts to strengthen preparedness NOT intended to be used to determine whether a hospital passes or fails in its planning endeavors

The Evaluation Modules Developed by a multi-disciplinary team of experts at JHU EPC Based upon systematic review of published reports on hospital disaster drills and literature relevant to bioterrorism preparedness Had “expert input” from federal, state, and local agencies Piloted the modules in two multi-hospital regional disaster drills in the summer of 2003

Guiding Principles in Developing Evaluation Modules Need for observing multiple hospital zones Evaluation of a disaster drill requires an understanding of drill activities in all areas of the hospital Four zones identified include: Incident Command Decontamination Triage Treatment Addendum if Biological or Radiological Scenario Decontamination Zone Module needed for radiation & chemical drills; not recommended in biological drill

Guiding Principles in Developing the Evaluation Modules Need for documentation of time points Recording time points of drill activities is a widely accepted method of evaluation Labor-intensive if excessive number of time points and may detract from overall evaluation Limited, specific time points thus predetermined

Guiding Principles in Developing the Evaluation Modules Need for documenting clinical care outcomes Track the volume of victims in each zone and adequacy of provisions made for them, including space, staff, supplies, etc. Modules are not intended to collect individual victim level data Modules monitor the zone and outcome for the zone as a whole, not for each victim

Guiding Principles in Developing the Evaluation Modules Need for debriefing (after-action review) Obtain feedback from participants, including organizers, staff, and victims Allows for discussion of issues that span more than one zone Evaluate and integrate cross-zone issues at a post-drill debriefing session

Guiding Principles in Developing the Evaluation Modules Need for ease and flexibility of care Designed to be readily understood, easy to use, and applicable to many different drill scenarios Items on form are arranged by subject, and this is the same across all modules to facilitate analysis Decision to include a module is dependent upon the specific scenario and drill

Guiding Principles in Developing the Evaluation Modules Need for safety and security Consider planned drill activities; i.e., use of unfamiliar equipment Protect actual patients on the premises Safety of drill victims and healthcare workers Contingency plan to stop the drill in case of an actual emergency Consider designating a safety officer to monitor the drill and its participants

Recommended Use of Evaluation Modules and Addenda ( X = USE IT!) Type of DrillModulesAddenda FireICTriage TreatmentDECONBiological RAD Explosion XXX Natural Disaster (Earthquake, flood, drought) XXX Internal Hospital Failure XXX BiologicalXXXX ChemicalXXXX RadiationXXXXX

Internal Structure of Evaluation Modules Zone forms for Incident Command, Decontamination, Triage, and Treatment have same structure and subject headers: Time points Zone description Personnel Zone operations Communications Information flow Security Victim decontamination and tracking Victim flow Personal protective equipment (PPE) and Safety Equipment and Supplies Rotation of Staff Zone disruption

Internal Structure of Evaluation Modules Numbering of questions on the forms “C” denotes questions that are common to several zones Questions that are zone-specific are identified by a unique letter code 120 questions per module Coding on the forms and use of comment boxes Yes = Y; No = N; Unclear = U; Not applicable = NA NA is not a substitute for missing information, negative information, or to avoid writing a comment Color coding for the modules and addenda Assists in organizing and tracking modules and addenda before and after a drill

Common Sample Questions from an Evaluation Module (Time Points) C1. Time the drill began:___AM/PM/U C2. Time the hospital disaster plan was initiated in this zone:___AM/PM/U/Not initiated C3. Time this zone was ready to accept victims: ___AM/PM/U C4. Time when this zone was notified that incident command was operational:___AM/PM/U/Not notified C5. Time the drill ended in this zone: ___AM/PM/U

Common Sample Questions from an Evaluation Module (Zone Description) C8. Was the boundary for this zone defined? Y/N/U C9. If this zone had a defined boundary, how was it defined? (Check all that apply) Barricade Security personnel Sign Tape Vehicle Wall (permanent or temporary) No boundary Other (specify): __________________

Common Sample Questions from an Evaluation Module (Personnel) C11. Did someone take charge of this zone? Y / N / U C12. If someone took charge of this zone, how many minutes after the drill activities in this zone began did this person take charge? (Check one) O < 10 min O min O min O hrs O > 2 hrs O NA C13. If someone took charge of this zone, was it the officially designated person? Y / N / U / NA

Common Sample Question from an Evaluation Module (Personnel) C14. How was the person in charge of the zone identified? (Check all that apply) a. [ ] Arm band b. [ ] Hat c. [ ] Name tag d. [ ] Verbal statement e. [ ] Vest f. [ ] Not identified g. [ ] Other physical identification (specify): __________________________________________

Common Sample Questions from Evaluation Module (Zone Operations) C29. Was the space allocated for the zone adequate? Y/N/U C30. If not enough space for the zone, where did zone activities overflow to? (Check all that apply) Adequate space allotted Conference room Hallways Outside hospital Treatment/victim care areas Waiting rooms No overflow NA Other (specify): ______________________________ C31. Was this zone used for the same functions during non- drill operations?Y/N/U

Common Sample Questions from Evaluation Module (Zone Operations) C33. Did clinical staff interact directly with families of victims?Y/N/U/NA C34. Were families of victims referred to specially designated staff? Y/N/U/NA C35. How was victims’ privacy ensured? (Check all that apply) Curtains Individual areas Privacy screens Not ensured Other (specify): ____________________________

Common Sample Questions from Evaluation Module (Communications) Communication device (s): If device not present, circle “N” in column “a” and go to the next line. A. Was device present?Y/N/U B. If present, # available: C. If present, was it used in drill?Y/N/U D. Comments (note problems) C36. 2 way radio/phones C37. Direct line C42. Numeric paging C44. Text paging C45. and internet access C47. Intercom C49. Runner

Common Sample Questions from Evaluation Module (Information Flow) C54. How was this zone notified of the event? (Check all that apply) FAX Runner Telephone Not notified Other (Specify):__________ C55. Who notified this zone of the event? Drill organizer Incident command center Media Other hospital staff Outside source Victims arriving Not notified Other (specify):____________

Common Sample Questions from Evaluation Module (Security) C60. Were security present in this zone?Y/N/U C62. If security personnel were present, what type of security? (Check all that apply and provide approximate numbers) FBI ___ Hospital Security ___ Local Police ___ State Police ___ NA Other (specify): ______________

Common Sample Questions from Evaluation Module (Victim Documentation and Tracking) C74. Were all incoming victims registered and given a unique identification or medical record number? (check one) Yes, before entering this zone Yes, on entering this zone No, not while in this zone Unclear C76. Was a central list of victims generated for this zone? Y/N/U C77. Were the triage markers on the victims clearly visible? Y/N/U

Common Sample Questions from Evaluation Module (Victim Flow) C81. Did a bottleneck develop in this zone? Y/N/U C84. Were the paths leading to the next zone marked? Y/N/U C86. Were the lowest acuity victims directed by staff to an area separate from higher acuity victims? Y/N/U C87. What proportion of victims had treatment delayed because of zone staffing shortage? (Check one) None Less than half At least half (but not all) All Unclear

Common Sample Questions from Evaluation Module (Personal Protective Equipment and Safety) If needed, were these items for standard precautions available for the healthcare workers? Y/N/U Used by staff? Y/N/U Adequate supply?Y/N/U Eye protection Waterproof gowns Isolation gowns Gloves

Common Sample Questions from Evaluation Module (Equipment and supplies) TX15. Were medications needed for the treatment of victims available within the hospital?Y/N/U/NA TX16. Were medications requested from and outside source? Y/N/U Were needed medical supplies available? TX19. BandagesY/N/U/NA TX20. Basic airway equipmentY/N/U/NA TX22. Blood drawing suppliesY/N/U/NA TX23. Burn PacksY/N/U/NA TX24. Cleaning supplies for contaminated equipment Y/N/U/NA TX25. Crash cartsY/N/U/NA TX26. Intravenous fluids Y/N/U/NA

Common Sample Questions from Evaluation Module (Rotation of staff) C98. Was there a staff rotation /shift change?Y/N/U C99. If there was a staff rotation, did the officially designated person in charge of the zone change? Y/N/U/NA C101. What method of shift changing was used? (Check one) Group shift change Staggered shift change NA Other (specify): _____________ C102. How were incoming staff updated? (Check all that apply) Group briefing Individual briefing Written notes Not updated NA Other (Specify):_____________

Common Sample Questions from Evaluation Module (Zone disruption) C103. Was there a plan to relocate this zone if necessary? Y/N/U C104. Did this zone close at any time during the drill?Y/N/U If no, disregard the remainder of this section. STOP. This zone module is complete… C105. If the zone closed during the drill, what was the reason for closing? (Check all that apply) Contamination Other safety concerns Space Other (Specify): ________________

Description of Modules and Objectives Pre-drill Module Incident Command Center Zone Module Decontamination Zone Module Triage Zone Module Treatment Zone Module Group Debriefing Module

Pre-drill Module Should be used in all disaster drills during the planning stages Form is designed to collect the following: Goals and objectives for the scope of evaluation Background information Information on areas that hospital wishes to evaluate Resources required If multi-hospital or regional drill, each site must work closely with overall coordinators

Incident Command Center Zone Module Designed to reliably collect information about operations of the incident command system (ICS) Should be used in all disaster drills when evaluating the ICS Form is designed to assess the following: Command structure in the zone Adequacy of staffing in the ICS Communication and information flow from hospital areas to the ICS Communication with outside agencies Adequacy of security, safety provisions, and physical space

Decontamination Zone Module Designed to collect information re: functioning of decontamination area Used if scenario involves radiation or chemical exposure and decontamination is needed Form assesses the following: Command structure in the zone Communication and information flow in the zone Victim and staff safety in the zone Adequacy of staffing and physical space in the zone Appropriateness of equipment and PPE Victim flow in the zone

Triage Zone Module Designed to collect information re: functioning of treatment areas Should be used whenever drill objectives include evaluation of patient care activities beyond triage Appropriate for use in emergency department-based treatment areas or in other clinical areas

Triage Zone Module Form designed to assess the following: Command structure in the zone Communication and information flow in the zone Victim and staff safety in the zone Relation of physical characteristics of zone to treatment activities Efficacy of treatment operations Adequacy of materials and supplies in the zone Victim flow in the zone

Debriefing Debriefing is integral part of drill process Main objective of debriefing is to identify issues not captured by evaluation modules Facilitators should create an open, non- judgmental atmosphere Should occur in all drills to obtain feedback from participants and observers

Debriefing Different approaches to debriefing exist One method: conduct a session with all participants and observers present and ask a series of general questions about the drill Another method: conduct a group debriefing session with the participants from that zone

Group Debriefing Module Contains open-ended questions designed to facilitate discussion after completion of drill Questions may be added or deleted Designed to cover all issues, including incident command structure, communications, security, decontamination, triage, and treatment Documenting the debriefing A scribe should be assigned to record Videotaping / audiotaping debriefing may help capture comments but should not hinder open exchange

Group Debriefing Module Questions Did you feel you were notified of the disaster in a timely fashion? Did the incident command center work effectively? Did anyone receive incorrect information from the incident command center? If not correct, what specifics do you recall about incorrect information? Was the information from the incident command center received by other zones in a timely way?

Group Debriefing Module Questions Were there problems with information flow within the hospital? Were memorandums of understanding with outside agencies (e.g., police) activated? Did nurses and physicians respond quickly to the disaster call? Was the zone set up when the first mock victim arrived? Was security in place before the first mock victim arrived?

Group Debriefing Module Questions Did people have a good understanding of their roles, as defined in the disaster plan? Did the decontamination system work effectively? Did you have any problems with the decontamination equipment? Functioning properly? Adequate number of units? Participants used correctly? Were there delays in decontamination? If so, what triggered those delays?

Description of Addenda Four addenda are part of the hospital disaster drill evaluation: Biological Incident Addendum Radiation Incident Addendum General observation and documentation addendum Victim tracking addendum. Used to supplement the zone forms Example: for a radiation exposure drill, the Radiation Incident Addendum is added to Incident Command Center, Decontamination, Triage, and Treatment Zone Modules

Biological Incident Addendum Designed to collect additional information that address response to a biological incident Should be added to end of each Incident Command, Triage, and Treatment Zone modules Should be used in all drills that address a biological incident

Biological Incident Addendum Form is designed to assess the following: Awareness that biological agent cause of illness Whether appropriate personnel were contacted Whether health and safety needs of staff were met Whether health and safety needs of existing patients were met Whether health and safety needs of victims were met Availability of special medications and supplies

Biological Incident Addendum BA11. Was the Center for Disease Control and Prevention (CDC) informed that a biological illness was present? BA14. Was occupational health informed that a biological illness was present? BA22. If the cause of the illness was not known prior to victim arrival, how long after the first victim arrived was the cause of the illness identified? (Check one) <1 hour 1-4 hours 5-8 hours >8 hours Never identified NA

Biological Incident Addendum BA24. What resources were used to make the diagnosis? (Check all that apply) Consultation with an in-hospital expert Consultation with an expert from another hospital Consultation with an expert from local health department Consultation with an expert from state health department Consultation with the CDC History and physical exam by the treating physician Microbiological data Radiological data NA Other (specify): ______________

Biological Incident Addendum BA26. Was isolation required for the suspected biological illness involved? Y/N/U (Isolation is required for smallpox, plague, viral hemorrhagic fever, certain pneumonias or rashes, and other symptoms suggestive of a contagious infectious outbreak) BA27. If isolation was required, were victims transported into an isolation room?Y/N/U BA28. If they were transported to an isolation room, was the room under negative pressure? Y/N/U/NA

Biological Incident Addendum BA29. Were there enough isolation rooms? BA30. If not enough isolation rooms, how were victims isolated? (Check all that apply) Conversion of other rooms/area (specify):___________ Existing isolation room in other area (specify):__________ Overflow victims not isolated Victims with the same suspected biological illness placed in the same isolation room NA Other

Radiation Incident Addendum Designed to gather information in response to radiation-related incident Should be added to end of each zone module Should be used in all drills that address radiation exposure

Radiation Incident Addendum Form is designed to assess the following: Awareness that radiation was cause of illness Whether appropriate monitoring personnel were contacted Whether health and safety needs of staff were met Whether health and safety needs of existing patients were met Whether health and safety needs of victims were met Availability of special supplies

General Observation and Documentation Addendum Designed for use by an additional observer to document detailed activities in a single unit Example: During a chemical drill, an additional observer could be assigned to area where PPE is donned to document time required to dress, appropriateness of dress, etc. Has a front page and a continuation page, which may be copied as necessary

General Observation and Documentation Addendum Instructions: Enter time and describe any activities in this zone related to drill. Include: Response to patients Information received Any activities (such as real emergencies) that may delay or prevent drill activities in this zone TimeActivity or Observation

Victim Tracking Addendum Designed for use by an additional observer to track victims through the drill May be used within one zone for a large group of victims, OR Observer can follow victims across zones from beginning to end of drill disposition from medical perspective Useful if there is victim descriptions and emergency medical procedures are well delineated Has front page & continuation page

Victim Tracking Addendum Victim IDTime arriving in zone Time exiting from zone Disposition

Observers Value and success of drill, depend on observers, who must be able to understand and record events Observer selection is therefore critically important Observers must be trained to use evaluation modules Documentation by observers provides information for evaluation Record the type and number of victims, as well as the care given or not given to victims May record personal views but should note it as opinion

Role of Observer Observers must not have any role other than that of evaluating the drill Individual who normally works in zone may function as an observer, but during the drill, he/she may not assist with any activities Must not respond to questions from drill participants about the drill To qualify as observers, volunteers drawn from outside the hospital must have knowledge of hospital functions

Observers Background knowledge required General knowledge of operations of zone Specific medical knowledge not required

Observers Number of observers needed in each zone One observer should be present in each zone continuously Additional observers may be needed to observe numerous staff or victims Should use victim tracking or general observation and documentation addendum

Observers Number of observers needed in each zone If extensive time point data collection is needed, an additional observer should be assigned to this task Specific time points and reasons for collection should be outlined before drill If clinical process or outcome data is needed, additional observers will be needed Observers must have sufficient clinical knowledge to report on decision making

Observer Training Training session Receive zone assignments Review relevant zone modules Achieve familiarity with content of evaluation modules and zone configuration Explain all questions and response sets Address questions about forms Instruct how to be an effective observer

Final Observer Training Points All observations are confidential Observer should evaluate drill without obstructing flow of drill Observers may ask participants questions to clarify observations Questions should be asked in an unobtrusive manner Observers should refrain from asking questions that may alter drill actions

Final Observer Training Points Observers must not participate in drill activities If questioned about a drill issue by a drill participant, the observer should state that they are evaluating and unable to answer Each question on each module should have a response The response NA should be indicated only when the question does not apply

References Cosgrove SE, Jenckes MW, Kohri D, Hsu EB, Green G, Feuerstein CJ, Catlett CL, Robinson KA, Bass EB. Evaluation of Hospital Disaster Drills: A Module-Based Approach. Prepared by Johns Hopkins University Evidence-based Practice Center under Contract No AHRQ Publication No Rockville, MD: Agency for Healthcare Research and Quality. April 2004.