1. Interrater agreement and time it takes to assign a Canadian Triage and Acuity Scale score in 7 emergency departments
McLeod, S.L.1,2 McCarron, J.3 Stein, K.3 Al-Mujtaba, M.3 Scott, S.3 Ovens, H.1,2 Mittmann, N.3 Borgundvaag, B.1,2
1) Schwartz/Reisman Emergency Medicine Institute, Mount Sinai Hospital
2) Department of Family and Community Medicine, University Of Toronto
3) Analytics & Informatics, Cancer Care Ontario
Reliability Study
Purpose
Preliminary Results
Figure 1: Generating a CTAS Score
The Canadian Triage and Acuity Scale (CTAS) is used in all Canadian emergency departments (EDs) for establishing the priority by which patients should be assessed
In 2010, 38% of patient cases in Ontario were under-triaged1
In 2015, the Ministry of Health and Long-Term Care engaged CCO to develop and implement a provincial electronic Canadian Triage Acuity Scale (eCTAS) system to improve consistency in CTAS application
In 2016, in preparation of the implementation of eCTAS, CCO conducted two evaluation studies to establish baseline data regarding 1) reliability of CTAS assessments, and 2) triage times at representative hospitals using both single and double stage triage
Reliability Study Results (Table 1) indicates:
738 consecutive patient CTAS assessments were audited
Exact modal agreement was achieved for 554 (75.0%) patients
On-duty triage nurses:
Over-triaged 89 (12.1%) patients
Under-triaged 95 (12.9%) patients
Interrater agreement was “good”
Unweighted kappa of 0.63 (95% CI: 0.58, 0.67)
Quadratic-weighted kappa of 0.79 (95% CI: 0.67, 0.90)
Time Study Results (Table 2) indicates:
Overall median (IQR) triage time was 5.2 (3.8, 7.3) minutes
Triage time varied from 3.9 (3.1, 4.8) minutes to 7.5 (5.8, 10.8) minutes
Median (IQR) time for:
Single-stage triage: 4.4 (3.5, 6.0) minutes
Two-stage triage: 7.3 (5.6, 10.0) minutes
Table 1: Interrater Agreement
Table 1: Interrater Agreement
Expert CTAS Auditor 1 2 3 4 5
Total
Triage Nurse
121 27
151
247 43
318 55
168 13
238 7 18
154
331
220 31
738
Methods
Study Design:
Observational studies in 7 hospital EDs, pre-post eCTAS implementation.
Pre-Implementation
Reliability Study (Fig. 1):
Study periods covered three 7-hour shifts per hospital (21 shifts total)
Triage nurse and expert auditor blinded to each other’s CTAS decisions, co-triaged the same patients and assigned CTAS scores
Expert auditor documented:
Presenting complaint captured by the triage nurse
Canadian Emergency Department Information Systems (CEDIS) presenting complaint recommended by the expert auditor
Vital signs captured at triage with patient age
CTAS level determined by the duty triage nurse
Modifiers observed by the expert auditor
CTAS level determined by the expert auditor
Interrater agreement was estimated using unweighted and quadratic-weighted kappa statistics with 95% confidence intervals (CIs).
Time Study (Fig. 2):
Study periods covered approx. ten 6-hour shifts per hospital (69 shifts total)
Triage time elapsed from first patient-nurse interaction to the time the triage assessment is complete
Triage timestamps were captured by research students
Post-Implementation
For both studies:
Observations in each hospital ED will be conducted at a minimum of three months after eCTAS implementation (May to October 2017)
Each site will serve as it's own control when pre-post implementation data will be compared
Conclusion
Variability in the accuracy and time taken to perform CTAS assessments suggest that a standardized approach to performing the CTAS process would improve both clinical decision making and administrative data accuracy.
Further Analyses:
The eventual comparison of pre-post eCTAS implementation data will provide evidence on how the eCTAS process could significantly improve CTAS reliability without increasing triage time.
Yellow – under triage, Green – Agreement, Red – over triage
Time Study
Figure 2: Triage Time Evaluation Setting
References
1. Office of the Auditor General of Ontario, Annual Report (3.05):
Table 2: Median Triage Time by Hospital
Hospital
#Visits
#Patients
Median (IQR) Triage Time (min) A
10 (14.5%)
440 (11.6%)
5.5 (4.0, 7.4) B
582 (15.3)
4.3 (3.5, 5.7) C
9 (13.0%)
472 (12.4%)
7.5 (5.8,10.8) D
300 (7.8%)
5.9 (4.5, 7.7) E
384 (10.1%)
4.4 (3.5, 5.7) F
804 (21.1%)
3.9 (3.1, 4.8) G
826 (21.7%)
6.9 (5.2, 9.2)
Total 69
3808
5.2 (3.8, 7.3)
Acknowledgement
This work is supported by the Ministry of Health and Long-Term Care.