1. Gottlieb AB14, Christensen R2,15, Jemec GBE1
Interrater agreement and reliability of outcome measurement instruments and staging systems used in hidradenitis suppurativa
Thorlacius L1,2, Garg A3, Theut Riis P1, Bettoli V4, Ingram JR5, del Marmol V6, Matusiak L7, Pascual JC8, Revuz J9, Sartorius K10 , Tzellos T11,van der Zee HH12, Zouboulis CC13, Saunte DM1
Gottlieb AB14, Christensen R2,15, Jemec GBE1
Department of Dermatology, Zealand University Hospital , Roskilde, Denmark.
Musculoskeletal Statistics Unit, The Parker Institute, Bispebjerg and Frederiksberg Hospital, The Capital Region of Denmark. Copenhagen, Denmark.
Department of Dermatology, Hospital General Universitario de Alicante, Alicante, Spain
Department of Dermato-Venereology, Henri Mondor Hospital, France
Department of Dermatology, Hofstra North Shore-LIJ School of Medicine, Hempstead, NY,USA
Karolinska Institutet, Department of Clinical Science and Education, Södersjukhuset, Sweden
Dermatology and Venereology, University of Ferrara, Italy
University Hospital of North Norway, Thomsø, Norway
Institute of Infection & Immunity, University Hospital of Wales, Heath Park, Cardiff, U.K.
Eramus University Rotterdam, Rotterdam, the Netherlands
Departments of Dermatology, Venereology, Allergology and Immunology, Dessau Medical Center, Dessau, Germany
Université Libre de Bruxelles, Department of Dermatology and Venereology, Brussels, Belgium
Department of Dermatology, ICahn School of Medicine at Mt Sinai, NY, NY, USA
Department of Dermatology, Venereology and Allergology, Medical University, Wrocław, Poland.
Department of Rheumatology, Odense University Hospital, Denmark.
British Journal of Dermatology. DOI: /bjd.17716

2. Thorlacius L1,2
Jemec GBE1
Department of Dermatology, Zealand University Hospital , Roskilde, Denmark.
Musculoskeletal Statistics Unit, The Parker Institute, Bispebjerg and Frederiksberg Hospital, Denmark

3. Introduction What’s already known?
Without valid and reliable instruments to measure outcomes, researchers and clinicians lack the necessary benchmarks to assess primary and secondary endpoints of interventional trials properly
Hidradenitis suppurativa (HS) is a chronic, inflammatory skin disease
Several outcome measure instruments exist for HS, but their validation is generally incomplete or of relatively low methodological quality

4. Objective
To investigate interrater reliability and agreement of both the most frequently used outcome measurement instruments (OMIs) and staging systems in HS

5. Methods (1)
The study was designed in accordance with the ‘Guidelines for Reporting Reliability and Agreement Studies’ (GRRAS) (Kottner et al., 2011)
Raters were dermatologists from 12 different countries, with a clinical background with a least 10 years of experience with HS
24 patients with HS severity from mild to severe were recruited at random

6. Methods (2)

7. Methods (3)
All patients were examined and scored with nine different instruments by 12 different raters on the same day
Raters were also asked to count the number of typical HS lesions in each anatomical area
Raters were not allowed any communication with each other
The order in which each rater met each patient was based on a pre-specified patient-by-rater visit algorithm

8. Methods (4): Statistical analysis
For the quantification of the interrater reliability the intraclass correlation coefficient (ICC) type ICC (2,1)
For quantification of the interrater agreement for variables with >4 categories, a modified Bland-Altman method was applied
For variables with ≤4 categories, the observed agreement was calculated as the number of patients for which all the raters agreed

9. Results (1)

10. Results (2)

11. Discussion (1)
For the staging systems, we found good interrater reliability for Hurley staging in the axillae and gluteal region and moderate results in the groin
Accordingly, Hurley staging seems to be an acceptable instrument for classification in HS
Experts in clinimetrics emphasise that agreement parameters are preferred when evaluating instruments designed to measure changes in health status
Our observed limits of agreement intervals were very wide compared to the usual ranges, of all the included OMI scales

12. Discussion (2)
None of the included OMIs can be recommended under the given conditions in terms of interrater reliability
Our results illustrate how difficult it is, even for experts, to measure HS physical signs of HS by a simple rater counting
Consequently, other assessment methods of physicals signs, such as ultrasound evaluation, do need consideration.
More research is needed to examine intrarater agreement and to examine if interrater agreement can be improved e.g. by training

13. Conclusions What does this study add?
In a prospective completely balanced design the study examined interrater reliability with HS-experienced dermatologists as the rater population of interest
The study did not find very good reliability for any included instrument or lesion counts
The study illustrates how difficult it is even for experts to agree on the type and number of lesions
The results question whether physical signs are best measured by a traditional physician lesion count instrument

15. Call for correspondence
Why not join the debate on this article through our correspondence section?
Rapid responses should not exceed 350 words, four references and one figure
Further details can be found here