Professor Jeffrey Braithwaite Director, Institute of Health Innovation Director, Centre for Clinical Governance Research University of New South Wales, Australia EPSO Meeting Copenhagen, 23 October 2008 Accreditation research

Research partnership

The ARC Linkage Team Professor Jeffrey Braithwaite Dr David Greenfield Dr Marjorie Pawsey Professor Johanna Westbrook Professor Bill Runciman Professor Sally Redman Professor Robert Gibberd Conjoint A/Professor Mary Westbrook Dr Justine Naylor Ms Sally Nathan Ms Maureen Robinson Ms Judie Lancaster Mr Brian Johnston Dr Desmond Yen Ms Lena Low Ms Heather McDonald Ms Darlene Hennessey Mrs Margaret Jackson ACHS staff ACHS surveyors ACHS member organisations Consumer Reference Group

Systematic review 4

5

Research categorySummary of findings Professions attitudes to accreditationMixed Accreditation promotes changeYes it does Accreditation has an organisational impactUnclear impact Accreditation has a financial impactYes; but cost-benefits unknown Quality measures eg Cis and accreditationNot clearly related Program assessment: are accreditation programs credible and worthwhile? Mixed results Consumers’ views/patient satisfaction and accreditation No clear relationships Public disclosure of accreditationResults support doing this Does accreditation promote professional development? Generally yes Surveyor issuesNot much research

The research: four studies A prospective, multi-method, multi-disciplinary, multi-level, collaborative project for researching health sector accreditation

The research: four studies Research aim A: To explore the relationships between accreditation, clinical performance, organisational culture, and consumer participation

The research: four studies Research aim B: To examine the influence of surveyors on both the accreditation process and outcome and their own health organisations

The research: four studies Objectives Aim B: Examine the influence of surveyors by assessing: 1) the reliability of the accreditation process; and 2) the organisational influence of accreditation surveyors. Aim A: Examine the relationships between accreditation status and processes, and individual and organisational performances, notably clinical performance and organisational culture. Study 3: Prospective study of intra- and inter-rater reliability of EQuIP surveyors and survey teams. Study 4: Prospective and retrospective study of organisational influence of accreditation surveyors. Study 1 Prospective study of the relationships between accreditation and clinical and organisational performances, and consumer participation. Study 2 Prospective study of health services participating and not participating in accreditation

Study 1 To determine whether there are relationships between EQuIP performance and organisational culture organisational climate consumer participation leadership clinical performance

Study 1 12 This is a three year ARC grant which conducted four major studies of accreditation One key study, study 1, examined 19 randomly sampled health care organisations looking at accreditation performance, organisational climate, organisational culture, consumer involvement, leadership, and clinical indicator performance

Study 1 13 We took six variables And measured them in the 19 randomly sampled health care organisations, each of which had participated in accreditation in Australia through ACHS EQuIP Each of the variables was measured and the data interpreted by a research team blinded from the other research teams

Study 1 Small: 7 Medium: 6 Large: 6 Public: 13 Private: 6 Metropolitan: 8 Regional: 3 Rural: 7 Remote: 1 Each Australian state and a territory represented 14 Characteristics of the sampled organisations

Study 1 15 Data and procedures Participant organisations were ranked 1 …19 on the basis of performance: Accreditation: statistical ranking of performance based on ACHS EQuIP surveyor reports Organisational culture: ~1,000 semi structured interviews with organisational members

Study 1 16 Data and procedures Organisational climate: ethnographic, non-participant observations and informal interviews, one week at each site Consumer participation: semi- structured interviews Leadership: semi-structured interviews

Study 1 17 Data and procedures Clinical indicators: proportion of clinical indicators for that organisation that were better than the national average In summary: independent measures of the six variables were taken, and on each variable the data were subject to a rank order correlation

Study 1 18 A note on rank order correlation Correlation coefficients can range from -1 to +1; they measure the degree of relationship between two variables A perfect linear relationship between two variables gives a correlation coefficient of 1

Study 1 Summary: Random sample n = 19 organisations ACHS: EQuIP outcome + survey research tool UNSW research team: cultural, climate, consumer participation, leadership assessment + clinical data analysis Research outcome sought: understanding organisational relationships

Study 1 A CultureClimateCons- umer Leader- ship CI A *** ***0.450 p (2- tailed) n19 16 

Study 1: Outcomes Braithwaite et al. Health service accreditation as a predictor of clinical and organisational performance: a blinded, random, stratified study. Submitted to International Journal for Quality in Health Care, 13/10/08. Braithwaite, J., Greenfield, D., Johnston, B., Scrivens, E. and Shaw, C. (2008) Recent results analysed from large accreditation studies with new research results on accreditation, standards and surveying. Abstract in ISQua Twenty-fifth International Safety and Quality Conference: Healthcare quality and safety: meeting the next challenges, Copenhagen, Denmark: International Society for Quality in Health Care, October 19-22

Study 1: Outcomes Nathan, S., McLure, K., Greenfield, D., Pawsey, M. and Braithwaite, J. (2008) “Taking our first steps together: the experiences of a research team and a consumer advisory panel in a national study of health service accreditation.” Involving People in Research Symposium, Perth: 5-6 March. Braithwaite, et al. (2007) Accreditation: assessing the evidence. Transforming patient safety, accreditation and risk in health care in international context. Abstract in the ISQua Twenty-fourth International Safety and Quality Conference: Transforming health care in the electronic age, Boston, United States of America, International Society for Quality in Health Care, October 2.

Study 2 A comparison between health services participating and not participating in an accreditation program

Study 2 Details: Purposive sample N = 3 organisations UNSW research team: cultural assessment + clinical data Research outcome sought: understanding accreditation effects

Study 2 Measure\ Acc: Non-Acc ZHZHCYCYFQFQ Accreditation performance Organisational culture Organisational climate Leadership Consumer

Study 2: Outcomes Of the twelve indicators, in 9/12, or 75%, the health services participating in accreditation outranked those health services that did not participate in accreditation

Study 2 Other research findings: One study failed to find any differences between accredited and non-accredited (rehabilitation) programs (Mazmanian et al. 1993) Another study found improved outcomes when a (trauma) service accredited (Simons et al. 2002)

Study 2 Outcome: Paper being drafted

Study 3 To assess the intra- and inter- rater reliability of ACHS-EQuIP surveyors and survey teams

Study 3 Details Five parts: Scenario exercise: intra-reliability Scenario exercise: inter-reliability (team) Focus groups: ACHS staff + surveyors Focus group: member organisations, An examination of two teams in practice Research outcome sought: reliability of survey teams

Study 3: Outcomes What promotes reliability of surveyors and survey teams? –A defined accreditation program where members participate in developing standards –Trained, experienced surveyors –Trained survey team leaders –Shared expectations of accreditation, standards, surveying processes –A system of self-governance with checks and balances along the way –= reliability

Study 3: Outcomes Greenfield, D., Pawsey, M., Naylor, J. and Braithwaite, J. (2008) Are healthcare accreditation surveys reliable?, International Journal of Health Care Quality Assurance, in press (Accepted 14/01/08). Greenfield, D., Braithwaite, J. and Pawsey, M.P. (2008) Health care accreditation surveyor style typology, International Journal of Health Care Quality Assurance, 21 (5), Greenfield, D., Pawsey, M., Naylor, J. and Braithwaite, J. (2008) Improving the reliability of an accreditation program: using research to educate and to align practice. Poster in ISQua Twenty-fifth International Safety and Quality Conference: Healthcare quality and safety: meeting the next challenges, Copenhagen, Denmark: International Society for Quality in Health Care, October 19-22

Study 4 To examine the influence of ACHS-EQuIP surveyors on their own health services

Study 4 Details: Two PhD studies in progress: Retrospective study of accreditation outcomes with surveyors Prospective in-depth case studies of health services with surveyors Research outcome sought: understanding surveyor influence

Study 4: Outcomes Details: –Becoming a surveyor has benefits for both the surveyor and the surveyor’s host organisation –Surveyors get exposed to other organisations’ processes, how they do things, what they do well and not so well –This allows them to take back that learning to their home organisation –They do this formally and informally

Study 4: Outcomes PhD expected to be completed 2010 and Lancaster, J., Braithwaite, J. and Greenfield, D. Benefits of participating in accreditation surveying. Submitted to International Journal of Health Care Quality Assurance, 9/9/08.

Other research outcomes Selected papers: Greenfield, D., Braithwaite, J., Pawsey, M.P., Johnston, B. and Robinson, M. (2009) Distributed leadership to mobilise capacity for accreditation research, Journal of Health Organisation and Management, in press (Accepted 20/6/08). Greenfield, D. and Braithwaite, J. (2008) Health care accreditation research: a systematic review, International Journal of Quality in Health Care, 20:

Other research outcomes Selected papers: Greenfield, D. and Braithwaite, J. (2007) Researching accreditation, E-Hospital, Journal of the Association of European Hospital Managers, 9 (5): Braithwaite, J., Westbrook, J.I., Pawsey, M., Greenfield, D., Naylor, J., Iedema, R.A., Runciman, B., Redman, S., Jorm, C., Robinson, M., Nathan, S. and Gibberd, R. (2006) A prospective, multi-method, multi-disciplinary, multi-level, collaborative, social-organisational design for researching health sector accreditation [LP ], BMC Health Services Research, 6,

Other research outcomes Selected presentations: Braithwaite, J., Greenfield, D., Johnston, B., Scrivens, E. and Shaw, C. (2008) Recent results analysed from large accreditation studies with new research results on accreditation, standards and surveying. Abstract in ISQua Twenty-fifth International Safety and Quality Conference: Healthcare quality and safety: meeting the next challenges, Copenhagen, Denmark: International Society for Quality in Health Care, October Braithwaite, J., Greenfield, D. and Westbrook, M. (2008) Contrasting and converging perspectives on organisational culture and climate. In Conference Proceedings of Culture and Climate: Cracking the Code. The Sixth International Conference on Organisational Behaviour in Health Care [obhc2008], Sydney: Society for the Study of Organising in Health Care, March

Other research outcomes Selected presentations: Greenfield, D., Travaglia, J., Pawsey, M., Lloyd, J. and Braithwaite, J. (2008) Who is accountable for quality and safety? Poster in Bold aims, bold outcomes: The Sixth Australasian Conference on Safety and Quality in Health Care, Christchurch, New Zealand: Australian Association for Quality in Health Care, September 1-3. Greenfield, D., Braithwaite, J. and Pawsey, M. (2007) Mobilising academic, industry and government stakeholders in collaborative research partnerships for improved patient safety through accreditation research. Abstract in the Patient Safety Research Conference: Shaping the European Agenda, Porto, Portugal: European Commission Sixth Framework Programme for Research and the Portuguese Ministry of Health, September

Summary and conclusion –12 papers published or in press – 7 papers in production –10 presentations to international conferences – 4 further presentations planned – Australian Accreditation Research Network established Research project, comprising four studies, successfully completed International interest stimulated

Questions?