1. London, England 7-8 July 2011 International Congress on Professional and Occupational Regulation Applying what evidence there is to a new continuing competence programme: An individually tailored, high-trust approach. Anne Goodhead and Steve Osborne, New Zealand Psychologists Board Promoting Regulatory Excellence

2. Presentation Outline 1.Introduction to our Continuing Competence Programme (CCP) 2.Rationale 3.Development process 4.Principles 5.Evidence for our approach 6.Where to from here

3. The CCP Introduced April 2009. Compulsory for all active practitioners to participate. Just completed our second year, including audits of participation.

4. The CCP (six step) model 1 Against Core Competencies 2 With Supervisor's involvement Self Reflective Review 1,2 Document each step Strengths and Weaknesses Learning Objectives Learning Activities Self-directed Learning Plan Year-end Review of Benefits 2

5. Rationale The HPCA Act requires that the Board be satisfied that a practitioner is competent before issuing a practising certificate. Complaints data suggest older and more experienced practitioners are more likely to attract complaints, consistent with research in other disciplines. Our aim is to lift the competence of all.

6. How can we lift performance? Best practice guidelines. Newsletters to raise awareness of pertinent issues. Well-established culture of supervision at all levels of practice. And most recently, mandatory CCP.

7. Development of the CCP Consultation - widespread support. Pilot study helped refine the model. Informed by literature on CPD. Newsletter updates on instructions, also conference and stakeholder group presentations. Audit has informed further refinement, as has our new learning outcomes survey.

8. CCP principles Lift competence standards overall. Individually tailored: Practitioners benchmark themselves against competence standards, then strive to improve on own areas of need. Assure the public of high standards. Flexible: To encompass the diversity of practice.

9. CCP principles Valid and relevant for practitioners. Support self-regulation, and the natural incentives for high quality practice. Increase self-efficacy.

10. Why try to lift overall standard? Need to counter the trend of knowledge and performance declining with years of experience. Choudhry 2005 review Traditional input models to CME and CE known to have only a small impact. Marinopoulos et al 2007, Mansouri and Lockyer 2007, Forsetlund et al (Cochrane review) 2009, Davis and Galbraith 2009 Research on output models of CPD “sparse” but seen as way forward. PARN 2008

11. Will the CCP lift the overall standard? It is informed by what works: Improved by multiple media, mixed interactive. Mansouri and Lockyer 2007 Multiple events on single topic. Davis and Galbraith 2009, Mazmanian et al 2009 Practitioner assessed learning needs. Mazmanian and Davis 2002 Integrated with practice.

12. Will the CCP lift the overall standard? What works (continued): Self-awareness, self-reflection, and self-evaluation. Kaslow et al 2007 Seek feedback. Eva and Regehr 2008 Guided self-assessment. Duffy and Holmboe 2006 Evaluation of outcomes - participant satisfaction, knowledge, behaviour, patient safety. Tian et al 2007

13. And what of individual improvement? CCP requires a Self Reflective Review against prescribed Core Competencies. Eva and Regehr caution against self- evaluation. (2005, 2006) We require supervisor or colleague involvement - but is this enough? Face validity: Set goals, plans, and evaluate at end of the CCP year.

14. Learning outcomes, CCP vs. opportunistic

15. Does CCP assure safety of the public? Meeting legislative obligations. Consumer and policy “watchdogs” on public safety groups promote a similar approach to professional development. Citizens Advocacy Centre (2006), Institute of Medicine (2010) CCP includes the quality assurance step of evaluating outcomes and seeking feedback re impact on practice.

16. IS the CCP valid and relevant? Intention is that each psychologist shapes their programme to be meaningful, useful, relevant, and valid for them. Avoids tokenism. Does not block opportunistic professional development or that driven by unanticipated professional challenge.

17. Does the CCP facilitate self-regulation? Our pilot study suggests most psychologists do a considerable amount of PD without being directed to. The majority hold high ethical standards and strive for excellence. Natural incentives reward high standards. But, a US survey showed ~ 20% did little if not mandated. Neimeyer,2009,2010a,2010b

18. Does CCP support self-efficacy? Our pilot study indicated amount of PD was correlated with confidence in being up-to-date with psychology practice. Undertaking the SRR also appeared to be associated with more confidence and higher professional self esteem.

19. Do the benefits outweigh the costs? In an ideal world: - cost-benefit analysis. - demonstrate cost-effective. - compared to status quo. - compared to other CE models. But hard to measure this in PD generally - Few studies, evidence weak, hard to research. Brown et al 2002

20. Costs Personal/Employer costs: Bureaucratic costs of record keeping. Opportunity costs in supervision. For some, increased PD. Organisational costs (e.g., employee’s time). Stress.

21. Costs For the Board : Labour costs of development and instruction. The financial, time, and other resource costs of the annual audit. Potential loss of goodwill.

22. Benefits Potential impact on client outcomes. Mixed reports whether it changes what individuals would have done anyway. Long term impact unknown (but we’re hopeful).

23. Benefits Meeting statutory obligations. Feedback from audit participants and others…

24. Audit participant feedback How burdensome or time consuming have you found the recording of your CCP? 1 = highly burdensome 10 = low burden

25. Audit participant feedback Has the CCP helped you focus your professional development activities? 1 = No 10 = Yes

26. Audit participant feedback Has CCP participation helped you to ensure that you are competent? 1 = No 10 = Yes

27. Audit participant feedback Have you found the CCP useful? 1 = not useful 10 = very useful

28. Where to from here? Continue to fine tune, have a framework we can tweak: Now offering more structured way of recording to reduce confusion and time (optional template). Clarifying role of supervisor, goal setting, and the end of year review. Attitudes remain mixed, ranging from enthusiastic to resentful.

29. Where to from here? Further research? We would be interested in any related research that you know of.

30. Thank you! And thanks to Dr Lois Surgenor for her assistance with the statistical analysis!

31. Research and development work by… Anne Goodhead Psychology Advisor New Zealand Psychologists Board PO Box 10-626 Wellington 6143 New Zealand anne.goodhead@nzpb.org.nz www.psychologistsboard.org.nz

32. Speaker Contact Information Steve Osborne MSc ClinPsych MASPPB MIPGA Chief Executive and Registrar New Zealand Psychologists Board PO Box 10-626 Wellington 6143 New Zealand steve.osborne@nzpb.org.nz www.psychologistsboard.org.nz

33. Bibliography Brown, C., C. Belfield, et al. (2002). "Cost effectiveness of continuing professional development in health care: a critical review of the evidence." British Medical Journal 324: 652-655. CAC - refer to Swankin et all below. Choudhry, N., R. Fletcher, et al. (2005). "Systematic Review: The relationship between clinical experience and quality of health care." Annals of Internal Medicine 142(4): 260 - 273. Davis, D. and R. Galbraith (2009). "Continuing Medical Education Effect on Practice Performance." Chest 135(3): 42S-48S. Duffy, F. D. and E. Holmboe (2006). "Self-assessment in lifelong learning and improving performance in practice: Physician know thyself." JAMA 296(9): 1137-1139. Eva, K. and G. Regehr (2005). "Self-assessment in the Health Professions: A reformulation and research agenda." Academic Medicine 80(10): S46-S54. Eva, K. and G. Regehr (2008). ""I'll never play professional football" and other fallacies of self-assessment." Journal of Continuing Education in the Health Professions 28(1): 14-19. Forsetlund, L., A. Bjorndal, et al. (2009). "Continuing education meetings and workshops: effects on professional practice and health care outcomes (review)." The Cochrane Library(2). Institute of Medicine (2010). Redesigning continuing education in the health professions. The National Academies Press. Washington, DC, Downloaded from: http://www.nap.edu/catalog/12704.html

34. Professional Associations Research Network (PARN) (2008). Approaches to Continuing Professional Development (CPD) Measurement. International Accounting Education Standards Board. New York, International Federation of Accountants. Downloaded from: http://www.ifac.org Kaslow, N., N. Rubin, et al. (2007). "Recognising, assessing and intervening with problems of professional competence." Professional Psychology: Research and Practice 38(5): 479- 492. Marinopoulos, S., Dorma,T. et al (2007)”Effectiveness of Continuing Medical Education, Evidence report/technology assessment no 149”. Rockville, MD: Agency of Healthcare Research and Quality. Mazmanian, P. (2009). "Continuing Medical Education Costs and Benefits: Lessons for competing in a changing health care economy." J of Continuing Education in the Health Professions 29(3): 133-134. Mazmanian, P. and D. Davis (2002). "Continuing Medical Education and the physician as a learner." JAMA 288(9): 1057-1060. Monsouri, M. and J. Lockyer (2007). "A meta-analysis of continuing medical education effectiveness." Journal of Continuing Education in the Health Professions 27(1): 6-15.

35. Neimeyer, G. and J. Taylor (2009). "Continuing education in psychology: Outcomes, evaluations and mandates." Professional Psychology: Research and Practice 40(6): 617-624. Neimeyer, G., J. Taylor, et al. (2010). "Continuing education in psychology: Patterns of participation and aspects of selection." Professional Psychology: Research and Practice 41(4): 281-287. Neimeyer, G., J. Taylor, et al. (2010). "Continuing education in psychology: Patterns of participation and perceived outcomes among mandated and nonmandated psychologists." Professional Psychology: Research and Practice 41(5): 435-441. Regehr, G. and K. Eva (2006). "Self-assessment, Self-direction, and the Self-regulating Professional." Clinical Orthopaedics and Related Research Number 449: 34-38. Regehr, G. and M. Mylopoulos (2008). "Maintaining competence in the field: Learning about practice, through practice, in practice." J of Continuing Education in the Health Professions 28(S1): S20-S23. Swankin, D., R. Le Buhn, et al. (2006) Implementing continuing competency requirements for health care practitioners. Citizens Advocacy Center, AARP Public Policy Institute. Downloaded from: http://www.aarp.org/ppi Tian, J., N. Atkinson, et al. (2007). "A systematic review of evaluation in formal continuing medical education." J of Continuing Education in the Health Professions 27(1): 16- 27.