Holistic Assessment of Social Work Students’ Professional Practice Marion Bogo Joint Social Work Education Conference and Social Work Research Conference University of Hertfordshire 2010

Overview Link between social work education and practice Conceptualization of professional practice Assessment Cognition Interpersonal Contextual

Social Work Education and Practice The purpose of social work education is to prepare social work practitioners

Professional Work: Dimensions Thinking Performing Acting with integrity (Lee Shulman, 2005)

Signature Pedagogy “…pedagogical norms with which to connect and integrate theory and practice.” (Lee Shulman, 2005)

Social Work Education and Practice United States: Council on Social Work Education, Educational Policy and Accreditation Standards 2008 England: Social Work Task Force 2009 Canada: Canadian Council of Social Work Regulators; Project - Social Worker Competency Profile 2010

Social Work Education and Practice Concept ASSESS Content APPROACH Method

Conceptualization of Professional Practice Critique of competency as dimensions and skills – a checklist approach How to capture holistic nature of practice and performance not mechanistic context professionalism knowledge competencies, internal cognitive processes for judgment, decision making

“In their own words…” Design a practice-based evaluation tool Reflect practice teachers’ tacit or implicit conceptualization of student competence – dimensions they use In practice teachers’ language

Procedural competencies Holistic Competence Meta-competencies Procedural competencies

Meta-Competencies Higher order overarching abilities and qualities - of a different order and nature than procedural or operational behaviors and skills. Related to individuals’ ability to: use discrete behaviors in a purposeful, integrated, and professional manner. learn the specific role competencies of particular professions.

Procedural competencies Behavior in the Organization Conceptualization of practice Collaborative relationship Meta-competencies Relational capacity/Intentional use of self Procedural competencies Conducts assessment and intervention Professional Communication Learning and growth as a professional DIRECT PRACTICE

Procedural competencies Behavior in the Organization/ Relationships Critical Thinking and Analysis Planning Meta-competencies Leadership Procedural competencies Project management Professional Communication Learning and growth as a professional MACRO PRACTICE

Social Work Education and Practice Concept ASSESS Content APPROACH Method

Challenges to Assessment Cognition Interpersonal Contextual

Levels of Performance High agreement on individual rankings of 20 vignettes- interclass correlation coefficient: 0.83 Five categories generated Exceptional - - - - - - - - - - - - - - - - - 5 Ready for practice - - - - - - - - - - - - 4 On the cusp - - - - - - - - - - - - - - - - - 3 Needs more training - - - - - - - - - - -2 Unsuitable for the profession - - - - 1

From Descriptions to a Tool Dimensions of holistic competence Learning and growth Behavior in the organization Conceptualizing practice Collaborative relationships Assessment and intervention Professional Communication

From Descriptions to a Tool For each dimension Extracted descriptors from the ranked vignettes for each of 5 levels of performance In practice teachers’ language.

Behavior in the Organization EXEMPLARY READY TO PRACTICE ON THE CUSP MORE TRAINING UNSUITABLE Work in a responsible and experienced manner within the agency. Show respect to other staff and work well within the agency’s culture. Understand the student role and maintain professional boundaries with other team members. Respected and valued by agency staff. Other staff regard her/him as a valued colleague. Highly flexible and adaptable. Relate to agency staff with humility and respect. Viewed as co-workers by other members of the team. Work well with colleagues, but can be intimidated by members of other professions. Follow agency procedures and are accepted by the team. Tend to work well in the student role. Regarded as a student/learner by other members of the team. Experience difficulty in building relationships with organizational staff. Struggles with the student role, as she/he wants more clients and responsibility, but does not possess or demonstrate the needed skills and abilities to take on this extra responsibility. Viewed primarily as a student. Lack appreciation of the student role and self-awareness about interactions with organizational staff. No respect for the agency and its procedures. Experience difficulty in fitting into the agency and engage in problematic behaviours, such as being highly critical of and argumentative with other staff and the field instructor, and/or becoming involved in agency politics. Demonstrate an overall inflexibility in the workplace.

Competency-Based Evaluation Intervention Planning and Implementation Identify and confront a reluctance to recognize viable options. 1 2 3 4 5 N Appropriately challenge the client system when required. Give feedback on above in a manner, which conveys respect and understanding.

Results New PBE tool was not more effective at discriminating students than older CBE tool. CBE Tool PBE Tool Mean 4.08 4.20 Std Dev 0.53 0.63 Pct <3.00 2.4% 4.8% 23

Using all Tools Recall most recent student 20 vignettes ordered randomly Select vignettes “similar” to their student. Select from “similar vignettes” one or two vignettes that are “most similar” to their student. Evaluate same student using Practice-Based Evaluation (PBE) Tool Competency-Based Evaluation (CBE) Tool

Distribution of Student Scores for the Three Measures

Distribution of Student Scores for the Three Measures

Distribution of Student Scores for the Three Measures

Scales Holistic impression of students Deconstruct Relate to a skill Convert to a point on a scale

Challenges to Assessment Cognition Interpersonal Contextual

Reflections on Assessment When values collide Student response and the inter-personal relationship Negotiating Where is ‘the school?’

Conclusion In instructors’ own language – holistic, global, authentic - reflects their conceptualization of practice competence. A tool without numbers. Approaches that empower the practice assessor/field instructor and are not negotiated – exercise her expertise.

An Online Tool

Dimensions of Practice Six dimensions Learning and growth Behavior in the organization Clinical relationships Conceptualizing practice Assessment and intervention Professional Communication For each dimension, descriptors provided for each of 5 levels of performance.

Levels of Performance Five categories Exceptional - - - - - - - - - - - - - - - - - 5 Ready for practice - - - - - - - - - - - - 4 On the cusp - - - - - - - - - - - - - - - - - 3 Needs more training - - - - - - - - - - -2 Unsuitable for the profession - - - - 1

Conceptualizing Practice EXEMPLARY READY TO PRACTICE ON THE CUSP MORE TRAINING UNSUITABLE Think about and understand practice on multiple levels from instrumental tasks to deeper, therapeutic interventions. Possess strong critical thinking skills. Competent in using numerous theories in practice, and apply them in a flexible manner. Understand and carry out the mandate of the organization, while remaining sensitive to clients’ situations and needs. Maintain an ethical practice, but are not rigid or inflexible. View practice on multiple levels, but in some cases this develops only with field instruction. Knowledgeable about theory and are capable of applying it to practice. Some students, however tend to know one approach very well and work primarily from that framework. Understand and maintain professional, ethical practice. Some ability to think critically. View practice as simply the process of collecting factual or affective information and demonstrate only a limited ability to link affect, behaviour, and cognition. Experience difficulty integrating theory, but some growth is observed. Understand and maintain an ethical practice. Give attention primarily to the concrete aspects of clients’ situations. Tend to think in the here and now and experience difficulties in making deeper connections. Able to identify and discuss ethical matters. Work well within established frameworks, but demonstrate inflexibility in using different practice approaches. Struggle with understanding the difference between personal relationships and worker-client relationships. View practice at a concrete level only and experience extreme difficulty in going beyond this level. Have difficulty applying theoretical understanding to practice situations. Do not understand ethical practice and client-worker boundaries. Some students have the potential to be harmful to clients. Demonstrate rigidity in her/his thinking and belief systems. Struggle in understanding social work values. Personal issues and difficulties are not kept out of her/his work and practice.

Conceptualizing Practice Descriptor Rank Able to begin to identify and discuss ethical matters. 2 Competent in using numerous theories in practice, and applying them in a flexible manner. 5 Culturally competent and actively incorporates factors relevant to diverse clients. Culturally competent, understands and begins to incorporate factors relevant to diverse clients. 4 Demonstrates rigidity in his/her thinking and belief systems 1 Demonstrates some ability to think critically. Displays some understanding of the impact of cultural factors. 3 Gives attention primarily to the concrete aspects of clients' situations. Has difficulty applying theoretical understanding to practice situations

2008-9 Comparisons between CBE Tool (480) and PBE Tool (152)

Challenges to Assessment Cognition Interpersonal Contextual

OSCE: Objective Structured Clinical Examination Developed in medicine and used in health professions (Harden et al., 1975) Students interview series of standardized, simulated “clients” “Clients” are trained actors Students are given brief information – role, presenting issues, time frame and goal.

Adapted for Social Work Holistic competence - in social work two components: interview for 12-15 minutes for procedural competencies followed by a structured reflective dialogue for 12-15 minutes to elicit meta-competencies Both observed and rated by a trained rater.

Iterative Process to Develop OSCE Competence Conceptualize Design Scenarios Define specific behaviors and rating scale Define components and scale for reflection Train and Pilot

The Examination

Reading Instructions

One of Five “Clients”

Hospital Social Work

Child Protection

Rater and Widowed Mother

Initial Study: Participants 11 students completed Year 1 of the MSW Program 7 recent graduates of the MSW Program 5 experienced social workers with 5-7 years of experience

Results Developed reliable tools – dimensions in each tool were assessing a common ability The two aspects – performance and reflection were related but conceptualized differently by raters.

Results Moderate reliability across stations : Cronbach’s alpha of 0.55 for Practice and 0.48 for Reflection Some evidence for generalizability of individual scores across stations Need more stations, better stations, better raters?

Results 30% of variance on performance 23% of variance on reflection Experienced practitioners had significantly higher scores 23% of variance on reflection Marginally significant effect of experience

Assessing Performance and Reflection Range of student ability to engage in meaningful reflective dialogues about: Conceptualizing practice and using frameworks about diversity Intentional use of self and own reactions Integrating learning into practice. Performing well and in depth reflection not necessarily at same level in individual students.

Is OSCE in our Future? Cautious optimism about the merits of using OSCE adapted for social work Appears useful for evaluation of both performance and critical thinking, conceptualization, reflection, self-assessment Scores not at high range as in field evaluations Participants and students very enthusiastic.

Concluding Thoughts Need approaches that assess holistic competence Need to assess at different points in time Need multiple assessment methods- authenticity and standardized In university program and in practice setting

Concluding Thoughts Need to study assessment methods and links to pedagogy Social work education researchers Institutional arrangements with practice settings

Acknowledgements These studies were funded by the Social Sciences and Humanities Council of Canada. Co-investigators: Cheryl Regehr, Professor, Factor-Inwentash Faculty of Social Work and Vice Provost, University of Toronto. Glenn Regehr, Professor and Associate Director, Research, Centre for Health Education Scholarship, Faculty of Medicine, University of British Columbia. Researchers: Maria Mylopoulos, Educational Researcher, SickKids Learning Institute, Hospital for Sick Children, Assistant Professor, Department of Paediatrics, Faculty of Medicine, University of Toronto Carmen Logie and Ellen Katz, doctoral candidate, and Rachael Walliser, research assistant, Factor-Inwentash Faculty of Social Work, University of Toronto.

Selected References About Field Education Bogo, M. (in press) Achieving competence in social work through field education. Toronto, ON: University of Toronto Press. Bogo, M. (2005). Field instruction in social work: A review of the research literature 1999-2005. The Clinical Supervisor, 24(1/2), 163-193. Bogo, M., & Vayda, E. (1998). The practice of field instruction in social work: Theory and process (2nd ed.). New York, NY: Columbia University Press. Fortune, A. E., & Kaye, L. (2002). Learning opportunities in field practica: Identifying skills and activities associated with MSW students' self-evaluation of performance and satisfaction. The Clinical Supervisor, 21, 5-28. Fortune, A. E., McCarthy, M., & Abramson, J. S. (2001). Student learning processes in field education: Relationship of learning activities to quality of field instruction, satisfaction, and performance among MSW students. Journal of Social Work Education, 37(1), 111-124. Homonoff, E. (2008). The heart of social work: Best practitioners rise to challenges in field instruction. The Clinical Supervisor, 27(2), 135-169. Shulman, L. S. (2005a). Signature pedagogies in the profession. Daedalus, 134(3), 52-59.

Selected References About Competence and Holistic Competence Bogo, M., Regehr, C., Woodford, M., Hughes, J., Power, R., & Regehr, G. (2006). Beyond competencies: Field instructors' descriptions of student performance. Journal of Social Work Education, 42(3), 191-205. Cheetham, G., & Chivers, G. (2005). Professions, competence and informal learning. Cheltenham, UK: Edward Elgar. Eraut, M. (1994). Developing professional knowledge and competence. London, UK: Falmer Press. Kelly, J., & Horder, W. (2001). The how and why: Competences and holistic practice. Social Work Education, 20(6), 689-699. Schon, D. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass. Skinner, K., & Whyte, B. (2004). Going beyond training: Theory and practice in managing learning. Social Work Education, 23(4), 365-381.

Selected References About Evaluating Competence Bogo, M., Regehr, C., Hughes, J., Power, R., & Globerman, J. (2002). Evaluating a measure of student field performance in direct service: Testing reliability and validity of explicit criteria. Journal of Social Work Education, 38(3), 385-401. Bogo, M., Regehr, C., Power, R., Hughes, J., Woodford, M., & Regehr, G. (2004). Toward new approaches for evaluating student field performance: Tapping the implicit criteria used by experienced field instructors. Journal of Social Work Education, 40(3), 417-426. Bogo, M., Regehr, C., Power, R., & Regehr, G. (2007). When values collide: Providing feedback and evaluating competence in social work. The Clinical Supervisor, 26(1/2), 99-117. Regehr, G., Bogo, M., Regehr, C., & Power, R. (2007). Can we build a better mousetrap? Improving the measures of practice performance in the field practicum. Journal of Social Work Education, 43(2), 327-343. About OSCE Harden, R. M., Stevenson, M., Downie, W. W., & Wilson, G. M. (1975). Assessment of clinical competence using objective structured examination. British Medical Journal, 1(5955), 447-451. Rating Scales Bogo, M., Katz, E., Logie, C., Regehr, C.,& Regehr, G. (2008). OSCE for social work: Practice performance rating scale. Unpublished scale. Factor-Inwentash Faculty of Social Work, Toronto, ON. Bogo, M., Mylopoulos, M., Katz, E., Logie, C., Regehr, C., & Regehr, G. (2009). Reflective dialogue rating scale. Unpublished scale. Factor-Inwentash Faculty of Social Work, Toronto, ON.

Website Factor-Inwentash Faculty of Social Work Research Institute for Evidence-Based Practice Competency for Professional Practice http://www.socialwork.utoronto.ca/research/initiatives/competency.htm Knowledge building – studies, articles, presentations Knowledge dissemination – fact sheets, assessment and evaluation tools.