4 Overview Link between social work education and practice Conceptualization of professional practiceAssessmentCognitionInterpersonalContextual
5 Social Work Education and Practice The purpose of social work education is to prepare social work practitioners
6 Professional Work: Dimensions ThinkingPerformingActing with integrity(Lee Shulman, 2005)
7 Signature Pedagogy“…pedagogical norms with which to connect and integrate theory and practice.”(Lee Shulman, 2005)
8 Social Work Education and Practice United States: Council on Social Work Education, Educational Policy and Accreditation Standards 2008England: Social Work Task Force 2009Canada: Canadian Council of Social Work Regulators; Project - Social Worker Competency Profile 2010
9 Social Work Education and Practice ConceptASSESSContentAPPROACHMethod
10 Conceptualization of Professional Practice Critique of competency as dimensions and skills – a checklist approachHow to capture holistic nature of practice and performancenot mechanisticcontextprofessionalismknowledge competencies, internal cognitive processes for judgment, decision making
11 “In their own words…” Design a practice-based evaluation tool Reflect practice teachers’ tacit or implicit conceptualization of student competence – dimensions they useIn practice teachers’ language
13 Meta-CompetenciesHigher order overarching abilities and qualities - of a different order and nature than procedural or operational behaviors and skills.Related to individuals’ ability to:use discrete behaviors in a purposeful, integrated, and professional manner.learn the specific role competencies of particular professions.
14 Procedural competencies Behavior in the OrganizationConceptualization of practiceCollaborative relationshipMeta-competenciesRelational capacity/Intentionaluse of selfProcedural competenciesConducts assessment and interventionProfessional CommunicationLearning and growth as a professionalDIRECT PRACTICE
15 Procedural competencies Behavior in the Organization/RelationshipsCritical Thinkingand AnalysisPlanningMeta-competenciesLeadershipProcedural competenciesProject managementProfessional CommunicationLearning and growth as a professionalMACRO PRACTICE
16 Social Work Education and Practice ConceptASSESSContentAPPROACHMethod
17 Challenges to Assessment CognitionInterpersonalContextual
18 Levels of PerformanceHigh agreement on individual rankings of 20 vignettes- interclass correlation coefficient: 0.83Five categories generatedExceptionalReady for practiceOn the cuspNeeds more trainingUnsuitable for the profession
19 From Descriptions to a Tool Dimensions of holistic competenceLearning and growthBehavior in the organizationConceptualizing practiceCollaborative relationshipsAssessment and interventionProfessional Communication
20 From Descriptions to a Tool For each dimensionExtracted descriptors from the ranked vignettes for each of 5 levels of performanceIn practice teachers’ language.
21 Behavior in the Organization EXEMPLARYREADY TO PRACTICEON THE CUSPMORE TRAININGUNSUITABLEWork in a responsible and experienced manner within the agency.Show respect to other staff and work well within the agency’s culture.Understand the student role and maintain professional boundaries with other team members.Respected and valued by agency staff. Other staff regard her/him as a valued colleague.Highly flexible and adaptable.Relate to agency staff with humility and respect.Viewed as co-workers by other members of the team.Work well with colleagues, but can be intimidated by members of other professions.Follow agency procedures and are accepted by the team.Tend to work well in the student role.Regarded as a student/learner by other members of the team.Experience difficulty in building relationships with organizational staff.Struggles with the student role, as she/he wants more clients and responsibility, but does not possess or demonstrate the needed skills and abilities to take on this extra responsibility.Viewed primarily as a student.Lack appreciation of the student role and self-awareness about interactions with organizational staff.No respect for the agency and its procedures.Experience difficulty in fitting into the agency and engage in problematic behaviours, such as being highly critical of and argumentative with other staff and the field instructor, and/or becoming involved in agency politics.Demonstrate an overall inflexibility in the workplace.
22 Competency-Based Evaluation Intervention Planning and ImplementationIdentify and confront a reluctance to recognize viable options.NAppropriately challenge the client system when required.Give feedback on above in a manner, which conveys respect and understanding.
23 ResultsNew PBE tool was not more effective at discriminating students than older CBE tool.CBE ToolPBE ToolMean4.084.20Std Dev0.530.63Pct <3.002.4%4.8%23
24 Using all Tools Recall most recent student 20 vignettes ordered randomlySelect vignettes “similar” to their student.Select from “similar vignettes” one or two vignettes that are “most similar” to their student.Evaluate same student usingPractice-Based Evaluation (PBE) ToolCompetency-Based Evaluation (CBE) Tool
25 Distribution of Student Scores for the Three Measures
26 Distribution of Student Scores for the Three Measures
27 Distribution of Student Scores for the Three Measures
28 Scales Holistic impression of students Deconstruct Relate to a skill Convert to a point on a scale
29 Challenges to Assessment CognitionInterpersonalContextual
30 Reflections on Assessment When values collideStudent response and the inter-personal relationshipNegotiatingWhere is ‘the school?’
31 ConclusionIn instructors’ own language – holistic, global, authentic - reflects their conceptualization of practice competence.A tool without numbers.Approaches that empower the practice assessor/field instructor and are not negotiated – exercise her expertise.
33 Dimensions of Practice Six dimensionsLearning and growthBehavior in the organizationClinical relationshipsConceptualizing practiceAssessment and interventionProfessional CommunicationFor each dimension, descriptors provided for each of 5 levels of performance.
34 Levels of Performance Five categories ExceptionalReady for practiceOn the cuspNeeds more trainingUnsuitable for the profession
35 Conceptualizing Practice EXEMPLARYREADY TO PRACTICEON THE CUSPMORE TRAININGUNSUITABLEThink about and understand practice on multiple levels from instrumental tasks to deeper, therapeutic interventions.Possess strong critical thinking skills.Competent in using numerous theories in practice, and apply them in a flexible manner.Understand and carry out the mandate of the organization, while remaining sensitive to clients’ situations and needs.Maintain an ethical practice, but are not rigid or inflexible.View practice on multiple levels, but in some cases this develops only with field instruction.Knowledgeable about theory and are capable of applying it to practice. Some students, however tend to know one approach very well and work primarily from that framework.Understand and maintain professional, ethical practice.Some ability to think critically.View practice as simply the process of collecting factual or affective information and demonstrate only a limited ability to link affect, behaviour, and cognition.Experience difficulty integrating theory, but some growth is observed.Understand and maintain an ethical practice.Give attention primarily to the concrete aspects of clients’ situations.Tend to think in the here and now and experience difficulties in making deeper connections.Able to identify and discuss ethical matters.Work well within established frameworks, but demonstrate inflexibility in using different practice approaches.Struggle with understanding the difference between personal relationships and worker-client relationships.View practice at a concrete level only and experience extreme difficulty in going beyond this level.Have difficulty applying theoretical understanding to practice situations.Do not understand ethical practice and client-worker boundaries. Some students have the potential to be harmful to clients.Demonstrate rigidity in her/his thinking and belief systems.Struggle in understanding social work values.Personal issues and difficulties are not kept out of her/his work and practice.
36 Conceptualizing Practice DescriptorRankAble to begin to identify and discuss ethical matters.2Competent in using numerous theories in practice, and applying them in a flexible manner.5Culturally competent and actively incorporates factors relevant to diverse clients.Culturally competent, understands and begins to incorporate factors relevant to diverse clients.4Demonstrates rigidity in his/her thinking and belief systems1Demonstrates some ability to think critically.Displays some understanding of the impact of cultural factors.3Gives attention primarily to the concrete aspects of clients' situations.Has difficulty applying theoretical understanding to practice situations
39 2008-9 Comparisons between CBE Tool (480) and PBE Tool (152)
40 Challenges to Assessment CognitionInterpersonalContextual
41 OSCE: Objective Structured Clinical Examination Developed in medicine and used in health professions (Harden et al., 1975)Students interview series of standardized, simulated “clients”“Clients” are trained actorsStudents are given brief information – role, presenting issues, time frame and goal.
42 Adapted for Social Work Holistic competence - in social work two components:interview for minutes for procedural competenciesfollowed by a structured reflective dialogue for minutes to elicit meta-competenciesBoth observed and rated by a trained rater.
43 Iterative Process to Develop OSCE CompetenceConceptualizeDesign ScenariosDefine specific behaviors and rating scaleDefine components and scale for reflectionTrain and Pilot
50 Initial Study: Participants 11 students completed Year 1 of the MSW Program7 recent graduates of the MSW Program5 experienced social workers with 5-7 years of experience
51 ResultsDeveloped reliable tools – dimensions in each tool were assessing a common abilityThe two aspects – performance and reflection were related but conceptualized differently by raters.
52 ResultsModerate reliability across stations : Cronbach’s alpha of 0.55 for Practice and 0.48 for ReflectionSome evidence for generalizability of individual scores across stationsNeed more stations, better stations, better raters?
53 Results 30% of variance on performance 23% of variance on reflection Experienced practitioners had significantly higher scores23% of variance on reflectionMarginally significant effect of experience
54 Assessing Performance and Reflection Range of student ability to engage in meaningful reflective dialogues about:Conceptualizing practice and using frameworks about diversityIntentional use of self and own reactionsIntegrating learning into practice.Performing well and in depth reflection not necessarily at same level in individual students.
55 Is OSCE in our Future?Cautious optimism about the merits of using OSCE adapted for social workAppears useful for evaluation of both performance and critical thinking, conceptualization, reflection, self-assessmentScores not at high range as in field evaluationsParticipants and students very enthusiastic.
56 Concluding Thoughts Need approaches that assess holistic competence Need to assess at different points in timeNeed multiple assessment methods- authenticity and standardizedIn university program and in practice setting
57 Concluding ThoughtsNeed to study assessment methods and links to pedagogySocial work education researchersInstitutional arrangements with practice settings
58 AcknowledgementsThese studies were funded by the Social Sciences and Humanities Council of Canada.Co-investigators: Cheryl Regehr, Professor, Factor-Inwentash Faculty of Social Work and Vice Provost, University of Toronto.Glenn Regehr, Professor and Associate Director, Research, Centre for Health Education Scholarship, Faculty of Medicine, University of British Columbia.Researchers: Maria Mylopoulos, Educational Researcher, SickKids Learning Institute, Hospital for Sick Children, Assistant Professor, Department of Paediatrics, Faculty of Medicine, University of TorontoCarmen Logie and Ellen Katz, doctoral candidate, and Rachael Walliser, research assistant, Factor-Inwentash Faculty of Social Work, University of Toronto.
59 Selected References About Field Education Bogo, M. (in press) Achieving competence in social work through field education. Toronto, ON: University of Toronto Press.Bogo, M. (2005). Field instruction in social work: A review of the research literature The Clinical Supervisor, 24(1/2),Bogo, M., & Vayda, E. (1998). The practice of field instruction in social work: Theory and process (2nd ed.). New York, NY: Columbia University Press.Fortune, A. E., & Kaye, L. (2002). Learning opportunities in field practica: Identifying skills and activities associated with MSW students' self-evaluation of performance and satisfaction. The Clinical Supervisor, 21, 5-28.Fortune, A. E., McCarthy, M., & Abramson, J. S. (2001). Student learning processes in field education: Relationship of learning activities to quality of field instruction, satisfaction, and performance among MSW students. Journal of Social Work Education, 37(1),Homonoff, E. (2008). The heart of social work: Best practitioners rise to challenges in field instruction. The Clinical Supervisor, 27(2),Shulman, L. S. (2005a). Signature pedagogies in the profession. Daedalus, 134(3),
60 Selected ReferencesAbout Competence and Holistic Competence Bogo, M., Regehr, C., Woodford, M., Hughes, J., Power, R., & Regehr, G. (2006). Beyond competencies: Field instructors' descriptions of student performance. Journal of Social Work Education, 42(3), Cheetham, G., & Chivers, G. (2005). Professions, competence and informal learning. Cheltenham, UK: Edward Elgar. Eraut, M. (1994). Developing professional knowledge and competence. London, UK: Falmer Press. Kelly, J., & Horder, W. (2001). The how and why: Competences and holistic practice. Social Work Education, 20(6), Schon, D. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass. Skinner, K., & Whyte, B. (2004). Going beyond training: Theory and practice in managing learning. Social Work Education, 23(4),
61 Selected References About Evaluating Competence Bogo, M., Regehr, C., Hughes, J., Power, R., & Globerman, J. (2002). Evaluating a measure of student field performance in direct service: Testing reliability and validity of explicit criteria. Journal of Social Work Education, 38(3),Bogo, M., Regehr, C., Power, R., Hughes, J., Woodford, M., & Regehr, G. (2004). Toward new approaches for evaluating student field performance: Tapping the implicit criteria used by experienced field instructors. Journal of Social Work Education, 40(3),Bogo, M., Regehr, C., Power, R., & Regehr, G. (2007). When values collide: Providing feedback and evaluating competence in social work. The Clinical Supervisor, 26(1/2),Regehr, G., Bogo, M., Regehr, C., & Power, R. (2007). Can we build a better mousetrap? Improving the measures of practice performance in the field practicum. Journal of Social Work Education, 43(2),About OSCEHarden, R. M., Stevenson, M., Downie, W. W., & Wilson, G. M. (1975). Assessment of clinical competence using objective structured examination. British Medical Journal, 1(5955),Rating ScalesBogo, M., Katz, E., Logie, C., Regehr, C.,& Regehr, G. (2008). OSCE for social work: Practice performance rating scale. Unpublished scale. Factor-Inwentash Faculty of Social Work, Toronto, ON.Bogo, M., Mylopoulos, M., Katz, E., Logie, C., Regehr, C., & Regehr, G. (2009). Reflective dialogue rating scale. Unpublished scale. Factor-Inwentash Faculty of Social Work, Toronto, ON.
62 Website Factor-Inwentash Faculty of Social Work Research Institute for Evidence-Based PracticeCompetency for Professional PracticeKnowledge building – studies, articles, presentationsKnowledge dissemination – fact sheets, assessment and evaluation tools.