1. Holistic Assessment of Social Work Students’ Professional Practice Marion Bogo
Joint Social Work Education Conference and
Social Work Research Conference
University of Hertfordshire
2010

4. Overview Link between social work education and practice
Conceptualization of professional practice
Assessment
Cognition
Interpersonal
Contextual

5. Social Work Education and Practice
The purpose of social work education is to prepare social work practitioners

6. Professional Work: Dimensions
Thinking
Performing
Acting with integrity
(Lee Shulman, 2005)

7. Signature Pedagogy
“…pedagogical norms with which to connect and integrate theory and practice.”
(Lee Shulman, 2005)

8. Social Work Education and Practice
United States: Council on Social Work Education, Educational Policy and Accreditation Standards 2008
England: Social Work Task Force 2009
Canada: Canadian Council of Social Work Regulators; Project - Social Worker Competency Profile 2010

9. Social Work Education and Practice
Concept
ASSESS
Content
APPROACH
Method

10. Conceptualization of Professional Practice
Critique of competency as dimensions and skills – a checklist approach
How to capture holistic nature of practice and performance
not mechanistic
context
professionalism
knowledge competencies, internal cognitive processes for judgment, decision making

11. “In their own words…” Design a practice-based evaluation tool
Reflect practice teachers’ tacit or implicit conceptualization of student competence – dimensions they use
In practice teachers’ language

12. Procedural competencies
Holistic Competence
Meta-competencies
Procedural competencies

13. Meta-Competencies
Higher order overarching abilities and qualities - of a different order and nature than procedural or operational behaviors and skills.
Related to individuals’ ability to:
use discrete behaviors in a purposeful, integrated, and professional manner.
learn the specific role competencies of particular professions.

14. Procedural competencies
Behavior in the Organization
Conceptualization of practice
Collaborative relationship
Meta-competencies
Relational capacity/Intentional
use of self
Procedural competencies
Conducts assessment and intervention
Professional Communication
Learning and growth as a professional
DIRECT PRACTICE

15. Procedural competencies
Behavior in the Organization/
Relationships
Critical Thinking
and Analysis
Planning
Meta-competencies
Leadership
Procedural competencies
Project management
Professional Communication
Learning and growth as a professional
MACRO PRACTICE

16. Social Work Education and Practice
Concept
ASSESS
Content
APPROACH
Method

17. Challenges to Assessment
Cognition
Interpersonal
Contextual

18. Levels of Performance
High agreement on individual rankings of 20 vignettes- interclass correlation coefficient: 0.83
Five categories generated
Exceptional
Ready for practice
On the cusp
Needs more training
Unsuitable for the profession

19. From Descriptions to a Tool
Dimensions of holistic competence
Learning and growth
Behavior in the organization
Conceptualizing practice
Collaborative relationships
Assessment and intervention
Professional Communication

20. From Descriptions to a Tool
For each dimension
Extracted descriptors from the ranked vignettes for each of 5 levels of performance
In practice teachers’ language.

21. Behavior in the Organization
EXEMPLARY
READY TO PRACTICE
ON THE CUSP
MORE TRAINING
UNSUITABLE
Work in a responsible and experienced manner within the agency.
Show respect to other staff and work well within the agency’s culture.
Understand the student role and maintain professional boundaries with other team members.
Respected and valued by agency staff. Other staff regard her/him as a valued colleague.
Highly flexible and adaptable.
Relate to agency staff with humility and respect.
Viewed as co-workers by other members of the team.
Work well with colleagues, but can be intimidated by members of other professions.
Follow agency procedures and are accepted by the team.
Tend to work well in the student role.
Regarded as a student/learner by other members of the team.
Experience difficulty in building relationships with organizational staff.
Struggles with the student role, as she/he wants more clients and responsibility, but does not possess or demonstrate the needed skills and abilities to take on this extra responsibility.
Viewed primarily as a student.
Lack appreciation of the student role and self-awareness about interactions with organizational staff.
No respect for the agency and its procedures.
Experience difficulty in fitting into the agency and engage in problematic behaviours, such as being highly critical of and argumentative with other staff and the field instructor, and/or becoming involved in agency politics.
Demonstrate an overall inflexibility in the workplace.

22. Competency-Based Evaluation
Intervention Planning and Implementation
Identify and confront a reluctance to recognize viable options. N
Appropriately challenge the client system when required.
Give feedback on above in a manner, which conveys respect and understanding.

23. Results
New PBE tool was not more effective at discriminating students than older CBE tool.
CBE Tool
PBE Tool
Mean
4.08
4.20
Std Dev
0.53
0.63
Pct <3.00
2.4%
4.8% 23

24. Using all Tools Recall most recent student
20 vignettes ordered randomly
Select vignettes “similar” to their student.
Select from “similar vignettes” one or two vignettes that are “most similar” to their student.
Evaluate same student using
Practice-Based Evaluation (PBE) Tool
Competency-Based Evaluation (CBE) Tool

25. Distribution of Student Scores for the Three Measures

26. Distribution of Student Scores for the Three Measures

27. Distribution of Student Scores for the Three Measures

28. Scales Holistic impression of students Deconstruct Relate to a skill
Convert to a point on a scale

29. Challenges to Assessment
Cognition
Interpersonal
Contextual

30. Reflections on Assessment
When values collide
Student response and the inter-personal relationship
Negotiating
Where is ‘the school?’

31. Conclusion
In instructors’ own language – holistic, global, authentic - reflects their conceptualization of practice competence.
A tool without numbers.
Approaches that empower the practice assessor/field instructor and are not negotiated – exercise her expertise.

32. An Online Tool

33. Dimensions of Practice
Six dimensions
Learning and growth
Behavior in the organization
Clinical relationships
Conceptualizing practice
Assessment and intervention
Professional Communication
For each dimension, descriptors provided for each of 5 levels of performance.

34. Levels of Performance Five categories
Exceptional
Ready for practice
On the cusp
Needs more training
Unsuitable for the profession

35. Conceptualizing Practice
EXEMPLARY
READY TO PRACTICE
ON THE CUSP
MORE TRAINING
UNSUITABLE
Think about and understand practice on multiple levels from instrumental tasks to deeper, therapeutic interventions.
Possess strong critical thinking skills.
Competent in using numerous theories in practice, and apply them in a flexible manner.
Understand and carry out the mandate of the organization, while remaining sensitive to clients’ situations and needs.
Maintain an ethical practice, but are not rigid or inflexible.
View practice on multiple levels, but in some cases this develops only with field instruction.
Knowledgeable about theory and are capable of applying it to practice. Some students, however tend to know one approach very well and work primarily from that framework.
Understand and maintain professional, ethical practice.
Some ability to think critically.
View practice as simply the process of collecting factual or affective information and demonstrate only a limited ability to link affect, behaviour, and cognition.
Experience difficulty integrating theory, but some growth is observed.
Understand and maintain an ethical practice.
Give attention primarily to the concrete aspects of clients’ situations.
Tend to think in the here and now and experience difficulties in making deeper connections.
Able to identify and discuss ethical matters.
Work well within established frameworks, but demonstrate inflexibility in using different practice approaches.
Struggle with understanding the difference between personal relationships and worker-client relationships.
View practice at a concrete level only and experience extreme difficulty in going beyond this level.
Have difficulty applying theoretical understanding to practice situations.
Do not understand ethical practice and client-worker boundaries. Some students have the potential to be harmful to clients.
Demonstrate rigidity in her/his thinking and belief systems.
Struggle in understanding social work values.
Personal issues and difficulties are not kept out of her/his work and practice.

36. Conceptualizing Practice
Descriptor
Rank
Able to begin to identify and discuss ethical matters. 2
Competent in using numerous theories in practice, and applying them in a flexible manner. 5
Culturally competent and actively incorporates factors relevant to diverse clients.
Culturally competent, understands and begins to incorporate factors relevant to diverse clients. 4
Demonstrates rigidity in his/her thinking and belief systems 1
Demonstrates some ability to think critically.
Displays some understanding of the impact of cultural factors. 3
Gives attention primarily to the concrete aspects of clients' situations.
Has difficulty applying theoretical understanding to practice situations

39. 2008-9 Comparisons between CBE Tool (480) and PBE Tool (152)

40. Challenges to Assessment
Cognition
Interpersonal
Contextual

41. OSCE: Objective Structured Clinical Examination
Developed in medicine and used in health professions (Harden et al., 1975)
Students interview series of standardized, simulated “clients”
“Clients” are trained actors
Students are given brief information – role, presenting issues, time frame and goal.

42. Adapted for Social Work
Holistic competence - in social work two components:
interview for minutes for procedural competencies
followed by a structured reflective dialogue for minutes to elicit meta-competencies
Both observed and rated by a trained rater.

43. Iterative Process to Develop OSCE
Competence
Conceptualize
Design Scenarios
Define specific behaviors and rating scale
Define components and scale for reflection
Train and Pilot

44. The Examination

45. Reading Instructions

46. One of Five “Clients”

47. Hospital Social Work

48. Child Protection

49. Rater and Widowed Mother

50. Initial Study: Participants
11 students completed Year 1 of the MSW Program
7 recent graduates of the MSW Program
5 experienced social workers with 5-7 years of experience

51. Results
Developed reliable tools – dimensions in each tool were assessing a common ability
The two aspects – performance and reflection were related but conceptualized differently by raters.

52. Results
Moderate reliability across stations : Cronbach’s alpha of 0.55 for Practice and 0.48 for Reflection
Some evidence for generalizability of individual scores across stations
Need more stations, better stations, better raters?

53. Results 30% of variance on performance 23% of variance on reflection
Experienced practitioners had significantly higher scores
23% of variance on reflection
Marginally significant effect of experience

54. Assessing Performance and Reflection
Range of student ability to engage in meaningful reflective dialogues about:
Conceptualizing practice and using frameworks about diversity
Intentional use of self and own reactions
Integrating learning into practice.
Performing well and in depth reflection not necessarily at same level in individual students.

55. Is OSCE in our Future?
Cautious optimism about the merits of using OSCE adapted for social work
Appears useful for evaluation of both performance and critical thinking, conceptualization, reflection, self-assessment
Scores not at high range as in field evaluations
Participants and students very enthusiastic.

56. Concluding Thoughts Need approaches that assess holistic competence
Need to assess at different points in time
Need multiple assessment methods- authenticity and standardized
In university program and in practice setting

57. Concluding Thoughts
Need to study assessment methods and links to pedagogy
Social work education researchers
Institutional arrangements with practice settings

58. Acknowledgements
These studies were funded by the Social Sciences and Humanities Council of Canada.
Co-investigators: Cheryl Regehr, Professor, Factor-Inwentash Faculty of Social Work and Vice Provost, University of Toronto.
Glenn Regehr, Professor and Associate Director, Research, Centre for Health Education Scholarship, Faculty of Medicine, University of British Columbia.
Researchers: Maria Mylopoulos, Educational Researcher, SickKids Learning Institute, Hospital for Sick Children, Assistant Professor, Department of Paediatrics, Faculty of Medicine, University of Toronto
Carmen Logie and Ellen Katz, doctoral candidate, and Rachael Walliser, research assistant, Factor-Inwentash Faculty of Social Work, University of Toronto.

59. Selected References About Field Education
Bogo, M. (in press) Achieving competence in social work through field education. Toronto, ON: University of Toronto Press.
Bogo, M. (2005). Field instruction in social work: A review of the research literature The Clinical Supervisor, 24(1/2),
Bogo, M., & Vayda, E. (1998). The practice of field instruction in social work: Theory and process (2nd ed.). New York, NY: Columbia University Press.
Fortune, A. E., & Kaye, L. (2002). Learning opportunities in field practica: Identifying skills and activities associated with MSW students' self-evaluation of performance and satisfaction. The Clinical Supervisor, 21, 5-28.
Fortune, A. E., McCarthy, M., & Abramson, J. S. (2001). Student learning processes in field education: Relationship of learning activities to quality of field instruction, satisfaction, and performance among MSW students. Journal of Social Work Education, 37(1),
Homonoff, E. (2008). The heart of social work: Best practitioners rise to challenges in field instruction. The Clinical Supervisor, 27(2),
Shulman, L. S. (2005a). Signature pedagogies in the profession. Daedalus, 134(3),

60. Selected References
About Competence and Holistic Competence Bogo, M., Regehr, C., Woodford, M., Hughes, J., Power, R., & Regehr, G. (2006). Beyond competencies: Field instructors' descriptions of student performance. Journal of Social Work Education, 42(3), Cheetham, G., & Chivers, G. (2005). Professions, competence and informal learning. Cheltenham, UK: Edward Elgar. Eraut, M. (1994). Developing professional knowledge and competence. London, UK: Falmer Press. Kelly, J., & Horder, W. (2001). The how and why: Competences and holistic practice. Social Work Education, 20(6), Schon, D. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass. Skinner, K., & Whyte, B. (2004). Going beyond training: Theory and practice in managing learning. Social Work Education, 23(4),

61. Selected References About Evaluating Competence
Bogo, M., Regehr, C., Hughes, J., Power, R., & Globerman, J. (2002). Evaluating a measure of student field performance in direct service: Testing reliability and validity of explicit criteria. Journal of Social Work Education, 38(3),
Bogo, M., Regehr, C., Power, R., Hughes, J., Woodford, M., & Regehr, G. (2004). Toward new approaches for evaluating student field performance: Tapping the implicit criteria used by experienced field instructors. Journal of Social Work Education, 40(3),
Bogo, M., Regehr, C., Power, R., & Regehr, G. (2007). When values collide: Providing feedback and evaluating competence in social work. The Clinical Supervisor, 26(1/2),
Regehr, G., Bogo, M., Regehr, C., & Power, R. (2007). Can we build a better mousetrap? Improving the measures of practice performance in the field practicum. Journal of Social Work Education, 43(2),
About OSCE
Harden, R. M., Stevenson, M., Downie, W. W., & Wilson, G. M. (1975). Assessment of clinical competence using objective structured examination. British Medical Journal, 1(5955),
Rating Scales
Bogo, M., Katz, E., Logie, C., Regehr, C.,& Regehr, G. (2008). OSCE for social work: Practice performance rating scale. Unpublished scale. Factor-Inwentash Faculty of Social Work, Toronto, ON.
Bogo, M., Mylopoulos, M., Katz, E., Logie, C., Regehr, C., & Regehr, G. (2009). Reflective dialogue rating scale. Unpublished scale. Factor-Inwentash Faculty of Social Work, Toronto, ON.

62. Website Factor-Inwentash Faculty of Social Work
Research Institute for Evidence-Based Practice
Competency for Professional Practice
Knowledge building – studies, articles, presentations
Knowledge dissemination – fact sheets, assessment and evaluation tools.