1. School of Health and Social Care Can We Measure & Track the Acquisition of Competency? Evaluating learning in relation to the NOS competencies using mixed methods Dr Sharon Vitali Evaluating the Outcomes of Social Work Education Symposium 15 May 2009

2. School of Health and Social Care What do we want to know?  Is this student academically eligible for a BA (Hons) social work degree?  Is this student competent to practice as a novice social worker?  Are our assessment (academic & practice) mechanisms reliable?

3. School of Health and Social Care What do we want to know?  Where do our students acquire the knowledge skills and values that are required of them to qualify?  Is our curriculum fit for purpose?  What are the weaknesses in our curriculum ? (content and or timing of delivery)

4. School of Health and Social Care Competency Based Education & Training  From the Latin word ‘competens’ to “…be fit, proper or qualified” (The Oxford Library of Words and Phrases,1998)  a competent learner is "…one who has reached a defined standard of performance" (Freeman & Lewis 1998:314)  outcomes = workplace expectations of specific knowledge and skills, which are believed to be essential and the required level of performance  what a person can do; rather than length of type of training (allows APEL)

5. School of Health and Social Care Competency Based Education & Training  refers to skills and knowledge that can be transferred and applied to new situations and environments  results in a recognisable, consistent and portable qualification (Toohey, Ryan, McLean & Hughes, 1995)  market and government driven - attempting to ensure ability and accountability thereby enabling efficient and global recruiting of crucial segments of the workforce

6. School of Health and Social Care Competency Based Education & Training  an amalgam of knowledge, skills and values that must be demonstrated and measured against a benchmark of standards that have been agreed upon.  No penalty for barely reaching adequacy, neither is there any reward in excelling.  Licensed to legally perform a specific activity in society.  License holder subject to disciplinary measures upon infractions, and the license may be revoked should the misdemeanour be significant enough

7. School of Health and Social Care Challenges to Evaluating Outcomes of Educational Programmes:  Potential for bias in ‘satisfaction outcomes’  Potential for bias in ‘ipsative’ (self appraisal) evaluations  Competency Based Education vs. Competency Based Assessment  Methodology Strategy: = Mixed Methods & Triangulation of Measures

8. School of Health and Social Care What did we do?  Cohort Sample (n=38 at beginning n=33 at completion = 87%)  Longitudinal - Baseline to Graduation  Repeated Measures  Triangulation of Measurements  Mapping Against Learning Activities

9. School of Health and Social Care What measurement methods?  Fixed Outcomes  (National Occupational Standards of Competency for SW)  RICET  (Reflective Ipsative/self Competency Evaluation Tool)  Pre-placement interview performance  (Voices of Experience & Independent Researcher)  Portfolios inclusive of written reflective performance  Academic Performance  social work & non-social work modules  coursework, computer based tests, presentations

10. School of Health and Social Care Evaluation Sources & Schedule: Y1-S1RICET Self LEARNING PLAN Tutor RICET Self Grades SW & Other Y1-S2INTERVIEW Voices of Experience RICET Self PORTFOLIO Tutor INTERVIEW Researcher Grades SW & Other Y2-S1RICET Self RICET & PORTFOLIO P/Teacher PORTFOLIO P/Teacher & Tutor Grades SW & Other Y2-S2RICET Self RICET & PORTFOLIO P/Teacher PORTFOLIO P/Teacher & Tutor Grades SW & Other Y3-S1RICET Self RICET & PORTFOLIO P/Teacher PORTFOLIO P/Teacher & Tutor CASE VIVA Grades SW & Other Y3-S2RICET Self RICET & PORTFOLIO P/Teacher PORTFOLIO P/Teacher & Tutor CRITICAL CAREER REVIEW Grades SW & Other

11. School of Health and Social Care Curriculum of Learning Activities Y1-S1Fit for Practice OneHuman Growth & Development One Professional Communication Partnerships in Practice One Y1-S2Fit for Practice Two & Shadowing a Social Worker Human Growth & Development Two Social PolicySocial Work Law Y2-S1Placement in Voluntary Independent or Private Sector Critical Evaluation of Research Literature SW with Children + SW with Adults + Mental Health for Social Workers Partnerships in Practice Two Y2-S2Placement in Voluntary Independent or Private Sector SW with AdultsMental Health for Social Workers Y3-S1Placement in Statutory Sector Placement in Statutory Sector Practice Learning Interventions Dissertation Y3-S2Placement in Statutory Sector Alternative (LA Children or...) Professional Consolidation Dissertation

12. School of Health and Social Care What else did/could we collect ?  Demographic features of Students  Age, gender, country of origin, first language, previous education, previous work / volunteer experience, mode of entry, dyslexia/disability, childcare responsibilities, distance travelled to attend  Practice Setting  Type, experience level, student audit (PLEAT)  Demographics of Practice Teachers  Age, gender, country of origin, first language, previous education, previous work experience, practice teaching experience

13. School of Health and Social Care Caution about Findings  Previous study had larger sample and found statistically significant changes BUT…  This sample too small to run any tests of significance…so just descriptive findings  TODAY …some trends and demonstrate to you how the methodology works to answer your questions

14. School of Health and Social Care What did we find?  Inflation of Ipsative Measurement:  Inflation at ipsative baseline measure = “don’t know what they don’t know”  Correction of inflated measure except in one subset = cultural artefact  Students over-rated themselves compared to tutors & practice teachers throughout study

15. School of Health and Social Care

16. What did we find?  Performance by Mode of Entry :  Widening participation  Lowering of age at entry requirement  “Added value”  Slightly higher average graduation grades in traditional mode of entry students  Grade range very similar

17. School of Health and Social Care 51.8% to 75.5%53.3% to 77%

18. School of Health and Social Care What did we find?  Performance Trajectories  Academically Strong  ‘Natural’ in Practice  Confidence  3 trajectories (L to L) (H to H) (L to M)

19. School of Health and Social Care

20. What did we find?  Measurement of acquisition of competencies possible  Mapping of acquisition of competencies possible  Identification of gaps & weaknesses possible OBSERVATIONS:  Incremental over the 3 years  Practice teachers and esp. tutors tended to score lower than students’ ipsative scoring  Regression in measurement when shifting into second year (statutory) placement

21. School of Health and Social Care

22. F4P Test of Knowledge F4P Test of Knowledge & Skill

23. School of Health and Social Care Social Policy & Law

24. School of Health and Social Care ENHANCED: Assessments Social Policy & Law

25. School of Health and Social Care ENHANCED: Crisis Intervention

26. School of Health and Social Care

27. F4P2 Theory to Practice

28. School of Health and Social Care F4P2 Theory to Practice, Social Policy & Law

29. School of Health and Social Care F4P2 Theory to Practice HG&D 2 Group Work

30. School of Health and Social Care MOVED: Mental Health for Social Workers

31. School of Health and Social Care Social Policy & Law

32. School of Health and Social Care INCREASED: Individual Presentations & Posters Social Policy & Law

33. School of Health and Social Care MOVED: Mental Health for Social Workers

34. School of Health and Social Care MOVED: Mental Health for Social Workers

35. School of Health and Social Care ENHANCED IN: Practice Learning Interventions

36. School of Health and Social Care ENHANCED IN: Practice Learning Interventions

37. School of Health and Social Care

38. Professional Consolidation

39. School of Health and Social Care ENHANCED IN: Practice Learning Interventions Critical Evaluation of Research Literature

40. School of Health and Social Care Professional Consolidation

41. School of Health and Social Care Professional Consolidation

42. School of Health and Social Care ENHANCED IN: Practice Learning Interventions & Professional Consolidation

43. School of Health and Social Care What have we done?  Analysis of non-completers:  This cohort = 8  Delayed (Temporary Withdrawal) =2  Unsuitable = 2  Academically Incapable = 1  Analysis of Added Value & Exit Velocity Trajectories

44. School of Health and Social Care What have we done?  Redesigned curriculum and timing of module delivery  Redesigned module content & assessment methods  Enhanced support for ESL & non-traditional mode of entry students  Enhanced support for students in first year followed by proactive ‘counseling out’ for unsuitable/incapable

45. School of Health and Social Care What have we done?  Increased specific learning activities  Theory to practice  Interviewing skills  Improved quality of PLO – workshops training, etc  Increased participation of Practice Educators on PAP  Placement transition tutorials & workshops by Practice Leads

46. School of Health and Social Care What might be of use to you?  A bit about the RICET  Free instrument (just ask for request first & possible later collaboration)  Valid & reliable  Cheap to administer, easy, quick  Can be administered via V L Environments /email  Can be modified to suit any competency based evaluations

47. School of Health and Social Care What might be of use to you?  A bit more about the RICET  Useful for developing learning contracts  Useful for mid-placement evaluations  Useful for negotiating differences of opinion  Useful aide to self-reflection and critical awareness  Identifies ‘rogue’ students and/or practice teachers

48. School of Health and Social Care What might be of use to you?  Audit process so no need for:  research budget  extensive ethics applications  data analysis  Excellent for rolling curriculum reviews & refinements  Excellent for Validations & Quality Assurance Reviews  MY HOPE:…..collate larger sample for analysis