Presentation is loading. Please wait.

Presentation is loading. Please wait.

EVIDENCE-BASED RESEARCH IN CANADA  National research agenda to better understand “what works” in career services  3 calls for proposals since 2004 

Similar presentations


Presentation on theme: "EVIDENCE-BASED RESEARCH IN CANADA  National research agenda to better understand “what works” in career services  3 calls for proposals since 2004 "— Presentation transcript:

1

2 EVIDENCE-BASED RESEARCH IN CANADA  National research agenda to better understand “what works” in career services  3 calls for proposals since 2004  Source of major international envy among Career Development Practitioners and Researchers 2

3 WHY EVIDENCE-BASED PRACTICE?  Ethics: Clients deserve interventions that have demonstrated success  Confidence: Practitioners should know that a given intervention will result in a predictable outcome  Security: Being able to provide evidence of success increases ability to obtain and keep funding for services and secure jobs for CDPs/Employment Counsellors 3

4 WHY EVIDENCE-BASED PRACTICE?  Outcome-focused intervention  Evidence of client change …contrast with…  Client flow  Ease of access  Practitioner time use It’s easy to measure variables that don’t address client change 4

5 WHY EVIDENCE-BASED PRACTICE?  Some successful programs are discontinued because there are no data indicating client change  Client flow client change  There is a need for  Evidence of client change  Evidence of impact of service on…  Client  Society  Economy 5

6 6

7 SESSION OBJECTIVES  Highlight recent research projects and what has been learned (what evidence do we have?), with a focus on the tools used and their impact  Explore how tools used in the research could be used to support evidence based practice  Identify the key characteristics/practices of the Evidence- Based Practitioner  Identify action steps to infuse Evidence-Based Practice into work with clients, into professional growth and into office culture 7

8  What evidence do you currently collect?  What is done with the evidence currently collected? 8

9 EVIDENCE-BASE: A RESEARCH QUESTION  3 recent Canadian research studies  Front-line research conducted with employment offices in Alberta, Manitoba, Québec and New Brunswick  Consistent and strong results 9

10 QUESTIONS TO KEEP IN MIND  What did the research do?  What did we learn?  What was most surprising?  What are the implications for practice? 10

11 RESEARCH PROJECT #1: THE IMPACT OF LABOUR MARKET INFORMATION ON CAREER DECISION MAKING Research Questions If client needs are assessed and clients are given LMI consistent with their needs,  To what extent does assistance by a service provider enhance their effective use of LMI? OR  To what extent is independent self-help a sufficient process for clients to use LMI effectively? 11

12 METHOD  We prepared “guided” LMI packages (take- home booklets and Resource Centre binders) on:  Career Decision Making:  Know yourself  Know the Labour Market  Put it all Together  Job Search:  Check for “Fit”  Get Ready  Search for Work  Get a Job 12

13 INTERVENTION All participants in the study:  Received a needs assessment interview & completed an initial, pre-program survey  Received an LMI package specific to their identified employability need  Were randomly assigned to either a self- directed intervention delivery method or an assisted self-directed intervention delivery method.  Were given an orientation to the Resource Centre which they could freely use on their own  Completed a pre and then and now post survey  Received a cash honorarium and certificate of participation 13

14 INTERVENTION (CONT) The self-directed group:  worked independently for 3 weeks, making use of the materials and the Resource Centre  returned in Week 4 for their exit interview 14

15 INTERVENTION (CONT) The assisted self-help clients received:  two additional AIS (Advice and Information) interviews (20-30 minutes) in weeks 1 and 3 focused on helping them understand, interpret and apply the LMI to their own situations and /or access additional LMI  returned in Week 4 for their exit interview 15

16 WHAT DID WE MEASURE  General ability to use LMI  Knowledge  Clear vision of what I want in my career future  Knowledge of print and online resources  Skill  Have effective strategies for keeping myself motivated  Have a realistic action plan  Personal Attributes  Optimism about what lies ahead re meeting my career goals  Confidence in my ability to manage future career transitions 16

17 DIFFERENTIAL RESULTS-TOTAL SCORE  Both CDM and JS groups had significant increases across time  Change in CDM group was significantly larger than in JS group  Participants in the JS group had higher scores than participants in the CDM group, likely indicating that JS participants were more familiar with using LMI before the project began.  Participants receiving assistance demonstrated greater change across time than did those in the independent mode  Participants receiving assistance demonstrated greater change across time than did those in the independent mode Similar pattern for all subscales 17 For group as a whole: significant increase in overall ability to use LMI significant increase in overall ability to use LMI neither intervention was more conducive to one manner of delivery compared to the other neither intervention was more conducive to one manner of delivery compared to the other

18 ATTRIBUTION FOR CHANGE To what extent would you say that any changes in the ratings on the previous pages are a result of your participation in this research project, and to what extent were they a function of other factors in your life? 18 mostly other factors somewhat other factors uncertain somewhat this program mostly this program English French Program

19 FOLLOW UP INTERVIEWS (1 WEEK AND 4 MONTHS AFTER INTERVENTION)  Week 1: 103 clients  4 months: 65 clients  LMI overall relevant  Clients overall optimistic and confident  Clients continued to use LMI but less often 19

20 WHAT DID CLIENTS SAY? THEMES:  I was surprised at how much information there was  Initially I thought I could not do this; but I found out I could  I got more focused on my goal  I got clearer about what I wanted to do  There was too much information and that made it difficult….very hard to do this alone (Those who were working –85-90% responses in these categories)  Did not help very much (only from those still unemployed at 4 months) 20

21 WHAT DID PRACTITIONERS SAY?  I never really thought about clients as assisted or self-help—now it is built into all my assessments  Now I give more homework and I am more specific. The clients felt more focused and so did I  I used to be more maternal and I got more involved than I needed to be. Now I ask clients to be more responsible  I found the checklists inhibiting and awkward 21

22  What did we learn?  What was most surprising?  What are the implications for practice? 22

23 APPLICATIONS TO SERVICE DELIVERY  Tailored LMI embedded in a learning process results in knowledge and skill acquisition as well as the capacity for self-management  LMI appropriate for a client’s specific need (opposed to general LMI) appears to support engagement and action  For many clients, a little (or no) professional support is enough  Structure and timelines appear to motivate action and a sense of progress  Giving clients hands-on tools appears to motivate more than money 23

24 TOOLS/ SUPPORTS FOR THE EVIDENCE- BASED PRACTITIONER  Initial learning needs survey  Checklist for Employment Assessment Interview  Advice and Information Interview Checklist  LMI Resource Guide for Career Decision Making  LMI Resource Guide for Job Search 24

25 RESEARCH PROJECT #2: ASSESSING THE IMPACT OF CAREER RESOURCE AND SUPPORTS ACROSS THE EMPLOYABILITY DIMENSIONS  This project builds substantially on the LMI Impact study  Self-Help Index and Labour Market Attachment Index 25

26 THIS STUDY  Also began with a needs assessment, but expanded to 4 needs:  Career decision making  Skills Enhancement  Work Search  Job Maintenance (and pre-employability)  Each client received a tailored resource package, but well beyond LMI to include coaching activities to help them to reflect and personalize. 26

27 THIS STUDY  Clients randomly assigned to one of two treatment conditions:  Practitioner assessed (in which they had their needs assessed, were oriented to the appropriate Guide and then worked independently with their Resource Guide for 4 weeks)  Practitioner assessed and supported (in which they also worked with their Career Consultant for 4 weeks).  Examined:  Differential impact across treatment conditions (independent/supported)  Differential impact across labour market attachment & self- help ability (2 new variables) 27

28 RESEARCH QUESTION If clients are given a comprehensive needs assessment to determine their employability need(s), what is the differential effect of independent and consultant- supported career resources on clients who are weakly attached to the labour market versus those who are strongly attached to the labour market? 28

29 RESULTS  After just 4 weeks of intervention:  Tenfold increase in competencies rated in acceptable range  Percentage of employed clients rose from 27% to 45% (a 69% increase)  81% of participants reported that their employment was a “good fit” with their career vision (a 200% increase)  91% attributed the positive changes either partly (40%) or mostly (51%) to the intervention  92% had a clear plan for next steps  98% planned to use the Guides agai n 29

30 RESULTS  Statistically and clinically significant positive impact of career interventions across ALL sub- scores and across ALL employability dimensions – compelling evidence of positive changes in clients as a result of career interventions  Although see more positive trends for supported group, differences are not statistically significant 30

31 RESULTS  Career practitioners care very much about their clients, so much so that they were reluctant to ask clients who might need support to join the study— “in case” they were randomly assigned to the independent group  Therefore the sample was too homogeneous to test properly the Self-help and Labour Market attachment indices  Career practitioners care very much about their clients…BUT…in the next study, they CANNOT! 31

32  What did we learn  What was most surprising?  What are the implications for practice? 32

33 WHAT DID THE STUDY TELL US?  Career Development Interventions Work!  Clients demonstrated substantial positive changes in skills, knowledge, personal attributes, employment and fit of employment.  Many clients can benefit significantly from self-help resources when they are matched to their need and they are “launched” – moreso than we anticipated!  This is especially noteworthy given the short intervention period of 4 weeks and the “real-life” setting in which the changes occurred. 33

34  The Tools and Guides are “Ready to Go”  Virtually no adjustments are needed for use in typical employment centres across Canada. Self-help guides provided after a thorough needs assessment and orientation are effective and can be used as a first line of intervention, saving valuable practitioner time for clients who really need it, or potentially opening a promising perspective on online career services. This finding has considerable practical significance. 34

35 EMPLOYABILITY DIMENSIONS STUDY: TOOLS  Employment Goal and Action Planner  Client Checklists/Tracking Sheets (all employability dimensions)  Comprehensive Coaching Guides (all employability dimensions) 35

36 RESEARCH PROJECT #3: COMMON INDICATORS  What had we learned?  Where did we need to go next? 36

37 37 RESOURCES AVAILABLE CLIENT NEEDS ASSESSMENT CLIENT EMPLOYMENT HISTORY SELF-HELP INDEX LABOUR MARKET ATTACHMENT INDEX ACTIVITIES THAT LINK TO OUTCOMES OR DELIVERABLES Generic interventions  Working alliance, Specific interventions LMI Booklets Advice and Information Interviews Employability Dimension Coaching Guides Practitioner Coaching using the Guides External Referral 1.Learning outcomes  Ability to use LMI resources alone and with assistance  Ability to use Coaching Resources alone and with assistance  Having an employment goal/vision  Skills in employability dimension indicators 2.Personal attribute outcomes  Changes in intrapersonal variables e.g., attitudes, self-esteem, motivation, etc. 3.Impact Outcomes  Changes in the client’s life resulting from application of learning OUTCOMES INPUTS PROCESSES Client  Context  Life vision  Needs  Goals

38 UNANSWERED QUESTIONS  We have data/evidence of positive impacts from specific interventions for certain kinds of clients  We do not have data/evidence of positive impacts from employment services overall  We do not have a data gathering tool that can be used to gather common data across divergent employment service settings 38

39 THEREFORE THE NEXT STEP: COMMON INDICATORS PROJECT Key research questions:  What common indicators are applicable across different contexts, different client groups, different agencies and different interventions?  What statements of service effectiveness can be made by tracking common indicators? and 39

40 IF NUMBERS PERMIT… THE ULTIMATE QUESTION:  What kinds of interventions in what contexts produce what kinds of results? 40

41 HOW WE INVESTIGATED  Literature Review  Focus Groups  Compare the two  Accept those in common and most robust 41

42 FOCUS GROUP QUESTIONS:  If clients have benefitted from career and employment services, they have experienced some changes in in their lives. They are not in the same position they were when they started. What indicates that clients or their situations have changed as a result of career and employment services?  How do you know? What are your clues?  What do you do to bring about change? What influences your ability to do this?  What about clients? What influences their ability to change?  What factors are beyond both of you to control? 42

43 43 RESOURCES AVAILABLE  Availability of and access to community resources Availability of and access to employment opportunities and training  Practitioner Background and Experience  Client Employment History and Potential  Client Life Circumstances ACTIVITIES THAT LINK TO OUTCOMES OR DELIVERABLES Generic interventions  Working alliance, client engagement. Specific interventions Needs Assessment Employment Counselling Resources/Guides Goal setting and Action Planning Programs & Workshops Progress Tracking External Referral CLIENT CHANGE 1.Learning outcomes  Changes in knowledge and skills linked to the program or intervention used  Progress Indicators End Result Indicators 2.Personal attribute outcomes  Changes in intrapersonal variables e.g., self-confidence, self-esteem etc.. 3.Progress Indicators and Impact Outcomes  Changes in the client’s life resulting from application of learning OUTCOMES INPUTS PROCESSES Client  Context  Life vision  Needs  Goals

44 PRACTITIONER/RESEARCH PARTNER TASKS  Recruit clients into the study (NO criteria);  Follow all research protocols  Use PRIME each time you see a client to assess and record PROGRESS/CHANGE in:  Goals and Action Plans  Employability Learning Needs/Changes  Personal Attribute Needs/Changes  Life Circumstance Needs/Changes  Working Alliance and Client Engagement 44

45 PRACTITIONER/RESEARCH PARTNER TASKS  Work with each client for 6 weeks following Assessment interview and client entry into the research  At exit interview also gather data on… 45

46 PRACTITIONER/RESEARCH PARTNER TASKS  Labour Market Outcomes:  Employment or Training Status  Fit of Employment or Training with skills, qualifications, vision  Adequacy of salary  Linked to future opportunity  Ongoing Progress Evidence: Actions Underway to move forward:  Have clients complete then and now questionnaire 46

47 PRE-EMPLOYABILITY/JOB READINESS Client needs assistance to : Not at all Not much A little Quite a lot A lot  Identify and clarify future direction (e.g., training, education, employment or change in life circumstances goal)  Identify personal strengths/resources that support future direction (e.g., training, education, employment or change in life circumstances goal)  Resolve specific challenges/vulnerabilities that may impact on future direction (e.g., mortgage, public transit, day care etc.) 47

48 LIFE CIRCUMSTANCES Indicators of challenges in client life circumstances are in evidence: Not at all Not much A little Quite a lot A lot  Improved housing is needed  Improved transportation is needed  Improved capacity to work and/or study is needed  Increase in sense of responsibility for own choices and behaviours is needed  Increase in ability to set short and long term goals is needed  Increase in understanding expectations and demands of employers is needed  Reduction in destructive behavior is needed  Following medical and medication protocols is needed  Increased openness to change is needed  Increased access to constructive and positive support systems is needed  Improved relationships with family and friends are needed  Increased trust in other people is needed  Other: _________________________________ 48

49 PERSONAL ATTRIBUTES EXAMPLE Client needs assistance in the following areas: Not at all Not much A little Quite a lot A lot Ability to Self-Manage:  More info Developing stronger Self-Esteem:  More info Improving sense of Well-Being: i.e.:  More info Developing stronger Self-Efficacy: i.e.:  More info Increasing Self-Awareness:  More info  Other (Please specify): _______________________________ _ 49

50 SELF-ESTEEM ATTRIBUTE  more info: Self-Esteem: “a person’s overall emotional evaluation of his or her own worth, an attitude to self and a judgement of oneself. Self- esteem encompasses beliefs (e.g., I am competent; I am worthy) and emotions (pride, shame). Self-esteem is the positive or negative evaluation of the self”  Demonstrates an optimistic outlook  Sees self as competent and able  Has confidence in ability to interact with others  Uses positive self-talk  Acts with little hesitation  Raises few objections  Asks questions  Articulates skills assertively  Makes eye contact  Is hopeful  Takes reasonable risks 50

51 WORKING ALLIANCE 51 The client and I: Not at all Not much A little Quite a lot A lot  established a climate of trust and comfort in working together  arrived at a goal that is owned by the client  agreed on the action plan steps to help achieve the client’s goal

52 CLIENT ENGAGEMENT: 52 The client: Not at all Not much A little Quite a lot A lot  participated actively in the interview  was focused on achieving results

53 QUESTION #1: APPLICABLE COMMON INDICATORS The following items were shown in this study to have statistically significant connections to employment outcomes: 53 Input IndicatorsProcess Indicators Employability Dimension need/ competence (composite & 5 dimension scores) Responsibility and Access to Supports Personal attributes Working alliance/ client engagement

54 WHAT WE LEARNED  Clients improve their employability (acquire specific knowledge and skills) as a result of services;  Positive change in the level of a client’s personal attributes (self-esteem; self-efficacy; self- awareness) appears to predict both client learning and employment outcomes;  The quality of the working alliance between client and practitioner appears to predict both client learning and employment outcomes;  Client capacity to take personal responsibility and ability to access a support system appear to be among the most influential life circumstances with respect to employment outcomes; 54

55 WHAT WE LEARNED …  Practitioners tend to rate levels of change much more conservatively than client self ratings;  Practitioners overall found rating more complex variables, including client progress, challenging but helpful;  Tracking client progress indicators encouraged reflection on appropriateness of services;  It is important to use multiple sources of information to assess client employability;  Practitioners acquired an increased appreciation for the overall complexity of evaluating the impact of their services; statistical analysis is limited. 55

56 WHAT WE LEARNED…  We saw “relationships”, not causality;  Sample sizes were too small to perform some critical analyses;  Data collection time was too short to track sustained change;  Practitioner training, support and supervision in using the tool would be helpful  The tool can be enhanced AND simplified 56

57 WHAT WE LEARNED….  We CAN track common indicators across settings and contexts  The model has the capacity to begin to answer:  What works for whom under what circumstances?  The model has the capacity to strengthen practitioner competencies and practitioner practice 57

58 AT THE SAME TIME…  Study has shown a pathway to measuring client progress in a meaningful way and provided a line of sight to be able to connect interventions with changes in clients and with successful labour market outcomes  There remains much to do but there are strong indications that the approach can move us from a “what does not work” evaluation culture to a “what works”—a real transformation 58

59 COMMON INDICATORS STUDY: TOOLS  Life Circumstances Checklist  Personal Attribute Checklist  Working Alliance and Client Engagement Checklist  Progress/Change Indicators  Labour Market Outcomes Checklist  Final Then and Now Survey 59

60 DISCUSSION  What stands out for you as the most important findings?  What stands out for you as important to follow up in more detail and depth? 60

61 TRANSFORMATION FROM………………………..TOWARD FF  Data……………………………  What doesn’t work…………..  Working alone  Preoccupation with Employment Outcomes….  Program driven……….  Focus on client numbers…….  One-off’s inform practice…...  Action Oriented  Feedback……….  What works……  Working collaboratively  Preoccupation with Context  Progress and Outcomes  Evidence driven and client need driven  Focus on client change  Patterns inform practice  Action and Reflection Oriented 61

62 EVIDENCE-BASED PRACTICE Review of Research Tools How can I use one or more of these tools to move from gathering data to getting feedback How can I use one of more of these tools to move toward more collaboration with clients? 62

63 REFLECTIVE TOOLS FOR PRACTICE  Initial learning needs survey  Checklist for Employment Assessment Interview  Advice and Information Interview Checklist  Employment Goal and Action Planner  Client Checklists/Tracking Sheets (all employability dimensions)  Life Circumstances Checklist  Personal Attribute Checklist  Working Alliance and Client Engagement Checklist  Progress/Change Indicators  Labour Market Outcomes Checklist  Final Then and Now Survey 63

64 EVIDENCE-BASED PRACTICE  Choose at least one tool you might want to use.  Why did you choose it?  How might will you begin to try the tool out with clients?  What supports if any would you need? 64

65 EVIDENCE BASE: A PRACTITIONER QUESTION AND A CLIENT QUESTION  Who is the Evidence-Based Practitioner?  How does the Evidence-Based Practitioner work differently?  What does it mean for clients – the services they receive and the changes that result from service? 65

66 CHARACTERISTICS OF THE EVIDENCE- BASED PRACTITIONER  Client-Centered (client need, working alliance)  Learning/Change-Centered  Reflective  Positively Uncertain  Experimental 66

67 BEHAVIOUR OF THE EVIDENCE- BASED PRACTITIONER  Intervenes in a systematic manner  Documents what they did  Pays attention to what happened  Track the effects  Looks for associations between what they did and the effects of what they did  Sees their own practice as their data source for predicting client outcomes  Each client is a n=1 experiment  Looks for patterns across time and across clients  Has a focus on trying to make predictions linking interventions and outcomes 67

68 THE REFLECTIVE EVIDENCE-BASED PRACTITIONER  What is reflection?  How do we integrate it into practice? 68

69 REFLECTION IS… “the examination of our own thinking and action” 69

70 REFLECTION REQUIRES…  Consistent time commitment to reflect  The ability to see our “part”  An ability to evaluate processes in a non-judgemental way  A commitment to put learning into action 70

71 REFLECTIVE QUESTIONS  What happened today? What did I do? What caught my attention?  What worked? What energized me?  What were the low points? What frustrated me?  What can I learn from today? What are the implications?  How will I be different? What will I do differently as a result? 71

72 BUILDING EVALUATION INTO PRACTICE WITH MYSELF: AFTER A CLIENT SESSION…  What happened today? What did I do? What caught my attention?  What worked? What energized me?  What were the low points? What frustrated me?  What can I learn from today? What are the implications for my practice?  How will I be different? What will I do differently as a result? 72

73 BUILDING EVALUATION INTO PRACTICE WITH COLLEAGUES  Evidence-Based Team Meetings  What was my biggest success last week?  What contributed to that success and how do I know it was effective?  What was my biggest challenge? What specifically made it so challenging?  What can I learn from this? What else might I do differently?  Is there anything we (individuals/team) want to do more of or differently based on this discussion? 73

74 EVIDENCE-BASED PRACTICE: A POLICY/FUNDING QUESTION  What are the challenges/pitfalls of evidence- based practice to be addressed or avoided?  What are the advantages/benefits of evidence- based practice in terms of policy and funding? 74

75 MULTIPLE EFFECTS OF EVIDENCE-BASED PRACTICE  On client: + and –  Increased motivation  Increased awareness of effectiveness and progress  On CDPs/Employment Counsellors: + and –  Increased effectiveness  Increased confidence  Increased motivation  Increased job satisfaction 75

76 MULTIPLE EFFECTS OF EVIDENCE-BASED PRACTICE  On agency/department: + and –  Increased capacity to plan for and ensure that services meet client needs  Increased capacity to justify expenditures/budget  On Policy Makers/Funders: + and –  Increased rationale for funding decisions  Increased certainty that funding of career services is a strategic investment  On the Career Development Profession: + and –  Increased professionalism, respect and pride 76

77 DON’T WORRY ABOUT GETTING IT RIGHT OR DOING IT ALL  Small steps are great!  Several small steps = a BIG STEP  Do it together – you can learn from and support one another  Share your success stories  With everyone!  In language they can understand  Be persistent  Be proud! 77

78  Our field is amassing a growing body of evidence that our services have IMPACT far beyond employment/entry into training  There is VERY strong interest internationally…and Canada is an undisputed leader!  In times of fiscal restraint, evidence is critical  Evidence based research serves to enhance practice and inform policy – it is a mirror for the field and a lever to influence OUR EVIDENCE BASE

79 THANK YOU! Lynne Bezanson Sareena Hopkins All tools, booklets and resources are available free of charge on the CRWG web site: and/or on the CCDF website at 79


Download ppt "EVIDENCE-BASED RESEARCH IN CANADA  National research agenda to better understand “what works” in career services  3 calls for proposals since 2004 "

Similar presentations


Ads by Google