Presentation on theme: "Career Development: What’s our Proof?"— Presentation transcript:
1 Career Development: What’s our Proof? Lynne Bezanson & Sareena HopkinsCanadian Career Development Foundation IAEVG Québec CityJune 2014
2 Evidence-Based Research in Canada National research agenda to better understand “what works” in career services3 calls for proposals since 2004Source of major international envy among Career Development Practitioners and Researchers
3 Why Evidence-Based Practice? Ethics: Clients deserve interventions that have demonstrated successConfidence: Practitioners should know that a given intervention will result in a predictable outcomeSecurity: Being able to provide evidence of success increases ability to obtain and keep funding for services and secure jobs for CDPs/Employment Counsellors
4 Why Evidence-Based Practice? Outcome-focused interventionEvidence of client change…contrast with…Client flowEase of accessPractitioner time useIt’s easy to measure variables that don’t address client change
5 Why Evidence-Based Practice? Some successful programs are discontinued because there are no data indicating client changeClient flow client changeThere is a need forEvidence of client changeEvidence of impact of service on…ClientSocietyEconomy
7 Session ObjectivesHighlight recent research projects and what has been learned (what evidence do we have?), with a focus on the tools used and their impactExplore how tools used in the research could be used to support evidence based practiceIdentify the key characteristics/practices of the Evidence- Based PractitionerIdentify action steps to infuse Evidence-Based Practice into work with clients, into professional growth and into office culture
8 What evidence do you currently collect? What is done with the evidence currently collected?
9 Evidence-Base: A Research Question 3 recent Canadian research studiesFront-line research conducted with employment offices in Alberta, Manitoba, Québec and New BrunswickConsistent and strong results
10 Questions to keep in mind What did the research do?What did we learn?What was most surprising?What are the implications for practice?
11 Research Project #1: The Impact of Labour Market Information on career decision making Research QuestionsIf client needs are assessed and clients are given LMI consistent with their needs,To what extent does assistance by a service provider enhance their effective use of LMI?ORTo what extent is independent self-help a sufficient process for clients to use LMI effectively?
12 MethodWe prepared “guided” LMI packages (take- home booklets and Resource Centre binders) on:Career Decision Making:Know yourselfKnow the Labour MarketPut it all TogetherJob Search:Check for “Fit”Get ReadySearch for WorkGet a JobQuestion for panel:What would you say was the general response of clients to the LMI packages?Why do you think the response of clients was so positive?How were these packages different from what clients usually are given?
13 Intervention All participants in the study: Received a needs assessment interview & completed an initial, pre-program surveyReceived an LMI package specific to their identified employability needWere randomly assigned to either a self- directed intervention delivery method or an assisted self-directed intervention delivery method.Were given an orientation to the Resource Centre which they could freely use on their ownCompleted a pre and then and now post surveyReceived a cash honorarium and certificate of participation
14 Intervention (cont) The self-directed group: worked independently for 3 weeks, making use of the materials and the Resource Centrereturned in Week 4 for their exit interview
15 Intervention (cont) The assisted self-help clients received: two additional AIS (Advice and Information) interviews (20-30 minutes) in weeks 1 and 3 focused on helping them understand, interpret and apply the LMI to their own situations and /or access additional LMIreturned in Week 4 for their exit interview
16 What did we Measure General ability to use LMI Knowledge Clear vision of what I want in my career futureKnowledge of print and online resourcesSkillHave effective strategies for keeping myself motivatedHave a realistic action planPersonal AttributesOptimism about what lies ahead re meeting my career goalsConfidence in my ability to manage future career transitions
17 Differential Results-Total Score For group as a whole:significant increase in overall ability to use LMIneither intervention was more conducive to one manner of delivery compared to the otherBoth CDM and JS groups had significant increases across timeChange in CDM group was significantly larger than in JS groupParticipants in the JS group had higher scores than participants in the CDM group, likely indicating that JS participants were more familiar with using LMI before the project began.Participants receiving assistance demonstrated greater change across time than did those in the independent mode Similar pattern for all subscales
18 Attribution for Change To what extent would you say that any changes in the ratings on the previous pages are a result of your participation in this research project, and to what extent were they a function of other factors in your life?mostly other factorssomewhat other factorsuncertainsomewhat this programmostly this programProgramEnglish3154780French12526Question to Panel:The results are very positive. Did the results surprise you?You saw the clients at the end of the study when they came to complete the exit surveys and collect their honorarium. What sorts of comments did clients make about the study? What impact do you think the honorarium had on them?
19 Follow up interviews (1 week and 4 months after intervention) Week 1: 103 clients4 months: 65 clientsLMI overall relevantClients overall optimistic and confidentClients continued to use LMI but less often
20 What did clients say? Themes: I was surprised at how much information there wasInitially I thought I could not do this; but I found out I couldI got more focused on my goalI got clearer about what I wanted to doThere was too much information and that made it difficult….very hard to do this alone(Those who were working –85-90% responses in these categories)Did not help very much (only from those still unemployed at 4 months)
21 What did practitioners say? I never really thought about clients as assisted or self-help—now it is built into all my assessmentsNow I give more homework and I am more specific. The clients felt more focused and so did II used to be more maternal and I got more involved than I needed to be. Now I ask clients to be more responsibleI found the checklists inhibiting and awkward
22 What did we learn?What was most surprising?What are the implications for practice?
23 Applications to Service Delivery Tailored LMI embedded in a learning process results in knowledge and skill acquisition as well as the capacity for self-managementLMI appropriate for a client’s specific need (opposed to general LMI) appears to support engagement and actionFor many clients, a little (or no) professional support is enoughStructure and timelines appear to motivate action and a sense of progressGiving clients hands-on tools appears to motivate more than money
24 Tools/ Supports for the evidence-based practitioner Initial learning needs surveyChecklist for Employment Assessment InterviewAdvice and Information Interview ChecklistLMI Resource Guide for Career Decision MakingLMI Resource Guide for Job Search
25 Research Project #2: Assessing the Impact of Career Resource and Supports Across the Employability DimensionsThis project builds substantially on the LMI Impact studySelf-Help Index and Labour Market Attachment Index
26 This StudyAlso began with a needs assessment, but expanded to 4 needs:Career decision makingSkills EnhancementWork SearchJob Maintenance (and pre-employability)Each client received a tailored resource package, but well beyond LMI to include coaching activities to help them to reflect and personalize.
27 This StudyClients randomly assigned to one of two treatment conditions:Practitioner assessed (in which they had their needs assessed, were oriented to the appropriate Guide and then worked independently with their Resource Guide for 4 weeks)Practitioner assessed and supported (in which they also worked with their Career Consultant for 4 weeks). Examined:Differential impact across treatment conditions (independent/supported)Differential impact across labour market attachment & self- help ability (2 new variables)
28 Research QuestionIf clients are given a comprehensive needs assessment to determine their employability need(s), what is the differential effect of independent and consultant- supported career resources on clients who are weakly attached to the labour market versus those who are strongly attached to the labour market?
29 Results After just 4 weeks of intervention: Tenfold increase in competencies rated in acceptable rangePercentage of employed clients rose from 27% to 45% (a 69% increase)81% of participants reported that their employment was a “good fit” with their career vision (a 200% increase)91% attributed the positive changes either partly (40%) or mostly (51%) to the intervention92% had a clear plan for next steps98% planned to use the Guides againOf 227 clients who provided work status information, 61 were working part time or full time before the intervention, and 166 were not working (see Table 11 for a breakdown by province). 27% of the sample, therefore, was working. By the end of the intervention, 103, or 45% of the original 227 were working – a 69% increase (see Table 12).
30 ResultsStatistically and clinically significant positive impact of career interventions across ALL sub- scores and across ALL employability dimensions – compelling evidence of positive changes in clients as a result of career interventionsAlthough see more positive trends for supported group, differences are not statistically significant
31 ResultsCareer practitioners care very much about their clients, so much so that they were reluctant to ask clients who might need support to join the study— “in case” they were randomly assigned to the independent groupTherefore the sample was too homogeneous to test properly the Self-help and Labour Market attachment indicesCareer practitioners care very much about their clients…BUT…in the next study, they CANNOT!
32 What did we learnWhat was most surprising?What are the implications for practice?
33 What did the study tell us? Career Development Interventions Work!Clients demonstrated substantial positive changes in skills, knowledge, personal attributes, employment and fit of employment.Many clients can benefit significantly from self-help resources when they are matched to their need and they are “launched” – moreso than we anticipated!This is especially noteworthy given the short intervention period of 4 weeks and the “real-life” setting in which the changes occurred.
34 The Tools and Guides are “Ready to Go” Virtually no adjustments are needed for use in typical employment centres across Canada. Self-help guides provided after a thorough needs assessment and orientation are effective and can be used as a first line of intervention, saving valuable practitioner time for clients who really need it, or potentially opening a promising perspective on online career services. This finding has considerable practical significance.
36 Research Project #3: Common Indicators What had we learned?Where did we need to go next?
37 Activities that link to outcomes or deliverables Resources availableClient needs assessmentClient employment historySelf-help indexLabour market attachment indexActivities that link to outcomes or deliverablesGeneric interventionsWorking alliance,Specific interventionsLMI BookletsAdvice and Information InterviewsEmployability Dimension Coaching GuidesPractitioner Coaching using the GuidesExternal ReferralPROCESSESINPUTSClientContextLife visionNeedsGoalsOUTCOMESLearning outcomesAbility to use LMI resources alone and with assistanceAbility to use Coaching Resources alone and with assistanceHaving an employment goal/visionSkills in employability dimension indicatorsPersonal attribute outcomesChanges in intrapersonal variables e.g., attitudes, self-esteem, motivation, etc.Impact OutcomesChanges in the client’s life resulting from application of learning
38 Unanswered QuestionsWe have data/evidence of positive impacts from specific interventions for certain kinds of clientsWe do not have data/evidence of positive impacts from employment services overallWe do not have a data gathering tool that can be used to gather common data across divergent employment service settings
39 Therefore the next step: Common Indicators Project Key research questions:What common indicators are applicable across different contexts, different client groups, different agencies and different interventions?What statements of service effectiveness can be made by tracking common indicators?and
40 IF numbers permit… the ultimate question: What kinds of interventions in what contexts produce what kinds of results?
41 How we investigated Literature Review Focus Groups Compare the two Accept those in common and most robust
42 Focus Group Questions: If clients have benefitted from career and employment services, they have experienced some changes in in their lives. They are not in the same position they were when they started. What indicates that clients or their situations have changed as a result of career and employment services?How do you know? What are your clues?What do you do to bring about change? What influences your ability to do this?What about clients? What influences their ability to change?What factors are beyond both of you to control?
43 Activities that link to outcomes or deliverables Resources availableAvailability of and access to community resources Availability of and access to employment opportunities and trainingPractitioner Background and ExperienceClient Employment History and PotentialClient Life CircumstancesActivities that link to outcomes or deliverablesGeneric interventionsWorking alliance, client engagement.Specific interventionsNeeds AssessmentEmployment CounsellingResources/GuidesGoal setting and Action PlanningPrograms & WorkshopsProgress TrackingExternal ReferralPROCESSESINPUTSClientContextLife visionNeedsGoalsOUTCOMESclient changeLearning outcomesChanges in knowledge and skills linked to the program or intervention usedProgress Indicators End Result IndicatorsPersonal attribute outcomesChanges in intrapersonal variables e.g., self-confidence, self-esteem etc..Progress Indicators and Impact OutcomesChanges in the client’s life resulting from application of learning
44 Practitioner/Research Partner Tasks Recruit clients into the study (NO criteria);Follow all research protocolsUse PRIME each time you see a client to assess and record PROGRESS/CHANGE in:Goals and Action PlansEmployability Learning Needs/ChangesPersonal Attribute Needs/ChangesLife Circumstance Needs/ChangesWorking Alliance and Client Engagement
45 Practitioner/Research Partner Tasks Work with each client for 6 weeks following Assessment interview and client entry into the researchAt exit interview also gather data on…
46 Practitioner/Research Partner Tasks Labour Market Outcomes:Employment or Training StatusFit of Employment or Training with skills, qualifications, visionAdequacy of salaryLinked to future opportunityOngoing Progress Evidence: Actions Underway to move forward:Have clients complete then and now questionnaire
47 Pre-Employability/Job Readiness Client needs assistance to :Not at allNot muchA littleQuite a lotA lotIdentify and clarify future direction (e.g., training, education, employment or change in life circumstances goal)Identify personal strengths/resources that support future direction (e.g., training, education, employment or change in life circumstances goal)Resolve specific challenges/vulnerabilities that may impact on future direction (e.g., mortgage, public transit, day care etc.)
48 Life CircumstancesIndicators of challenges in client life circumstances are in evidence:Not at allNot muchA littleQuite a lotA lotImproved housing is neededImproved transportation is neededImproved capacity to work and/or study is neededIncrease in sense of responsibility for own choices and behaviours is neededIncrease in ability to set short and long term goals is neededIncrease in understanding expectations and demands of employers is neededReduction in destructive behavior is neededFollowing medical and medication protocols is neededIncreased openness to change is neededIncreased access to constructive and positive support systems is neededImproved relationships with family and friends are neededIncreased trust in other people is neededOther: _________________________________
49 Personal Attributes Example Client needs assistance in the following areas:Not at allNot muchA littleQuite a lotA lotAbility to Self-Manage:More infoDeveloping stronger Self-Esteem:Improving sense of Well-Being: i.e.:Developing stronger Self-Efficacy: i.e.:Increasing Self-Awareness:Other (Please specify): ________________________________
50 Self-Esteem Attribute more info: Self-Esteem:“a person’s overall emotional evaluation of his or her own worth, an attitude to self and a judgement of oneself. Self- esteem encompasses beliefs (e.g., I am competent; I am worthy) and emotions (pride, shame). Self-esteem is the positive or negative evaluation of the self”Demonstrates an optimistic outlookSees self as competent and ableHas confidence in ability to interact with othersUses positive self-talkActs with little hesitationRaises few objectionsAsks questionsArticulates skills assertivelyMakes eye contactIs hopefulTakes reasonable risks
51 Working Alliance The client and I: Not at all Not much A little Quite a lotA lotestablished a climate of trust and comfort in working togetherarrived at a goal that is owned by the clientagreed on the action plan steps to help achieve the client’s goal
52 Client Engagement: The client: Not at all Not much A little Quite a lotA lotparticipated actively in the interviewwas focused on achieving results
53 Question #1: Applicable Common Indicators The following items were shown in this study to have statistically significant connections to employment outcomes:Input IndicatorsProcess IndicatorsEmployability Dimension need/ competence (composite & 5 dimension scores)Responsibility and Access to SupportsPersonal attributesWorking alliance/ client engagement
54 What we learnedClients improve their employability (acquire specific knowledge and skills) as a result of services;Positive change in the level of a client’s personal attributes (self-esteem; self-efficacy; self- awareness) appears to predict both client learning and employment outcomes;The quality of the working alliance between client and practitioner appears to predict both client learning and employment outcomes;Client capacity to take personal responsibility and ability to access a support system appear to be among the most influential life circumstances with respect to employment outcomes;
55 What we learned …Practitioners tend to rate levels of change much more conservatively than client self ratings;Practitioners overall found rating more complex variables, including client progress, challenging but helpful;Tracking client progress indicators encouraged reflection on appropriateness of services;It is important to use multiple sources of information to assess client employability;Practitioners acquired an increased appreciation for the overall complexity of evaluating the impact of their services; statistical analysis is limited.
56 What we learned… We saw “relationships”, not causality; Sample sizes were too small to perform some critical analyses;Data collection time was too short to track sustained change;Practitioner training, support and supervision in using the tool would be helpfulThe tool can be enhanced AND simplified
57 What we learned….We CAN track common indicators across settings and contextsThe model has the capacity to begin to answer:What works for whom under what circumstances?The model has the capacity to strengthen practitioner competencies and practitioner practice
58 At the same time…Study has shown a pathway to measuring client progress in a meaningful way and provided a line of sight to be able to connect interventions with changes in clients and with successful labour market outcomesThere remains much to do but there are strong indications that the approach can move us from a “what does not work” evaluation culture to a “what works”—a real transformation
59 Common Indicators Study: Tools Life Circumstances ChecklistPersonal Attribute ChecklistWorking Alliance and Client Engagement ChecklistProgress/Change IndicatorsLabour Market Outcomes ChecklistFinal Then and Now Survey
60 Discussion What stands out for you as the most important findings? What stands out for you as important to follow up in more detail and depth?
61 Transformation From………………………..toward Data……………………………What doesn’t work…………..Working alonePreoccupation with Employment Outcomes….Program driven……….Focus on client numbers…….One-off’s inform practice…...Action OrientedFFFeedback……….What works……Working collaborativelyPreoccupation with ContextProgress and OutcomesEvidence driven and client need drivenFocus on client changePatterns inform practiceAction and Reflection Oriented
62 Evidence-based Practice Review of Research ToolsHow can I use one or more of these tools to move from gathering data to getting feedbackHow can I use one of more of these tools to move toward more collaboration with clients?
63 Reflective Tools for Practice Initial learning needs surveyChecklist for Employment Assessment InterviewAdvice and Information Interview ChecklistEmployment Goal and Action PlannerClient Checklists/Tracking Sheets (all employability dimensions)Life Circumstances ChecklistPersonal Attribute ChecklistWorking Alliance and Client Engagement ChecklistProgress/Change IndicatorsLabour Market Outcomes ChecklistFinal Then and Now Survey
64 Evidence-based Practice Choose at least one tool you might want to use.Why did you choose it?How might will you begin to try the tool out with clients?What supports if any would you need?
65 Evidence Base: A Practitioner Question and a Client Question Who is the Evidence-Based Practitioner?How does the Evidence-Based Practitioner work differently?What does it mean for clients – the services they receive and the changes that result from service?
66 Characteristics of the Evidence-Based Practitioner Client-Centered (client need, working alliance)Learning/Change-CenteredReflectivePositively UncertainExperimental
67 Behaviour of the Evidence-Based Practitioner Intervenes in a systematic mannerDocuments what they didPays attention to what happenedTrack the effectsLooks for associations between what they did and the effects of what they didSees their own practice as their data source for predicting client outcomesEach client is a n=1 experimentLooks for patterns across time and across clientsHas a focus on trying to make predictions linking interventions and outcomes
68 The Reflective Evidence-Based Practitioner What is reflection?How do we integrate it into practice?
69 “the examination of our own thinking and action” Reflection is…“the examination of our own thinking and action”
70 Reflection requires… Consistent time commitment to reflect The ability to see our “part”An ability to evaluate processes in a non-judgemental wayA commitment to put learning into action
71 Reflective questionsWhat happened today? What did I do? What caught my attention?What worked? What energized me?What were the low points? What frustrated me?What can I learn from today? What are the implications?How will I be different? What will I do differently as a result?
72 Building Evaluation into Practice with Myself: After a client session… What happened today? What did I do? What caught my attention?What worked? What energized me?What were the low points? What frustrated me?What can I learn from today? What are the implications for my practice?How will I be different? What will I do differently as a result?
73 Building Evaluation into Practice with Colleagues Evidence-Based Team MeetingsWhat was my biggest success last week?What contributed to that success and how do I know it was effective?What was my biggest challenge? What specifically made it so challenging?What can I learn from this? What else might I do differently?Is there anything we (individuals/team) want to do more of or differently based on this discussion?
74 Evidence-Based Practice: A Policy/Funding Question What are the challenges/pitfalls of evidence- based practice to be addressed or avoided?What are the advantages/benefits of evidence- based practice in terms of policy and funding?
75 Multiple Effects of Evidence-Based Practice On client: + and –Increased motivationIncreased awareness of effectiveness and progressOn CDPs/Employment Counsellors: + and –Increased effectivenessIncreased confidenceIncreased job satisfaction
76 Multiple Effects of Evidence-Based Practice On agency/department: + and –Increased capacity to plan for and ensure that services meet client needsIncreased capacity to justify expenditures/budgetOn Policy Makers/Funders: + and –Increased rationale for funding decisionsIncreased certainty that funding of career services is a strategic investmentOn the Career Development Profession: + and –Increased professionalism, respect and pride
77 Don’t worry about getting it right or doing it all Small steps are great!Several small steps = a BIG STEPDo it together – you can learn from and support one anotherShare your success storiesWith everyone!In language they can understandBe persistentBe proud!
78 Our Evidence BaseOur field is amassing a growing body of evidence that our services have IMPACT far beyond employment/entry into trainingThere is VERY strong interest internationally…and Canada is an undisputed leader!In times of fiscal restraint, evidence is criticalEvidence based research serves to enhance practice and inform policy – it is a mirror for the field and a lever to influence
79 Thank you!Lynne Bezanson Sareena Hopkins All tools, booklets and resources are available free of charge on the CRWG web site: and/or on the CCDF website at