Career Development: What’s our Proof? Lynne Bezanson & Sareena Hopkins Canadian Career Development Foundation IAEVG Québec City June 2014

Evidence-Based Research in Canada National research agenda to better understand “what works” in career services 3 calls for proposals since 2004 Source of major international envy among Career Development Practitioners and Researchers

Why Evidence-Based Practice? Ethics: Clients deserve interventions that have demonstrated success Confidence: Practitioners should know that a given intervention will result in a predictable outcome Security: Being able to provide evidence of success increases ability to obtain and keep funding for services and secure jobs for CDPs/Employment Counsellors

Why Evidence-Based Practice? Outcome-focused intervention Evidence of client change …contrast with… Client flow Ease of access Practitioner time use It’s easy to measure variables that don’t address client change

Why Evidence-Based Practice? Some successful programs are discontinued because there are no data indicating client change Client flow client change There is a need for Evidence of client change Evidence of impact of service on… Client Society Economy

Session Objectives Highlight recent research projects and what has been learned (what evidence do we have?), with a focus on the tools used and their impact Explore how tools used in the research could be used to support evidence based practice Identify the key characteristics/practices of the Evidence- Based Practitioner Identify action steps to infuse Evidence-Based Practice into work with clients, into professional growth and into office culture

What evidence do you currently collect? What is done with the evidence currently collected?

Evidence-Base: A Research Question 3 recent Canadian research studies Front-line research conducted with employment offices in Alberta, Manitoba, Québec and New Brunswick Consistent and strong results

Questions to keep in mind What did the research do? What did we learn? What was most surprising? What are the implications for practice?

Research Project #1: The Impact of Labour Market Information on career decision making Research Questions If client needs are assessed and clients are given LMI consistent with their needs, To what extent does assistance by a service provider enhance their effective use of LMI? OR To what extent is independent self-help a sufficient process for clients to use LMI effectively?

Method We prepared “guided” LMI packages (take- home booklets and Resource Centre binders) on: Career Decision Making: Know yourself Know the Labour Market Put it all Together Job Search: Check for “Fit” Get Ready Search for Work Get a Job Question for panel: What would you say was the general response of clients to the LMI packages? Why do you think the response of clients was so positive? How were these packages different from what clients usually are given?

Intervention All participants in the study: Received a needs assessment interview & completed an initial, pre-program survey Received an LMI package specific to their identified employability need Were randomly assigned to either a self- directed intervention delivery method or an assisted self-directed intervention delivery method. Were given an orientation to the Resource Centre which they could freely use on their own Completed a pre and then and now post survey Received a cash honorarium and certificate of participation

Intervention (cont) The self-directed group: worked independently for 3 weeks, making use of the materials and the Resource Centre returned in Week 4 for their exit interview

Intervention (cont) The assisted self-help clients received: two additional AIS (Advice and Information) interviews (20-30 minutes) in weeks 1 and 3 focused on helping them understand, interpret and apply the LMI to their own situations and /or access additional LMI returned in Week 4 for their exit interview

What did we Measure General ability to use LMI Knowledge Clear vision of what I want in my career future Knowledge of print and online resources Skill Have effective strategies for keeping myself motivated Have a realistic action plan Personal Attributes Optimism about what lies ahead re meeting my career goals Confidence in my ability to manage future career transitions

Differential Results-Total Score For group as a whole: significant increase in overall ability to use LMI neither intervention was more conducive to one manner of delivery compared to the other Both CDM and JS groups had significant increases across time Change in CDM group was significantly larger than in JS group Participants in the JS group had higher scores than participants in the CDM group, likely indicating that JS participants were more familiar with using LMI before the project began. Participants receiving assistance demonstrated greater change across time than did those in the independent mode Similar pattern for all subscales

Attribution for Change To what extent would you say that any changes in the ratings on the previous pages are a result of your participation in this research project, and to what extent were they a function of other factors in your life? mostly other factors somewhat other factors uncertain somewhat this program mostly this program Program English 3 15 47 80 French 1 25 26 Question to Panel: The results are very positive. Did the results surprise you? You saw the clients at the end of the study when they came to complete the exit surveys and collect their honorarium. What sorts of comments did clients make about the study? What impact do you think the honorarium had on them?

Follow up interviews (1 week and 4 months after intervention) Week 1: 103 clients 4 months: 65 clients LMI overall relevant Clients overall optimistic and confident Clients continued to use LMI but less often

What did clients say? Themes: I was surprised at how much information there was Initially I thought I could not do this; but I found out I could I got more focused on my goal I got clearer about what I wanted to do There was too much information and that made it difficult….very hard to do this alone (Those who were working –85-90% responses in these categories) Did not help very much (only from those still unemployed at 4 months)

What did practitioners say? I never really thought about clients as assisted or self-help—now it is built into all my assessments Now I give more homework and I am more specific. The clients felt more focused and so did I I used to be more maternal and I got more involved than I needed to be. Now I ask clients to be more responsible I found the checklists inhibiting and awkward

What did we learn? What was most surprising? What are the implications for practice?

Applications to Service Delivery Tailored LMI embedded in a learning process results in knowledge and skill acquisition as well as the capacity for self-management LMI appropriate for a client’s specific need (opposed to general LMI) appears to support engagement and action For many clients, a little (or no) professional support is enough Structure and timelines appear to motivate action and a sense of progress Giving clients hands-on tools appears to motivate more than money

Tools/ Supports for the evidence-based practitioner Initial learning needs survey Checklist for Employment Assessment Interview Advice and Information Interview Checklist LMI Resource Guide for Career Decision Making LMI Resource Guide for Job Search

Research Project #2: Assessing the Impact of Career Resource and Supports Across the Employability Dimensions This project builds substantially on the LMI Impact study Self-Help Index and Labour Market Attachment Index

This Study Also began with a needs assessment, but expanded to 4 needs: Career decision making Skills Enhancement Work Search Job Maintenance (and pre-employability) Each client received a tailored resource package, but well beyond LMI to include coaching activities to help them to reflect and personalize.

This Study Clients randomly assigned to one of two treatment conditions: Practitioner assessed (in which they had their needs assessed, were oriented to the appropriate Guide and then worked independently with their Resource Guide for 4 weeks) Practitioner assessed and supported (in which they also worked with their Career Consultant for 4 weeks).  Examined: Differential impact across treatment conditions (independent/supported) Differential impact across labour market attachment & self- help ability (2 new variables)

Research Question If clients are given a comprehensive needs assessment to determine their employability need(s), what is the differential effect of independent and consultant- supported career resources on clients who are weakly attached to the labour market versus those who are strongly attached to the labour market?

Results After just 4 weeks of intervention: Tenfold increase in competencies rated in acceptable range Percentage of employed clients rose from 27% to 45% (a 69% increase) 81% of participants reported that their employment was a “good fit” with their career vision (a 200% increase) 91% attributed the positive changes either partly (40%) or mostly (51%) to the intervention 92% had a clear plan for next steps 98% planned to use the Guides again Of 227 clients who provided work status information, 61 were working part time or full time before the intervention, and 166 were not working (see Table 11 for a breakdown by province). 27% of the sample, therefore, was working. By the end of the intervention, 103, or 45% of the original 227 were working – a 69% increase (see Table 12).

Results Statistically and clinically significant positive impact of career interventions across ALL sub- scores and across ALL employability dimensions – compelling evidence of positive changes in clients as a result of career interventions Although see more positive trends for supported group, differences are not statistically significant

Results Career practitioners care very much about their clients, so much so that they were reluctant to ask clients who might need support to join the study— “in case” they were randomly assigned to the independent group Therefore the sample was too homogeneous to test properly the Self-help and Labour Market attachment indices Career practitioners care very much about their clients…BUT…in the next study, they CANNOT!

What did we learn What was most surprising? What are the implications for practice?

What did the study tell us? Career Development Interventions Work! Clients demonstrated substantial positive changes in skills, knowledge, personal attributes, employment and fit of employment. Many clients can benefit significantly from self-help resources when they are matched to their need and they are “launched” – moreso than we anticipated! This is especially noteworthy given the short intervention period of 4 weeks and the “real-life” setting in which the changes occurred.

The Tools and Guides are “Ready to Go” Virtually no adjustments are needed for use in typical employment centres across Canada. Self-help guides provided after a thorough needs assessment and orientation are effective and can be used as a first line of intervention, saving valuable practitioner time for clients who really need it, or potentially opening a promising perspective on online career services. This finding has considerable practical significance.

Employability Dimensions Study: Tools Employment Goal and Action Planner Client Checklists/Tracking Sheets (all employability dimensions) Comprehensive Coaching Guides (all employability dimensions)

Research Project #3: Common Indicators What had we learned? Where did we need to go next?

Activities that link to outcomes or deliverables Resources available Client needs assessment Client employment history Self-help index Labour market attachment index Activities that link to outcomes or deliverables Generic interventions Working alliance, Specific interventions LMI Booklets Advice and Information Interviews Employability Dimension Coaching Guides Practitioner Coaching using the Guides External Referral PROCESSES INPUTS Client Context Life vision Needs Goals OUTCOMES Learning outcomes Ability to use LMI resources alone and with assistance Ability to use Coaching Resources alone and with assistance Having an employment goal/vision Skills in employability dimension indicators Personal attribute outcomes Changes in intrapersonal variables e.g., attitudes, self-esteem, motivation, etc. Impact Outcomes Changes in the client’s life resulting from application of learning

Unanswered Questions We have data/evidence of positive impacts from specific interventions for certain kinds of clients We do not have data/evidence of positive impacts from employment services overall We do not have a data gathering tool that can be used to gather common data across divergent employment service settings

Therefore the next step: Common Indicators Project Key research questions: What common indicators are applicable across different contexts, different client groups, different agencies and different interventions? What statements of service effectiveness can be made by tracking common indicators? and

IF numbers permit… the ultimate question: What kinds of interventions in what contexts produce what kinds of results?

How we investigated Literature Review Focus Groups Compare the two Accept those in common and most robust

Focus Group Questions: If clients have benefitted from career and employment services, they have experienced some changes in in their lives. They are not in the same position they were when they started. What indicates that clients or their situations have changed as a result of career and employment services? How do you know? What are your clues? What do you do to bring about change? What influences your ability to do this? What about clients? What influences their ability to change? What factors are beyond both of you to control?

Activities that link to outcomes or deliverables Resources available Availability of and access to community resources Availability of and access to employment opportunities and training Practitioner Background and Experience Client Employment History and Potential Client Life Circumstances Activities that link to outcomes or deliverables Generic interventions Working alliance, client engagement. Specific interventions Needs Assessment Employment Counselling Resources/Guides Goal setting and Action Planning Programs & Workshops Progress Tracking External Referral PROCESSES INPUTS Client Context Life vision Needs Goals OUTCOMES client change Learning outcomes Changes in knowledge and skills linked to the program or intervention used Progress Indicators End Result Indicators Personal attribute outcomes Changes in intrapersonal variables e.g., self-confidence, self-esteem etc.. Progress Indicators and Impact Outcomes Changes in the client’s life resulting from application of learning

Practitioner/Research Partner Tasks Recruit clients into the study (NO criteria); Follow all research protocols Use PRIME each time you see a client to assess and record PROGRESS/CHANGE in: Goals and Action Plans Employability Learning Needs/Changes Personal Attribute Needs/Changes Life Circumstance Needs/Changes Working Alliance and Client Engagement

Practitioner/Research Partner Tasks Work with each client for 6 weeks following Assessment interview and client entry into the research At exit interview also gather data on…

Practitioner/Research Partner Tasks Labour Market Outcomes: Employment or Training Status Fit of Employment or Training with skills, qualifications, vision Adequacy of salary Linked to future opportunity Ongoing Progress Evidence: Actions Underway to move forward: Have clients complete then and now questionnaire

Pre-Employability/Job Readiness Client needs assistance to : Not at all Not much A little Quite a lot A lot Identify and clarify future direction (e.g., training, education, employment or change in life circumstances goal)   Identify personal strengths/resources that support future direction (e.g., training, education, employment or change in life circumstances goal) Resolve specific challenges/vulnerabilities that may impact on future direction (e.g., mortgage, public transit, day care etc.)

Life Circumstances Indicators of challenges in client life circumstances are in evidence: Not at all Not much A little Quite a lot A lot Improved housing is needed   Improved transportation is needed Improved capacity to work and/or study is needed Increase in sense of responsibility for own choices and behaviours is needed Increase in ability to set short and long term goals is needed Increase in understanding expectations and demands of employers is needed Reduction in destructive behavior is needed Following medical and medication protocols is needed Increased openness to change is needed Increased access to constructive and positive support systems is needed Improved relationships with family and friends are needed Increased trust in other people is needed Other: _________________________________

Personal Attributes Example Client needs assistance in the following areas: Not at all Not much A little Quite a lot A lot Ability to Self-Manage: More info   Developing stronger Self-Esteem: Improving sense of Well-Being: i.e.: Developing stronger Self-Efficacy: i.e.: Increasing Self-Awareness: Other (Please specify): ________________________________ 

Self-Esteem Attribute more info: Self-Esteem:   “a person’s overall emotional evaluation of his or her own worth, an attitude to self and a judgement of oneself. Self- esteem encompasses beliefs (e.g., I am competent; I am worthy) and emotions (pride, shame). Self-esteem is the positive or negative evaluation of the self” Demonstrates an optimistic outlook Sees self as competent and able Has confidence in ability to interact with others Uses positive self-talk Acts with little hesitation Raises few objections Asks questions Articulates skills assertively Makes eye contact Is hopeful Takes reasonable risks

Working Alliance The client and I: Not at all Not much A little Quite a lot A lot established a climate of trust and comfort in working together   arrived at a goal that is owned by the client agreed on the action plan steps to help achieve the client’s goal

Client Engagement: The client: Not at all Not much A little Quite a lot A lot participated actively in the interview   was focused on achieving results

Question #1: Applicable Common Indicators The following items were shown in this study to have statistically significant connections to employment outcomes: Input Indicators Process Indicators Employability Dimension need/ competence (composite & 5 dimension scores) Responsibility and Access to Supports Personal attributes Working alliance/ client engagement  

What we learned Clients improve their employability (acquire specific knowledge and skills) as a result of services; Positive change in the level of a client’s personal attributes (self-esteem; self-efficacy; self- awareness) appears to predict both client learning and employment outcomes; The quality of the working alliance between client and practitioner appears to predict both client learning and employment outcomes; Client capacity to take personal responsibility and ability to access a support system appear to be among the most influential life circumstances with respect to employment outcomes;

What we learned … Practitioners tend to rate levels of change much more conservatively than client self ratings; Practitioners overall found rating more complex variables, including client progress, challenging but helpful; Tracking client progress indicators encouraged reflection on appropriateness of services; It is important to use multiple sources of information to assess client employability; Practitioners acquired an increased appreciation for the overall complexity of evaluating the impact of their services; statistical analysis is limited.

What we learned… We saw “relationships”, not causality; Sample sizes were too small to perform some critical analyses; Data collection time was too short to track sustained change; Practitioner training, support and supervision in using the tool would be helpful The tool can be enhanced AND simplified

What we learned…. We CAN track common indicators across settings and contexts The model has the capacity to begin to answer: What works for whom under what circumstances? The model has the capacity to strengthen practitioner competencies and practitioner practice

At the same time… Study has shown a pathway to measuring client progress in a meaningful way and provided a line of sight to be able to connect interventions with changes in clients and with successful labour market outcomes There remains much to do but there are strong indications that the approach can move us from a “what does not work” evaluation culture to a “what works”—a real transformation

Common Indicators Study: Tools Life Circumstances Checklist Personal Attribute Checklist Working Alliance and Client Engagement Checklist Progress/Change Indicators Labour Market Outcomes Checklist Final Then and Now Survey

Discussion What stands out for you as the most important findings? What stands out for you as important to follow up in more detail and depth?

Transformation From………………………..toward Data…………………………… What doesn’t work………….. Working alone Preoccupation with Employment Outcomes…. Program driven………. Focus on client numbers……. One-off’s inform practice…... Action Oriented FF Feedback………. What works…… Working collaboratively Preoccupation with Context Progress and Outcomes Evidence driven and client need driven Focus on client change Patterns inform practice Action and Reflection Oriented

Evidence-based Practice Review of Research Tools How can I use one or more of these tools to move from gathering data to getting feedback How can I use one of more of these tools to move toward more collaboration with clients?

Reflective Tools for Practice Initial learning needs survey Checklist for Employment Assessment Interview Advice and Information Interview Checklist Employment Goal and Action Planner Client Checklists/Tracking Sheets (all employability dimensions) Life Circumstances Checklist Personal Attribute Checklist Working Alliance and Client Engagement Checklist Progress/Change Indicators Labour Market Outcomes Checklist Final Then and Now Survey

Evidence-based Practice Choose at least one tool you might want to use. Why did you choose it? How might will you begin to try the tool out with clients? What supports if any would you need?

Evidence Base: A Practitioner Question and a Client Question Who is the Evidence-Based Practitioner? How does the Evidence-Based Practitioner work differently? What does it mean for clients – the services they receive and the changes that result from service?

Characteristics of the Evidence-Based Practitioner Client-Centered (client need, working alliance) Learning/Change-Centered Reflective Positively Uncertain Experimental

Behaviour of the Evidence-Based Practitioner Intervenes in a systematic manner Documents what they did Pays attention to what happened Track the effects Looks for associations between what they did and the effects of what they did Sees their own practice as their data source for predicting client outcomes Each client is a n=1 experiment Looks for patterns across time and across clients Has a focus on trying to make predictions linking interventions and outcomes

The Reflective Evidence-Based Practitioner What is reflection? How do we integrate it into practice?

“the examination of our own thinking and action” Reflection is… “the examination of our own thinking and action”

Reflection requires… Consistent time commitment to reflect The ability to see our “part” An ability to evaluate processes in a non-judgemental way A commitment to put learning into action

Reflective questions What happened today? What did I do? What caught my attention? What worked? What energized me? What were the low points? What frustrated me? What can I learn from today? What are the implications? How will I be different? What will I do differently as a result?

Building Evaluation into Practice with Myself: After a client session… What happened today? What did I do? What caught my attention? What worked? What energized me? What were the low points? What frustrated me? What can I learn from today? What are the implications for my practice? How will I be different? What will I do differently as a result?

Building Evaluation into Practice with Colleagues Evidence-Based Team Meetings What was my biggest success last week? What contributed to that success and how do I know it was effective? What was my biggest challenge? What specifically made it so challenging? What can I learn from this? What else might I do differently? Is there anything we (individuals/team) want to do more of or differently based on this discussion?

Evidence-Based Practice: A Policy/Funding Question What are the challenges/pitfalls of evidence- based practice to be addressed or avoided? What are the advantages/benefits of evidence- based practice in terms of policy and funding?

Multiple Effects of Evidence-Based Practice On client: + and – Increased motivation Increased awareness of effectiveness and progress On CDPs/Employment Counsellors: + and – Increased effectiveness Increased confidence Increased job satisfaction

Multiple Effects of Evidence-Based Practice On agency/department: + and – Increased capacity to plan for and ensure that services meet client needs Increased capacity to justify expenditures/budget On Policy Makers/Funders: + and – Increased rationale for funding decisions Increased certainty that funding of career services is a strategic investment On the Career Development Profession: + and – Increased professionalism, respect and pride

Don’t worry about getting it right or doing it all Small steps are great! Several small steps = a BIG STEP Do it together – you can learn from and support one another Share your success stories With everyone! In language they can understand Be persistent Be proud!

Our Evidence Base Our field is amassing a growing body of evidence that our services have IMPACT far beyond employment/entry into training There is VERY strong interest internationally…and Canada is an undisputed leader! In times of fiscal restraint, evidence is critical Evidence based research serves to enhance practice and inform policy – it is a mirror for the field and a lever to influence

Thank you! Lynne Bezanson (l.bezanson@ccdf.ca) Sareena Hopkins (s.hopkins@ccdf.ca) All tools, booklets and resources are available free of charge on the CRWG web site: www.crwg-gdrc.ca/ and/or on the CCDF website at www.ccdf.ca