1. Career Development: What’s our Proof?
Lynne Bezanson & Sareena Hopkins
Canadian Career Development Foundation IAEVG Québec City
June 2014

2. Evidence-Based Research in Canada
National research agenda to better understand “what works” in career services
3 calls for proposals since 2004
Source of major international envy among Career Development Practitioners and Researchers

3. Why Evidence-Based Practice?
Ethics: Clients deserve interventions that have demonstrated success
Confidence: Practitioners should know that a given intervention will result in a predictable outcome
Security: Being able to provide evidence of success increases ability to obtain and keep funding for services and secure jobs for CDPs/Employment Counsellors

4. Why Evidence-Based Practice?
Outcome-focused intervention
Evidence of client change
…contrast with…
Client flow
Ease of access
Practitioner time use
It’s easy to measure variables that don’t address client change

5. Why Evidence-Based Practice?
Some successful programs are discontinued because there are no data indicating client change
Client flow client change
There is a need for
Evidence of client change
Evidence of impact of service on…
Client
Society
Economy

7. Session Objectives
Highlight recent research projects and what has been learned (what evidence do we have?), with a focus on the tools used and their impact
Explore how tools used in the research could be used to support evidence based practice
Identify the key characteristics/practices of the Evidence- Based Practitioner
Identify action steps to infuse Evidence-Based Practice into work with clients, into professional growth and into office culture

8. What evidence do you currently collect?
What is done with the evidence currently collected?

9. Evidence-Base: A Research Question
3 recent Canadian research studies
Front-line research conducted with employment offices in Alberta, Manitoba, Québec and New Brunswick
Consistent and strong results

10. Questions to keep in mind
What did the research do?
What did we learn?
What was most surprising?
What are the implications for practice?

11. Research Project #1: The Impact of Labour Market Information on career decision making
Research Questions
If client needs are assessed and clients are given LMI consistent with their needs,
To what extent does assistance by a service provider enhance their effective use of LMI? OR
To what extent is independent self-help a sufficient process for clients to use LMI effectively?

12. Method
We prepared “guided” LMI packages (take- home booklets and Resource Centre binders) on:
Career Decision Making:
Know yourself
Know the Labour Market
Put it all Together
Job Search:
Check for “Fit”
Get Ready
Search for Work
Get a Job
Question for panel:
What would you say was the general response of clients to the LMI packages?
Why do you think the response of clients was so positive?
How were these packages different from what clients usually are given?

13. Intervention All participants in the study:
Received a needs assessment interview & completed an initial, pre-program survey
Received an LMI package specific to their identified employability need
Were randomly assigned to either a self- directed intervention delivery method or an assisted self-directed intervention delivery method.
Were given an orientation to the Resource Centre which they could freely use on their own
Completed a pre and then and now post survey
Received a cash honorarium and certificate of participation

14. Intervention (cont) The self-directed group:
worked independently for 3 weeks, making use of the materials and the Resource Centre
returned in Week 4 for their exit interview

15. Intervention (cont) The assisted self-help clients received:
two additional AIS (Advice and Information) interviews (20-30 minutes) in weeks 1 and 3 focused on helping them understand, interpret and apply the LMI to their own situations and /or access additional LMI
returned in Week 4 for their exit interview

16. What did we Measure General ability to use LMI Knowledge
Clear vision of what I want in my career future
Knowledge of print and online resources
Skill
Have effective strategies for keeping myself motivated
Have a realistic action plan
Personal Attributes
Optimism about what lies ahead re meeting my career goals
Confidence in my ability to manage future career transitions

17. Differential Results-Total Score
For group as a whole:
significant increase in overall ability to use LMI
neither intervention was more conducive to one manner of delivery compared to the other
Both CDM and JS groups had significant increases across time
Change in CDM group was significantly larger than in JS group
Participants in the JS group had higher scores than participants in the CDM group, likely indicating that JS participants were more familiar with using LMI before the project began.
Participants receiving assistance demonstrated greater change across time than did those in the independent mode Similar pattern for all subscales

18. Attribution for Change
To what extent would you say that any changes in the ratings on the previous pages are a result of your participation in this research project, and to what extent were they a function of other factors in your life?
mostly other factors
somewhat other factors
uncertain
somewhat this program
mostly this program
Program
English 3 15 47 80
French 1 25 26
Question to Panel:
The results are very positive. Did the results surprise you?
You saw the clients at the end of the study when they came to complete the exit surveys and collect their honorarium. What sorts of comments did clients make about the study? What impact do you think the honorarium had on them?

19. Follow up interviews (1 week and 4 months after intervention)
Week 1: 103 clients
4 months: 65 clients
LMI overall relevant
Clients overall optimistic and confident
Clients continued to use LMI but less often

20. What did clients say? Themes:
I was surprised at how much information there was
Initially I thought I could not do this; but I found out I could
I got more focused on my goal
I got clearer about what I wanted to do
There was too much information and that made it difficult….very hard to do this alone
(Those who were working –85-90% responses in these categories)
Did not help very much (only from those still unemployed at 4 months)

21. What did practitioners say?
I never really thought about clients as assisted or self-help—now it is built into all my assessments
Now I give more homework and I am more specific. The clients felt more focused and so did I
I used to be more maternal and I got more involved than I needed to be. Now I ask clients to be more responsible
I found the checklists inhibiting and awkward

22. What did we learn?
What was most surprising?
What are the implications for practice?

23. Applications to Service Delivery
Tailored LMI embedded in a learning process results in knowledge and skill acquisition as well as the capacity for self-management
LMI appropriate for a client’s specific need (opposed to general LMI) appears to support engagement and action
For many clients, a little (or no) professional support is enough
Structure and timelines appear to motivate action and a sense of progress
Giving clients hands-on tools appears to motivate more than money

24. Tools/ Supports for the evidence-based practitioner
Initial learning needs survey
Checklist for Employment Assessment Interview
Advice and Information Interview Checklist
LMI Resource Guide for Career Decision Making
LMI Resource Guide for Job Search

25. Research Project #2: Assessing the Impact of Career Resource and Supports Across the Employability Dimensions
This project builds substantially on the LMI Impact study
Self-Help Index and Labour Market Attachment Index

26. This Study
Also began with a needs assessment, but expanded to 4 needs:
Career decision making
Skills Enhancement
Work Search
Job Maintenance (and pre-employability)
Each client received a tailored resource package, but well beyond LMI to include coaching activities to help them to reflect and personalize.

27. This Study
Clients randomly assigned to one of two treatment conditions:
Practitioner assessed (in which they had their needs assessed, were oriented to the appropriate Guide and then worked independently with their Resource Guide for 4 weeks)
Practitioner assessed and supported (in which they also worked with their Career Consultant for 4 weeks).
 Examined:
Differential impact across treatment conditions (independent/supported)
Differential impact across labour market attachment & self- help ability (2 new variables)

28. Research Question
If clients are given a comprehensive needs assessment to determine their employability need(s), what is the differential effect of independent and consultant- supported career resources on clients who are weakly attached to the labour market versus those who are strongly attached to the labour market?

29. Results After just 4 weeks of intervention:
Tenfold increase in competencies rated in acceptable range
Percentage of employed clients rose from 27% to 45% (a 69% increase)
81% of participants reported that their employment was a “good fit” with their career vision (a 200% increase)
91% attributed the positive changes either partly (40%) or mostly (51%) to the intervention
92% had a clear plan for next steps
98% planned to use the Guides again
Of 227 clients who provided work status information, 61 were working part time or full time before the intervention, and 166 were not working (see Table 11 for a breakdown by province). 27% of the sample, therefore, was working. By the end of the intervention, 103, or 45% of the original 227 were working – a 69% increase (see Table 12).

30. Results
Statistically and clinically significant positive impact of career interventions across ALL sub- scores and across ALL employability dimensions – compelling evidence of positive changes in clients as a result of career interventions
Although see more positive trends for supported group, differences are not statistically significant

31. Results
Career practitioners care very much about their clients, so much so that they were reluctant to ask clients who might need support to join the study— “in case” they were randomly assigned to the independent group
Therefore the sample was too homogeneous to test properly the Self-help and Labour Market attachment indices
Career practitioners care very much about their clients…BUT…in the next study, they CANNOT!

32. What did we learn
What was most surprising?
What are the implications for practice?

33. What did the study tell us?
Career Development Interventions Work!
Clients demonstrated substantial positive changes in skills, knowledge, personal attributes, employment and fit of employment.
Many clients can benefit significantly from self-help resources when they are matched to their need and they are “launched” – moreso than we anticipated!
This is especially noteworthy given the short intervention period of 4 weeks and the “real-life” setting in which the changes occurred.

34. The Tools and Guides are “Ready to Go”
Virtually no adjustments are needed for use in typical employment centres across Canada. Self-help guides provided after a thorough needs assessment and orientation are effective and can be used as a first line of intervention, saving valuable practitioner time for clients who really need it, or potentially opening a promising perspective on online career services. This finding has considerable practical significance.

35. Employability Dimensions Study: Tools
Employment Goal and Action Planner
Client Checklists/Tracking Sheets (all employability dimensions)
Comprehensive Coaching Guides (all employability dimensions)

36. Research Project #3: Common Indicators
What had we learned?
Where did we need to go next?

37. Activities that link to outcomes or deliverables
Resources available
Client needs assessment
Client employment history
Self-help index
Labour market attachment index
Activities that link to outcomes or deliverables
Generic interventions
Working alliance,
Specific interventions
LMI Booklets
Advice and Information Interviews
Employability Dimension Coaching Guides
Practitioner Coaching using the Guides
External Referral
PROCESSES
INPUTS
Client
Context
Life vision
Needs
Goals
OUTCOMES
Learning outcomes
Ability to use LMI resources alone and with assistance
Ability to use Coaching Resources alone and with assistance
Having an employment goal/vision
Skills in employability dimension indicators
Personal attribute outcomes
Changes in intrapersonal variables e.g., attitudes, self-esteem, motivation, etc.
Impact Outcomes
Changes in the client’s life resulting from application of learning

38. Unanswered Questions
We have data/evidence of positive impacts from specific interventions for certain kinds of clients
We do not have data/evidence of positive impacts from employment services overall
We do not have a data gathering tool that can be used to gather common data across divergent employment service settings

39. Therefore the next step: Common Indicators Project
Key research questions:
What common indicators are applicable across different contexts, different client groups, different agencies and different interventions?
What statements of service effectiveness can be made by tracking common indicators?
and

40. IF numbers permit… the ultimate question:
What kinds of interventions in what contexts produce what kinds of results?

41. How we investigated Literature Review Focus Groups Compare the two
Accept those in common and most robust

42. Focus Group Questions:
If clients have benefitted from career and employment services, they have experienced some changes in in their lives. They are not in the same position they were when they started. What indicates that clients or their situations have changed as a result of career and employment services?
How do you know? What are your clues?
What do you do to bring about change? What influences your ability to do this?
What about clients? What influences their ability to change?
What factors are beyond both of you to control?

43. Activities that link to outcomes or deliverables
Resources available
Availability of and access to community resources Availability of and access to employment opportunities and training
Practitioner Background and Experience
Client Employment History and Potential
Client Life Circumstances
Activities that link to outcomes or deliverables
Generic interventions
Working alliance, client engagement.
Specific interventions
Needs Assessment
Employment Counselling
Resources/Guides
Goal setting and Action Planning
Programs & Workshops
Progress Tracking
External Referral
PROCESSES
INPUTS
Client
Context
Life vision
Needs
Goals
OUTCOMES
client change
Learning outcomes
Changes in knowledge and skills linked to the program or intervention used
Progress Indicators End Result Indicators
Personal attribute outcomes
Changes in intrapersonal variables e.g., self-confidence, self-esteem etc..
Progress Indicators and Impact Outcomes
Changes in the client’s life resulting from application of learning

44. Practitioner/Research Partner Tasks
Recruit clients into the study (NO criteria);
Follow all research protocols
Use PRIME each time you see a client to assess and record PROGRESS/CHANGE in:
Goals and Action Plans
Employability Learning Needs/Changes
Personal Attribute Needs/Changes
Life Circumstance Needs/Changes
Working Alliance and Client Engagement

45. Practitioner/Research Partner Tasks
Work with each client for 6 weeks following Assessment interview and client entry into the research
At exit interview also gather data on…

46. Practitioner/Research Partner Tasks
Labour Market Outcomes:
Employment or Training Status
Fit of Employment or Training with skills, qualifications, vision
Adequacy of salary
Linked to future opportunity
Ongoing Progress Evidence: Actions Underway to move forward:
Have clients complete then and now questionnaire

47. Pre-Employability/Job Readiness
Client needs assistance to :
Not at all
Not much
A little
Quite a lot
A lot
Identify and clarify future direction (e.g., training, education, employment or change in life circumstances goal)
Identify personal strengths/resources that support future direction (e.g., training, education, employment or change in life circumstances goal)
Resolve specific challenges/vulnerabilities that may impact on future direction (e.g., mortgage, public transit, day care etc.)

48. Life Circumstances
Indicators of challenges in client life circumstances are in evidence:
Not at all
Not much
A little
Quite a lot
A lot
Improved housing is needed
Improved transportation is needed
Improved capacity to work and/or study is needed
Increase in sense of responsibility for own choices and behaviours is needed
Increase in ability to set short and long term goals is needed
Increase in understanding expectations and demands of employers is needed
Reduction in destructive behavior is needed
Following medical and medication protocols is needed
Increased openness to change is needed
Increased access to constructive and positive support systems is needed
Improved relationships with family and friends are needed
Increased trust in other people is needed
Other: _________________________________

49. Personal Attributes Example
Client needs assistance in the following areas:
Not at all
Not much
A little
Quite a lot
A lot
Ability to Self-Manage:
More info
Developing stronger Self-Esteem:
Improving sense of Well-Being: i.e.:
Developing stronger Self-Efficacy: i.e.:
Increasing Self-Awareness:
Other (Please specify): ________________________________ 

50. Self-Esteem Attribute
more info: Self-Esteem:
“a person’s overall emotional evaluation of his or her own worth, an attitude to self and a judgement of oneself. Self- esteem encompasses beliefs (e.g., I am competent; I am worthy) and emotions (pride, shame). Self-esteem is the positive or negative evaluation of the self”
Demonstrates an optimistic outlook
Sees self as competent and able
Has confidence in ability to interact with others
Uses positive self-talk
Acts with little hesitation
Raises few objections
Asks questions
Articulates skills assertively
Makes eye contact
Is hopeful
Takes reasonable risks

51. Working Alliance The client and I: Not at all Not much A little
Quite a lot
A lot
established a climate of trust and comfort in working together
arrived at a goal that is owned by the client
agreed on the action plan steps to help achieve the client’s goal

52. Client Engagement: The client: Not at all Not much A little
Quite a lot
A lot
participated actively in the interview
was focused on achieving results

53. Question #1: Applicable Common Indicators
The following items were shown in this study to have statistically significant connections to employment outcomes:
Input Indicators
Process Indicators
Employability Dimension need/ competence (composite & 5 dimension scores)
Responsibility and Access to Supports
Personal attributes
Working alliance/ client engagement

54. What we learned
Clients improve their employability (acquire specific knowledge and skills) as a result of services;
Positive change in the level of a client’s personal attributes (self-esteem; self-efficacy; self- awareness) appears to predict both client learning and employment outcomes;
The quality of the working alliance between client and practitioner appears to predict both client learning and employment outcomes;
Client capacity to take personal responsibility and ability to access a support system appear to be among the most influential life circumstances with respect to employment outcomes;

55. What we learned …
Practitioners tend to rate levels of change much more conservatively than client self ratings;
Practitioners overall found rating more complex variables, including client progress, challenging but helpful;
Tracking client progress indicators encouraged reflection on appropriateness of services;
It is important to use multiple sources of information to assess client employability;
Practitioners acquired an increased appreciation for the overall complexity of evaluating the impact of their services; statistical analysis is limited.

56. What we learned… We saw “relationships”, not causality;
Sample sizes were too small to perform some critical analyses;
Data collection time was too short to track sustained change;
Practitioner training, support and supervision in using the tool would be helpful
The tool can be enhanced AND simplified

57. What we learned….
We CAN track common indicators across settings and contexts
The model has the capacity to begin to answer:
What works for whom under what circumstances?
The model has the capacity to strengthen practitioner competencies and practitioner practice

58. At the same time…
Study has shown a pathway to measuring client progress in a meaningful way and provided a line of sight to be able to connect interventions with changes in clients and with successful labour market outcomes
There remains much to do but there are strong indications that the approach can move us from a “what does not work” evaluation culture to a “what works”—a real transformation

59. Common Indicators Study: Tools
Life Circumstances Checklist
Personal Attribute Checklist
Working Alliance and Client Engagement Checklist
Progress/Change Indicators
Labour Market Outcomes Checklist
Final Then and Now Survey

60. Discussion What stands out for you as the most important findings?
What stands out for you as important to follow up in more detail and depth?

61. Transformation From………………………..toward
Data……………………………
What doesn’t work…………..
Working alone
Preoccupation with Employment Outcomes….
Program driven……….
Focus on client numbers…….
One-off’s inform practice…...
Action Oriented FF
Feedback……….
What works……
Working collaboratively
Preoccupation with Context
Progress and Outcomes
Evidence driven and client need driven
Focus on client change
Patterns inform practice
Action and Reflection Oriented

62. Evidence-based Practice
Review of Research Tools
How can I use one or more of these tools to move from gathering data to getting feedback
How can I use one of more of these tools to move toward more collaboration with clients?

63. Reflective Tools for Practice
Initial learning needs survey
Checklist for Employment Assessment Interview
Advice and Information Interview Checklist
Employment Goal and Action Planner
Client Checklists/Tracking Sheets (all employability dimensions)
Life Circumstances Checklist
Personal Attribute Checklist
Working Alliance and Client Engagement Checklist
Progress/Change Indicators
Labour Market Outcomes Checklist
Final Then and Now Survey

64. Evidence-based Practice
Choose at least one tool you might want to use.
Why did you choose it?
How might will you begin to try the tool out with clients?
What supports if any would you need?

65. Evidence Base: A Practitioner Question and a Client Question
Who is the Evidence-Based Practitioner?
How does the Evidence-Based Practitioner work differently?
What does it mean for clients – the services they receive and the changes that result from service?

66. Characteristics of the Evidence-Based Practitioner
Client-Centered (client need, working alliance)
Learning/Change-Centered
Reflective
Positively Uncertain
Experimental

67. Behaviour of the Evidence-Based Practitioner
Intervenes in a systematic manner
Documents what they did
Pays attention to what happened
Track the effects
Looks for associations between what they did and the effects of what they did
Sees their own practice as their data source for predicting client outcomes
Each client is a n=1 experiment
Looks for patterns across time and across clients
Has a focus on trying to make predictions linking interventions and outcomes

68. The Reflective Evidence-Based Practitioner
What is reflection?
How do we integrate it into practice?

69. “the examination of our own thinking and action”
Reflection is…
“the examination of our own thinking and action”

70. Reflection requires… Consistent time commitment to reflect
The ability to see our “part”
An ability to evaluate processes in a non-judgemental way
A commitment to put learning into action

71. Reflective questions
What happened today? What did I do? What caught my attention?
What worked? What energized me?
What were the low points? What frustrated me?
What can I learn from today? What are the implications?
How will I be different? What will I do differently as a result?

72. Building Evaluation into Practice with Myself: After a client session…
What happened today? What did I do? What caught my attention?
What worked? What energized me?
What were the low points? What frustrated me?
What can I learn from today? What are the implications for my practice?
How will I be different? What will I do differently as a result?

73. Building Evaluation into Practice with Colleagues
Evidence-Based Team Meetings
What was my biggest success last week?
What contributed to that success and how do I know it was effective?
What was my biggest challenge? What specifically made it so challenging?
What can I learn from this? What else might I do differently?
Is there anything we (individuals/team) want to do more of or differently based on this discussion?

74. Evidence-Based Practice: A Policy/Funding Question
What are the challenges/pitfalls of evidence- based practice to be addressed or avoided?
What are the advantages/benefits of evidence- based practice in terms of policy and funding?

75. Multiple Effects of Evidence-Based Practice
On client: + and –
Increased motivation
Increased awareness of effectiveness and progress
On CDPs/Employment Counsellors: + and –
Increased effectiveness
Increased confidence
Increased job satisfaction

76. Multiple Effects of Evidence-Based Practice
On agency/department: + and –
Increased capacity to plan for and ensure that services meet client needs
Increased capacity to justify expenditures/budget
On Policy Makers/Funders: + and –
Increased rationale for funding decisions
Increased certainty that funding of career services is a strategic investment
On the Career Development Profession: + and –
Increased professionalism, respect and pride

77. Don’t worry about getting it right or doing it all
Small steps are great!
Several small steps = a BIG STEP
Do it together – you can learn from and support one another
Share your success stories
With everyone!
In language they can understand
Be persistent
Be proud!

78. Our Evidence Base
Our field is amassing a growing body of evidence that our services have IMPACT far beyond employment/entry into training
There is VERY strong interest internationally…and Canada is an undisputed leader!
In times of fiscal restraint, evidence is critical
Evidence based research serves to enhance practice and inform policy – it is a mirror for the field and a lever to influence

79. Thank you!
Lynne Bezanson Sareena Hopkins All tools, booklets and resources are available free of charge on the CRWG web site: and/or on the CCDF website at