Investigating the I MPACT OF LMI 1 Research Team Canadian Research Working Group in Evidence-Based Practice (CRWG), Canadian Career Development Foundation (CCDF) In partnership with New Brunswick Post-Secondary Education & Labour Saskatchewan Advanced Education, Employment & Immigration

Overview 1.Background  Problem with LMI research  Our approach to evaluation 2.Project background  Participants  Results 3.Implications 2

Why this project? 1.Most LMI research focuses on usability of products  Readability  Accuracy of information  Easy to access  Amount of use 2.Very little research about the direct contribution of LMI to employment or career decision-making  Very little with adults in natural service settings 3.Several questions remain unanswered  How do people use LMI?  What (if any) assistance would be helpful? 3

Research Questions IF client needs are assessed and clients are given LMI tailored to meet their needs: 1.To what extent does assistance by a service provider enhance the effective use of LMI? + 2.To what extent is independent self-help a sufficient process in order for clients to use LMI effectively? The project isolated the use of LMI as a viable approach, independent of other interventions such as career counselling or psychoeducational workshops. 4

Incorporating Research Into Practice Research findings are seldom implemented in field settings  likely because the research has been viewed as not relevant by practitioners Participant-researcher approach (Action Research) Our study was implemented in real service delivery settings  utilizing service providers who work in those agencies delivering the interventions  Participants are people seeking assistance for career decision-making or job search. We believed that this approach would increase the likelihood that positive findings would be incorporated into daily service provision. 5

Scientist Practitioner Intervening in a systematic (scientific) manner  Documenting what you did Paying attention to what happened  Tracking the effects Looking for associations between what you did & the effects that happened Your own practice becomes your data source for predicting client outcomes  Each client is a n = 1 experiment Across time and across clients  Acquire ability to make predictions linking interventions & outcomes 6

7 Evidence-based Outcome-focused Practice Input  Process  Outcome Need to link process with outcome Framework developed by the Canadian Research Working Group on Evidence-Based Practice in Career Development

Outcome Focused Evidence-Based Practice 8 RESOURCES AVAILABLE  Context: Structure of opportunity. Labour market attachment  Staff: Number of staff, level of training, type of training  Funding: Budget  Service guidelines  Facilities  Infrastructure  Community resources ACTIVITIES THAT LINK TO OUTCOMES OR DELIVERABLES Generic interventions  Working alliance, microskills, etc. Specific interventions  Career Decision Making  Skill Enhancement  Job Search  Job Maintenance  Other Programs & Workshops  Facilitation guides  Intervention manuals External Referral INDICATORS OF CLIENT (LEARNER) CHANGE 1.Learning outcomes  Changes in knowledge and skills linked to the program or intervention used  Progress Indicators End Result Indicators 2.Personal attribute outcomes  Changes in intrapersonal variables e.g., attitudes, self-esteem, motivation, etc.  Progress Indicators End Result Indicators 3.Impact Outcomes  Changes in the client’s life resulting from application of learning OUTCOMES INPUTS PROCESSES Client  Context  Life vision  Needs  Goals

Sample by Province Province Intervention Type Delivery Total IndependentAssisted Saskatchewan CDM JS Province Total New Brunswick English CDM JS Province total New Brunswick French CDM JS11920 Province total26 52 Total for 3 groups

Dependent Measures Total Score  General ability to use LMI Knowledge  clear vision of what I want in my career future  knowledge of print and online resources Skill  have effective strategies for keeping myself motivated  have a realistic action plan Personal Attributes  optimism about what lies ahead in terms of meeting my career goals  confidence in my ability to manage future career transitions 10

11 Assessing Learning & Attribute Outcomes Post-Pre Assessment We would like you to compare yourself now and before the workshop. Knowing what you know now about using LMI, how would you rate yourself before the workshop, and how would you rate yourself now? Please use a two-step process:  Decide whether the characteristic in question is acceptable or unacceptable, then  assign the appropriate rating acceptable unacceptable

Inferential Results-Total Score Both CDM and JS groups had significant increases across time Change in CDM group was significantly larger than in JS group Participants in the JS group had higher scores than participants in the CDM group, likely indicating that JS participants were more familiar with using LMI before the project began. Participants receiving assistance demonstrated greater change across time than did those in the independent mode For group as a whole: significant increase in overall ability to use LMI neither intervention was more conducive to one manner of delivery compared to the other 12

Descriptive Results Looking at the 14 items in the survey instrument,  55% of the responses BEFORE participating in this research were OK (2 or 3 or 4),  95% of the responses AFTER participating in this research were OK. All of the mean scores on the Before Program responses were in the Not OK range. All of the mean scores on the After Program responses were greater than minimally OK (mean score 3 or greater) with 2 exceptions. The amount of change was similar across all three dimensions of the survey, knowledge, skills, and personal attributes all demonstrated about the same amount of change. 13

Descriptive Results (Continued) Of particular note are items that suggest increased ability to self-manage their careers, such as:  A clear understanding of what I need to do to move forward in my career.  A clear vision of what I want in my career future.  Knowledge of print and online resources that help me to research career/employment options.  The ability to access career resources that can help me implement my career vision.  Effective strategies for keeping myself motivated to achieve my career/employment goals.  A realistic action plan (or schedule) summarizing the main career/employment-related activities I want to pursue and the processes I am engaging in.  Confidence in your ability to manage future career transitions.  Confidence in my ability to research career, employment, and training options that are available 14

Other Noteworthy Results No significant differences for  Gender: Women and men responded equally well to all intervention-delivery combinations  Work history  Unemployment history Significant age-by-time interaction effect (English sample)  change across time for all age groups was statistically significant  participants 25 years or younger and those ages reported greater change than those 26-30, 36-40, and 51 years or older No significant age effects for French sample 15

Impact Outcomes Employment status Before 13 working full time, 22 working part time 6 = good fit, 11 = OK fit, 18 = poor fit After 50 working 24 = good fit, 11 = OK fit, 15= poor fit 16

LMI: General Summary of Results 1.All intervention-delivery combinations produced significant and meaningful change  General ability to access and use LMI  Knowledge about how to use LMI  Skills for using LMI and taking action  Personal attributes 2.Assisted use produced greater change across time than independent use 3.To what extent would you say that any changes in the ratings on the previous pages are a result of your participation in this research project, and to what extent were they a function of other factors in your life? 17 mostly other factors somewhat other factors uncertain somewhat this program mostly this program Total (French) Total (English)

Products and Generalizations Needs assessment checklist LMI resource guide for Career Decision Making LMI resource guide for Job Search Client learning outcomes survey Client resource tracking checklists Final Research Reports All are available free of charge on the CRWG web site: With minor adaptation to include local LMI, the results are generalizable across provinces 18

What have we learned? For Practitioners Structure and checklists are foreign at first  But helped practitioners be more focused later Practitioners are willing research partners  Most said they would do it again if given the opportunity For Clients  Tailored LMI embedded in a learning process results in knowledge and skill acquisition as well as the capacity for self-management  LMI appropriate for a client’s specific need (opposed to general LMI) supports engagement and action  For many clients, a little (or no) professional support is enough  Structure and timelines appear to motivate action and a sense of progress 19