New Ways to Evaluate Counselling Effectiveness Bryan Hiebert Adjunct Professor Department of Educational Psychology & Leadership Studies University of Victoria Professor Emeritus in Education University of Calgary 1

2 What is counselling all about? Counselling or Counselling Psychology Clinical Psychology Social Work etc. 1.Write down the words or phrases that come to your mind 1 minute

Overview 1.Background 2.An alternate to RTCs 3.A framework for demonstrating value 4.Alternate evidence gathering tools Purpose 1.An alternate way of looking at demonstrating the value of human services 2.Intended to stimulate discussion 3

4 Evaluation is important 1.But counsellors do not evaluate their work with clients study 40% never evaluated their work with clients 35% evaluated with client during interview study 33% did not answer 56% reported data  Client flow, counsellor time use, etc.

5 How to address the problem We need an approach that is: 1.Comprehensive enough to include what is needed 2.Simple enough for people to use 3.Incorporates evaluation into standard practice Your own practice becomes your data source for predicting client outcomes  Each client is a n = 1 experiment Across time and across clients, counsellors acquire ability to make predictions linking interventions & outcomes Local Clinical Scientist (Professional Practitioner)

6 Evidence-Based Outcome-Focused Practice Input  Process  Outcome Need to link process with outcome Resources Actions Client change

7 Evidence-Based Outcome-Focused Practice Input  Process  Outcome Indicators of client change 1. Learning outcomes Knowledge and skills linked to intervention 2. Personal attribute outcomes Changes in attitudes Intrapersonal variables (self-esteem, motivation, independence) 3. Impact outcomes Impact of #1 & #2 on client’s life, e.g., employment status, enrolled in training Societal, economic, relational impact

8 Outcomes of Counselling 1.Client learning outcomes Knowledge Skills 2.Impact on client’s life Client presenting problem Economic factors Third party factors + Precursors Personal Attributes

9 Precursors Intervene between learning outcomes & impact outcomes 1.Attitude 2.Motivation 3.Self-esteem 4.Stress 5.Internal locus of control 6.Belief that change is possible Changes counsellors mentioned That were being observed but not reported

10 Evidence-based Outcome-focused Practice Input  Process  Outcome Activities that link to outputs or deliverables Generic interventions Working alliance, microskills, etc. Specific interventions 1. Interventions used by service providers Skills used by service providers Home practice completed by students 2. Programs offered by school 3. Involvement by 3 rd parties 4. Quality of service indicators Stakeholder satisfaction, including students

11 Evidence-based Outcome-focused Practice Input  Process  Outcome Resources available 1. Staff Number of staff, level of training, type of training 2. Funding Budget 3. Service guidelines Agency mandate 4. Facilities 5. Infrastructure 6. Community resources

12 Outcome Focused Evidence-Based Practice Input  Process  Outcome Need to link process with outcome 1.What will I do? 2.What are the expected client changes?  What do I expect clients to learn?  What sorts of personal attributes do I want my clients to acquire?  What will be the impact on their lives? 3.How will I tell?

Outcome Focused Evidence-Based Practice 13 RESOURCES AVAILABLE  Context: Structure of opportunity  Staff: Number of staff, level of training, type of training  Funding: Budget  Service guidelines  Facilities  Infrastructure  Community resources ACTIVITIES THAT LINK TO OUTCOMES OR DELIVERABLES Generic interventions  Working alliance, microskills, etc. Specific interventions  Strategies linked to specific client problems (stress, grief, depression, career, etc.)  Client home practice  Other Programs & Workshops  Facilitation guides  Intervention manuals External Referral INDICATORS OF CLIENT (LEARNER) CHANGE 1.Learning outcomes  Changes in knowledge and skills linked to the program or intervention used  Progress Indicators End Result Indicators 2.Personal attribute outcomes  Changes in intrapersonal variables e.g., attitudes, self-esteem, motivation, etc.  Progress Indicators End Result Indicators 3.Impact Outcomes  Changes in the client’s life resulting from application of learning OUTCOMES INPUTS PROCESSES Client  Context  Needs  Goals

14 Evidence gathering 1.A Framework for evaluation Learning from the evaluation experts 2.Gathering evidence of our impact Documenting what works and why 3.Developing evidence gathering tools Need to use informal evidence Some examples

15 Assessment as Decision Making (vs. Judgement) Please use a two-step process 1.Would you say that your level of mastery of the attribute under considerations is 1.Then assign the appropriate rating  0 = really quite poor  1 = just about OK, but not quite  2 = OK, but just barely  4 = really very good  3 = in between barely OK and really good acceptable unacceptable

Problem with skill self-assessment 1.Participants asked to rate their skill (or knowledge) before and after a program 2.Often, pre-workshop scores are high and post-workshop scores are lower People find out as a result of the workshop that they knew less than they thought or had less skill than they thought Based on the new awareness, post-scores are lower 3.People don’t know what they don’t know 4.How can we get around this problem? 16

17 Assessing Learning & Attribute Outcomes Post-Pre Assessment 1.We would like you to compare yourself now and before the workshop. Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now? 2.Please use a two-step process: Decide whether the characteristic in question is acceptable or unacceptable, then assign the appropriate rating acceptable unacceptable

Applied Career Transitions Program (on-line program for unemployed university grads) For Module 1 All together there were 10 (items) x 29 (participants) = 290 ratings Pre: 144 Unacceptable Ratings – Post: 3 Unacceptable Ratings Unacceptable Ratings decreased from 50% to 14% Pre: 6 Exceptional Ratings – Post: 130 Exceptional Ratings Exceptional Ratings increased from 2 to 44% of the participants 18

Results: Impact outcomes Module 1 23 out of 29 had found a job 10 of the jobs lined up well with career vision Module 2 4 out of 6 had found a job 3 of the jobs lined up well with career vision 19

Post-Pre Assessment: Bridging Health Care with Self-Care 1.Knowledge of stress and stress control 2.Knowledge of how to manage personal change 3.Knowledge of nutrition and nutrition control Level of stress 2.Level of nutrition (high=healthy) 3.Level of fitness 4.Confidence in ability to manage personal change 20

21 Participant Self-Assessed Change Stress & Stress Control KnowledgeLevel Response categoryStartFinishStartFinish very low17%0%4%9% somewhat low30%0%4%35% neither high nor low26%0%13%26% somewhat high21%70%39%26% very high4%30%39%4%

Post-Pre Results Bridging Health Care with Self-Care Stress & Stress Control End of ProgramStart of Program Vertical axis indicates % of participants 22

Post-Pre Results Bridging Health Care with Self-Care Manage Personal Change Start of ProgramEnd of Program Vertical axis indicates % of participants 23

Attribution for Change 1.To what extent would you say that any changes depicted above were a function of participating in our program, and to what extent were they a function of other factors (e.g., job, personal circumstances, etc.) mostly other factors somewhat other factors uncertain somewhat this program mostly this program  24 0% 9%26%57%

Participant Comments 1.This program made me realize that change is possible for me to make, and that even a small effort can bring significant benefits. When I joined the program, I thought that only a superhuman effort would result in any positive benefits, and that thought itself was a barrier to making a behavioural change. Now I know that “I can do it” and that knowledge itself is half the battle 2.This is a fantastic program. The biggest benefit for me is that I am starting to see myself as being a more active person, rather than a “couch potato”. I appreciated the focus of the program being on life change and wise choices, rather than on diet and weight loss. I 25

Tracking client goal attainment 1.Clients are different Live in different contexts have different goals require different interventions 2.How can we consolidate the results? 26

27 Goal Attainment Scaling 1.Much better than expected 1.A bit better than expected 1.About what I expected 1.A bit worse than expected 1.Much worse than expected Observable Indicators

Thought Listing Stimulus question 1.For Clients: What are the main factors contributing to your presenting problem? What needs to happen in order for you to achieve your goals? What are the barriers you need to overcome? For counsellors:  What is counselling all about? What factors are responsible for client change? 2.Write down all the words or phrases that come to mind in response to this question. 28

29 Cognitive Mapping Task 1.Complete the thought listing task 2.Transfer each item to a “Post-it” sticker, 1 item per sticker. 3.Arrange the stickers on a large sheet of graph paper Place related concepts close to each other 4.Draw connecting lines between stickers that represent related concepts 5.Draw a circle around any clusters of concepts label each circle

30 Cognitive Mapping-Example What is counselling all about? Scoring scheme:

Relaxation Monitoring Sheet Date/Time Indicator StartFinishDifference pulse rate (beats per minute) breathing rate (beats per min.) finger temperature (degrees) Feelings, body sensations, etc. Thoughts, perceptions, imagery, etc.. 31

Informal Evaluation of Headache, Pain, and Related Affective States 32

Self-Monitoring Headache 0 -No headache 1 -Low level, only enters awareness when you think about it 2 -Aware of headache most of the time, but it can be ignored at times 3 -Painful headache, but still able to continue job 4 -Severe headache, difficult to concentrate with demanding tasks 5 -Intense incapacitating headache 33

Headache Monitoring Grid (time of day) Level Before treatment After treatment 34

Headache: Indicators of Success MeasureBeginningEnd Average headache level Number of intense (level 4 or 5 ) headaches 30 Hours of intense (level 4 or 5 ) headache 130 Hours headache free010 35

Self-Monitoring Stress 0 -No stress 1 -Low level, only enters awareness when you think about it 2 -Aware of stress level most of the time, but still able to stay focused on job 3 -Very stressful, beginning to interfere with what you are doing 4 -Severe stress, creates major interference with what you are doing 5 -The most stress you ever feel 36

37 Self-Monitoring Confidence 0 - No confidence at all in your ability to handle the situation 1 - Low confidence, but you might get lucky and it will turn out OK 2 - Sort of confident, you think you might be able to handle it, but probably not 3 - Pretty confident, you think you have a chance of this working out OK 4 - Very confident, you’ll probably be able to pull this off 5 - Completely confident about what you are doing

38 Self-Monitoring Job Satisfaction 0 - No satisfaction at all with your job 1 - Somewhat satisfied, there are some things about your job that are OK 2 - Satisfied enough – as jobs go, this one is as good as any 3 - Mostly satisfied, occasionally there are things about your job that you actually like 4 - Very satisfied, but there are occasional things bother you 5 - Completely satisfied with your job, feeling a sense of self-fulfillment

39 Self-Monitoring Motivation 5 -Very motivated-it's the most important thing for me to do today 4 -Very motivated, but something might come up to interfere 3 -Quite a bit motivated, I think I will end up doing some of it 2 -I sort of care and I might get around to doing some work today 1 -If I run out of things to do, I'll try looking in some of my books 0 -I couldn't care less if I ever did my school work

Using Self-Talk to Control Anger With cognitively-challenged adults in residential settings Hiebert, B., & Malcolm, D. (1988). Cognitive strategies for mentally handicapped clients. In B. O'Byrne, (Ed.). Natcon-14 (pp ). Toronto, On: Ontario College Counsellor's Association. (Reprinted in Resources In Education, June 1989, ERIC Document Reproduction Service No. Ed ) 40

41 Picture Stimulus: Pushing

42 Picture Stimulus: Hitting

43 Picture Stimulus: Kicking

44 Interview Protocol for Assessment 1.I am going to show you 3 pictures and ask you some questions about each picture. 2.Look at this picture. 3.Tell me how many people are in the picture. 4.What are the people doing in the picture? 5.Point to the person that is being pushed/kicked/hit 6.If you were the person being pushed/kicked/hit, how would you feel? 7.If you were the person being pushed/kicked/hit, what would you feel? 8.If you were the person being pushed/kicked/hit, what would you think? 9.If you were the person being pushed/kicked/hit, what would you do?

45 Interview Scoring Sheet StatementFeelingThinkingAction Anger Non anger Non facilitative FacilitativeAggressive Non aggressive 1 – 5 6 … how would you feel? 7 … what would you feel? 8 … what would you think? 9 … what would you do?

46 Field Test Results: Picture Stimuli Test time Group Response Category FeelingThinkingAction Anger Non anger Facilitative Non facilitative Aggressive Non aggressive Pre test Male Female Total Post test Male Female Total

Frequency of Anger Outbursts 10 Mentally Handicapped Residents 47

48 Monitoring Your Self-Talk ConPro

49 Changing Your Self-Talk ConPro

50 Self-monitoring Stress Subjective Units of Disturbance

51 Self-monitoring Stress Subjective Units of Disturbance

52 Self-monitoring Stress Subjective Units of Disturbance

53 Emotional Thermometer

54 Outcome-Focused Evidence-Based Practice Outcomes LearningPersonalImpact Processes Decision making Skill training Stress control Self-management Other

Linking Process and Outcome We have evidence that the facilitator followed the program We have evidence about which participants followed the program more (vs. less) closely We could compare the results of those who follow the program compared to those who did not We have evidence of how much change took place We have a connection counsellor process  client engagement  outcomes Participants attribute change to the program A solid link between program and outcome Impact Learning Attribute 55

Professional Identity: What we do defines who we are 1.Most practitioners define their job as delivering services So … they do not evaluate the impact of their services on clients 2.What is counselling all about? 3.Needs to include BOTH process and outcome What will I do to facilitate client change? + How well is it working? 56

57 To demonstrate value, we need to develop Culture of evaluation: We need to reach the state where Identification of outcomes is an integrated part of providing services  Without efficacy data, counselling services are vulnerable  It is in our best interest to gather evidence attesting to the value of the services we provide Measuring & reporting outcomes is integrated into practice  Informal data are acceptable Reporting outcomes is a policy priority Outcome assessment is a prominent part of practitioner education This needs to be a priority in all sectors

Multiple Effects of Relevant Evaluation Practices 1.On client Increased motivation Increased awareness of effectiveness How am I doing at this? 2.On counsellor Increased effectiveness Increased confidence Increased motivation Increased job satisfaction 3.On agency Value of services 58

59 What is counselling all about? Counselling or Counselling Psychology Clinical Psychology Social Work etc. 1.Write down the words or phrases that come to your mind 1 minute

New Ways to Evaluate Counselling Effectiveness Questions? Comments? Bryan Hiebert Adjunct Professor Department of Educational Psychology & Leadership Studies University of Victoria Professor Emeritus in Education University of Calgary 60