Some Problems evaluating large programs or program going in many different directions 1. the objectives are too general; they are really goals many large program such as mental health outpatient programs have objectives such as “help clinets live satisfying lives”. Nlw it could be that they have poorly written objectives OR When there are objectives like this, they may have been written this broadly on purpose; because there are many clients and each client has particulalrized objectives that are peculiar to his or her own situation. In other words, each clients objectives are different. 2. there are no objectives only very general goals. The same is true here. It is possible that your job might be to help them write objectives as we discussed earlier OR they have purposefully chosen to NOT write narrow and specific objectives because there are many clients with a broad range of problems and each client has particularized objectives that are peculiar to his or her own situation. 3. even when very large program do have specific objectives, often there are so many clients each with his or her own focus, purpose goals and objectives, that it is impossible to use the goals and objectives of the program to evaluate the clients success or achievement in the program. What do I do then? In program evaluation, if you are evaluating each client with a different set of standard, then there is really no COHESIVE PROGRAM or at least no way to evaluate it as a PROGRAMMATIC WHOLE111

Evaluators must find some way to measure clients a. On either the same factor or b. Using some kind of standardized measure that allows different problems to be measured using the same measure

Solution a: A note on client satisfaction: client satisfaction is often the factor that is used for solution a. We will get more into client satisfaction in the future. However it is important to know that many evaluators consider “client satisfaction” a hollow measure. That means, it really does not measure anything meaningful. It would be like asking you “who is your favorite teacher?”. Although the answer would be nice to know, it is in no way, a measure of your learning! Solution b: Goal Attainment Scaling. GAS is a way to measure very different objectives, using the same measure. It is not a perfect solution!!!

Gas does not measure specific outcomes or objectives, as much as it converts each client’s progress toward their specific objective into a standard measure; that is GAS standardizedes levels of outcomes rather than specific outcomes.. in this way it is able to montor client achievement within a program EVEN WHEN THEY HAVE DIFFERENT KINDS OF PROBLEMS Steps in developing a GAS system 1. using numbers that mean something 2. writing individualized outcome anchors for each number on each goal 3. training staff to use GAS.

1. Using numbers that mean something In GAS measurement, a. each treatment objective for a single client is measured on the degree to which the client has achieved or under-achieved or exceeded that objective. b. each treatment objective is scored using a series of numbers that are indicative of a level of achievement toward that objective c. each number indicates a clients achievement on each of his or her particular objectives and also allows the evaluator to sompile all clients’ degree of achievement together, even to the point of giving a mean achievement score!!

1. Using numbers that mean something-con’t Usually, goal attainment scales use 5 or 7 measurement points. This number allows the researcher to create a ‘mid-point’ ( 3 on a 5 point scale and 5 on a 7 point scale, to establish the expected outcome The mid-point is the most important In goal attainment scaling. It establishes the Center for all the other measures. In other words, there is an equal number of points Below the mid-point and an equal number above. OR

1. Using numbers that mean something-con’t Think of the mid-point as the center Far below below average average above average far above Average average The mid-point is the most important In goal attainment scaling. It establishes the Center for all the other measures. In other words, there is an equal number of points below the mid-point and an equal number above. OR Much less less than meets expected better than much better Than expected expected outcome. expected than expected Outcomeoutcome outcome outcome

1. Using numbers that mean something-con’t This example, with variations, is found on likert type ratings used on many standardized scale or RAIs Far below below average average above average far above Average average In the 1 st example, the mid-point is the average or ‘neutral point’. In the 2 nd Example, the mid-point is the point where the expected outcome is actually achieved. This example is found on many “Goal Attainment Scales” Much less less than meets expected better than much better Than expected expected outcome. expected than expected Outcomeoutcome outcome outcome

1. Using numbers that mean something-con’t As you can see, both examples show a progression from ‘worst’ to ‘best’ But with goal attainment scales, the standard for meeting the objective is the mid-point. Goal attainment scales allow you to a. measure achievement of an objective (midpoint), b. lack of achievement (-1, -2), as well as c. surpassing an objective (+1, +2) Thus you can measure different objective for different clients, using the same scale of measurement

2. writing individualized outcome anchors for each number on each goal Although the numbers now mean something, they must also be meaningful for each individual client objective. An anchor is a description of what each number (level on the GAS) actually looks like for each client objective Anchors connect each client particular objective with the numbers on the scale Thus each client objective is “operationalized” on the same scale of measurement

2. writing individualized outcome anchors for each number on each goal Lets take a client who is depressed. Some of her individual symptoms/problems are: –Sleeps all the time she is not working –Continual thoughts of worthlessness –Trouble concentrating at work 7 finishing tasks Finally her overall score on the BDI upon admission is 37 (37= severe dep is the clinical cutoff for mild dep.)

2. writing individualized outcome anchors for each number on each goal Let’s write “expected outcomes” for those. Normally we would do this with the client. –Sleeps all the time she is not working = goes to bed at 9P.M. and takes no naps –Continual thoughts of worthlessness = no more than 1 thought /day re worthless and dismisses –Trouble concentrating at work & finishing tasks = focused completely on work and produces as as before depression And For the 37 on the BDI we might have scores no higher than 18 on the BDI We might typically expect that if a client achieved these outcomes, she would no longer be terribly affected by depression.

2. writing individualized outcome anchors for each number on each goal Now we would use each of those outcomes as the mid-point anchors on the GAS. Lets use negative thoughts & BDI no more than 1 thought /day re worthless and dismisses it score on BDI

2. writing individualized outcome anchors for each number on each goal As you can see, there is now room for both greater improvement as well as failure. The next step would be to write individualized anchors for the other attainment points; for example no more than 1 thought /day re worthless and dismisses it score on BDI Has occasional + Thoughts about Self along with no – Scores On BDI Has + thoughts About self Much of time & No neg. thought Scores 9 or below on BDI Still has same Thoughts of Worthlessness Score on BDI Remains thoughts of Worthlessness Have increased or Become more Severe BDI score worse 38 or above

As you can see, the GAS has problems –It is not really a true likert scale and it is not designed to be treated as interval level data (although people do). –It is highly subjective and depends on peoples (staff & clients) ability to come up with precise operational definitions of each number. –Its reliability and validity can be questioned –Using the scoring the way it is, small negative scores can be positive! –Te severity of clients individual problems and their ability to solve them, vary widely and this is NOT captured on GAS. It can make all problems look the same. (unless you “weight” each objective) –Training all staff to use it can be a major pain!

Although GAS can be a very useful tool when clients are not focused on program objectives, but individual ones, the key to its effective use is getting staff to use it. Training staff in the use of GAS is difficult and time-consuming.