1. Customer Satisfaction Research 2019 Q1 Results April 5, 2019
Florida Rehabilitation Council
Division of Vocational Rehabilitation Florida Department of Education
Customer Satisfaction Research 2019 Q1 Results April 5, 2019
Brian Robertson
Mark Noyes

2. Contents Consumer Survey Survey Methodology Trends to Watch
Customer Experience Core Metrics
Problems Experienced by VR Customers and Feedback
Understanding the Process if You Disagree with VR About Your Services
Satisfaction with Outside Vendors
Current Employment Status
Those Whose Cases Were Unsuccessfully Closed
Difficulties in Surveying Youth in Transition 2

3. Current Quarter Response Rates%
APPOR Response Rate (RR3)
18.5%
AAPOR Respondent Cooperation Rate (COOP3)
76.8%
AAPOR Respondent Refusal Rate (REF3)
7.5%
AAPOR – The American Association for Public Opinion Research.
Response Rate – The response rate is the number of completed interviews divided by the number of eligible reporting customers in the sample.
Cooperation Rate – This represents the proportion of all cases interviewed out of all eligible customers ever contacted. That is, the percent of identified respondents that ended up completing the interview.
Refusal Rate – The refusal rate represents the proportion of all cases in which the respondent refused to be interviewed, or broke off an interview, out of all respondents that were contacted and spoken with. 3

4. Trends to watch Trending Positive in 2019 Q1
Problems with FL VR or its services
Problems reported as resolved
Awareness of dispute resolution process
Job satisfaction
Trending Downward in 2019 Q1
Experience with Services Provided by VR
Communications with VR Staff
Modest but present downward trends - something to keep an eye on and potentially examine if the trends continue 4

5. 2019 Q1 Survey Results 5

6. Current Quarter Survey Methodology
Data collection took place from January 5, 2019 through April 4, 2019.
Over the course of data collection, 750 customers completed surveys:
290 general VR customers with open cases
182 general VR customers - closed successfully
161 general VR customers - closed unsuccessfully
117 transition youth customers 6

7. Customer Experience Core Metrics7

8. Florida VR Customer Experience Dashboard
Measure
2018 Q2
2018 Q3
2018 Q4
2019 Q1
Overall Satisfaction and Expectations 81 79 78
Experience with Services Provided by VR 83 80
Experience with VR Staff and Counselors 92 90
Communications with VR Staff 77
Customer Control and Involvement
Outcomes and Meeting Goals 84 82
Satisfaction with Current Employment
Satisfaction with Outside Vendors
All things considered, would you tell your friends with disabilities to go to the VR program for help? (% Yes) 95 94 91
Have you experienced any problems with VR or the services they have provided to you? (% No) 76 8

9. Overall Satisfaction and Expectations:
Q01: Overall, how satisfied are you with the Florida VR’s program? Q33: How satisfied would you say you are with the services provided by VR? Q35: Please rate how well VR services met your expectations. 9

10. Experience with Services Provided by VR:
Q09: How satisfied are you with the choice of services?
Q11: How satisfied are you with the choice of service providers? 10

11. Experience with VR Staff and Counselors:
Q19: How helpful is the staff of VR in helping you achieve your VR goal?
Q29: The VR staff treats me with dignity and respect. 11

12. Communications with VR Staff:
Q13: How satisfied are you with the information you were given about the choices you had?
Q15: How satisfied are you with how long it took your counselor to answer your questions and address your concerns?
Q21: How easy is it for you to contact your VR counselor? 12

13. Customer Control and Involvement:
Q05: How satisfied are you with your involvement in your VR experience?
Q07: How satisfied are you with your choice of a vocational goal? 13

14. Outcomes and Meeting Goals:
Q25: The VR services I am receiving will help me become more financially independent.
Q27: The VR services I am receiving will help me become more independent, in general.
Q31: The VR staff will help me reach my job goal. 14

15. Satisfaction with Current Employment:
Q51: If working, thinking about your current job, how satisfied are you with what you are doing? Would you say you are...? 15

16. Satisfaction with Outside Vendors:
Q46: How satisfied were you with the services provided by OUTSIDE VENDOR? 16

17. Recommend VR:
All things considered, would you tell your friends with disabilities to go to the VR program for help?
(% reporting yes) 17

18. Experienced Problems:
Have you experienced any problems with VR or the services they have provided to you?
(% reporting no) 18

19. Problems Experienced by VR Customers and Feedback
Customers were asked if they experienced problems specifically with VR or the services they received from VR and reasons for dissatisfaction. 19

20. 17% of customers experienced problems with VR or the services provided.20

21. The Most Commonly Mentioned Problems (2019 Q1):%
Counselor was not helpful or supportive
20%
Did not receive employment, VR could not find me a job
16%
Counselor did not return calls, s or follow up
13%
Didn't receive job search help
10%
Listen to customer, understand needs, wants, ability
Received no help in reaching plan or goals 9% 21

22. 46% of customers reporting problems indicated that VR worked to resolve their problem(s).22

23. Customer Feedback
Customers were asked to provide additional feedback when they were not positive in their assessment.
Feedback was collected throughout the survey in response to 25 questions.
Customers could provide more than one response.
The following table summarizes the most common responses from 527 customers that provided additional feedback. 23

24. Customer Feedback %
Counselor did not return calls, s or follow up
32%
Better communication needed
28%
Changing counselors, switching too much, causes problems
25%
Process is hard, difficult, complicated
20%
Counselor was not helpful or supportive
17%
Get voic , never answers the phone
Counselor would not listen, dismissed concerns
14% 24

25. Understanding the process if you disagree with VR about your services
Customers were asked if they were aware of the options available if they disagreed with VR’s decisions about the services they did or did not receive. 25

26. 83% of customers were aware they could contact their VR counselor in cases of disagreement.26

27. 72% of customers were aware they could contact the VR Unit Supervisor.27

28. 59% of customers were aware they could contact the VR Ombudsman Office.28

29. 62% of customers were aware they could contact Disability Rights Florida, Client Assistance Program.29

30. 73% of customers were aware they could appeal the decision(s) they disagreed with, if necessary.30

31. Satisfaction with Outside Vendors
Customer were asked to provide feedback on the services they received from outside vendors. 31

32. 79% of customers with an outside vendor were satisfied with the services provided.32

33. 18% of customers using an outside center reported problems with their vendor.33

34. Feedback about Vendors (2019 Q1):%
Need more guidance, support, explanation
30%
Received no help in reaching plan or goals
22%
Vendor did not return calls, s or follow up
18%
Vendor would not listen, dismissed concerns
Vendor no help or help not effective
16% 34

35. 71% of customers felt that both the VR and their vendor(s) kept them informed and up-to-date.35

36. What worked well during experiences with outside vendors:
VEND05: In thinking about your experience with the outside vendor what worked well for you?
(Multiple responses accepted) %
All, everything
14%
Communication and accessibility of counselor 8%
Having help finding a job and with applications
Having a job coach in general 7%
Receiving help from a counselor in general 36

37. Challenges customers faced while working with outside vendors:
VEND06: What has been the most challenging part of your experience working with the vendor?
(Multiple responses accepted) %
Finding desirable position or school
25%
Transportation 9%
Lack of communication 8%
Challenging testing 6%
Personal barriers, being independent, coping with disability
Unable to get ahold of counselor 37

38. Current Employment Status38

39. 41% of current and former customers are working full or part time.
Q50: Finally, these last few questions ask about what you are currently doing. Are you currently...? %
Working full time, more than 35 hours per week
13%
Working part time
28%
Currently looking for a job
30%
In school or receiving job training
36%
Keeping house 2%
Currently unable to work 6%
Volunteering my time 39

40. 86% of working customers were satisfied with their current job.40

41. Feedback on Employment
The below responses summarize feedback given to all questions about current job activity. %
Dissatisfied with aspects or parts of job
46%
Job does not provide benefits, does not provide good benefits
40%
Need a job aligned with skills, training
34%
Bored with job, been at job too long, burnt out, need change
31%
Varying hours, unreliable hours, scheduling problem
28%
Unpleasant work environment
25% 41

42. 87% of working customers find their job meaningful.42

43. Those Whose Cases Were Unsuccessfully Closed
Customers were asked if they requested their case be closed and reasons for doing so.
Questions asked beginning in November 2018. 43

44. 40% of customers with unsuccessful cases asked for their case to be closed.44

45. (Multiple responses accepted, most common responses)
Customers asked for their case to be closed most often due to finding employment or moving to a different location.
(Multiple responses accepted, most common responses) %
Found employment, have a job
16%
Moved away from VR
14%
Dissatisfied with VR, services not effective
13%
Unable to work due to injury or disability
11%
Have other priorities, needed a break
10% 45

46. UNSUC01B: Why do you believe your case was closed?
Among those that did not ask or their cases to be closed, the most frequent reason they felt their case was closed was a lack of communication.
UNSUC01B: Why do you believe your case was closed? %
Lack of communication
20%
Transportation issues, customer couldn't keep appointments 9%
VR couldn't help, never found jobs 8%
Conflict with counselor or VR staff 6%
Customer relocated 5%
Health issues 46

47. 54% of customers whose cases closed unsuccessfully received a closure letter with the reason for closure and their rights to appeal. 47

48. Difficulties in Surveying Youth in Transition48

49. A review: current quarter response rates%
APPOR Response Rate (RR3)
18.5%
AAPOR Respondent Cooperation Rate (COOP3)
76.8%
AAPOR Respondent Refusal Rate (REF3)
7.5%
AAPOR – The American Association for Public Opinion Research.
Response Rate – The response rate is the number of completed interviews divided by the number of eligible reporting customers in the sample.
Cooperation Rate – This represents the proportion of all cases interviewed out of all eligible customers ever contacted. That is, the percent of identified respondents that ended up completing the interview.
Refusal Rate – The refusal rate represents the proportion of all cases in which the respondent refused to be interviewed, or broke off an interview, out of all respondents that were contacted and spoken with. 49

50. Differences in FL VR 2019 Response Rate Cooperation Rate Refusal Rate
Recent quarters of the Florida VR Consumer Experience Survey have shown a declining response rate.
While there are multiple reasons for this, the primary driver is the addition of pre-ETS/youth in transition customers.
This population also has lower cooperation and higher refusals.
Response Rate
Cooperation Rate
Refusal Rate
General VR
18.4%
80.1%
7.2%
Youth in Transition
10.6%
62.9%
8.6%
Total
18.5%
76.8%
7.5% 50

51. Differences by case status
Previously, closed unsuccessful cases had been the most difficult to contact. YiT cases have response rates seven percentage points lower than unsuccessful cases.
Response Rate
Cooperation Rate
Refusal Rate
Open
23.2%
84.2%
5.8%
Successful
23.3%
82.2%
4.9%
Unsuccessful
17.4%
71.4%
11.5%
Youth in Transition
10.6%
62.9%
8.6% 51

52. Differences by age of YiT customers
Response Rate
Cooperation Rate
Refusal Rate
14 to 15
9.6%
60.7%
6.0% 16
9.9%
57.1%
9.5% 17
8.9%
66.7%
6.9% 18
12.0%
59.6%
13.1%
19-21
14.3%
73.3%
5.5% 52

53. Why is this so challenging?53

54. Non-working numbers?
YiT cases are actually slightly less likely to provide non-working phone number than other individuals in the sample.
% Non-working or wrong numbers
General VR
15.3%
Youth in Transition
14.2%
Total
15.0%
% Non-working or wrong numbers
Open
14.3%
Successful
13.6%
Unsuccessful
18.1%
Youth in Transition
14.2% 54

55. Refusals to completed surveys
YiT customers are more likely to refuse to do the survey than general VR customers.
For every 4 completes with YiT customers we get 1 refusal.
This is the same ratio as closed unsuccessful customers.
Ratio:
Refusals to completed surveys
General VR
0.14
Youth in Transition
0.26
Overall
0.16
Open
0.10
Successful
0.09
Unsuccessful 55

56. The main reason: “I never received services from FL VR.”
For every four completed surveys completed among YiT customers, another five indicate they have never received services through FL VR.
Ratio:
Respondents indicating they did not receive services from FL VR to completed surveys
General VR
0.09
Youth in Transition
1.26
Total
0.27 56

57. By case status
Ratio:
Respondents indicating they did not receive services from FL VR to completed surveys
Open
0.09
Successful
0.02
Unsuccessful
0.18
Youth in Transition
1.26 57

58. By age of YiT customers Ratio:
Respondents indicating they did not receive services from FL VR to completed surveys
14 to 15
1.94 16
1.63 17
1.46 18
0.96
19-21
0.45 58

59. Efforts to Improve Response among Youth in Transition59

60. Consulting with FL VR and FRC
MDR has spoken with FL VR/FRC in order to improve the language in the survey and potentially help YiT customers recognize FL VR.
MDR has added new prompt statements and clarifying language throughout the survey, most important in the introduction screen and for the specific YIT questions.
These changes are fully implemented in the April 2019 survey.
MDR will hopefully be discussing this at the upcoming 2019 Summit Conference. 60

61. New introduction
If an individual doesn’t recognize VR as providing services they’ve received:
Voc Rehab is the program that may have helped you look for a job, helped with a disability, go to back school, or to help you transition from school to the workplace. VR provides many services, some of which are conducted through schools or other outside sources.
These services may have included things such as talking about finding a job or career you like, trying out different jobs, talking about education or training after high school, help with resumes or interviews, or help speaking up for yourself at school or work. 61

62. New language for questions
Original wording
Suggested revised wording
(Final, ELT approved)
Job exploration counseling or career counseling.
Talked about finding a job or career I like.
Work based learning experiences including a job, internship, or volunteering.
Times you went to try different jobs.
Counseling on opportunities to attend college, learn a trade, get professional or industrial certification.
Talking about college or training after high school needed for the job I want.
Work place readiness to help get ready for the challenges of work.
Help with resumes, interviewing and how to do well at work.
Instruction in self-advocacy or peer-mentoring to help you live the life you want.
Help with speaking up for myself at school or work. 62

63. What else can we do? Your thoughts.63

64. I’d be happy to answer any questions you might have.
Thank You
I’d be happy to answer any questions you might have.