1. Evaluation of Government Contracting for Human Services Systems Reform
November 3, 2011
Evaluation 2011: Values and Valuing in Evaluation
Susan M. Wolfe
Susan Wolfe and Associates, LLC
and
Kathryn Race
Race & Associates, Ltd.

2. Background
Wallace Foundation Grant funding provided to the Donors Forum to reform government contracting in the State of Illinois and the City of Chicago
Partnership Principles were developed collaboratively with Policy Forum (providers, advocates, foundations, other stakeholders) and released in January 2010
6 Areas
16 Principles
40 Practices

3. Background: Example of Area, Principle, Practice
Area 1: CONTRACTED SERVICES: Based on a dynamic, data-driven system
Principle 1.1: Contracted services are based on a comprehensive and transparent planning process that defines and prioritizes services
Practice 1.1.a: Planning includes local and regional input

4. Objectives Assess/describe level of adoption of Partnership Principles
Develop a measurement protocol and tools to assess adoption and implementation.

5. Methods Multi-method, multi-source
Online Survey – Illinois Department of Human Services (DHS) and Illinois Department of Child and Family Services (DCFS)
Online Survey – Human Service providers with contracts with DHS and DCFS
Online Survey – Provider advocacy organizations
Site Visits to DHS and DCFS
Focus Groups
Interviews
Document Reviews
Memorandums
Legislation
Planning Documents
Presentations
Obtained from Agencies and the Internet

6. Adoption Framework Adoption Stage Description No Awareness
Stages of Principles Adoption
Adoption Stage
Description
No Awareness
The Department is unaware of a need for change and the current status of practices is not consistent with the partnership principles.
Awareness
Members of the Department have voiced an awareness of a need to change; meetings or discussions have been held.
Commitment
The Department has developed and committed to a plan of action for change.
Action
The Department has begun implementing or has implemented the plan of action for change.
Change
The changes that were made have been institutionalized and are now standard practice.
One of the first steps before measurement development was to create a framework that defined the stages of adoption.
They ranged from no awareness such that the department did not recognize a need for change and the current practices were not consistent with the partnership principles to the change stage which was defined as changes have been made and institutionalized, or that the current practices are consistent with the partnership principles.

7. From Framework to Measurement
Practice
Commitment
Action
Change
Area 1 – CONTRACTED SERVICES: Based on a dynamic, data-driven system
Principle 1.1 – Contracted services are based on a comprehensive and transparent planning process that defines and prioritizes services
1.1.a Planning includes local and regional input
Formal discussions are underway to plan for obtaining input from local and statewide groups – committees formed, ideas solicited; data sources for planning identified
Methods for obtaining stakeholder input at the local and statewide levels are being piloted – town hall meetings, surveys, focus groups, data sources, requests for comments issued
There is a routinized system in place for obtaining local and statewide stakeholder input and gathering relevant data during the planning process
From this framework, the next step was to create the measures. The first step in that process was to define the specific actions that would describe the stage for each practice. Of special interest was differentiating between commitment, action, and change. No awareness and awareness were fairly straightforward.
We developed this grid of indicators for each practice and then shared it with the Donors Forum, government representatives, and other policy forum volunteers who helped us to refine the statements and ensure that they were, in fact, accurate.
We used this grid to develop online surveys.
[1] Stakeholders include providers, citizens, partners, and all others with an interest in the results of the planning process
[2] "Department" refers to the department and its divisions

8. Government Survey
Practice 1.1.A Planning includes local and regional input
My department does not routinely obtain local and regional input
Individuals in my department are aware of and/or have discussed the need to
include local and regional input
Formal discussions are underway to plan for obtaining local and regional input – for
example, committees have been formed or ideas have been solicited.
Data sources for planning have been identified.
Methods for obtaining local and regional input are being piloted such as town hall meetings,
surveys, focus groups, data sources, or requests for comments.
My department has a routine system in place for obtaining local, regional, and statewide
stakeholder input and gathering relevant data during the planning process.
This does not apply to my job
I prefer not to answer this question
And here is how the statements transferred from the grid, to a question on the survey. Respondents were instructed to check all that applied. Each statement corresponded to a level of adoption, and we were able to select the statement that each respondent checked that represented the highest level as an indicator of the respondents’ ratings for level of adoption for each practice.

9. Provider Survey DHS DCFS Planning includes local and regional input.
Principle 1.1 Contracted services are based on a comprehensive and transparent planning process that defines and prioritizes services. Based on your experience, please check each practice DHS and/or DCFS has demonstrated during the last 12 months.
DHS
DCFS
Planning includes local and regional input.
Planning is coordinated across service and funding areas.
Planning is conducted every 5 to 10 years based on the most recent census data.
Public funding is allocated across services, geography, and populations based on needs, service gaps, and disparities.
None of these practices are relevant to our contractual relationship.
We took a little different approach with the provider survey. We asked providers to rate the agency they had contracts with, and that could be either DHS, DCFS, or both. A typical provider survey question looked like this. The objective was to obtain a measure of which practices providers were able to see that government had implemented.

10. Site Visits and Document Reviews
Focus Group
Middle managers
Individual Interviews
Senior officials
Documents
Memos
Legislation
Presentations
Reports and Implementation Plans
We also conducted site visits to both DHS and DCFS.
We first held a focus group with middle level managers to learn more about the successes and challenges they encountered with contracting processes in each of the 6 areas and gather their impressions of the Partnership Principles. We had originally proposed using a modified Delphi method to obtain consensus about the level of adoption, but there was so much inconsistency across divisions that would not have been possible.
We also conducted individual interviews with senior officials and asked them questions to learn more about successes, challenges, and contextual factors to be considered in interpreting the survey responses from both providers and government.
We gathered relevant documents as we became aware of them, including relevant legislation that had been passed, power point presentations that had been made, and memoranda and implementation plans that had been issued. We reviewed them for content to gather factual information to support or refute survey and interview data, and to inform recommendations.
Among our findings were that:
There were differences between government and provider ratings on the perceived extent of adoption of many of the practices
We saw progress with a lot of room for growth.
True collaboration between all stakeholders will be key to reforming the system to meet everyone’s needs to the extent that this is practical and realistic.
A significant portion of providers could benefit from capacity building assistance, especially in the area of measurement and evaluation.