1. Lynn Elinson, Ph.D. Project Director
Developmental Disabilities Program Independent Evaluation (DDPIE) Project
Lynn Elinson, Ph.D.
Project Director

2. Evaluation Standards*
Utility
Feasibility
Propriety
Accuracy
*Joint Committee on Standards for Educational Evaluation

3. Purpose of the Independent Evaluation
Demonstrate impact of DD Network programs on:
Individuals
Families
Service providers
State systems
Provide feedback to ADD to help improve the effectiveness of its programs and policies
Promote accountability to the public

4. Purpose for UCEDDs
Identification of accomplishments from external organization
Identification of areas that need improvement

5. DDPIE Project Independent evaluation 2 phases
Phase 1 – development and testing of tools
Phase 2 – full-scale evaluation
Westat – contracted by ADD to implement Phase 1

6. Evaluation Standards Indicators What do we hope to achieve?
What do we observe
(measurement of indicators)?
Comparison
Are there differences/discrepancies?
What is the nature and extent of the differences?
What action needs to be taken?

7. Open Systems Model Structure (Input) Process Output (Product) Outcome
Effectiveness
Structure
(Input)
Process
Output
(Product)
Outcome
Have Hoda re-create this so it looks better on the slide.
Inputs are those resources that are needed to set processes in motion and keep them running. Some examples of inputs are staff, policies, resource networks, facilities, and funding. Inputs must be in place before proposed processes can function properly.
Processes are those event sequences and arrangements of staff, services, and resources needed to achieve the intended result(s). When inputs are in place and processes are functioning as intended, then outputs and outcomes are produced.
Outputs, often referred to as products, are the “units” produced by processes supported by given inputs. An example is the number of staff trained to use particular strategies targeted at meeting the employability needs of people with disabilities.
Outcomes refer to the intended results of creating certain outputs/products. While the product is the trained staff, the relevant outcome is the increased capacity of the staff to serve people with disabilities who seek their services. Of course, the ultimate outcome is improvement in the employment outcomes of people with disabilities.
Effectiveness is defined as the relationship between the outcomes achieved and the processes used to affect those outcomes. One can ask the question, “Do the unique capacity-building strategies used by a grantee (or the grantees in general) in fact produce the desired outcomes?”
The efficiency of capacity-building efforts is evaluated through a comparison of inputs and outputs. Given that outcomes are satisfactory, or remain constant, the relative efficiency can be assessed among different approaches to building system capacity to serve people with disabilities. In this model, efficiency is only relevant if positive outcomes are realized.
Efficiency

8. Basic Evaluation Approach
Performance-based approach
Development of standards and indicators
Measurement of indicators to determine level at which standards are being met
Development of measurement matrices that contain standards, indicators, and performance levels (not developed; limited development; adequate development)
Determination of overall performance at the national level

9. Evaluation Tools Measurement matrices - standards
- indicators (structures, processes, outputs,
outcomes)
- performance levels
Data collection instruments

10. INDEPENDENT EVALUATION FRAMEWORK OF INDICATORS
DD Act
Administered by ADD
Programs
DD Councils
P&As
UCEDDs
Collaboration
KEY FUNCTIONS
FRAMEWORK OF INDICATORS
STANDARDS
PERFOR-
MANCE LEVELS
Standards will be used as benchmarks to evaluate the indicators, as measured.
DD Councils
Outreach
Informing
policymakers
UCEDDs
Training
Comm. serv.
Research
Dissemination
P&As
Individual advocacy
Outreach/public education
Collab.
Proj. develop.
Proj. implement.
Structure
Process
Output
Outcome
Level
at which a
standard is
met.
Three
performance
levels for
each
standard.
MEASUREMENT MATRICES
Will be developed to organize each program’s key functions, framework of indicators, standards and corresponding indicators, and performance levels for each standard.

12. Project Tasks and Timing
October, 2005 – December, 2006
Collect and review background information.
Establish Advisory Panel, Working Groups, and Validation Panels.
Develop draft performance standards, indicators, and data collection tools.
Work with Validation Panels to finalize matrices for pilot study.
Train pilot study staff.
January, 2007 – September, 2007
Conduct pilot study in up to 5 states.
Write report to ADD with recommendations.

13. Advisory Panel Self-advocates Family members
Representatives from 3 programs - UCEDD: Richard Carroll from Arizona
Child/disability advocates
Evaluation expert
Federal representative (for PAIMI evaluation)

14. Working Groups
4 Working Groups (UCEDD, P&A, DD Council, Collaboration)
Role: To assist Westat in developing draft measurement matrices that will be reviewed and endorsed by Validation Panels
Process: In-person and telephone meetings; work offline

15. Criteria for Selection of UCEDD Working Group Members
Director/Associate Director
At least one state/jurisdiction has more than one UCEDD program
Rural/urban
Geographic distribution
Reflects different types of UCEDDS (e.g., placement in medical school, school of education, standalone)

16. Working Group Activities
Description of program
Discussion and identification of key functions
Identification of structures, processes, outputs, and outcomes for each key function
Discussion of standards for structures, processes, outputs, and outcomes

17. What can UCEDDs do? Stay informed Open process - input welcome
Participation in pilot study – random selection

18. UCEDD Working Group Members
Carl Calkins, Ph.D.
Kansas City, MO
Tawara Goode, M.A.
Washington, DC
Gloria Krahn, Ph.D.
Portland, OR
David Mank, Ph.D.
Bloomington, IN
Fred Orelove, Ph.D.
Richmond, VA
Fred Palmer, M.D.
Memphis, TN
Lucille Zeph, Ed.D.
Orono, ME

19. Westat Contact Information
Lynn Elinson -
Cynthia Thomas -
Priyanthi Silva -
Bill Frey -