1. 35 Years of Evaluation Learning in 5 Minutes
David J. Bernstein, Ph.D.
Owner, DJB Evaluation Consulting
2018 American Evaluation Association Conference
Cleveland, OH
November 1, 2018
I am David Bernstein, owner of DJB Evaluation Consulting and an inaugural AEA member.
I am also Director of Evaluation for an agency of the United Methodist Church.
Here are some observations from my 35 year career as an evaluator.

2. 35 Years in 5 Minutes? That’s….
I’ve been a Federal evaluator, a program developer, a local government evaluation coordinator, a performance auditor, a school system survey manager, and a consultant.
The humor and sarcasm that follow are intentional, whether it works or not.

3. Accountability
Accountability is one of the primary uses of evaluation.
That said, as Dilbert notes, I too have a vague feeling that I have not been all that I can be.
That hasn’t stopped me yet, and it won’t stop me now.
© United Features Syndicate, 3/13/99

4. Accountability Elections Internal & External Processes
The Public:
Voters, Taxpayers,
Citizens, Clients
Strategic Plans
Elections
Performance Measurement
Public Budget Documents
Program Evaluations/Audits
Elected &
Appointed Officials
Internal Processes
Internal Management Reports
I have an MPA and a Ph.D. in public administration, so accountability is important to me.
I’ve done a lot of research on accountability, evaluation sponsors, and how to keep stakeholders informed and involved.
Budget & Decision Documents
Personnel Operations
Program Managers
Government Staff
Policy Analysis

5. PERFORMANCE MEASUREMENT SYSTEMS
Montgomery County, MD Department of Finance
EXTERNAL
ACCOUNTABILITY:
Report Outcomes &
Efficiency
USE OF P E R F O M A N C S U
INTERNAL
MANAGEMENT:
Program/Resource
Management
Employee Appraisal
FEWER
NUMBER OF P E R F O M A N C S U
GREATER
External
Reporting
of Finance
Performance Measures
Executive Use of Finance
Performance Measures for Decision Making
Departmental Use of
Finance Performance Measures
I’ve researched how a lot of governments and nonprofits have implemented performance measurement systems.
That research influenced how Montgomery County Maryland’s Finance Department used performance measures in the late 1990s and 2000s.
Finance Department Divisional Use of Program Indicators to
Manage Programs

6. Best Practices Standard
Benchmarking and Performance Measurement
Legal Standard
Historical Standard
Strategic Standard
Peer-based Standard
Best Practices Standard
Benchmarking gets performance measurement closer to evaluation since it includes a comparison.
However, performance measurement is NOT evaluation, but instead complements and contributes to it.
Benchmarking can help you determine when evaluations may be needed.

7. PM Versus Evaluation
“You can measure without evaluating, but you cannot evaluate without measuring.”
© David J. Bernstein, Ph.D.
This one is pretty good, so I copyrighted it.
You can measure without drawing evaluative conclusions, but you cannot evaluate without data.
I have long argued that evaluation is dependent on performance measurement to provide data.

8. Performance Measurement
Performance Measurement vs. Evaluation
Temporal Order
Co-Variation
Non-spuriousness (Elimination of extraneous variables, aka internal validity)
Performance Measurement
When I took research methods, we started each class by identifying the 3 elements of causal inference.
The first two elements are available by examining performance measures.
The third element requires evaluation to help identify variables affecting the others.
EVALUATION

9. Intended/Unintended Users and Uses
Unintended Uses
Intended Uses
Unintended Users
Unintended Trouble
Unintended Success
Intended Users
Need More Technical Assistance and Training
WINNER:
Utilization-focused (Patton)
Patton’s Utilization-focused Evaluation had a huge influence on my evaluation work.
What happens when we don’t anticipate how evaluations will be used?
In the Unintended Uses column, I see both threats and opportunities.

10. Patton and Epstein on “Use”
Primary Uses
of Evaluation
Uses of Performance Measurement
1. Judge Merit and Worth
1. Improve Public Accountability/ Communication
2. Improve Programs
2. Improve Service Performance
3. Knowledge Generation
3. Improve Decision Making
My doctoral dissertation on local government performance measurement focused on systematic and sustained use of performance measures.
I’ve done comparative analysis of Epstein, whose model influenced my research framework, and Patton’s utilization-focus.
Patton, 2012
Epstein, 1988

11. Federal Evaluation Policy+ =
Federal Evaluation Policy
I founded the AEA Government Evaluation TIG because government evaluation policy is important.
Since 1964, every President has started at least one major evaluation policy that used performance measures and evaluation.

12. 1990s, National Performance Review
Paperwork Reduction Act
1990s, National Performance Review
The Paperwork Reduction Act (PRA) requires that even small data collections of 10 or more be reviewed.
I contributed to AEA’s position on Improving the PRA.
Reviews should be based on hours of burden, not the number of respondents.

13. Logic Model-Kellogg Foundation
I’ve helped a lot of clients develop logic models.
The Kellogg Foundation developed a simple illustration of a logic model.
A lot of organizations use this model to demonstrate results.
It’s a pretty good starting point for logic modeling.
Kellogg Foundation, 2004

14. Theory of Change - Not a Logic Model
This is a theory of change.
It is a starting point for understanding what a program hopes to accomplish, and how change will be achieved.
It is not a logic model, but looks like one because it includes arrows, activities, and results.

15. Logic Chart - NOT a Logic Model, but It’s Close
This is a logic chart, not a logic model.
It is a useful tool for capturing an inventory of activities, outputs, and outcomes.
It has a few arrows, and is a good tool to help organizations develop program logic models.

16. Logic Model This IS a logic model.
Each activity has its own set of results.
It shows the relationship between activities, outputs, and outcomes.
The client I developed this for still dislikes logic models, and thinks I am obsessed about them.

17. Are Random Control Trials the Gold Standard?
Are random control trials or RCTs the gold standard for measuring effectiveness?
Maybe.
I summarized 28 criteria from the evaluation literature to see if an RCT is feasible.
The client ignored my work.

18. Evaluation, Implementation, Recommendations
Evaluators can effectively evaluate what gets implemented, and how implementation may differ from what was intended.
I wish I could have been more involved before and after the evaluations I’ve done.
That would have improved the evaluations.
Source:

19. Kathy Newcomer (and me)
Mentors/Friends
Joe Wholey
Rakesh Mohan
Michael Quinn Patton
Kathy Newcomer (and me)
This will come as no surprise to anyone who knows my work.
I’ve learned a lot about evaluation from Joe Wholey, Kathy Newcomer, Michael Quinn Patton, and Rakesh Mohan.
They are my mentors, and I am forever in their debt.

20. David J. Bernstein, Ph.D. DJB Evaluation Consulting djbeval@gmail.com
For all your evaluation needs, please contact…
David J. Bernstein, Ph.D. DJB Evaluation Consulting
If you are a graduate student or new evaluator, pay attention to what I am about to say.
My career just started yesterday, and this is my 35th year.
As the late Warren Zevon said, “Enjoy every sandwich.”
Thanks!