1. Managing Evaluation Projects CRJS 4466EA

2. Evaluation Project Management  “a process of considerations, decisions, and activities engaged in by the leadership of the evaluation project to facilitate its conduct” (Bell, 1994)  “the evaluation mandate is the direction on the analytic purpose and intended use of evaluation findings given to evaluation management by the project sponsor” (Bell, 1994)

3.  Project management is more art than science  No prescription exists  This lecture focuses on five areas that are essential to effective project management – Clarifying the evaluation mandate – Staffing and organizing for success – Making assignments productive – Monitoring interim progress – Ensuring product quality and usefulness

4. Clarifying the evaluation mandate  Gaining initial agreement – Should be formalized orally or in writing – Evaluation management need to clarify the evaluation user, purpose, scope, design and method, resources, schedule, and other technical requirements from the sponsor – Both parties should check for congruence of mandate interpretation – Develop framework – When multiple purposes exist, evaluation management should discuss with the sponsor the relative importance of the different purposes  Checking the mandate during the evaluation – A new legislative proposal or executive initiative may cause sponsors to change their views about preferred measurements – Project managers need to involve sponsors in discussion about mandate throughout the evaluation – Sometimes shifts cause a complete mid-stream negotiation of mandate

5. Staffing and organizing for results  “the project staff should embody the qualifications needed to conduct the planned evaluation activities on schedule in a high quality and effective manner…they should be available and motivated….they should be organized into teams with well-defined roles” (Bell, 1994)

6. Selecting appropriate staff  First step – staffing matrix – Substantive and methodological qualifications needed to conduct the project are arrayed in rows in the matrix and the identities of candidate staff members are listed in the columns – Cells in the matrix are marked for individuals whose qualifications demonstrated in past similar or related evaluations match those needed to conduct the tasks and activities required for this evaluation (Bell, 1994)

7. Substantive qualifications  Knowledge, skills and experience that demonstrate familiarity with the program and subject area  The ideal substantive qualifications encompass knowledge of all pertinent aspects of the subject area within and around the scope of the evaluation (Bell, 1994)  Administration and management, financing, client intake, case management, and management information systems are functions in many social programs. Knowledge in these functional areas is transferable across programs (Bell, 1994)

8. Methodological qualifications  Should reflect the requirements of each stage of a project – evaluation design and instrument development, data collection, data analysis and interpretation and report-writing  Distinguishing candidates who have capabilities in quantitative and qualitative methods and techniques is also beneficial (Bell, 1994)

9. Interpersonal relations/communications capabilities  Should be appropriate to member roles on the team  Avoid staff who are doctrinaire or zealous  Seek neutral and objective staff

10. Level of training  Need mix of junior and senior staff appropriate to the problem-solving challenge of the project (Bell, 1994)

11. Organizing for results  Teams should be flat matrix work groups  Members can be on more than one team  Advisable to form a synthesis group to shepherd the efforts of the technical teams – Involved in most aspects of planning evaluation assignments, monitoring technical progress, and interpreting and integrating products and results from completed evaluation activities – In the synthesis group, the evaluation part are brought together and shaped through internal peer review into the information products that will present the evaluation results (Bell, 1994) – Some evaluation projects also benefit from an external advisor panel comprised of experts

12. Making assignments productive  Evaluation management must ensure that each staff member has at all times a clear and well-defined assignment governing his or her efforts on the evaluation  “Effort spent on gaining well-conceived and agreed upon assignments should not strain the evaluation resources and schedule. A project must maintain momentum by integrating the effort required for planning and initiating assignments into daily, weekly, and monthly activities. Only a very small share of total project labor and a moderate share of total evaluation management effort should be devoted solely to assignment making” (Bell, 1994)

13. Shaping individual assignments  Management determines set of interim projects needed for the next evaluation period  Management matches each assignment within the actual capabilities demonstrated on other evaluations by the individuals  In making assignments, management should encourage open discussion with project staff  Ground rules for reporting unanticipated problems should be established (Bell, 1994)

14. Formalizing assignments  Management and project staff should enter into an agreement specifying the expected product, the major milestones in product development, the resources set aside for the assignment, and the expected completion date (Bell, 1994)

15. Monitoring interim progress  In addition to the technical progress on evaluation products, the monitoring reports should inform evaluation management about the calendar time and labor spent and remaining for each product (Bell, 1994)

16. Well specified milestones  Progress milestones should exist for each evaluation product

17. Well-timed monitoring  Progress monitoring is appropriate when the forward momentum of the project is enhanced or maintained through monitoring  Monitoring milestones for each product should be defined by considering the steps in product development  A monitoring approach based solely on elapsed time should be avoided (Bell, 1994)

18. Ineffective use of monitoring  Managers who conduct the monitoring must be able to interpret monitoring information and respond appropriately  They need to know how to revise assignments accordingly  If concerns exist re. middle manager monitoring efforts, senior managers need to step up their presence  Milestones may be well defined but their indicators may be flawed (Bell, 1994)

19. Source Bell, J. (1994). Managing Evaluation Projects. Handbook of Practical Program Evaluation (Chapter 21). San Francisco, California: Jossey-Bass Inc.