1. Southeastern Association of Educational Opportunity Program Personnel 38 th Annual Conference January 30 – February 3, 2010 Upward Bound Internal & External Program Evaluation

2. Presenter Elizabeth C. Retamozo, Executive Director NOSOTROS Education Center www.nosotrosedu.org Phone: 469-450-7137 Email: eretamozo@nosotrosedu.orgeretamozo@nosotrosedu.org 2

3. Presentation Materials PowerPoint Slide Presentation & Handout Located at: www.nosotrosedu.org www.nosotrosedu.org –Go to Resources Page –Register Your Email Address –Click the Presentations & Links Tab –Click SAEOPP 2010 Folder –UB Internal & External Program Evaluation 3

4. 4 Types of TRIO Evaluation Plans  Outcome-Based Evaluation  Goals-Based Evaluation  Process-Based Evaluation

5. 5 Common Evaluation Plan Terminology Qualitative Values  Expressed by impressions, feelings & perceptions Ex. Surveys, Interviews, Focus Groups, etc. Quantitative Values  Expressed in numerical terms, counted, or compared on a scale Ex. Grade Point Averages, Test Scores, etc.

6. 6 Common Evaluation Plan Terminology Formative Methods  Continuous (daily, weekly, monthly, quarterly, semester) assessment to track progress Ex. Monthly Reports Summative Methods  End of year assessment that measures the results or outcomes of a Project Ex. Annual Performance Report

7. 7 Sample TRIO Evaluation Plans  Outcomes Based Evaluation External & Summative Looks at impacts/benefits/changes to your students as a result of your project’s efforts after their participation in your Project (PE OBJECTIVES).

8. 8 Sample TRIO Evaluation Plans  Goals Based Evaluation Internal & Formative Evaluates the extent to which projects are meeting predetermined internal goals or objectives.

9. 9 Sample TRIO Evaluation Plans  Process Based Evaluation External & Summative) Geared to fully understanding how a Project works How does it produce that results that it gets?

10. 10 Outcome Based Evaluation  Outcomes Based Evaluation (External & Summative)  Measures performance on the four (4) established UB Program objectives by the end of each year (summative).  DE Mandated Objectives = Prior Experience

11. 11 Outcome Based Evaluation Plan 1)Identify Major Outcomes (Mandatory Objectives) 2)What Types Of Data Will Be Collected? 3)When Will Various Types Of Data & Reports Be Collected? 4)What Instruments Will Be Developed? 5)How Will The Data Be Analyzed? 6)How Will The Data Collected Determine Accomplishment?

12. 12 Outcome Based Evaluation UB Outcome Based Evaluation Plan

13. 13 Goals Based Evaluation  Goals Based Evaluation (Internal & Formative)  Evaluates the extent to which Project is meeting predetermined internal goals  Project Developed INTERNAL GOALS based on DE Objectives  Includes quantifiable & qualitative measures to track progress during the Project year (formative)  Monitored and reviewed weekly/monthly

14. 14 Goals Based Evaluation 1)What Project goals were established? 2)What is the status of the project’s progress towards achieving the goals? 3)Will the goals be achieved according to the timelines specified in the program implementation or operations plan? 4)Do personnel have adequate resources (supplies, facilities, training, etc)?

15. 15 Goals Based Evaluation 5)How should priorities be changed to focus more on achieving the goals? 6)How should timelines be changed? 7)How should goals be established in the future?

16. 16 Goals Based Evaluation Plan 1)Develop Internal Goals based on External Objectives 2)Activities & Services Needed to Achieve Each Goal 3)Project Staff Responsible to Carry Out Activities & Services 4)Data that Needs to be Collected 5)Instruments (Forms) that Need to be Developed and Collected

17. 17 Goals Based Evaluation Plan 6)When is Data Collected? 7)How is Data Collected Analyzed? 8)Was the Goal Met? Yes/No 9)Needed or Recommended Revisions 10) Unanticipated Outcomes

18. 18 Outcome Based Evaluation Sample UB Internal Goals UB Goals Based Evaluation Plan

19. 19 Process Based Evaluation  Process Based Evaluation (External & Summative)  Geared to fully understanding how a project works  How does it produce the results that it gets?  Usually External Conducted by Program Evaluator

20. 20 External Evaluation Recommended in last grant proposal competitions Must be conducted if included in grant proposals Allowable cost for the project May be paid by the grantee agency/institution May be provided free of charge – in-kind/donation

21. 21 External Evaluation May be conducted by person working/employed by agency/institution BUT not for the Project Must have experience conducting evaluations for educational opportunity programs & TRIO like programs Must state what type of evaluation will be conducted by the evaluator.

22. 22 Process Based Evaluation Plan 1)How does the Project identify, recruit, & select participants? 2)How does the Project decide what specific services & activities to provide to its participants? 3)How does the Project monitor participants’ academic progress? 4)How does the Project deliver services to its participants?

23. 23 Process Based Evaluation Plan 5)How does the Project document services provided to participants? 6)What is the Project’s goals based evaluation plan? 7)What is the Project’s outcome based evaluation plan? 8)How does the Project plan for the next program year?

24. 24 Process Based Evaluation UB Process Based Evaluation Checklist External Process Based Evaluation Report

25. 25 Basic Guide to Program Evaluation Written by Carter McNamara, MBA, PhD, Authenticity Consulting, LLC. Copyright 1997-2007.Carter McNamara, MBA, PhD, Authenticity Consulting, LLC Adapted from the Field Guide to Nonprofit Program Design, Marketing and Evaluation.Field Guide to Nonprofit Program Design, Marketing and Evaluation Online Guide: A Basic Guide to Program Evaluation.pdf Bibliography & Resources

26. 26 Basic Guide to Goals-Based Evaluation for Nonprofit Organizations with Very Limited Resources Written by Carter McNamara, MBA, PhD, Authenticity Consulting, LLC. Copyright 1997-2007.Carter McNamara, MBA, PhD, Authenticity Consulting, LLC Online Guide: Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources.mhtBasic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources.mht Bibliography & Resources

27. 27 US Department of Health & Human Services Administration for Children & Families - Office of Planning, Research & Development The Program Manager's Guide to Evaluation ACF OPRE The Program Manager’s Guide to Evaluation website.mht This guide explains program evaluation -what it is, how to understand it, and how to do it. It answers your questions about evaluation and explains how to use evaluation to improve programs and benefit students, families, and staff. Bibliography & Resources

28. Questions & Answers 28