1. BPC-NCWIT Evaluation Workshop Boulder, CO May 16, 2007 Evaluation of the BPC Alliance Projects Daryl E. Chubin AAAS Capacity Center www.aaascapacity.org

2. BPC-NCWIT Evaluation - Boulder 05-16-07 AAAS Portfolio Assessment of BPC Alliances: Purpose The portfolio assessment will look across the Alliance projects and askboth in knowledge gained and impact on participantswhat does investment in the set yield? The portfolio will be constructed from the data collected and reported by the projects. AAAS will offer technical assistance to the project evaluators, while framing the respective efforts relative to the overall BPC mission. AAASs purpose is ultimately to help tell the story of BPC.

3. BPC-NCWIT Evaluation - Boulder 05-16-07 Guiding Questions How is BPC affecting the capacity of the community to broaden participation from education to workplace? What practices seem promising for whom and in what settings? How are relationships among partners maturing, being leveraged, and expanding the sphere of influence within the national computing community?

4. BPC-NCWIT Evaluation - Boulder 05-16-07 Constraints Seeking short-term outcomes from long-term interventions Measuring processes and organizational arrangements as well as readily-quantified outputs (e.g., enrollees, graduates, achievements) Admitting what is seemingly not working as well as what isand describing adjustments No AAAS accessyetto Alliance annual reports Special scrutiny of programs by the Administration due to the Academic Competitiveness Council report

5. BPC-NCWIT Evaluation - Boulder 05-16-07 Definition of Success for the Academic Competitiveness Council (2006) To fulfill the requirements of the Deficit Reduction Act, the council will: Identify all federal education programs with a STEM focus; Identify the target populations served by these programs; Identify the stated goals of these programs; Identify the extent to which the programs have undergone independent external evaluations based on sound, scientific principles; Identify the extent to which the programs have quantitative evidence of achieving stated goals; Identify or develop standards for measuring and evaluating these programs, including common measures as appropriate; and Recommend administrative or legislative action to efficiently integrate and coordinate federal spending on STEM education programs. Source: Report of the Academic Competitiveness Council, May 2007

6. BPC-NCWIT Evaluation - Boulder 05-16-07 NSF/BPC Alliance Projects: Strategies

7. BPC-NCWIT Evaluation - Boulder 05-16-07 NSF/BPC Alliance Projects: Populations Targeted

8. BPC-NCWIT Evaluation - Boulder 05-16-07 Suggested Variables for Baseline Data-Collection* Suggested Variable Illustrative Disaggregations Study Population gender, ethnicity, disability, education level/cohort, institutional affiliation Institutional Partners number, sector (e.g., academic, corporate, nonprofit), forms of participation (i.e., division of labor) Origin of Intervention number of pre-existing programs (e.g., enrichment, mentoring, internship), ways of distinguishing BPC participants from others (recruits, applicants, enrollees, majors, etc.) Change in Study Population attitudes, enrollment, performance (e.g., class/project grade, GPA, retention, skills/knowledge, aspirations, degree completion) Change in Strategy added/dropped activity, distribution of time/effort in activity, acquisition of new partner, addition of expertise on project staff *see BPC Alliance Methods of Evaluation (handout)

9. BPC-NCWIT Evaluation - Boulder 05-16-07 Evaluation of the BPC Alliance Projects Agenda 10 AMCohort 1 project presentations of baseline data (5-10 minutes each) 11 Cohort 2 project overviews of baseline data (5 minutes each) 11:30Discussion of common needs, challenges, & strategies for data- collection, analysis, & interpretation