1. Broadening Participation in Computing (BPC) Alliance Evaluation Workshop Patricia Campbell, PhD Campbell-Kibler Associates, Inc. campbell@campbell-kibler.com

2. Evaluation Basics: Soup, Cooks, Guests & Improvement When cooks taste the soup, its formative evaluation; the collection of information that can be used to improve the soup. If necessary, the cooks next step is to explore strategies to fix the problem. The cook makes some changes and then re-tastes the soup, collecting more formative evaluation data. When the guests taste the soup at the table, theyre doing summative evaluation. They are collecting information to make a judgment about the overall quality and value of soup. Once the soup is on the table and in the guests mouths, there is little that can be done to improve that soup. Thanks to Bob Stake for first introducing this metaphor.

3. Challenging Assumptions When I was a physicist people would often come and ask me to check their numbers, which were almost always right. They never came and asked me to check their assumptions, which were almost never right. Eli Goldratt

4. Pats Evaluation Assumptions The core evaluation question is What works for whom in what context? The core evaluation question is What works for whom in what context? Black hole evaluations are bad. Black hole evaluations are bad. If you arent going to use the data, dont ask for it. If you arent going to use the data, dont ask for it. A bad measure of the right thing is better than a good measure of the wrong thing. A bad measure of the right thing is better than a good measure of the wrong thing. Acknowledging WIIFM increases response rates. Acknowledging WIIFM increases response rates. Process is a tool to help understand outcomes. Process is a tool to help understand outcomes. Outcomes are at the core of accountability. Outcomes are at the core of accountability.

5. Some Thoughts on Measurement Dont reinvent the wheel; where possible use existing measures Dont reinvent the wheel; where possible use existing measures Share measures with other projects. Common questions can be useful. Share measures with other projects. Common questions can be useful. Look for benchmark measures that are predictors of your longer term goals. Look for benchmark measures that are predictors of your longer term goals. All self developed measures need some checking for validity and reliability. All self developed measures need some checking for validity and reliability. A sample with a high response rate is better than a population with a low one. A sample with a high response rate is better than a population with a low one.

6. Some Web-based Sources of Measures OERL, the Online Evaluation Resource Library. http://oerl.sri.com/home.html http://oerl.sri.com/home.html ETS Test Link (a library of more than 25,000 measures). http://www.ets.org/portal/site/ets/menuitem.148851 2ecfd5b8849a77b13bc3921509/?vgnextoid=ed462d3 631df4010VgnVCM10000022f95190RCRD&vgnextcha nnel=85af197a484f4010VgnVCM10000022f95190RCR D http://www.ets.org/portal/site/ets/menuitem.148851 2ecfd5b8849a77b13bc3921509/?vgnextoid=ed462d3 631df4010VgnVCM10000022f95190RCRD&vgnextcha nnel=85af197a484f4010VgnVCM10000022f95190RCR D http://www.ets.org/portal/site/ets/menuitem.148851 2ecfd5b8849a77b13bc3921509/?vgnextoid=ed462d3 631df4010VgnVCM10000022f95190RCRD&vgnextcha nnel=85af197a484f4010VgnVCM10000022f95190RCR D

7. Sample Under-represented Student Instruments From OERL http://oerl.sri.com/home.html http://oerl.sri.com/home.html Attitude Surveys Attitude Surveys Attitude Surveys Attitude Surveys Content Assessments Content Assessments Content Assessments Content Assessments Course Evaluations Course Evaluations Course Evaluations Course Evaluations Focus Groups Focus Groups Focus Groups Focus Groups Interviews Interviews Interviews Journal/Log Entries Journal/Log Entries Journal/Log Entries Journal/Log Entries Project Evaluations Project Evaluations Project Evaluations Project Evaluations Surveys Surveys Surveys Workshop Evaluations Workshop Evaluations Workshop Evaluations Workshop Evaluations

8. Compared to What? Evaluation Designs Experimental designs Experimental designs Quasi-experimental designs Quasi-experimental designs Mixed methods designs Mixed methods designs Case studies Case studies NSF does not promote one design, rather it wants the design that will do the best job answering your evaluation questions!

9. Making Comparisons: Why Bother

10. Web-based Sources of Comparisons: K- 12 By state, all public schools have web-based school report cards that include grade level student achievement test scores on standardized mathematics and language arts/reading tests, often disaggregated by race/ethnicity and by sex, for a period of years. The U.S. Department of Educations Common Core of Data (CCD) (http://www.nces.ed.gov/ccd/CCD) reports public school data including student enrollment by grade, student demographic characteristics, and the percent of students eligible for free or reduced price lunches. http://www.nces.ed.gov/ccd/CCD Comparison schools can be selected using the CCD and achievement data for both sets of schools over time can be downloaded from states report cards.

11. Caveats on using Web-based Sources of Comparisons: K-12 These data can be used only if: the goal of the strategy/project is to increase student achievement in a subject area tested by the state; the goal of the strategy/project is to increase student achievement in a subject area tested by the state; participating students have not yet taken their final state mandated test in that subject area; participating students have not yet taken their final state mandated test in that subject area; most of the teachers in a school teaching in that subject area are part of the strategy/project, and/or most of the students studying the subject area are part of that strategy/project. most of the teachers in a school teaching in that subject area are part of the strategy/project, and/or most of the students studying the subject area are part of that strategy/project.

12. Web-based Sources of Comparisons: College and University WebCASPAR database (http://caspar.nsf.gov) provides free access to institutional level data on students from surveys as Integrated Postsecondary Education Data System (IPEDS) and the Survey of Earned Doctorates. The Engineering Workforce Commission (http://www.ewc- online.org/) provides institutional level data (for members) on bachelors, masters and doctorate enrollees and recipients by sex by race/ethnicity for US students and by sex for foreign students. http://www.ewc- online.org/http://www.ewc- online.org/ Comparison institutions can be selected from the Carnegie Foundation for the Advancement of Teachings website, (http://www.carnegiefoundation.org/classifications/) based on Carnegie Classification, location, private/public designation, size and profit/nonprofit status. http://www.carnegiefoundation.org/classifications/

13. Some Web-based Sources of Resources OERL, the Online Evaluation Resource Library. http://oerl.sri.com/home.html http://oerl.sri.com/home.html User Friendly Guide to Program Evaluation http://www.nsf.gov/pubs/2002/nsf02057/start.htm http://www.nsf.gov/pubs/2002/nsf02057/start.htm AGEP Collecting, Analyzing and Displaying Data http://www.nsfagep.org/CollectingAnalyzingDisplayin gData.pdf http://www.nsfagep.org/CollectingAnalyzingDisplayin gData.pdf http://www.nsfagep.org/CollectingAnalyzingDisplayin gData.pdf American Evaluation Association http://www.eval.org/resources.asp http://www.eval.org/resources.asphttp://www.eval.org/resources.asp

14. OERL, the Online Evaluation Resource Library. http://oerl.sri.com/home.html http://oerl.sri.com/home.html Includes NSF project evaluation plans, instruments, reports and professional development modules on Designing an Evaluation Designing an Evaluation Developing Written Questionnaires Developing Written Questionnaires Developing Interviews Developing Interviews Developing Observation Instruments Developing Observation Instruments Data Collection Data Collection Instrument Triangulation and Adaptation. Instrument Triangulation and Adaptation.

15. User Friendly Guide to Program Evaluation http://www.nsf.gov/pubs/2002/nsf02057/start.htm User Friendly Guide to Program Evaluation http://www.nsf.gov/pubs/2002/nsf02057/start.htm Introduction Introduction Introduction Section I - Evaluation and Types of Evaluation Section I - Evaluation and Types of Evaluation Section I - Evaluation and Types of Evaluation Section I - Evaluation and Types of Evaluation Section II - The Steps in Doing an Evaluation Section II - The Steps in Doing an Evaluation Section II - The Steps in Doing an Evaluation Section II - The Steps in Doing an Evaluation Section III - An Overview of Quantitative and Qualitative Data Collection Methods Section III - An Overview of Quantitative and Qualitative Data Collection Methods Section III - An Overview of Quantitative and Qualitative Data Collection Methods Section III - An Overview of Quantitative and Qualitative Data Collection Methods Section IV - Strategies That Address Culturally Responsive Evaluations Section IV - Strategies That Address Culturally Responsive Evaluations Section IV - Strategies That Address Culturally Responsive Evaluations Section IV - Strategies That Address Culturally Responsive Evaluations Other Recommending Reading, Glossary, and Appendix A: Finding An Evaluator Other Recommending Reading, Glossary, and Appendix A: Finding An Evaluator Other Recommending Reading, Glossary, and Appendix A: Finding An Evaluator Other Recommending Reading, Glossary, and Appendix A: Finding An Evaluator

16. AGEP Collecting, Analyzing and Displaying Data http://www.nsfagep.org/CollectingAnalyzi ngDisplayingData.pdf AGEP Collecting, Analyzing and Displaying Data http://www.nsfagep.org/CollectingAnalyzi ngDisplayingData.pdf I. Make Your Message Clear I. Make Your Message Clear II. Use Pictures, Where Appropriate II. Use Pictures, Where Appropriate III. Use Statistics and Stories III. Use Statistics and Stories IV. Be Responsive to Your Audience. IV. Be Responsive to Your Audience. V. Make Comparisons V. Make Comparisons VI. Find Ways To Deal With Volatile Data VI. Find Ways To Deal With Volatile Data VII. Use the Results VII. Use the Results

17. Some Thoughts for Discussion Some Thoughts for Discussion 1. What evaluation resources do you have you would like to share? What evaluation resources would you like to have from others? 1. What evaluation resources do you have you would like to share? What evaluation resources would you like to have from others? 2. What can you (and we) do to make your evaluation results known to and useful to: 2. What can you (and we) do to make your evaluation results known to and useful to: your project? your project? other projects? other projects? Jan? the broader world? the broader world? 3. Why do you think your strategies will lead to your desired outcomes? Use research, theory or just plain logic 3. Why do you think your strategies will lead to your desired outcomes? Use research, theory or just plain logic