1. Objectives of Session Three Focus Group and Survey Assignment Formative Design Proposal Construct Validity Reliability in Data In-class Data Exercise

2. Focus Group/Survey Project Select topic and moderator(s) ½ hour – no penalty if you do not finish your guide Short ice-breaker, background on topic, and questions Be prepared with follow-up questions Handout relating to survey design

3. Sources for Formative Design Proposal Conduct an internet search using keywords for your program area of interest and evaluation terms Points of Light Foundation - http://www.pointsoflight.org/ http://www.pointsoflight.org/ United Way - http://national.unitedway.org/ http://national.unitedway.org/ Electronic Policy Network - http://movingideas.org/ http://movingideas.org/ Evaluation Center - http://www.wmich.edu/evalctr/index.html http://www.wmich.edu/evalctr/index.html Michigan Non-profit link - http://comnet.org/index.html http://comnet.org/index.html Community Foundation of CNY - http://www.cnycf.org/nonprofit/grantees.cfm http://www.cnycf.org/nonprofit/grantees.cfm

4. Construct Validity Does the measure capture the construct of interest? How well does the measure capture the process at hand? -Face validity -Content validity -Convergent v. Discriminant validity

5. Achieving Construct Validity Propose different possibilities for valid measures or operationalizations of the following concepts: –Quality of an individual’s diet –Customer satisfaction with visit to the Dept. of Motor Vehicles –Racial or ethnic identity –Physical limitations

6. Reliability Free of error Consistency in measurement and in coding Data cannot be valid unless it is reliable, but reliable data can be invalid.

7. Internal and External Validity Internal Validity: Does the design allow us to reach causal conclusions? External Validity: Are your findings generalizable?

8. Improving Validity and Reliability of Data Pretest your questions or data collection techniques Use widely trusted measures Correlations between coders/respondents Develop multiple measures Use qualitative findings to check quantitative findings and vice versa

9. In-Class Data Exercise Data collected from Open Cupboard, nonprofit that opened 10 food pantries 6 months ago What can the data tell us about service delivery? Different measures of initial implementation and service delivery -- no impact or outcomes Frequencies Means and variance Correlation Which measures are most valid? Most reliable?

10. Readings for Next Time: Weiss, Chapters 4 and 7