The schedule moving forward… Today – Evaluation Research – Remaining time = start on quantitative data analysis Thursday – Methodology section due – Quantitative analysis continued (start Exercise 10) Next Tuesday – Exercise 10 Due – Maahs returns Exercise #9 Research proposal (everything is now online) is due on Dec 21 st (Monday)

Evaluation Research PURPOSE rather than specific method Much more popular in last 20 years A form of “applied” research Results, though designed to impact decision- making, often have no impact

Types of Evaluation Research Needs assessment Cost-benefit study Monitoring study Program evaluation

Doing evaluation (program) research Research questions / measurement as Job #1 – Political context  how do measure – Operationalize “outcome” of interest “Response” variable NEED AGREEMENT: what is the program trying to do? – Operationalize “processes” Intermediate objectives – Context of – ”

Research designs for evaluation Experimental – Problems, ethics (placebo) – The “black box” issue – Quasi-experimental designs Time Series (multiple time series) Qualitative evaluations – Low birth weight study

Evaluation Research ISSUES Evaluation research as “MESSY” – Administrative control, context of “real life” Importance of looking at “black box” Ethics – The intervention itself can raise issues – Control group members – Violating Ethics 101 (Tuskegee) – Evaluation impacts people’s lives

The (non) use of findings Purely rational/scientific world vs. the one we inhabit – Nixon panel on porn – Scared Straight – 3 Strikes and You’re Out legislation Studies gone wrong… – Fire the evaluator (or dismiss as pointy headed idiot) and keep the program – Why not trust/use the results? Scientist/practitioner gap “True believers” Vested interests

Bivariate Analysis Backed up with a little inferential statistics……yeah baby!

Review Descriptive statistics – Purpose? – Types? – These are “univariate” statistics Explanatory research – Attempt to demonstrate cause-effect Necessary Criteria?

Demonstrating Associations Bivariate (2 variables) analysis – There are a number of ways to do this The method you choose depends largely on characteristics of the two variables – Levels of Measurement?

Contingency Tables (cross tabs) Some find these very intuitive…others struggle – It is very easy to misinterpret these critters Convention: the independent variable is on the top of the table (dictates columns) and the dependent variable is on the side (dictates rows). – What is in the individual “cells?” Frequencies (number of cases that fit criteria) Convert to Percentages: a way to standardize cells and make relationships more apparent

Example A survey of 10,000 U.S. residents Research question: is one’s political view related to attitudes towards police? – What is the DV and IV? – In constructing a table, what goes where?

An Example Attitude Towards Police Political Party Total RepubDemocratLibertarianSocialist Favorable Unfavorab Total

The Percentages of Interest Attitude Towards Police Political Party Total RepubDemocratLibertarianSocialist Favorable2900 (60%) 2100 (54%) 180 (53%) 30 (52%) 5210 Unfavorab Total

Inferential Statistics Researchers are typically not interested in whether there is a relationship in the sample – Want to know about the population – Why might there be a difference between what you find in the sample and what actually exists in the pop? Even with a probability sample, there is always sampling error Without a probability sample = error + bias

Inferential Statistics II Cannot assume that the relationship in your sample is true in population – BUT: probability theory allows us to estimate: “The likelihood of obtaining a particular finding if in the population, there was no relationship” OR “The likelihood of obtaining a particular finding assuming the null hypothesis”

Test Statistics and Significance Tests What does “statistically significant” mean? – Not due to sampling error What do you need to do to test for statistical significance? 1.A “test statistic” An indicator of how different the sample is from “no relationship” 2.The level of error that is acceptable (.05,.01) 3.Degrees of Freedom

The Test Statistic for Contingency Tables Chi Square, or χ 2 – Calculation Observed frequencies Expected frequencies – Can get confusing calculating these critters by hand – Intuitive: how different are the observed cell frequencies from the expected (under null hypothesis) cell frequencies – Degrees of Freedom: (# of Rows -1) (# of Columns -1)

The Chi-Square Sampling Distribution (Assuming Null is True)

Conventional Significance Testing 1.Calculate the test statistic 2.Set your “Type II” error level – The risk of being wrong that you are willing to live with 3.Find the “critical region” within the distribution for your sample statistic – How far out on the curve does your test statistic have to get before you can reject the null hypothesis 4.Decide whether to reject, or fail to reject the null hypothesis

Significance Testing in SPSS Within your crosstabs, select “Chi-Square” – SPSS gives you the χ2 value What you would calculate based on the observed and expected cell frequencies – SPSS also gives you “p” The exact probability of obtaining this χ2, if indeed the null hypothesis was correct In newer versions of SPSS, the p values are listed under “sig” or “significance” or something similar

Review of Contingency Tables Level of Measurement – Both IV and DV are Nominal or Ordinal (Categorical data) Constructing a table – IV on top (columns), DV on bottom (rows) With this format, select “column percentages”

Review of Inferential Statistics Purpose of inferential statistics – Figure out the odds that a finding from sample is due to “chance” or “sampling error” Process – Assume null hypothesis is correct Calculate the odds of obtaining your particular finding under this assumption If the odds are very low, you may be suspicious that the null hypothesis is incorrect At some point (research sets level), you say that the odds are so low that you are going to go ahead and reject the null hypothesis

Review of Chi-Square A measure of how different observed (from your sample data) cell frequencies are from what would be expected under the null hypothesis (e.g., no relationship) – The Chi-square distribution changes shape with different degrees of freedom This is because, by definition, the more cells you have, the higher χ2 gets – df = (R-1)(C-1)

Interpreting Chi-Square Chi-square has no intuitive meaning, it can range from zero to very large – The real interest is the “p value” associated with the calculated chi-square value – This is the exact probability of obtaining the chi- square if in the population there was no relationship In other words, the exact probability of finding that chi- square, under the null hypothesis