1. Statistical Methods in Computer Science Hypothesis Testing III: Categorical dependence and Ido Dagan

2. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 2 Testing Categorical Data Single-factor, with categorical dependent variable Determine effect of independent variable values (nominal/ categorical too) on the dependent variable treatment 1 Ind 1 & Ex 1 & Ex 2 &.... & Ex n ==> Dep 1 treatment 2 Ind 2 & Ex 1 & Ex 2 &.... & Ex n ==> Dep 2 control Ex 1 & Ex 2 &.... & Ex n ==> Dep 3 Compare performance of algorithm A to B to C.... Cannot use the typical ANOVA testing

3. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 3 Testing Categorical Data Single-factor, with categorical dependent variable Determine effect of independent variable values (nominal/ categorical too) on the dependent variable treatment 1 Ind 1 & Ex 1 & Ex 2 &.... & Ex n ==> Dep 1 treatment 2 Ind 2 & Ex 1 & Ex 2 &.... & Ex n ==> Dep 2 control Ex 1 & Ex 2 &.... & Ex n ==> Dep 3 Compare performance of algorithm A to B to C.... Dep1, Dep2, Dep3,.... are categorical data Cannot use the numerical ANOVA testing Values of independent variable Values of dependent variable

4. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 4 Contingency tables We collect categorical data in contingency tables Different screens and their hue bias

5. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 5 Contingency tables We collect categorical data in contingency tables Different screens and their hue bias Coffee and tea drinkers in a sample of 100 people

6. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 6 Contingency tables We collect categorical data in contingency tables Different screens and their hue bias Coffee and tea drinkers in a sample of 100 people Operating System used at home

7. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 7 Is there a relation? We want to know whether there exists a dependence between the variables, or between what we observed and some expectation e.g., are tea-drinkers more likely to be coffee-drinkers? e.g., is the selection of operating systems different from expected? To find out, cannot ask about means, medians,... Focus instead on proportions/distributions: Does the observed distribution differ from the expected one?

8. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 8 Single Sample Suppose we have a-priori notion of what's expected For instance, selection of OS is expected to be uniform How do we tell whether the observed data is different?

9. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 9 Hypotheses Refers to the number of data points of each value H 0 : Expected distribution = underlying distribution of observed data H 1 : Expected != Observed Need a method to measure likelihood of null hypothesis

10. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 10 Difference in Expectations Focus on the difference between expected and observed For instance: Expected (out of 256 sampled homes) 25% each operating system = 64 homes each operating system Observed (out of 256 sampled homes)

11. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 11 Chi-square The chi-square ( ) of given data is: Where: f e is the expected frequency of the value f o is the observed frequency of the value The sum is over all values Recall from Cramer correlation Chi-square values of random samples are distributed according to the chi-square distribution A family of distributions, according to degrees-of-freedom Large enough chi-square indicates significance – reject null hypothesis Too low values are unlikely – check for error

12. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 12 In our example This is compared to chi-square distribution with df = 3 Result: highly significant

13. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 13 Features of chi-square Cannot be negative (sum of squares) Only tests for difference from expected distribution But here it is a one-tailed test (only tests for greater value) Equals 0 when all frequencies are exactly as expected It's not the size of the discrepancy, but its relative size Relative to the expected frequency Depends on the number of discrepancies This is why we have to consider degrees of freedom – the more df, the more discrepancies we might get by chance

14. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 14 Comparing against arbitrary distribution Do we always need to assume population is uniform/known? No: Instead of expecting: Can have different expectations (from past), for instance:

15. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 15 Comparing against arbitrary distributions This is compared to chi-square distribution with df = 3 Result: not significant Same procedure may be used for testing against any expected distribution (e.g. Normal, to decide on t-test applicability)

16. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 16 Multiple-variable contingency tables Correlation-testing Two numerical variables Single-factor testing (one-way ANOVA) Independent variable: Categorical Dependent variable: Numerical Two-way contingency table: two categorical variables As seen for Cramer correlation Different screens and their hue bias

17. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 17 Hypotheses about contingency tables Null hypothesis: values are independent of each other Analogous to correlation and ANOVA tests Alternative (H 1 ): values of variables are dependent How do we use chi-square here? Calculate expected frequency from margin probabilities Calculate chi-square value Compare to chi-square distribution

18. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 18 Example Independent probabilities: Any display has probability of 140/400 of being bluish, 160/400 of being reddish, 100/400 of being greenish Any hue has probability of 200/400 of being of display 1, 100/400 of display 2, 100/400 of display 3

19. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 19 Example Given the above table, what is the likelihood of: Being a blueish display 1 ? Being a greenish display 2?

20. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 20 Example: Expected Frequencies Given the above table, what is the likelihood of: Being a blueish display 1 ? Being a greenish display 2? 25 cases 70 cases

21. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 21 Translating into expected frequencies Observed Expected

22. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 22 Chi-square of contingency table In this example the chi-square is: Observed-expected Squared, summed... chi-square

23. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 23 Degrees of freedom of contingency table There are two variables One has 2 degrees of freedom (3 possible values) Other has 2 degrees of freedom (3 possible values) Total: 2 * 2 = 4 degrees of freedom Knowing the marginal values, setting 4 values in the table enables to derive the 5 remaining ones In general: df = (#rows – 1) · (#columns – 1) Comparison against chi-square distribution with df=4: Significant: p < 0.01

24. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 24 Chi-square in Spreadsheet Excel, openoffice spreadsheets have chi-square testing chitest(observed,expected) Where observed, expected are data-arrays of same size Give back p value of null hypothesis

25. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 25 Interpreting dependence Knowing variables are dependent does not tell us how Analogical to single-factor ANOVA testing we learned (but unlike Pearson correlation, which is directional) Have to consider interpretation carefully For instance, consider the following data Suppose we are investigating the following relation: Are people drinking tea likely to drink coffee? (i.e., tea-drinking correlated with coffee-drinking)

26. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 26 Interpreting Dependence We gather the following results: The chi-square test is significant: p = 0.05 But what is the direction of dependence?

27. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 27 Interpreting Dependence We gather the following results: Naively: Out of 25 tea drinkers, 20 also drink coffee (80% ! )

28. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 28 Interpreting Dependence We gather the following results: Naively: Out of 25 tea drinkers, 20 also drink coffee (80% ! ) But: 90% of people drink coffee (a-priori) 93% of non-tea drinkers drink coffee

29. Empirical Methods in Computer Science © 2006-now Gal Kaminka/Ido Dagan 29 Interpreting Dependence We gather the following results: Naively: Out of 25 tea drinkers, 20 also drink coffee (80% ! ) But: 90% of people drink coffee (a-priori) 93% of non-tea drinkers drink coffee So: Negative dependence between tea and coffee! (can be revealed by comparing observed to expected)