1. Techniques expérimentelles 2 Barbara Hemforth Most of this is stolen from a lecture by Chuck Clifton http://webcache.googleusercontent.com/search?q=cache:nPJ6Gh wZ2NkJ:people.umass.edu/cec/Experimental%2520Design%2520f or%2520Linguists.ppt+Clifton+experiments+linguists&cd=5&hl=fr &ct=clnk&gl=fr&client=safari&source=www.google.fr

2. II: How to do experiments. Part 1, General design principles Dictum 1: Formulate your question clearly Dictum 2: Keep everything constant that you don t want to vary Dictum 3: Know how to deal with unavoidable extraneous variability Dictum 4: Have enough power in your experiment Dictum 5: Pay attention to your data, not just your statistical tests C. Clifton Jr

3. Dictum 1: Formulate your question clearly Independent variable: variation controlled be experimenter, not by what subject does Dependent variable: variation observed in subject s behavior, perhaps dependent on IV Operationalization of variables C. Clifton Jr

4. Dictum 2: Try to keep everything constant except what you want to vary Try to hold extraneous variables constant through norms, pretests, corpora… When you can t hold them constant, make sure they are not associated (confounded) with your IV

5. What happens when there is unavoidable variation? Subdictum B: When in doubt, randomize – Random assignment of subjects to conditions – Questionnaire: order of presentation of items? Single randomization: problems Different randomization for each subject Constrained randomizations Equate confounds by balancing and counterbalancing – Alternative to random assignment of subject to conditions: match squads of subjects

6. Counterbalancing of materials Counterbalancing – Ensure that each item is tested equally often in each condition. – Ensure that each subject receives an equal number of items in each condition. Why is it necessary? – Since items and subjects may differ in ways that affect your DV, you can t have some items (or subjects) contribute more to one level of your IV than another level.

7. Sometimes you don t have to counterbalance If you can test each subject on each item in each condition, life is sweet E.g., Ganong effect (identification of consonant in context) – Vary VOT in 8 5-ms steps /dais/ - /tais/ /daip/ - /taip/ – Classify initial segment as /d/ or /t/ Present each of the 80 items to each subject 10 times Ganong effect: biased toward /t/ in type, /d/ in dice Connine, C. M., & Clifton, C., Jr. (1987). Interactive use of information in speech perception. Journal of Experimental Psychology: Human Perception and Performance, 13, 291-299.

8. If you have to counterbalance… Simple example – Questionnaire, 2 conditions, N items – Need 2 versions, each with N items, N/2 in condition 1, remaining half in condition 2 Versions 1 and 2, opposite assignment of items to conditions More general version – M conditions, need some multiple of M items, and need M different versions Embarrassing if you have 15 items, 4 conditions… That means that some subjects contributed more to some conditions than others did; bad, if there are true differences among subjects

9. Counterbalancing things besides items Order of testing – Don t test all Ss in one condition, then the next condition… – At least, cycle through all combinations of conditions (all lists) before testing a second subject with the same list – Fancier, latin square Avoid minor confound if always test cond 1 before cond 2 etc. N x n square, sequence x squad, containing condition numbers, such that each condition occurs once in each column, each order Location of testing – E.g., 2 experiment stations

10. Latin Squares (Euler, 1773) Latin square of order 2Latin square of order 3 a bx y z b az x y y z x A latin square of order n is an n by n array of n symbols in which every symbol occurs exactly once in each row and column of the array.

11. Variance in an experiment Systematic variance: variability due to manipulation of IV and other variables you can identify Random variance: variability whose origin you re ignorant of Point of inferential statistics: is there really variability associated with IV, on top of other variability? – Is there a signal in the noise?

12. Best way to deal with extraneous variability: Minimize it! Keep everything constant – Reduce experimental noise See the signal easier – Keep environment, instructions, distractions, experimenter, response manipulanda, etc. constant – Pretest subjects and select homogeneous ones, if that suits your purposes

13. One way to minimize extraneous variance: Within-subject designs Subjects differ – …a lot, in some measures, eg. Reading speed, reaction time Present all levels of your IV to each subject – Assume the subject effect is a constant across all the levels. – Differences among conditions thus abstracted from subject differences Counterbalancing necessary – Test each item in each condition for an equal number of subjects. Worry about experience changing what your subject did – E.g., will reading an unreduced relative clause (The horse that was raced past the barn fell) affect reading of a reduced relative clause sentence?

14. Dictum 4: Have enough power to overcome extraneous variability Add more data! – Minimizes noise component of differences among condition means Law of large numbers – The larger the sample size, the more probable it is that the sample mean comes arbitrarily close to the population mean – If you re (almost) looking at population means, any differences have to be real – not sampling error

15. Dictum 5: Pay attention to your data, not just your statistical tests Look at your data, graph them, try to make sense out of them – Don t just look for p <.05! Examine confidence intervals Look at your data distributions – Stem and leaf graphs – By subjects…

16. Magnitude estimation: Stevens Power Law http://www.cis.rit.edu/people/faculty/mon tag/vandplite/pages/chap_6/ch6p10.html

17. Stevens Power Law http://www.cis.rit.edu/people/faculty/mon tag/vandplite/pages/chap_6/ch6p10.html

18. Stevens Power Law http://www.cis.rit.edu/people/faculty/mon tag/vandplite/pages/chap_6/ch6p10.html

19. Magnitude estimation: an example Which man did you wonder when to meet? Assign an arbitrary number to that item, greater than zero. Now, for each of the following items, assign a number. If the item is better than the first one, use a larger number; if it s worse, smaller. Make the item proportional to how much better or worse the item is than the original – if twice as good, make the number 2x the start; if 1/3 as good, make the number 1/3 as big as the start.

20. Magnitude estimation : an example Which man did you wonder when to meet? – Assign an arbitrary number, greater than 0, to this first item. – Now, for each successive item, assign a number – bigger if the item is better, smaller if worse, and proportional – if the item is 2x as good, make the number 2x the original; if ¼ as good, make the number ¼ as big as the original. Which book would you recommend reading? When do you know the man whom Mary invited? This is a paper that we need someone who understands. With which pen do you wonder when to write. Who did Bill buy the car to please? Bard, E. G., Robertson, D., & Sorace, A. (1996). Magnitude estimation of linguistic acceptability. Language, 72.