1. Experimental design and analysis Experimental design  Gerry Quinn & Mick Keough, 1998 Do not copy or distribute without permission of authors.

2. What is an experiment? Test of hypothesis (prediction) Create conditions under which various predictions (hypotheses) should occur Compare observations under these conditions to observations under original (unaltered) conditions

3. Principles of experimental design Replication Controls Randomisation Reducing unexplained variation Power analysis

4. Replication Biological systems inherently variable –particularly ecological systems Measuring that variation requires replicate experimental or sampling units Avoids confounding: –effect of different factors (variables) cannot be distinguished

5. Replication and pseudoreplication Experimental units must be at a spatial and temporal scale appropriate for the treatments Subsampling of experimental units does not provide true replication, only pseudoreplication (Hurlbert 1984)

6. Effects of fire on small mammals 1 2 3 4 5 6 1 2 3 4 5 6 Burnt Unburnt One burnt location and one unburnt location surveyed for mammals after a fire Each location divided into 6 smaller plots –mammals sampled from each plot Mean no. mammals between locations compared with t test

7. Burning treatment applied to location, not to small plots Plots are only pseudoreplicates –not true replicates True replicates are locations –only 1 replicate per treatment

8. Locations may be different in small mammal numbers irrespective of fire Effect of fire cannot be distinguished from spatial differences between locations –confounding of two factors (burning and location) Differences between locations cannot be attributed to burning

9. No conclusions possible about effect of fire Experiment requires replicate burnt and unburnt areas Burnt Unburnt

10. Constant temperature rooms The effects of increased temperature on growth rate of Acacia terminalis seedlings Two constant temperature rooms set up: –one at 15 o C and one at 30 o C –20 “replicate” seedlings in each CT room

11. Temperature treatment applied to whole CT rooms, not to individual plants Plants are only pseudoreplicates –not true replicates True replicates are CT rooms –only 1 replicate per temperature treatment

12. CT rooms different in other ways that affect growth besides temperature Effect of temperature cannot be distinguished from other differences between CT rooms. –confounding of two factors (temperature and CT room) Differences between CT rooms cannot be attributed to temperature.

13. Other solutions Replicate CT rooms required for each temperature What if only two CT rooms are available? –CT rooms usually expensive and heavily used

14. “Crossover” design possible –run experiment with CT room one at 15 o C and CT room two at 30 o C –then swap CT room and temperature combination so CT room one at 30 o C and CT room two at 15 o C –confounding of temperature and “CT room” less likely Better but still “unreplicated”

15. Point-source pollution study Effects of sewage outfall on species richness of benthic invertebrates on sandy beach –ten cores taken randomly from beach near outfall –another ten cores taken from unaffected beach away from outfall Effect of sewage outfall tested with t- test on mean species richness

16. Sewage applied to whole beach, not to small cores Cores are only pseudoreplicates –not true replicates True replicates are beaches –only 1 replicate per treatment

17. Beaches different in invertebrate diversity irrespective of sewage Effect of sewage cannot be distinguished from spatial differences between beaches –confounding of two factors (sewage and beach) Differences between beaches cannot be attributed to sewage

18. Solutions? Multiple sewage beaches not usually possible (or desirable) - multiple unaffected beaches required! –Is impact beach outside range of natural variation of unaffected beaches? Before-After-Control-Impact (BACI) design?

19. Controls Many factors that could influence the outcome of experiment are not under our control –allowed to vary naturally What would happen if the experimental manipulation had not been performed? –controls

20. Salamander competition Hairston (1989) Hypothesis –2 species of salamanders (Plethodon jordani and P. glutinosus) in the Great Smoky Mountains compete Experiment: –Treatment = P. glutinosus removed from plots –Control = P. glutinosus not removed

21. Treatment: –population of P. jordani increased following P. glutinosus removal Control: –population of P. jordani on control plots (with P. glutinosus not removed) showed similar increase

22. Without controls: –increase in P. jordani might have been incorrectly attributed to P. glutinosus removal –some other factor affecting P. jordani abundance besides P. glutinosus

23. Physiological experiments Effects of a drug on physiology of animal (e.g. lab rat) –compare response of animals injected with drug to response of control animals not injected Differences between control and injected animals may be due to injection procedure –handling effects, injury from needle etc.

24. Suitable control is to inject animals with inert substance –handling effects, injury etc. same for control and treatment animals Any difference between treatment and control animals more likely due to effect of drug

25. Ecological experiments The effect of predatory fish on marine benthic communities (eg. mudflats) –compare areas of mud with fish exclusion cages to areas of mud with no cages. Differences between 2 types of areas may be due to cage effects –shading, reduced water movement, presence of hard structure etc.

26. Must use cage controls –cages with small gaps to allow in fish –shading, water movement etc. same for cages and cage controls Any difference between exclusion and control areas more likely due to effect of fish

27. Randomisation I Experimental units within each treatment must be random sample from population of possible experimental units Haphazard (faster but still unbiased) sampling often used

28. Drugged rats must be random sample of all possible drugged rats about which we wish to draw conclusions –same for control rats Caged plots on mudflat must be random sample of all possible caged plots on mudflat –similarly for control plots

29. Random sampling ensures: –estimates of population parameters (means, treatment effects, variances) are unbiased –statistical inference (conclusions from statistical test) are reliable

30. Randomisation II Experimental units must be randomly allocated to treatment groups. Ensures systematic biases that could influence estimates of treatment effects are less likely.

31. Allocation of replicates to treatments Effect of exercise on [Hb] in blood of flathead Fish sampled from population of fish in aquarium First six fish caught: –used for treatment group –forced to swim rapidly before [Hb] is measured Next six fish: –used as the controls –left undisturbed before [Hb] is measured

32. BUT first 6 fish caught are probably slower or more stupid or less healthy, and hence easier to catch Bias/confound our estimates of the effects of exercise

33. E E E E E C C C C C Randomisation vs interspersion Experiment on effects of fish predation on mudflats. –randomly choose 10 plots –randomly allocate 5 as fish exclusion (E) plots and 5 as cage-control (C) plots Possible random arrangement:

34. Interspersed of treatments important –avoids confounding fish effects with other spatial differences Re-randomise –to get reasonable interspersion –decide a priori unacceptable degree of spatial clumping –single rerandomisation usually improves interspersion

35. Why not systematic? Why not arrange the treatments in a systematic way to guarantee perfect interspersion: E E E E E CCC CC

36. Problems with systematic arrangement Positions of plots not random: –not random sample of any population? Non-random spacing interval between neighbouring treatments: –may coincide with unknown environmental fluctuation –confound treatment effects

37. Experimental design and analysis Statistical decisions and power analysis  Gerry Quinn & Mick Keough, 1998 Do not copy or distribute without permission of authors.

38. Statistical decisions Conclusions of statistical hypothesis test –Reject null hypothesis –Do not reject null hypothesis

39. Statistical conclusion Biological situation Reject Ho (Effect) Retain Ho (No effect) Effect H O false Correct decision Effect detected No effect H O true Correct decision No effect detected None exists Type I error False rejection Type II error Failure to detect an effect Decision errors

40. Type I error (  ) –probability of false rejection of H O –significance level (usually 0.05) Type II error (  ) –probability of false retention of H O Power of test

41. Probability of detecting an effect if it exists Probability of rejecting incorrect H O 1 - 

42. Statistical power depends on Effect size (ES) –size of difference between treatments –large effects easier to detect than small effects Background variation: –variation between experimental units (  2 estimated by s 2 ) –greater background variability, less likely to detect effects

43. Sample size (n) for each treatment group: –increasing sample size makes effects easier to detect Significance level (  ): –Type I error rate –as  decreases,  increases, power decreases

44. ES  n Power (1-  )  s 2 If  is fixed (usually at 0.05), then ES  n Power (1-  )  s 2 Power analysis

45. Exact formula depends on statistical test (i.e. different for t, F etc.) but power charts and tables available in many books and some software will do these calculations.

46. If conclusion is non-significant: report power of experiment to detect relevant effect size. Solve power equation for specific ES: Post-hoc (a posteriori) power analysis ES  n Power (1-  )  s 2

47. Karban (1993) Ecology 74:1-8 Plant growth and reproduction in response to reduced herbivores. Two treatments: –normal herbivore damage –reduced herbivore damage –n = 31 plants in each treatment For plant growth: F 1,60 = 0.51, P = 0.48 ns

48. Karban (1993) Ecology 74:1-8 Power to detect effects: Small effect (ES = 0.1)0.11 Medium effect (ES = 0.25)0.50 Large effect (ES = 0.40)0.88 Effect size (ES) =  MS Groups /  MS Residual ie. SD Groups / SD Reps - see Cohen (1992)

49. A priori power analysis - sample size determination To determine appropriate sample size a priori, we need to know: what power we want background variation (from pilot study or previous literature) what ES we wish to be able to detect if it occurs

50. Solve power equation for n Power * s 2 n   ES

51. Example of a priori power analysis Effects of fish predation on mudflat crabs Two treatments: –cage vs cage control Pilot study: –number of crabs in 3 plots –variance was 19 (so s 2 = 19) –mean was 20

52. Aims: –to detect 50% increase in crab numbers due to caging, ie. an increase from 20 to about 30; so ES = 50% (or 10 crabs per plot) –to be 80% sure of picking up such an effect if it occurred; so power = 0.80 How many replicate plots required for each treatment? –what is required n?

53. Use version of power relationship for 1 factor ANOVAs: Sample sizePower 20.269 30.575 40.781 50.893 50% increase (mean 20 to mean 30) We need 5 replicate plots per treatment to be 80% sure of detecting a 50% change in crab numbers if it occurs.

54. 25% increase (mean 20 to mean 25) Sample sizePower 20.110 80.579 100.689 120.775 140.822 We need about 14 replicate plots per treatment to be 80% sure of detecting a 25% change in crab numbers if it occurs.

55. Effect size How big: –what size of effect is biologically important? –how big an effect do we want to detect if it occurs? Where from? –biological knowledge –previous work/literature –compliance requirements (e.g. water quality)

56. Specification of effect size Easy for 2 groups: –difference between 2 means Harder for more than 2 groups: Consider 3 groups: –50% difference from smallest to largest –1 = 2 < 3? –1 < 2 < 3? –1 < 2 = 3?

57. Spare stuff

58. Principles of experimental design Replication –using more than one experimental or sampling unit per treatment group Controls –what would have happened without the experimental treatment

59. Randomisation –populations must be randomly sampled –treatments must be randomly allocated to experimental units

60. Reducing unexplained (residual) variation –recording additional independent (explanatory or predictor) variables –multiple regression, multi-factor ANOVAs, analysis of covariance, blocking

61. Power analysis –Type II errors –determine appropriate sample size –help interpret nonsignificant results?

62. Temporal confounding Effects of floods in streams on drifting insects Six artificial stream channels with drift nets Two treatments: –high flow –normal flow Six replicates per treatment required

63. Experiment at two times –six replicates of high flow at time 1 –six replicates of normal flow at time 2

64. Effects of flow are completely confounded with differences between the two times. Solution: –three replicates of each treatment at each time –two factor experiment (treatment & time)

65. Ecological experiments II The effect of height on shore on gastropods –compare gastropods translocated to lower levels with those left where they were. Differences between 2 groups of gastropods may be due to translocation effects –time off the rock, unfamiliar surroundings etc.

66. Must use better controls: –control gastropods picked up and handled in same way as transplocated animals –replaced at original height but in new place Additional control: –gastropods at original height that are not moved –test for the effects of handling