1. Experimental Research I Day 4

2. For Tomorrow Email first draft of introduction by class tomorrow. –Follow framework from day 1 and in syllabus –Use published articles as a guide –Feel free to cite authors that support your statements – Non-research articles are OK for this. –Before the purpose statement, explain the problem or the issues involved. This material will lead to your purpose statement.

3. Review Standard Error of the Mean Confidence –Level –Interval –Limits Upper bound Lower bound

4. Experimental Research Only type of research with an intervention A direct attempt to influence a particular variable Only method that can truly begin to untangle cause and effect hypotheses Directional Hypothesis = Theory statement predicting the outcome [directional] (There will be a significant difference…). Reflects researcher’s expectations. Bilingual 3 rd graders taught with the Kodaly method will demonstrate significantly higher musical achievement than bilingual 3 rd graders taught with a traditional eclectic method.

5. Null Hypothesis Null Hypothesis = Theory statement predicting the outcome stated in the negative [non- directional] (There will be no significant difference…) The statistical hypothesis that states that there are no differences between observed and expected data. Does not reflect researcher’s expectations (value free) –There will be no significant difference in musical achievement of bilingual 3 rd graders taught with the Kodaly methods and bilingual 3 rd graders taught with a traditional eclectic method. –The goal is to REJECT the Null Hypothesis based (on 95% Confidence level or above) –Cannot prove the null hypothesis (a negative)

6. Type I and Type II Error Type I Error is erroneously claiming statistical significance or rejecting the null hypothesis when in fact, it’s true (claiming success when experiment failed to produce results) –Possible w. incorrect statistical test Type II Error is when a researcher fails to reject the null hypothesis when it is in fact false (claiming failure when successful) –The smaller the sample size, the more difficult it is to detect statistical significance –In this case, a researcher could be missing an important finding because of study design

7. Group Comparisons Experimental Group –Receives a particular treatment specified by the researcher Control/Comparison Group –Does not receive that particular treatment Sometimes difficult in educational research to have a strict no-treatment, control –Example: Any instruction is likely to be more effective than no instruction

8. Randomization Random assignment to groups –Every individual has an equal chance of being in the experimental or control/comparison group Supposed to help eliminate extraneous sources of variance –For example… if the groups are sufficiently large, any differences between groups on extraneous variables are likely to be due to chance or randomly distributed among the groups Quasi-Experimental=non-randomized groups –Most ed. research –Intact classes & convenience samples –Impacts ability to generalize to whole population

9. Variables Independent variable (IV) –The experimental or treatment variable –This variable is manipulated by the researcher –Examples: instructional approach, environmental condition, the introduction of a particular musical element –Participant attribute Dependent variable (DV) – Compared b/w groups –The criterion or outcome variable –Examples: student attitudes, student achievement, teacher effectiveness as measure by ? Experiments can be expressed as “The effect of the ‘IV’ on the ‘DV’” Extraneous Variables –Those that are not specifically included in the study but never the less may effect the outcome –Object is to control for extraneous variables –The researcher may not know them all

10. Manipulating the IV Presence of the variable vs. absence of the variable –Kodaly instruction (treatment group) vs. Kodaly instruction (control group) One form of the variable vs. another –modeling vs. verbal music instruction (vs. control group) Varying degrees of the same variable –100% positive feedback, no negative feedback vs. 50% positive feedback, no negative feedback

11. Controlling for Extraneous Variables Best case scenario – all individuals are as similar as possible on all variables other than the independent variable Methods to control: –Randomization & large sample –Holding variables constant (freeze private lessons) –Build variable into the design (compare private lessons w/ no private lessons) –Matching pairs – one to control, other to exper. –Statistical control – analysis of covariance (ANCOVA)

12. Design and Experiment [Effect of Colored note heads on Music Reading] State Hypothesis and Null Hypothesis Select sample and assign to group (control and treatment). How many in each? Identify independent and dependent variables. Any possible extraneous variables? Describe experiment. What will you do w/ each group and for how long? How will you know what they already know?

13. Discussion of Projects On task Practice explaining your project –Background; State the problem –Purpose statement –Research questions –Methodology (research design)

14. Experimental Research Designs

15. Nomenclature/Abbreviations When looking at the symbols used to describe various experimental design approaches: –R = random assignment –O = testing (pre- or post-) –X = treatment –C = control/comparison –M = matched

16. Pre-Experimental Designs [Pilot Studies – Generally Weak] One Shot Case Study (X O) –No random assignment, No control/comparison, no pre- test One-Group Pre-test, Post-test (O X O) –No random assignment, No control/comparison group Static/Intact-group Comparison X O –No random assignment O Static/Intact-group Pre-test, Post-test –No random assignment, possible pre-test effects O X O O

17. True Experimental Designs Stronger – not always possible in educ. Randomized Post-test Only, R X O Control Group R O –Still not sure about pre-test levels Randomized Pre-Test, R O X O Post-test, Control Group R O O –Good checking whether smaller groups are actual similar at the start of the study and possible effects of pretest

18. Randomized Solomon Four-Group & Posttest Only, Control Group Solomon 4–Group controls for possible sensitization effects due to testing or maturation. 1. R O X O 2. R O O (maturation or pretesting?) 3. R X O (effect of pretest?) 4. R O (control group) In a successful experiment, what would we expect for each group? What if the Post Test scores for group 2 were as high as the Post Test for group 1? What if the Post Test scores for group 3 were lower than group 1? What if the Post Test scores for group 4 were the same as groups 1 & 2?

19. Counterbalanced Design (Latin Square) Order effect Ex: Introducing note values: 1=tatiti; 2=num/no subdiv; 3=num/sub div; 4 dotted notes

20. Quasi-Experimental Design So called b/c there is no randomization… Matching Only –Participants matched in pairs to control for an extraneous variable rather than randomly assigned Counterbalanced Design (next slide) –Multiple groups receive all treatment types in different order –Average post-test scores across groups are compared to determine effectiveness/effect of the treatment order –Vulnerable to multiple-treatment interference Time-series Design –Outcome measured several times before and after introduction of the treatment O O O O X O O O O

21. Quasi-Experimental Design Factorial Design (2 or more factors (IVs) at different levels) –Examining the effect of more than one independent variable –Allows for examination of attribute variables (i.e. gender, age) and interaction effects b/w combinations of IVs –Example: Kodaly vs. trad. instruction for bilingual and non-bilingual students Possible outcome showing interaction of two IV’s Non-bilingual students may do equally well w/ Kod. and traditional methods, while bilingual students may do better w/ Kod. vs. traditional methods. What if you had not separated these groups out?

22. Factorial Example (Two Way - 2x2) IVs = Language classification (biling. vs. non-biling.) & method (Kodaly vs. trad.) DV = musical achievement test? Groups (Six 3 rd gr. Sections-3 Kodaly; 3 Trad; Bilingual & Non-Bilingual in all groups.) –Bilingual & Kodaly –Non-Bilingual & Kodaly –Bilingual & Traditional –Non-Bilingual & Traditional

23. 2 Way Factorial Designs (2 independent variables [often one manipulated, one attribute)

24. Internal Validity (Usefulness/Meaningfulness) - Control of Extraneous Variables: Time Bound Factors What happens within the experiment –History – What happens b/w pretest and posttest (private lessons, change in practice routine) –Maturation – is change result of treatment natural result of repetition and improvement over time?) –Mortality – Loss of participants may cause imbalance b/w groups

25. Internal Validity – Sampling & Measurement Factors Testing – pretest affect posttest. Ceiling and floor effects (eliminate outliers?) Instrumentation – changes in measurement or observers (judges at contest from one site to the next) Statistical regression – students who score extremely high (ceiling) or low (floor) on pretest may regress to the mean on posttest Selection – participants do not represent normal population (also affects external validity) Interactions – influence of a combination of the above factors

26. Internal Validity John Henry Effect –Control group performs beyond usual level because they perceive they are in competition with the experimental group

27. External Validity – Generalizability Population Validity –Extent sample is representative of the population to which the researcher wishes to generalize the results. Ecological –Study conditions and setting are representative of the setting in which the researcher would like to apply the findings Replication –Results can be reproduced (problem w/ Mozart effect) Detailed description of the sample needed in study –Important regardless of sampling method –‘Next best thing’ if not a large, random sample – often the case in music ed. research –Consider demographic questions in descriptive research

28. Other Threats to External Validity Effect or interaction of testing (testing will not occur in natural setting) Sample does not reflect population –Discuss in research report Reactive effects of sample –Hawthorne Effect Effects due simply to subjects’ knowledge of being in a study –Teacher or Researcher interactions different than in population Subconsciously encouraging or discouraging a group Research setting does not reflect typical settings (ecological validity) –A university lab school