CONDUCTING EDUCATIONAL RESEARCH Guide to Completing a Major Project Daniel J. Boudah Chapter 5 Designing and Conducting Experimental Research Lucy B. Houston, Presenter

Chapter Objectives: reader will understand... Characteristics of experimental and quasi-experimental research design. The link between research questions and research design. Commonly used experimental and quasi-experimental design. The characteristics of single-subject designs. Common ways to measure dependent variables and types of data. How to align research questions with design, measurement, and data. How to be prepared to implement experimental research interventions.

Key Points Validity and outcome of the study hinge on selecting the appropriate design. Experimental designs are the “blueprint” for the study. The fit of a design must be linked to the research questions and hypothesis. Know the purpose and characteristics of the common designs used in experimental research.

Review: Distinguish Descriptive from Experimental Research Descriptive Describes a condition or phenomenon - Descriptive Identifies a potential cause/effect relationship – Casual Comparative Explores relationship among variables, i.e. strength of relationship - Correlational Experimental Makes inferential connections between variables Attempts to “establish cause and effect relationships” between variables Intervenes to determine if X (the independent) variable will effect Y (dependent variable). Participants are a part of an experimental group (with or without a control group) and single subject.

Types of Experimental Research Experimental Attempts to determine if a one variable has an effect on the other. Independent variable, the intervention and dependent variable, the outcome. Random assignment to sample, higher level of control of extraneous variables, and generalization. Quasi-experimental Attempts to determine if a one variable has an effect on the other. Independent variable, the intervention and dependent variable, the outcome. Do not randomize sample, limitation to generalizing outcome.

Characteristics of Experimental Designs Figure 5.1: page 95 Characteristics of Experimental Designs Figure 5.1: page 95 Selection of experimental participants Direct manipulation of an independent variable Control of extraneous variables Measurement of outcomes Also quantitative research

Steps in Experimental Research Design Figure 5.2, page 96 Define what variables you are interested in and how you might address issues of validity. Select a design that will enable you to investigate the variables. Decide how you want to measure the impact of independent variable on the dependent variable. Think about the kind of data will want to collect through your measurement. Decide how you want to analyze the effects of the intervention

Step 1: Define what variables you are interested in and how you might address issues of validity. Questions determine the Design

Questions are the basis for determining the research design because... stem from interest in a problem or phenomenon, i.e. the outcome or dependent variable includes a statement of the intervention, i.e. the independent variable. names the target population or participants provides an idea of the methodology or measurement

Interactive Question: Do teachers participant more in profession development when they receive a stipend? Possible hypothesis: There is a significant difference in participation in staff development when teachers receive a stipend. What is the problem? What is the dependent variable? What is the intervention or treatment? What is the independent variable?

Choosing an intervention Draft a timeline Materials and resources needed Identify the who, what, where Consider the characteristics, planning, conditions and activities Focus on a specific outcome Brainstorm and weigh the pros and cons

Revisiting validity Construct Measurement tool measures what it claims to measure. Examples: Memory, intelligence, reading fluency Internal Accuracy of conclusion that an intervention made a significant difference Most often questioned when differences can be explained by demographics External Extent to which results can be generalized beyond investigation. Adequate sample of population Feasibility of replication

Step 2: Select a design that will enable you to investigate the variables. Group Designs and Single-Subject Designs

Group Design Posttest Only Design: group → intervention → Posttest Pretest-Posttest Design/Paired Sample: Group → Pretest → Intervention → Posttest / Threats to external validity due to lack of a comparison group and using same test for pre and post test Comparison Group Design/Control group or static group:Experimental Group→ Pretest→ Intervention→ Posttest Comparison Group→ Pretest →no intervention → Posttest/Threats to validity is comparability of two groups (Homegenity of variance) and only difference in experiences of participants is the intervention (confounding variables).

Group Designs FactorialExperimental Group → Pretest → Intervention → Posttest (1, 2, 3, 4): Enables research to study multiple independent variables that may cause observed problem or effect. Example: study of the effect of four variables with two conditions (control and experimental), 4 x 2 factorial design. Time-Series Group → M → M → M → I → M → M → M: Multiple assessments prior to intervention to determine a “valid baseline”. Threats to valid is the treatment of one group.

Interactive Mrs. McGrady attended a workshop on vocabulary development. She decided to test the effectiveness of the multiple word meaning strategy by using it as an intervention for her Bubble kids during guided reading. In order to determine if the new vocabulary strategy worked she administered a pretest before the treatment and a posttest after six weeks. True or False Mrs. McGrady used a quasi-experiemental design.

Single-Subject Designs Characteristics Study of an individual Repeated measurement of dependent variable before and after the intervention Baseline and intervention results charted or graphed rather than statistical analysis. Variations A-B-A design – Baseline/Intervention/Baseline Multiple-baseline design – intervention begins at different points in time for two or more participants

Step 3: Decide how you want to measure the impact of independent variable on the dependent variable. Measurement tools must be valid, reliable and meaningful

Selecting a measurement tool page 106 Align tightly to the intervention. Use validated measurement tools along with those you develop. A priori calculation of coefficient alpha for new measures and observations. Balance of global and specific measures. Learn about construct validity as you conduct intervention research

Measurement Tools Observations Formal – identify and record targeted behavior using event-based observations (frequency of behavior) or time-based observations (duration of behavior), example: using tally marks to determine exact number of times behavior occurred with a set time frame or counting time intervals. Excellent records for conferences with parents or principles about student behavior. Informal – anecdotal records and field notes, less structured Test and Permanent Records – State performance test, attendance, office referrals, and grade point average

Measurement Tools continued Criterion-Referenced Assessments Predetermined level of mastery set by school district, state education agency or teacher. % correct indicates mastery, e.g. 90% compared to norm referenced, i.e. same age/same grade level average. Continuous Assessment curriculum-based measurement (CBM) Commercial products developed using state and district curricula e.g. TPRI, DRIBELS, DRA Teacher administered/ Personal digital assistants (PSAs) Interviews ask questions to determine participants’ personal perceptions. To determine student’s cognition or strategies used. “Think Aloud” technique In combination with other measures, strengthens validity.

Clearly state the type of measurement used to measure your dependent variable in you research proposal or report.

Text to World Connection

Step 4: Think about the kind of data will want to collect through your measurement. Step 4: Think about the kind of data will want to collect through your measurement.

Step 5: Decide how you want to analyze the effects of the intervention Step 5: Decide how you want to analyze the effects of the intervention