1. Experimental Design
Week 9 Lecture 1

2. ISYS3015 Analytical Methods for IS Professionals
Agenda
Purpose of experimental design
Key elements in experimental design
Various types of experimental design
Causal-comparative design
Field and controlled laboratory experiment
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

3. Purpose of experimental design
Examine the possible influences that one factor or condition may have on another factor or condition
Build on positivist approach
Research questions appropriate for an experiment
Issues that have a narrow scope or small scale
Can rarely address question that may require looking at conditions across an entire society or across decade
Small number of participants (subjects) compared with survey research
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

4. Independent variables
Cause
Research “measure” (manipulate) independent variable by creating a condition or situation
Manipulation of independent variable create different treatments.
Event manipulation
Affecting the independent variable by altering the events that subjects experience
Presence versus absence
Instructional manipulation
Varying the independent variable by giving different sets of instructions to the subjects
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

5. ISYS3015 Analytical Methods for IS Professionals
Dependent variables
Effect (outcome)
Physical conditions, behaviors, attitudes, feelings, or beliefs of subjects that change in response to a treatment.
How to measure
Questionnaire, interviews
Observation
Test
Direct outcome
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

6. The importance of control
Internal validity -- The extent to which we can accurately state that the independent variable produced the observed effect
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

7. Strategies to achieve control
Keep some things constant
What are variables that need to be held constant in most experiments?
Include a control group
Treatment group (experimental group)
Between-subjects design
Randomly assign people to groups
Use matched pairs
Matched-subject design
Expose participants to both or all treatment conditions
Within group design
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

8. Between and matched-subjects design1 8 3 2 6 7 10 4 9 5
Random assignment
treatment
control DV 2 3 7 5 9 2 8 1 10 6 4
Randomly assign
one member of
each pair to each group
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

9. Steps in conducting an experiment
State hypotheses
Decide on an experimental design
Decide the way to manipulate independent variables
Develop a valid and reliable measure for dependent variable
Pilot testing the treatment and dependent variable measures
Recruit subjects (or locate cases)
Assign subject to groups
Introduce treatment to treatment groups
Gather data for measure of the dependent variables
Hypotheses testing
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

10. ISYS3015 Analytical Methods for IS Professionals
Mini-workshop
Identify the IV and DV in the following hypotheses
Adults find it easier to remember a list of meaningful words than to remember a list of nonsense syllables
Nationality of salesperson will affect customers’ intention to buy a foreign made product
Perceptions of the characteristics of the “good” or effective teacher are in part determined by the perceiver’s attitudes toward education.
Determine how might you manipulate the IV and measure the DV
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

11. Design elements and notations
Observations or Measures
Symbolized by O in design notation
Treatments or Programs
A manipulated independent variable
Symbolized by X in design notation
Time
Used to indicate the time you make the observation or take the measure
Time moves from left to right. Elements that are listed on the left occur before elements that are listed on the right
Assignment to group
R = random assignment
N = Nonequivalent groups
C = assignment by cutoff
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

12. Classic true experimental design
pretest-posttest
Treatment Versus control group
Randomized
Experimental design
Vertical alignment shows two
Pretests are measured at same time
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

13. Pre-experimental design
One-shot case study
Result usually not valid
Seldom used in serious research
One-group pretest-posttest design
Some effect after the treatment
Can not rule out alternative explanations for the effect
Static group comparison
Any posttest outcome difference between the groups could be due to group differences prior to the experiment instead of to the treatment X O O X N X O
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

14. Quasi-experimental design
Non-equivalent control group design
What does the term “nonequivalent” mean?
Assignment to group was not random
The groups may be different prior to the study
Susceptible to internal validity threat of selection
Selection bias: groups were not comparable before the study N O1 X O2
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

15. ISYS3015 Analytical Methods for IS Professionals
Time-series design
Interrupted time-series design
Why do we use time-series design
Difficult to find an equivalent group of subjects to serve as a control group
Single group pretest-posttest design is susceptible to lots of internal validity threat O1 O2 O3 X O4 O5 O6
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

16. ISYS3015 Analytical Methods for IS Professionals
An example
Research question:
The impact of distance learning on university students’ learning outcome and cognitive skills
Units of analysis – college students
Independent variables
Distance learning (2 level)
Dependent variables:
Learning outcome
Cognitive skills
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

17. ISYS3015 Analytical Methods for IS Professionals
An example (cont)
Hypotheses
H1. Students using distance learning system will achieve better learning outcome than students with traditional classroom learning do
H2. The cognitive skills of students using distance learning system will be lower than those of students with traditional classroom learning
Experimental design?
Subject recruitment?
Manipulation of independent variable
Measurement of dependent variables
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

18. Causal-comparative designs
What can we do if we are not able to manipulate the independent variable
Consider the following hypotheses
Girls who follow science courses in Year 12 are more aggressive than girls following non-science courses.
Working class children will learn nonsense syllables slower than middle-class children
Causal-comparative designs provide the means by which a research can examine how specific IVs (personal trait, history of family violence, etc) affect the dependent variable (s) of interest
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

19. Causal-comparative designs (cont)
Look at conditions that have already occurred and then collects data to investigate a possible relationship between these conditions and dependent variables of interest (ex post facto research)
Difference between causal-comparative design and correlational design
Less able to draw firm conclusion about cause and effect
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

20. Field and controlled laboratory experiment
Field experiment
Experiments conducted in real-life or field settings
Researcher has less control over the experimental condition
Greater external validity but lower internal validity
Controlled laboratory experiment
Conducted under controlled conditions of a laboratory
Greater internal validity but lower external validity
Practical consideration
Planning and pilot testing
Instruction to subjects
Post experiment interview
Thursday, May 13, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney