1. Research Experience for Teachers (RET)
Research Methods
Dr. Randa L. Shehab
Dr. Chen Ling

2. Overview Research Methodology Technical Writing and Presentation
Problem Formulation
Development and Measurement of Performance Criteria
Experimental Design
Data Collection and Analysis
Presentation of Results
Technical Writing and Presentation
Introductions
Sign-up List
Review Syllabus: Note Project Deadlines
Alias for Class Members
Office Info
Course Description
The purpose of this course is to introduce the student to the techniques of experimental methodology through the vehicles of ergonomics and human performance measurement. Experimental methodology includes problem formulation, the development and measurement of human performance criteria, experimental design, data collection, data analysis, and the presentation of results. Problems in the study of human performance will be discussed along with the proper techniques for writing and presenting a technical report. As part of the course, students will be required to conduct an experimental investigation of some relevant aspect of human performance. This research project will require a significant contribution of time and the amount of credit given will reflect this contribution.
RET 2006 Summer

3. The Scientific Method
A systematic approach to answering questions using empirical investigation
Steps
Form a hypothesis - an idea that can be tested through observation or experimentation
Collect data through observation
Analyze the data - test the hypothesis
“Do not reject” or “reject” the hypothesis
Confirm results by repeating the experiment
RET 2006 Summer

4. The Research Environment
Naturalistic Observation
Systematic, detailed observation of behavior
Purely descriptive, no causality
Laboratory Research
Arbitrary environment where elements in environment can be controlled
Control allows statements about causality
Naturalistic Observation
Journal, time sample, etc.
e.g., observe wild animals in natural environment, children on playground, guards at a prison, assembly workers on the job
RET 2006 Summer

5. Basic Research
Acquisition of knowledge for developing or validating a theory
Not necessarily for practical application
Used to set forth general design principles
Examples
Nature of human performance?
How is performance affected by variable(s)?
Relationship between variables?
Impact on system performance?
Example: Working in a Heat Stress Environment
Basic Research:
Exercise at various levels of heat stress and monitor core temperature and heart rate for sole purpose of learning physiological responses (physiologist).
Set Forth General Design Principles
Applied Research:
Perform actual work task while monitoring core temperature and heart rate with goal of setting standards such as OSHA (NIOSH) exposure limits.
Evaluate Specific Design Alternatives
Example: Texas Instruments LED Watch
Flick wrist to turn on product.
Perform experimentation to evaluate various response times.
Other elements: display brightness, on time, size of digits
RET 2006 Summer

6. Applied Research Examples Finding answers to practical problems
Highly specific hypothesis
Absence of general knowledge or theory advancement
Used to evaluate specific design alternatives
Examples
Effect of technology on human performance?
Effect of behavioral variables on use of technology?
Most effective behavioral/technological variable to achieve desired performance?
Example: Working in a Heat Stress Environment
Basic Research:
Exercise at various levels of heat stress and monitor core temperature and heart rate for sole purpose of learning physiological responses (physiologist).
Set Forth General Design Principles
Applied Research:
Perform actual work task while monitoring core temperature and heart rate with goal of setting standards such as OSHA (NIOSH) exposure limits.
Evaluate Specific Design Alternatives
Example: Texas Instruments LED Watch
Flick wrist to turn on product.
Perform experimentation to evaluate various response times.
Other elements: display brightness, on time, size of digits
RET 2006 Summer

7. Why Research?
To understand behavior and answer questions about people, environments, systems
To evaluate the research of others
We should change the system to make it more effective…
Determine if research is done correctly and if it applies
To conduct research of your own
Research is a tool for answering questions
RET 2006 Summer

8. Learning to Perform Scientific Research
Scientific Research is Unique
Acquire Research Experience
Perform Solid Literature Review
Use Proper Tactics and Strategy
Avoid Mistakes of Other Investigators
RET 2006 Summer

9. General Scientific Method: Steps
Development of Hypothesis
Body of Existing Knowledge
Define the problem
Controlled Experimentation
Observation of Phenomena
Scientific curiosity
Quantification of Observations
Analyze the problem
Logical Analysis
Test of Hypothesis
RET 2006 Summer

10. Research Experiments
Experiment: A series of controlled observations taken in an artificial situation with deliberate manipulation of variables to answer one or more specific hypotheses.
Controlled Observations
Conditions and events
Must be repeatable!
Artificial Situation
Allows accurate observations, but might affect behavior
Controlled Observations
- Control variables that might affect the results:
- Conditions: Environmental; Everything must be the same for each subject during each trial, No confounding factors
- Events: No outside influences, distractions; No differences in test methodology (time of day, noise levels, etc…)
- Must be able to replicate the experiment! Verify and Validate!
Artificial Situation
- Contrived Conditions and Circumstances; Not naturally occurring.
- Unfortunately, people tend to behave artificially when they know they are being observed. Might respond differently than they normally would. (Hawthorne Effect: Workers performance improves merely because of the added attention => increased motivation).
RET 2006 Summer

11. Research Experiments Manipulation of Variables Specific Hypotheses
Systematically vary conditions to see effects
Eliminate extraneous factors to determine causes
Specific Hypotheses
Allows clear definition of the experimental plan
Ideas can be obtained from observation
“Effects of (independent variable) on (measured variable)”
“Comparison of the effects of (different technologies / products) on (performance measure)”
Manipulation of Variables
- Systematically vary ONLY the factors that you wish to evaluate.
- Avoid CONFOUNDING - if you vary more than one factor at a time, you can’t be certain what was responsible for any effect you may observe.
- Control all other variables that might have an effect on the outcome.
Specific Hypotheses
- Starting point of all your research => Very Important!
- Proposes an explanation that can be tested.
- State clearly what your topic of interest is and what the independent and dependent variables are.
RET 2006 Summer

12. Developing a Hypothesis
A hypothesis states an idea that can be tested through experimentation or observation
The hypothesis proposes an explanation for some observed phenomenon
Initial
Interest
Literature
Review
Hypothesis guides inquiry and directs thinking
Working Hypothesis
Formal Hypothesis
RET 2006 Summer

13. Types of Hypotheses Working Hypothesis Exploration Formal Hypothesis
Preliminary statement based on limited information
Informal statement derived from initial observation
Subject to modification
Exploration
Obtain additional information on the topic
Books, journals, related knowledge, collaboration
Formal Hypothesis
Precise expression of a predicted relationship between or among events
Capable of being tested
Does not change
Working Hypothesis
- Oranges are larger than apples.
Formal Hypothesis
- Defines the concern of our research
- The average diameter of oranges is larger than that of apples
- In a statistical test, this becomes our alternate hypothesis
Null Hypothesis
- The average diameter of apples and oranges is the same
- You can prove a relationship does not exist, but you can never prove that one does.
RET 2006 Summer

14. Key Principles of Experimental Design
Provide a Measure of Random Error
Design must contain a measurable estimate of variation to determine statistical significance
Provided through replication
Avoid Systematic Bias
Effects due to time-ordering such as learning
Counterbalance - perform half the trials using one sequence and the other half using another
Randomize - assign units randomly
Do Not Confound Variables
Change only one variable at a time
Provide a Measure of Random Error
- Design must contain a measurable estimate of variation to determine statistical significance
- Provided through replication
Repeated measurement of the same experimental unit (less variability but not as generalizable)
Measurement of several experimental units (more generalizable, but higher variability)
Avoid Systematic Bias
- Effects due to time-ordering such as learning; Occurs when there is a dependence between measured values that follows some pattern
- Counterbalance - perform half the trials using one sequence and the other half using another
- Randomize - (if you cannot counterbalance) assign units and trials randomly
Do Not Confound Variables
- Two variables are confounded when their effects cannot be isolated. The levels of two factors change simultaneously.
- Change only one variable at a time
RET 2006 Summer

15. Defining the Problem Easiest way is to define the variables
A variable is a quantity that may assume any one of a set of values
Independent Variable
Variable that is deliberately manipulated
Must have at least 2 levels
Forms the basis of the hypothesis
RET 2006 Summer

16. Independent Variables
The factor or treatment
Presence/Absence
Does the variable impact behavior?
Quantitative
Continuous along a single dimension
Must have enough levels to describe expected relationship
Must consider range and spacing of levels
Presence / Absence:
Non-zero value in one condition, Zero in the other.
Does the variable impact behavior or performance?
- Quantitatively:
Choose levels along a single but continuous dimension
Select a wide enough range to differentiate effects between the levels -
RET 2006 Summer

17. Independent Variables
Qualitative
Combination of multiple factors not described with a single dimension
Vary one dimension and control the others
Select “optimal” values for all variables held constant
Test at range of values to identify the optimal points
Qualitatively:
Combination of multiple factors that cannot be described with a single dimension. (shape - three dimensional; color - contains both hue and shade characteristics (light))
RET 2006 Summer

18. Quantitative Variables
Number of Levels
Choose enough levels to get an accurate view of the relationship between the variables
Cost tradeoff when adding more levels
Total Trials = (# IV's)  (IV levels)  (repetitions)
Actual
Relationship
Dependent Variable
Independent Variable
Level 1
Level 2
Level 3
Suspected
Relationship
RET 2006 Summer

19. Quantitative Variables
Range of Values
Wide enough to cover region of interest
Start large and refine (pilot study)
Spacing
Should match phenomenon being investigated
Equal: Better for statistical analysis
Unequal: Often more representative of system response (e.g., decibels)
RET 2006 Summer

20. Qualitative Variables
Example
When comparing control shapes, select the best size, texture, etc. for each shape
Compare Round A with Square C
Fundamental Principle
Select the best levels of all other variables (usually quantitative) to be held constant, then use the best combination at each qualitative level
Alternatively, test across a broad range of values to locate optimum levels
Dependent Variable
Knob Size A B C
Round
Square
RET 2006 Summer

21. Dependent Variables The criterion measure Variable that is measured
Value “depends” on the value of the independent variable
Selection of the criterion determines the outcome
Does it answer the question?
Is it relevant to the task?
RET 2006 Summer

22. Control Variables Examples:
Variables that are held constant because they may affect the results
Avoid Confounding
Uncontrolled influences on experimental results
Interpret results within a narrow limit to determine causality
Examples:
Time of day
Attitudes/expectation
Instruction
RET 2006 Summer

23. Variable Examples Effect of Alcohol on Reaction Time
Independent Variables
Alcohol Level / Consumption (0%, .08%, 1.2% BAC)
Gender (Female, Male)
Dependent Variables
Errors
Processing Time
Motor Time
Control Variables
Food Intake
Environmental
RET 2006 Summer

24. Characteristics of Good Performance Measures
Appropriate Level of Detail
Measure appropriately reflects differences to detect
Adequate precision and range to detect differences
Reliability
Degree of repeatability (can be expressed as a correlation coefficient)
Interrater reliability – the degree to which multiple observers agree when scoring the same event
Validity
Degree to which a measure actually measures what it is supposed to measure
Ensures a measure tells you “what” your data mean and that you have selected an appropriate response variable
RET 2006 Summer

25. Characteristics of Good Performance Measures
Sensitivity
Can detect changes in the behavior of interest
Accuracy
More precise than the phenomena being measured
Minimize measurement error Non-Intrusiveness
Collection method does not affect performance
Observer should not be a distraction
Data collection should not interrupt the task (i.e., filling out forms)
Measuring device should not attract participant's attention
Implementation Requirements
Practical with respect to time, budget personnel requirements, ease and quality of collection, and measurement robustness
RET 2006 Summer

26. Populations and Samples
Sample: subset
of the population
Population: all
relevant cases
Population
- Represents all relevant cases in the entire data set
- Finite, but may be very large
Sample
- Manageable subset of the population.
- Very Important: Should represent the population
Sampling Methods
- To ensure representative samples
Sampling Methods
Random Sampling
Stratified Sampling
Matched Sampling
Fortuitous Sampling
Proportionate Stratified
RET 2006 Summer

27. Random Sampling Randomly select members of population
Independent
Representative of the population for large sample sizes
Reduces systematic bias
Randomly assign subjects to conditions
How would you randomly sample n=20 to study ability of drivers to localize warning sirens?
Random Sampling
- Define the population of test units and randomly sample
- Each element has an equal probability of being sampled
- Random numbers table: a large table of numbers used to assign participants randomly to treatment conditions
- Sample is representative of the population when a large number are selected
- Random assignment reduces bias
Fortuitous Sampling
- Take whatever you can get!
RET 2006 Summer

28. Fortuitous Sampling Use whatever is available
Does not guarantee independence
Not representative of the population
How would you fortuitously sample n=20 to study ability of drivers to localize warning sirens?
RET 2006 Summer

29. Stratified Sampling Identify critical subgroups of the population
Randomly select members for subgroup samples based on similar critical characteristics
Mirrors population characteristics
Accurate information is not always available
How would you develop a stratified sample of n=20 to study ability of drivers to localize warning sirens?
RET 2006 Summer

30. Proportionate Stratified Sampling
Identify critical subgroups of the population
Randomly select members for subgroup samples until proportionate to the population characteristics
Mirrors population characteristics and count
How would you develop a proportionately stratified sample of n=20 to study ability of drivers to localize warning sirens?
RET 2006 Summer

31. Matched Sampling
Experimental samples are identical with the exception of the independent variable
Controls for extraneous variables
Allows strong conclusions to be made about any significant differences
Equating groups
Precision matching - match identical participants
Range matching – match participants within a range
Average matching - match average group score
How would you develop a matched sample of n=20 to study the effects of different work hardening programs on material handling endurance?
Matched Sampling
- Controls for Extraneous Variables: variables not controlled that may affect the outcome of the experiment (i.e., age, gender)
- Match participants on these extraneous variables
- Attempts to create identical samples to ensure that observed differences are due only to treatment
- Equating Groups: Matches groups along a particular dimension (common features or participant characteristics)
- Precision Matching: Pairs of identical participants are randomly assigned to treatment conditions.
- Range Matching: Pairs of participants falling within the defined range are randomly assigned to treatment conditions.
Problems:
- Difficult to determine precise values.
- Can’t always find matches, so discard participants ==> - Participant selection bias could affect generalization.
- Yoking Procedure - Matches participant sequences to control for temporal and sequential effects.
- Randomized Block Designs - Includes an extraneous variable in the design as an independent variable. Randomized assignment to treatments within blocks (levels of the blocking factor).
RET 2006 Summer