1. Evaluation Procedures
Dr. Steve
Training & Development
INP6325

2. Perspectives on Training
Trainee – What if trainee does well in training, but gets fired or laid off for being ineffective on the job?
Trainer – What if trainee develops what s/he thinks is effective training, but supervisors won’t let workers use newly learned techniques/skills?
Organization – What if organization spent big money on training program, but employees are still not effective?

3. Applied Questions Training specialists interested in:
Whether criteria (performance, $ savings) indicate improvement following training
Whether improvement was a result of training
Whether training will be equally effective with different group of trainees
Whether training will be beneficial to other organizations.

4. Research Questions
Which of two or more training programs is the best and why?
Which type of training works best with which type of worker?
What type of organization can benefit most by this type of training?

5. Types of Training Researcher (Randall)
Negativists – Evaluating training is either impossible or unnecessary
Often decisions made based on anecdotal evidence
Positivists – Only rigorous scientific evaluation is worthwhile
If there’s no experimental data, it’s a waste of time
Often don’t have resources for full-blown experiment
Frustrates (Activists) – All training should be evaluated, but must recognize that evaluation quality varies depending on constraints.

6. Training Evaluations Training Evaluation only as good as the:
Question that is asked
Appropriateness of the criteria
Evaluation design

7. Training Evaluations: Ethical Issues
Ethical issues of empirical study
Use of control group means some may not get training
Eliminates promotion opportunities
Control group may feel slighted
Trainees act differently if know in experiment
Hawthorne Effect

8. Formative vs. Summative Evaluation
Formative Evaluation – evaluate whether training program is operating as planned BEFORE it is implemented
Focus on process criteria – how training should be implemented
Summative Evaluation – Does training produce the expected outcome?
Trained vs untrained group comparisons’
Comparative summative evaluation – which of 2 or more training approaches produces greatest benefit

9. Threats to Validity of Training
Internal Validity – did training make a difference in this situation?
Training Validity – Does trainee’s training performance meet criteria for training program?
Transfer Validity – Does trainee’s job performance meet criteria?
External Validity – Can training results be generalized to other trainees or settings?
Intraorganizational Validity – Will training be effective on new group of trainees?
Interorganizational Validity – Can training program be used successfully at other organizations?

10. Threats to Internal Validity
History – Events that happen between pre- & post- tests have nothing to do with training
Contamination of results (ex: layoffs, accidents, holidays, etc.)
Maturation – Biological or psychological effects of time on training
Effects of fatigue or disinterest

11. Threats to Internal Validity
Testing – pre-test sensitizes trainee to upcoming post-test
Trainee prepares for it, or learn from pretest
Ex: GRE score  GRE course  GRE score improves
Instrumentation – questionable reliability of test measure
If fluctuation in scores is normal, can’t attribute change to training

12. Threats to Internal Validity
Statistical Regression – Extreme (hi or lo) pre-test scores will often regress to the mean on post-test
Differential Selection of Participants – if allow participants to volunteer, experimental group may be more highly motivated
Random selection, random assign, or matched groups
Pretest Posttest
Experimental
Control

13. Threats to Internal Validity
Experimental Mortality – differential loss of participants between groups
Trainees who did poorly on pre-test may get discouraged, fired, quit, etc.
Training may coincide with other project deadlines
Interactions – differential effects that one threat might have on another
Ex: testing effects different in each group due to selection

14. Threats to Internal Validity
Diffusion or Limitation of Treatments – members of one group share information with the other
Compensatory Equalization of Treatments – to avoid perceived differential treatment, some alternative might be given to control group
No longer true control, but 2nd experimental group

15. Threats to Internal Validity
John Henry Effect – competition between experimental and control group increases motivation of control group to work harder
Demoralization of Control Groups – passive-aggressive response to not being selected for training

16. Threats to External Validity
Reactive Effect of Pre-testing (sensitization)- effects of pre-test lead to increase sensitivity of instruction
Increased attention to material that was seen in pre-test
Interaction of Selection & Experimental Treatment (representative sample) – characteristics of the group chosen for evaluation may be different than in future groups

17. Threats to External Validity
Reactive Effects of Experimental Settings – Experimental group knows they’re being observed causing them to react differently from future groups
Hawthorne, Guinea pig, or Pygmalion Effects
Multiple-Treatment Interference (carryover)– problem with within subjects experiments
participant is exposed to more than one treatment - previous treatments will affect later ones

18. Training Evaluation Designs
Pre-Experimental Designs
One Shot Case Study:
No control group
X T2
One Group Pre-Test/Post-Test:
Before and after comparison, no control group
T1 X T2
Static Group Comparison:
-- T2

19. Training Evaluation Designs
True Experimental Designs
Pre-test/Post-test Control Group:
Controls for most internal validity threats (except diff treatment)
R T1 X T2
R T1 -- T2
Solomon 4-Group Design:
Controls for both internal and external threats
R T T2
R X T2
R T2
Post-Test Only Control:
Like static group, but with randomization
R X T2
R T2

20. Training Evaluation Designs
Quasi-Experimental Designs
Time Series
Longitudinal method rules out maturation, testing, stat regression
T1 T2 T3 T4 X T5 T6 T7 T8
Non-Equivalent Control Group
Same as pre-test/post-test control, but not random selection
T1 X T2
T1 --- T2

21. Other Evaluation Methods
ANCOVA – use pre-test as covariate in comparing post-test means
Example covary measure of ability prior to training
Useful when small r between pre & post test
R T1 X T2
R T1 --- T2
Correlations – correlate training performance to on-the-job performance
Doesn’t necessarily mean training had effect
Content Validity – SMEs opinion that KSAs covered in training were those identified in needs assessment