Presentation on theme: "Evaluation Procedures"— Presentation transcript:
1Evaluation Procedures Dr. SteveTraining & DevelopmentINP6325
2Perspectives on Training Trainee – What if trainee does well in training, but gets fired or laid off for being ineffective on the job?Trainer – What if trainee develops what s/he thinks is effective training, but supervisors won’t let workers use newly learned techniques/skills?Organization – What if organization spent big money on training program, but employees are still not effective?
3Applied Questions Training specialists interested in: Whether criteria (performance, $ savings) indicate improvement following trainingWhether improvement was a result of trainingWhether training will be equally effective with different group of traineesWhether training will be beneficial to other organizations.
4Research QuestionsWhich of two or more training programs is the best and why?Which type of training works best with which type of worker?What type of organization can benefit most by this type of training?
5Types of Training Researcher (Randall) Negativists – Evaluating training is either impossible or unnecessaryOften decisions made based on anecdotal evidencePositivists – Only rigorous scientific evaluation is worthwhileIf there’s no experimental data, it’s a waste of timeOften don’t have resources for full-blown experimentFrustrates (Activists) – All training should be evaluated, but must recognize that evaluation quality varies depending on constraints.
6Training Evaluations Training Evaluation only as good as the: Question that is askedAppropriateness of the criteriaEvaluation design
7Training Evaluations: Ethical Issues Ethical issues of empirical studyUse of control group means some may not get trainingEliminates promotion opportunitiesControl group may feel slightedTrainees act differently if know in experimentHawthorne Effect
8Formative vs. Summative Evaluation Formative Evaluation – evaluate whether training program is operating as planned BEFORE it is implementedFocus on process criteria – how training should be implementedSummative Evaluation – Does training produce the expected outcome?Trained vs untrained group comparisons’Comparative summative evaluation – which of 2 or more training approaches produces greatest benefit
9Threats to Validity of Training Internal Validity – did training make a difference in this situation?Training Validity – Does trainee’s training performance meet criteria for training program?Transfer Validity – Does trainee’s job performance meet criteria?External Validity – Can training results be generalized to other trainees or settings?Intraorganizational Validity – Will training be effective on new group of trainees?Interorganizational Validity – Can training program be used successfully at other organizations?
10Threats to Internal Validity History – Events that happen between pre- & post- tests have nothing to do with trainingContamination of results (ex: layoffs, accidents, holidays, etc.)Maturation – Biological or psychological effects of time on trainingEffects of fatigue or disinterest
11Threats to Internal Validity Testing – pre-test sensitizes trainee to upcoming post-testTrainee prepares for it, or learn from pretestEx: GRE score GRE course GRE score improvesInstrumentation – questionable reliability of test measureIf fluctuation in scores is normal, can’t attribute change to training
12Threats to Internal Validity Statistical Regression – Extreme (hi or lo) pre-test scores will often regress to the mean on post-testDifferential Selection of Participants – if allow participants to volunteer, experimental group may be more highly motivatedRandom selection, random assign, or matched groupsPretest PosttestExperimentalControl
13Threats to Internal Validity Experimental Mortality – differential loss of participants between groupsTrainees who did poorly on pre-test may get discouraged, fired, quit, etc.Training may coincide with other project deadlinesInteractions – differential effects that one threat might have on anotherEx: testing effects different in each group due to selection
14Threats to Internal Validity Diffusion or Limitation of Treatments – members of one group share information with the otherCompensatory Equalization of Treatments – to avoid perceived differential treatment, some alternative might be given to control groupNo longer true control, but 2nd experimental group
15Threats to Internal Validity John Henry Effect – competition between experimental and control group increases motivation of control group to work harderDemoralization of Control Groups – passive-aggressive response to not being selected for training
16Threats to External Validity Reactive Effect of Pre-testing (sensitization)- effects of pre-test lead to increase sensitivity of instructionIncreased attention to material that was seen in pre-testInteraction of Selection & Experimental Treatment (representative sample) – characteristics of the group chosen for evaluation may be different than in future groups
17Threats to External Validity Reactive Effects of Experimental Settings – Experimental group knows they’re being observed causing them to react differently from future groupsHawthorne, Guinea pig, or Pygmalion EffectsMultiple-Treatment Interference (carryover)– problem with within subjects experimentsparticipant is exposed to more than one treatment - previous treatments will affect later ones
18Training Evaluation Designs Pre-Experimental DesignsOne Shot Case Study:No control groupX T2One Group Pre-Test/Post-Test:Before and after comparison, no control groupT1 X T2Static Group Comparison:-- T2
19Training Evaluation Designs True Experimental DesignsPre-test/Post-test Control Group:Controls for most internal validity threats (except diff treatment)R T1 X T2R T1 -- T2Solomon 4-Group Design:Controls for both internal and external threatsR T T2R X T2R T2Post-Test Only Control:Like static group, but with randomizationR X T2R T2
20Training Evaluation Designs Quasi-Experimental DesignsTime SeriesLongitudinal method rules out maturation, testing, stat regressionT1 T2 T3 T4 X T5 T6 T7 T8Non-Equivalent Control GroupSame as pre-test/post-test control, but not random selectionT1 X T2T1 --- T2
21Other Evaluation Methods ANCOVA – use pre-test as covariate in comparing post-test meansExample covary measure of ability prior to trainingUseful when small r between pre & post testR T1 X T2R T1 --- T2Correlations – correlate training performance to on-the-job performanceDoesn’t necessarily mean training had effectContent Validity – SMEs opinion that KSAs covered in training were those identified in needs assessment