1. BBA 229 Training and Development
Lecture 5
Training Evaluation
Dr. S. Chan
Department of Business Administration

2. Training Evaluation (1 of 2)
Training effectiveness – the benefits that the company and the trainees receive from training
Training outcomes or criteria – that the trainer and the company use to evaluate training programs

3. Training Evaluation (2 of 2)
Training evaluation – the process of collecting the outcomes needed to determine if training is effective
Evaluation design – from whom, what, when, and how information needed for determining the effectiveness of the training program will be collected

4. Reasons for Evaluating Training (1 of 2)
Companies are investing millions of dollars in training programs to help gain a competitive advantage
Training investment is increasing because learning creates knowledge
This differentiates between those companies and employees who are successful and those who are not

5. Reasons for Evaluating Training (2 of 2)
Because companies have made large dollar investments in training and education and view training as a strategy to be successful, they expect the outcomes or benefits related to training to be measurable

6. Training evaluation provides the data needed to demonstrate that training does provide benefits to the company.

7. Formative Evaluation
The evaluation of training that takes place during program design and development
Helps to ensure that:
the training program is well organized and runs smoothly
trainees learn and are satisfied with the program
Provides information about how to make the program better

8. Pilot Testing
The process of previewing the training program with potential trainees and managers or with other customers
It can be used:
as a “dress rehearsal” to show the program to managers, trainees, and customers
for formative evaluation

9. Summative Evaluation
Evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program
May also measure the return on investment (ROI) that the company receives from the training program

10. Why Should A Training Program Be Evaluated? (1 of 2)
To identify the program’s strengths and weaknesses
To assess whether content, organization, and administration of the program contribute to learning and the use of training content on the job
To identify which trainees benefited most or least from the program

11. Why Should A Training Program Be Evaluated? (2 of 2)
To gather data to assist in marketing training programs
To determine the financial benefits and costs of the programs
To compare the costs and benefits of training versus non-training investments
To compare the costs and benefits of different training programs to choose the best program

12. The Evaluation Process
Conduct a Needs Analysis
Develop Measurable Learning Outcomes and Analyze Transfer of Training
Develop Outcome Measures
Choose an Evaluation Strategy
Plan and Execute the Evaluation

13. Outcomes Used in Evaluating Training Programs: (1 of 4)
Affective
Outcomes
Results
Return on
Investment
Cognitive
Skill-Based
The following suggestions specify the types of changes in jobs that are most likely to lead to improvements in each of the five core dimensions.
(1) Combine tasks - managers should put existing fractionalized tasks back together to form a new, larger module of work. This increases skill variety and task identify.
(2) Create natural work units - managers should design tasks that form an identifiable and meaningful whole. This increases employee “ownership” of the work and encourages employees to view their work as meaningful and important rather than as irrelevant and boring.
(3) Establish client relationships - the client is the user of the product or service that the employee works on. Whenever possible, managers should establish direct relationships between workers and their clients. This increases skill variety, autonomy, and feedback for the employee.
(4) Expand jobs vertically - vertical expansion means giving employees responsibilities and controls that were formerly reserved for management. It partially closes the gap between the “doing” and “controlling” aspects of the job, and it increases employee autonomy.
(5) Open feedback channels - by increasing feedback, employees not only learn how well they are performing their jobs but also whether their performances are improving, deteriorating, or remaining at a constant level. Ideally, employees should receive performance feedback directly as they do their jobs rather than from management on an occasional basis. 9

14. Outcomes Used in Evaluating Training Programs: (2 of 4)
Cognitive outcomes
determine the degree to which trainees are familiar with the principles, facts, techniques, procedures, or processes emphasized in the training program
measure what knowledge trainees learned in the program
Skill-based outcomes
assess the level of technical or motor skills
include acquisition or learning of skills and use of skills on the job

15. Outcomes Used in Evaluating Training Programs: (3 of 4)
Affective outcomes
include attitudes and motivation
reaction outcomes – trainees’ perceptions of the program including the facilities, trainers, and content
Results
determine the training program’s payoff for the company

16. Outcomes Used in Evaluating Training Programs: (4 of 4)
Return on Investment (ROI)
comparing the training’s monetary benefits with the cost of the training
direct costs
indirect costs
benefits

17. Determining Whether Outcomes Are Good
Good training outcomes need to be:
Relevant
Reliable
Discriminative
Practical

18. Good Outcomes: Relevance
Criteria relevance – extent to which training programs are related to learned capabilities emphasized in the training program
Criterion contamination效標混淆 – extent that training outcomes measure inappropriate capabilities or are affected by extraneous conditions=> 當我們收集效標資料進行效度考驗時，常發現效標資料會受到測驗分數的影響，這就是效標混淆的問題。例如：一位教師知道學生的性向測驗分數很低，他對這位學生效標分數的評定，因受那個測驗分數影響，也給很低；反之，便給很高。這種誤差來源，稱為效標混淆，類似心理學上所稱的月暈效應（halo effect）。為了避免此種誤差，最好的方法是使參與效標評定者，不知道受試者的測驗分數。

19. Good Outcomes: Relevance
Criterion deficiency – failure to measure training outcomes that were emphasized in the training objectives

20. Good Outcomes (continued)
Reliability – degree to which outcomes can be measured consistently over time
Discrimination – degree to which trainee’s performances on the outcome actually reflect true differences in performance
Practicality – the ease with which the outcomes measures can be collected

21. Training Program Objectives and Their Implications for Evaluation:
Learning
Transfer
Outcomes
Reactions:
Did trainees like the program?
Did the environment help learning?
Was material meaningful?
Skill-Based:
Ratings by peers or managers based on observation of behavior
Cognitive:
Pencil-and-paper tests
Affective:
Trainees’ motivation or job attitudes
Performance on a work sample
Results:
Did company benefit through sales, quality, productivity, reduced accidents, and complaints?
Performance on work equipment

22. Factors That Influence the Type of Evaluation Design
How Factor Influences Type of Evaluation Design
Change potential
Can program be modified?
Importance
Does ineffective training affect customer service, product development, or relationships between employees?
Scale
How many trainees are involved?
Purpose of training
Is training conducted for learning, results, or both?
Organization culture
Is demonstrating results part of company norms and expectations?
Expertise
Can a complex study be analyzed?
Cost
Is evaluation too expensive?
Time frame
When do we need the information?

23. Conditions for choosing a rigorous evaluation design: (1 of 2)
The evaluation results can be used to change the program
The training program is ongoing and has the potential to affect many employees (and customers)
The training program involves multiple classes and a large number of trainees
Cost justification for training is based on numerical indicators

24. Conditions for choosing a rigorous evaluation design: (2 of 2)
You or others have the expertise to design and evaluate the data collected from the evaluation study
The cost of training creates a need to show that it works
There is sufficient time for conducting an evaluation
There is interest in measuring change from pre-training levels or in comparing two or more different programs

25. Importance of Training Cost Information
To understand total expenditures for training, including direct and indirect costs
To compare costs of alternative training programs
To evaluate the proportion of money spent on training development, administration, and evaluation as well as to compare monies spent on training for different groups of employees
To control costs