1. Establishing a strategic approach to determine the measures that reliably connect performance and training inputs
Bob Megens BMO Financial Group Institute For Learning STEP: May 3, 2007
23/11/2018

2. Why focus on measuring the impact of learning on performance?
BMO has set up an Our Way Model as a way of achieving the goal of a high performance culture. Tracking performance is an essential component of BMO’s enterprise strategy of becoming a high performing culture.
Harvard Business Review
Special Double issue – July/August 2005
The High Performance Organization
BMO Financial Group
Our Way Model
Editor Thomas A. Stewart’s
Key Things for a High Performance Organization
Leadership
Teams
Organizational Design
Learning
What are the outputs that will be used to valid the achievement of high performance?
To what degree of accuracy are we able to measure the contribution of these inputs to the achievement of the outputs being measured?
23/11/2018

3. The Conference Board of Canada
Learning and Development Outlook 2007 (May), Ch. 6, Learning Evaluation
“The lack of detailed evaluation may explain [training, learning and development] professionals’ inability to obtain higher training budgets: they simply do not have the analytical basis to make the business case for increasing budgets.
Although many [training, learning and development] professionals expect their budgets to increase, their evaluation efforts suggest that they are not fully engaged in demonstrating the impact of their current training investments.
This is likely to reduce spending: as evaluation efforts decrease, budgets might be expected to decrease.”
23/11/2018

4. Lexicons and Paradigms: Kirkpatrick or time for a change?
High-Impact Learning Measurement, Bersin and Associates, Nov. 2006
While many organizations make a valiant effort to measure specific high-value programs, most organizations do not have a repeatable, high-value measurement process for measuring training impact.
We believe that one of the reasons for this problem is the limitations of the existing well-known measurement “models,” primarily the Kirkpatrick Model – which describes four levels for measurement: learner satisfaction, learning, job impact and business impact. A fifth “level” (ROI or return on investment) was added through the work of Jack Phillips4. While Donald Kirkpatrick does not necessarily believe that ROI is the fifth level, it is often recognized as the ultimate measure of training impact.
While these models are widely understood and have done much to help training managers understand some of the possibilities of training measurement, we believe that they tend to limit an organization’s thinking and make the measurement process difficult to implement…
While we do not want to confuse the industry with yet another model, as a result of this research we found it necessary to develop one of our own.
Level
Measure 1
Reaction 2
Learning 3
Behaviour 4
Results 5
Investment (ROI)
23/11/2018

5. 3 Key Criteria for linking performance and training
There were 3 criteria (which stood out from the research) that needed to be in place to determine the impact of learning on performance.
1. Alignment of learning with enterprise priorities
2. Conditions that maximize the learning investment
Business performance measurement and reporting systems in place to identify the metrics that are to be used to isolate the business performance impact of learning
Systematic approach
Concerted monitoring
Consistent process
23/11/2018

6. What is measured and management reporting
Current Approach
Consistent collection and reporting on Level 1 evaluation. While there is collection of data at higher levels, evaluation and reporting are less consistent and not systematic.
Level 4: - ad hoc collection - late in the process - no management reporting
Level 3: - follow up survey - no management reporting - employee survey (Learning Index) reported annually
Level 4
Level 3
Level 2: - dependent upon initiative - no management reporting.
Level 2
Level 1: - evaluation consistently applied - quarterly reports.
Level 1
Approach Amount of activity
What is measured and management reporting
23/11/2018

7. 2 dimensions for capturing the impact of learning on performance
Establish processes that identify opportunities to measure the impact of learning at an enterprise level and consistently capture the impact of learning for specific initiatives and programs
What are the opportunities for the collection of non-survey data?
If survey data is collected, who would provide the most valid data?
Dimension
Metrics: measuring the impact of learning on business performance
1. Enterprise
Aggregate Enterprise Measurement
human capital assessment, measuring employee performance and capability for the entire BMO Workforce (e.g. customer service professionalism)
- influence Customer Satisfaction surveys to capture the impact of learning
2. Specific Initiatives
Specific Strategic Learning Initiatives
metrics to determine the impact of learning for strategic initiatives within specific businesses (e.g. sales and service training to increase revenue)
Increase Customer Service metrics for learning at enterprise level
- establish a link with Market Research to formalize a process to align with Market Research’s initiative in linking customer survey results to business results
- develop customer service metrics/questions that target specific learning and development outcomes for integration into customer surveys
23/11/2018

8. What is measured and management reporting
Proposed Approach
The proposed approach places more emphasis on identifying robust and appropriate metrics for determining the impact of learning on performance (Level 4) at the beginning of the engagement process , making the agreement to collect these metrics explicit components of learning initiatives.
Level 4: (financial, non-financial and survey data) - focus on strategically relevant metrics (no more than 10% of all learning initiatives) - leverage business performance measurement systems and coaching tools to isolate or associate the impact of learning - survey executives and managers to assess the impact of learning on business
- establish reporting that feeds into tracking systems
Level 4
Level 3
Level 3: (financial, non-financial and survey data) - employee performance metrics
- survey executives and managers to assess the impact of learning on employee performance - index of employee survey performance questions
- index of performance questions from follow-up evaluations
BENCHMARKING
Level 2
Level 1
Level 2: dependent upon initiative
Level 1: evaluation system reviewed and reported quarterly.
Approach Amount of activity
What is measured and management reporting
23/11/2018

9. 3 approaches for measuring the impact of learning on performance
Developing more rigorous metrics to demonstrate the impact of training on performance requires a declaration of priority, sustained by a focused and managed approach.
Track and aggregate (where possible) metrics for the impact of learning as a performance measure for the learning function
Develop the expertise for client engagement, analysis and collection of measurement data within the learning function
develop skills of individual project manager/course owners
establish a centre of competency for evaluation to work with projects/learning initiatives
Leverage performance measurement systems used by the business
learning function cannot create systems to measure performance within the business
must leverage the financial and non-financial data collected by the business
begin at the scoping stage of the project, start with Level 4 metrics and seek agreement with business client(s) on measures used
document measurement decisions into the project charter
Collect metrics that isolate or associate the impact of learning on business performance for relevant strategic initiatives
validate alignment with learning objectives
select initiatives that best demonstrate the business value of learning (up to 10% of all learning initiatives)
focus on sales and service, and leadership and performance management
Monitor the conditions that maximize the impact of learning on business performance by monitoring them 1.
Attention
and
Priority 2.
Process
and
Focus 3.
Monitor
Conditions
23/11/2018

10. Business Conditions Required to Maximize Learning Investment
Conditions directly affecting the impact of learning on performance
Research indicates that specific conditions need to be in place to realize the investment in learning as a sustainable performance lift.
Business Conditions Required to Maximize Learning Investment
Leadership Engagement (Funding and Support, e.g. Diagnostic)
Appropriate target audience properly prepared for learning
Follow up Coaching
Conditions for Business Performance
Reward and Recognition
Appropriate Measurement
Hiring Strategies
Business process improvement and technology
Potential to maximize learning’s impact on business performance.
Monitor to ensure presence.
Assumed to be in place.
Raised only if they present barriers to performance.
Conditions taken from Phillips, Jack. Return on Investment in Training and Performance Improvement Programs, 2nd Ed, 2003.
23/11/2018

11. Metrics that measure business and employee performance
Measuring the impact of learning on business performance will leverage business performance measurement and reporting systems.
The following table lists metrics at the business and employee performance levels that can be used to isolate the performance impact of learning. These metrics are particularly effective when applied to sales and service, especially in the area of customer service.
Business Performance
Employee Performance
Increased revenue
Increased customer service
Decreased time to performance
Increased quality
Decreased cost
Increased retention
Benefit to Cost Ratio
Return on Learning Investment
Other (payback period, audit results, etc.)
Increased customer satisfaction
23/11/2018

12. Dedication of Resources
Deriving verifiable and specific metrics
Research shows that deriving verifiable level 4 metrics to determine the performance impact of learning using specific figures is only viable and recommended about 10% of the time.
The following criteria can be used by the learning function and the business sponsors to determine when it is more appropriate to pursue robust data as opposed to a more associative level of evaluation for particular learning initiatives:
- Business leader engagement - Highly strategic initiative
- Large audience Greater allocation of resources
- Comprehensive and specific business - Appropriate baseline measurements performance measurement systems
Level 4 evaluations require greater engagement of the business and dedication of resources over an agreed upon time period.
Level 4
Business Engagement
Dedication of Resources
Level 3
Level 1 and 2 Evaluations will remain in high use due to the relative low dedication of resources and the ease of implementation immediately after the learning is completed
Level 2
Level 1
from Phillips, Jack. Return on Investment in Training and Performance Improvement Programs, 2nd Ed, 2003.
23/11/2018

13. Tracking impact on performance
Metrics on the impact of learning on business performance should be used as an input into tracking of the performance of the learning function.
Business Metrics
Performance Dashboard
Financial and non financial data
use business performance measurement systems
impact of learning isolated or associated with specific business performance
leverage business coaching tools used by business to evaluate employee performance
Business Performance Metrics (Level 4)
Customer Satisfaction Surveys (Level 4)
Sales and Service Index (Level 4)
Employee Performance Metrics (Level 4 and Level 3)
Survey Metrics
Managerial evaluation:
surveys sent to managers to assess the impact of learning on
business performance
employee performance
Employee evaluations
employee survey results
follow-up evaluations
course evaluations
Managerial Summary Reports
Business performance (Level 4)
Employee performance (Level 4 and Level 3)
Employee Summary Reports
Index of AES performance questions (Level 3)
Index of performance question for follow-up evaluations (Level 3)
Index of performance questions in course evaluations (Level 1)
23/11/2018