Managing Learning and Knowledge Capital Human Resource Development: Chapter 11 Evaluation Copyright © 2010 Tilde University Press

Misconceptions about evaluation Evaluation is a control system –Likely pay-off balanced against the cost –Costs unlikely to surface until a time in the future –Such risks will eventually translate into facts, and the dues will then have to be paid Most of the evaluation techniques discussed in this chapter apply to the legitimate system Some effort to evaluate development in the shadow system has to be made 2

Need for evaluation All HRD interventions will be evaluated anyway, either formally or informally Legal imperatives Essential to the survival of the HRD function Role of the evaluation –Measure what change has occurred –Improve the other three stages of the HRD system –See if the change is attributable to the learning episode –See if the change was worthwhile 3

Assessment of learning Gathering, interpreting and describing of information about learner achievement Constructive alignment – see Figure 11.1 Cannot directly observe what learning has occurred: –So, most assessment procedures ask the learner to perform a behaviour –It is this behaviour that is then measured. Several potential weaknesses. For example: –The behaviour is only a sample of the total learning –The behaviour represents only explicit knowledge –Measurement vs Indicator 4

Types of assessment From easiest to most difficult –skills testing –objective written –subjective written –performance tests –learning diaries –analytical critiques –portfolio assessment 5

Assessment and the HLO Constructive alignment –Relationship between the learning outcomes, the learning strategies of choice, and the assessment types of choice Figure 11.3 provides a logical starting point for deciding on appropriate assessment types. Initial decision may be adjusted –Can be a range of complexity within each of the assessment types –Tend to use a number of assessments in combination –Authentic assessment –The characteristics of the learner –Any issues raised in the HRDNI 6

Self-assessment Self-assessment is the best and richest form of assessment Need to develop the learners to such an extent that they value their own self-assessment above all else The learner finally becomes an independent learner 7

The HR developer’s dilemma Skills tests Reflect a simple situation Easy to show reliability and However, the observed behaviours of skills are blurred shadows of the quality that is being assessed Portfolio assessment, depends on the honesty of the learner Major strength when the development of the learner is the main objective (i.e., encourages self-assessment) –When assessment is for other purposes (e.g. monetary gain for the learner), the use of the more complex assessment types becomes more problematical 8

The meaning of scores A raw score - when assessment is gauged using a quantitative figure –What does the raw score mean? –Criterion ‑ referenced scoring –Norm-referenced scoring Use of scores –Formative assessment –Summative assessment 9

Use of assessment Feedback –To the learner –To the HR developer Evaluation –One of the levels of evaluation 10

Kirkpatrick’s four levels Reaction –Reactions of the learners to the learning episode –Usually measured with a questionnaire - happy sheets? Learning – discussed previously Behaviour –Change in on-the-job behaviour –Use performance appraisal process Results –Impact of the learning episode on the organisation as a whole –Tangible indicators (e.g. improved profits) –Success Case Method 11

The presage factors Stage I: evaluate needs and goals Stage II: evaluate HRD design Stage III: evaluate implementation Stage IV: evaluate learning - similar to Kirkpatrick Stage V: evaluate usage and endurance of learning - similar to Kirkpatrick Stage VI: evaluate payoff - similar to Kirkpatrick Could also add another – evaluate the HR developer SEE Table 11.1 for a discussion on the eight (8) levels of evaluation 12

Review of the four roles of evaluation The 8 levels of evaluation achieves first two: –Identify what change has occurred –Improve the other three stages—investigation, design and implementation—of the HRD system Need to go further to achieve the last two: –See whether the change is attributable to the learning episode – scientific model –See whether the amount of change was worthwhile – cost benefit analysis 13

Scientific Model Based on the experimental methods In order from the simple and less costly to the complex and more costly –Post-test –Pre-test–post-test –Time series –Control group –Solomon four A complex decision 14

Cost benefit Analysis Identify the costs, in dollar terms –Relatively straight forward –Cut off points? – e.g. apportioning electricity charges Identify the benefits accruing in dollar terms – Sometimes easy – e.g. reduction in accidents –Often difficult – e.g. value of change at the meta-ability level in a learner –Often use a ‘shadow value’ – e.g. promotions individual achieves –Cut off points? The ratio of costs/benefits –Should be in favour of the benefits 15

The evaluation plan Starts during the design stage Should include –Develop the assessment of learning first –Add further assessment of learning as required for the evaluation –Decide what presage variables will be evaluated and when –Review the investigating instruments and incorporate into plan –Design daily and overall course (or workshop) reaction sheets –Plan the scientific pre-test and post-test instruments, if needed. –Identify the methods for the behaviour and results levels –Decide if a cost–benefit analysis is to be used and plan for it –Prepare a budget for the evaluation plan –Send the evaluation plan to staff who are affected 16

The evaluation report Should at least include the following –Executive summary –Findings/recommendations section –Table of contents –Main body reasons for the evaluation list of the personnel involved discussion of - types of evaluations used and how the data was collected and analysed discussion of the findings, list and a discussion of the recommendations –Appendices 17