1. Using Measurements of Contractor Performance to make Informed Decisions Regarding Services 6th Annual Strategic Facilities Management Conference Conferenz, Auckland March 23, 2009 Garry Law Law Associates Ltd Consulting Engineers PO Box 87311 Meadowbank Auckland – 09 520 2152 - 0275 665764

2. Garry Law: Performance Measurement2 Road Map Using measurements of contractor performance to make informed decisions regarding services rendered. Quantifiably measuring contractor performance is not straight forward; yet doing so yields authoritative data upon which to base decisions on regarding your contractor’s performance. This session will reveal how best to link KPIs, Service Level Agreement and contracts to quantifiably determine your contractor’s performance. –Extracting metrics from their performance –Best processes for interlinking KPIs, SLAs and contracts –Examples and guidelines for using the metrics yielded to make decisions regarding contractors

3. Garry Law: Performance Measurement3 Why M easure ?  Truism – What you can’t measure you can’t manage  But – not everything is measurable  Relationships are hard to quantify: Tolstoy, Anna Karenina: “Happy families are all alike. Unhappy ones are each miserable in their own way"  Don’t expect the sum of a series of measures to capture the whole relationship

4. Garry Law: Performance Measurement4  Working on the relationship is not just about reviewing the performance metrics  They can be a good place to start – they should be an important part of a periodic review meeting – but they are not the whole part.  Relationships are two way – not one scoring the other  A breech of duty / trust / confidence is better directly addressed than taken into some subjective performance score Relationships

5. Garry Law: Performance Measurement5 Why Use KPIs?  The objective in service contracts is not just cost management  Want reliability / quality / service / information  On one-off work paid on time and materials - Want productivity  Want the service aligned to your broad needs  Multiple objectives are the norm  May link performance to rewards

6. Garry Law: Performance Measurement6 Setting KPIs  Need to start with your organisation’s objectives (Key Results Areas)  Only some of these will involve the contract  You will have some organisational KPIs. Some will measure broad things including the relevant service.  To get organisational objectives into a useful form, cascade the objectives to the relevant level and set a KPI on that.  If the contractor can’t control most of what influences them they can’t be used.

7. Garry Law: Performance Measurement7 Cascading Objectives LevelObjectiveKPI CorporateIncrease EVA> +1% for year 2009/10 ManufacturingIncrease asset utilisation and free assets for sale Output / asset value employed + 5% by Dec 2009 Plant xyzFree old widget line for sale Sold by Jan 2010 Service Contract at Plant xyz Increase plant availability New widget line availability > 95% The objectives are hierarchically linked - the KPIs relate to objectives not each other

8. Garry Law: Performance Measurement8 Sorts of KPIs  “Milestone” Achieve something by a date. e.g. Populate the Computerised Maintenance Management System by July 30 th 2011  “Measure” Meet a target on a defined measurement. e.g. Measure: Rework rate on class 3 tasks Target: < 5% A service contract will most likely have both.

9. Garry Law: Performance Measurement9 Good KPI Characteristics  Output oriented  Measured for a long time  Have external benchmarks to set good performance  Contractor has primary influence over achievement  Close to real time (contractor can act to correct)  Contractor can measure (but with auditability)  Balanced set across the scope of the contract  Limited natural variation  Not too many!

10. Garry Law: Performance Measurement10 Output oriented  The client’s need’s are outputs not inputs  The contractor’s inputs do not guarantee the outputs will be achieved  Leave the “how” to the contractor – that’s their job not yours

11. Garry Law: Performance Measurement11 Measured for a long time  - If Yes - Is it still relevant? –or is it an ingrained habit?  - If No – its not fatal –Look at what other people have been measuring (they may have benchmarks as well) –If its entirely new – do a slow start – don’t make it central / vital / all important until you have a history of knowing how to measure – knowing what makes it better or worse and know what a “good” score is.

12. Garry Law: Performance Measurement12 Have external benchmarks to set good performance  Great if you have these  But the compromise will often be you have to measure them in exactly the same way that others do (sample size / frequency etc.)  The price of sharing BM information may be revealing your own performance (not always a bad thing, but don’t do it unthinkingly)

13. Garry Law: Performance Measurement13 Close to real time (so the contractor can act to correct)  If you undertake surveys on a long period interval the contractor will be in the dark as to their performance for much of the time.  If your results have some sort of delay from when they were collected the contractor will be likely not to give them much credence

14. Garry Law: Performance Measurement14 Contractor can measure (but with auditability) If the client does all the measurement it can lead to problems:  Master / slave relationships /  “Gotcha” approach from client  Contractor duplicating measurements in their own defence and disputing the client’s measurements

15. Garry Law: Performance Measurement15 Limited Natural Variation  Common Causes - variations from the contractor’s systems - manageable by them)  Special Causes - variations external to the contractor’s systems (but will include variations caused by the client) (W E Deming on Quality) Using alliance methods may allow client side issues to be worked on as well.

16. Garry Law: Performance Measurement16 Balanced  Akin to the ‘balanced score card’ within an undertaking  Some examples:

17. Garry Law: Performance Measurement17 Roading Operations and Maintenance Contract (Extra work and extension reward) 12 KPIs (Monthly to Annual) Groups:  Communications and Customer Services and Emergency Response (20%)  Health, Safety, Quality & Environmental (20%)  Operations, Physical Works (40%)  Technical Support Services (20%) Hurdle scores for extra work and extensions

18. Garry Law: Performance Measurement18 Water/ Wastewater / Stormwater Operations and Maintenance Contract (Extra work and extension reward) 15 KPIs (Monthly to Annual) Groups:  Communications, Customer Care, Relationship & Emergency Management (20%)  Health & Safety, Quality, Environmental (20%)  Operations, Physical Works (40%,)  Reporting, Management of Assets, Innovation (20%) Hurdle scores for extra work and extensions

19. Garry Law: Performance Measurement19 Wastewater Treatment Plant Maintenance Contract (Sum at risk penalty) 9 KPIs (Monthly to annual) Groups:  Quality 70%  Reporting 20%  Health and Safety 10% Sum paid proportional to scores with range: awful / slight stretch

20. Garry Law: Performance Measurement20 Alliance Design Build Contract, Wastewater Treatment and Recycling Plant Construction (Quality pool reward / penalty) 12 KPIs (Monthly to whole of term) Groups:  Acceptance / commissioning 25%  Environmental compliance 15%  Operating costs 30%  Legacy 30% Sum paid + proportional to scores with range: awful / BAU / stretch

21. Garry Law: Performance Measurement21 Managing Contract, Water / Wastewater Reticulation Design / Construction (Quality pool reward / penalty) 12 KPIs (Monthly to annual) Groups:  Environmental outcomes 30%  Stakeholder management 30%  Legacy / quality 40% Sum paid + proportional to scores with range: awful / BAU / stretch

22. Garry Law: Performance Measurement22 Mathematically: Mathematically there are different sorts of measurement scales: Non- Parametric:  Nominal scales: milestones, pass/fail, yes/no, on/off, true/false, counts by classes  Ordinal scales: ranks, counts by ranks  Interval scales: defined units but no natural zero, differences are real but ratios not Parametric:  Ratio scales: defined units with a natural zero, ratios are real and unit-independent Engineers tend to be happiest with the last – but the others are perfectly respectable (and have their own branch of statistics)

23. Garry Law: Performance Measurement23 Defining KPIs  Units  Time window looked at  What counts  Who measures  When reported  Exceptions management  Who can correct errors A methodology datasheet per KPI is good practice

24. Garry Law: Performance Measurement24 Linking KPIs to Rewards / Penalties - 1  Termination  Extra work within the contract term  Extending the contract term  Renewing the contract  Performance payment incentives / disincentives

25. Garry Law: Performance Measurement25 Linking KPIs to Rewards / Penalties - 2  Use all the KPIs? – Probably not  Extra work within the contract term? – may not be an incentive if there are better opportunities elsewhere  Extending the contract term? – ditto, May be a distant reward at the beginning, in the final term extension the incentive is lost  Renewing? – Can only be one factor - a distant reward at the beginning

26. Performance Incentive Reward is:  Fixed fee for management / overheads  Fixed fee for specified scope and / or  Variable fee (time and materials) for unpredictable items  “At risk” element variable on performance

27. Garry Law: Performance Measurement27 KPIs linked to Payment: Sum at Risk Method  The at-risk element may be a fixed sum or a percentage of the base payment, or the profit revealed –If the client sets this too large the contractor will manage risk by bidding up the base payment  The actual payment is the “at risk” element factored by an overall performance measure  The overall performance measure is made up of a “basket” of performances on KPIs  Full reward is at a “Stretch Target” level for the KPIs.  Zero reward is an abysmal performance level

28. Garry Law: Performance Measurement28 KPIs linked to Payment: Quality Pool Method  The at-risk element is a fixed sum set by the client  The actual payment is the pool factored by an overall performance measure, additional if the performance score is favourable, a deduction if unfavourable  The overall performance measure is made up of a “basket” of performances on KPIs  “No payment” level is performance at a “Business as Usual” level  Full positive reward is at a “Stretch Target” level for the KPIs.  Full negative penalty is an abysmal performance level

29. Garry Law: Performance Measurement29 KPIs linked to Payment  Why offer more? (i.e. the quality pool method)  In a competitive environment it in fact makes little difference – the contractors will tailor their bid around the profit they are targeting, where they think they can perform on the KPIs and price accordingly

30. Garry Law: Performance Measurement30 KPIs within the overall performance score  Can use as “all – or – nothing” - appropriate for measures where failure has an external risk – environmental prosecution or such. Must be achievable. Don’t use this on matters which may be cause for contract termination  Can be proportional. Set the upper and lower limits and pro rata in between  With a proportional linking review if you actually want better than ‘business as usual’ for a particular KPI – don’t do it unless it has value to you

31. Garry Law: Performance Measurement31 Opportunities with Targets  Set them once measurement has run for a bit (on some - not too many)  Step them up – if way below a benchmark standard, get there progressively rather than pretend you will do it overnight.  Ratchet them – a new performance record becomes the new target (don’t do if there is much variation beyond the control of the Contractor i.e. Deming's special causes dominate )

32. Garry Law: Performance Measurement32 Risks with KPIs RiskMitigation Strategies Disputes on KPI calculation- Use methodology datasheets - Contractor self-production External measure discontinuedMinimise use of external measures Measure unavailable at start-upAllow full score for start up period Measure unavailability at paymentIncentive: Score zero for Contractor generated, full for Principal generated, no back calculation. Contractor mis-calculationAudit trail – keeping of records – and post term Disputes over applicability – unforeseen circumstances Have disputes procedure apply, with staged escalation if unresolved Declining relevance of KPIs over timeHave a periodic review – substitution by agreement. Declining relevance of targets over time Ratchet, re-evaluation against external bench marks with substitution by agreement. Tenderers sceptical on achievabilityHave history available Performance history treated as commercially sensitive Contract must make clear will be available to tenderers in renewal

33. Garry Law: Performance Measurement33 END Law Associates Ltd Consulting Engineers PO Box 87311 Meadowbank Auckland – 09 520 2152 - 0275 665764