Evaluating and Managing Performance …..it’s not just about Vendors Yukon Procurement Conference February 16, 2015

Purpose Provide an overview of: what “Vendor Performance Management” is, and is not Some examples from public sector agencies in Canada (and their different objectives)

What is “Vendor Performance Management”? It is: about improving success in meeting contract outcomes one element of “Vendor Management” Aligning procurement, contract terms, project management to meet organizational objectives Monitoring/mitigating vendor risks as well as Aggregating, assessing and communicating vendor performance information and Actively managing relationships It is NOT : code for simply increasing criticism of suppliers Surprise report cards Only about ‘bidder barring’

Office of the Procurement Ombudsman, 2010 Benefits Increase accountability Help achieve best value for taxpayers Office of the Procurement Ombudsman, 2010 Increase competitive advantage Improve stakeholder satisfaction Increase performance visibility Survey Analytics, 2011 “Measuring, monitoring, evaluating and reporting on vendor performance creates an atmosphere that fosters better communication and results in improved government-vendor relationships.” Office of the Procurement Ombudsman, Procurement Practices Review, 2009-10

How is it done? Most effective approaches integrate tools across all phases of the procurement and contract lifecycle For example, buyers can: Ensure clear statement of functional and performance requirements Establish key performance indicators Use in-contract evaluations, monitoring procedures, and measurement of performance against KPIs Evaluate and maintain records on supplier performance Provide feedback to suppliers

Which tool(s) to pick? Each tool: Has different advantages/disadvantages Requires a different level of investment to implement and maintain Has a different likelihood of success, depending on: Market characteristics (e.g. market size and health, maturity of vendors) Capacity/maturity of an organization to properly implement and support

Public Sector Activity Recent Survey (Ontario)* 92% of suppliers and buyers think VPM is an important activity 25% of organizations have VPM activities Becoming increasingly common across Canada, some examples include: PWGSC Canada Revenue Agency Correctional Service of Canada Nfld Department of Transportation and Works Infrastructure Ontario Defence Construction Canada Province of BC Ministry of Government Services (Ontario) *Malatest & Assoc. (2012), Vendor Performance Management Study The Malatest study found that of public sector organizations surveyed that did have vendor management tools in place, the majority used performance clauses in contracts to help manage vendor performance (94%), as well as a variety of templates and forms (75%), progress meetings (70%), performance documentation (65%), and third party verification (35%). In addition, approximately 80% of respondents (with virtually no difference between buyers and suppliers) thought performance should be measured on every contract (and not, for example, just those triggered by poor performance or by a dollar threshold).   This study also proposed that the most important elements of performance to be measured (in order of priority) were: Quality of deliverable Effective communication throughout the engagement Budget/cost control Availability of resources to carry out the contract Quality of resources Maintaining timelines/deadlines

Defence Construction Canada (DCC) Driver: “train (not penalize) contractors/consultants to meet expectations of DCC”  Overall rating = sum of points from equally weighted criteria Administration / Contract Management Execution / Project Management Quality of Workmanship Completion / Close Out / Time Health & Safety Scale of Unacceptable (0-5), Not satisfactory (6-10); Satisfactory (11-16) and Superior (17-20) Defence Construction Canada (DCC) is a federal crown agency that provides contracting, construction contract management and related infrastructure services to the Department of National Defence. Note slide shows the criteria for contractors, there is a slightly different list for consultants

DCC - continued Bidding privileges suspended for any score of 5 or less in one category, or a second occurrence of a total score less than 50% “failures” are relatively rare Heavy reliance on documentation – require clear evidence DCC’s own practices didn’t contribute Generally considered by staff and suppliers to be successful Staff believe contributes to their ability to be an attractive client in a competitive market Staff at DCC indicated that overall the system is considered to be working well. They rely on an evidence based approach to document performance concerns and will only pursue suspensions in cases where the documentation clearly indicates DCC’s own contract management practices have not contributed to poor vendor performance. In general, the system is considered to be working well, is appreciated by the majority of suppliers, and contributes to the organization’s ability to be an attractive client in a competitive market

Infrastructure Ontario Driven in part by desire to reduce time spent evaluating high volume of responses to RFPs/tenders Required for all contracts >$100,000 Vendor Performance Rating = average of that vendor’s Scorecards over a three year period Vendor performance is rated as : 1 – Consistently falls far below expectations (< 25% of expectations met) 2 – Frequently misses expectations (<50%) 3 – Mostly meets expectations but sometimes misses (<75%) 4 – Consistently meets expectations (100% of requirements) 5 – Exceeds expectations Ratings applied during final evaluation, worth minimum of 10% excellent service can lead to extended time on prequalified list A Vendor Performance Rating (“VPR”) is an average of all of that vendor’s (final) Scorecards over a three year period. If the vendor does not have a scorecard for each of the three years, its VPR will be based on the average ratings for the years for which scorecards are available, or – if it has none -- it will be assigned a rating based on the average of all IO vendors’ ratings from the previous fiscal year (the “Global Average”). 1 – Consistently falls far below expectations (< 25% of expectations met). Performance jeopardized the achievement of contract requirements, despite additional oversight. 2 – Frequently misses expectations (<50%). There are a number of performance issues that required IO to provide additional oversight to ensure that contract requirements were met. 3 – Mostly meets expectations but sometimes misses (<75%). There are very minor performance issues but vendor or service provider has otherwise met the contract requirements. 4 – Consistently meets expectations (100% of requirements). There are no performance issues and the vendor or service provider has met the contract requirements. 5 – Exceeds expectations. Vendor or service provider has demonstrated a performance level in measurable excess of contract requirements (e.g. provided tangible recommendations for improvement, proactively addressed issues before they arose, etc.).

BC – Internal Reference Check (2011) Required for services contracts >$10M Conducted at beginning of a prequalification process, result in pass/fail Common indicators include: Performance - contractual standards & service target levels Cost performance Schedule performance Team This internal reference check must review a vendor's performance on all contracts with a value of $1 million or greater that the vendor has had with the Province in the past three years.   The policy arose as a result of concerns that poor performance by a vendor on one government contract often had no consequences in terms of their ability to bid on or to be selected for future opportunities within government. The pressure was public/political – not driven by staff Performance - contractual standards and service target levels (has contractor met contractual standards and service target levels? Was the client satisfied with the contractor’s performance of the services? Other considerations can be professional behaviour, client management, timely meeting of administrative requirements eg. reporting, record keeping etc.) Cost performance (were the services provided within budget or close to the cost estimates? Did the contractor manage expenses appropriately? How did the contractor manage cost overruns?) Schedule performance (timeliness of completion, meeting of target levels, interim and final milestones) Team (evaluation of parent companies, affiliates, major subcontractors, management, key personnel, predecessors) BC experiences suggests staff support for the tools is a critical consideration The internal reference check procedure is not considered to be particularly successful significant reluctance among staff to both record and rely on documentation of poor performance, at least in the case of these high value and typically highly visible strategic contracts, to eliminate a vendor from a procurement process. The policy does not appear to have had significant impact on the procurements it is intended to affect. The Vendor Scorecards, however, appear to be off to a more successful start. SPO has been using a ‘deal health check’ measurement tool in relation to the major strategic deals for the past two years in an informal way – i.e. not required in the contracts themselves and not intended for public reporting purposes. The ‘health check’ has some significant similarities to the Vendor Report Card in that it assesses similar areas and involves some two-way ratings. Through this approach they have gained the input and acceptance of the vendors involved, which will help with the formal launch of the Report Card as a formal performance management and public reporting tool in the coming year. The usefulness of the tool in supporting a proactive and constructive approach to identifying and managing performance issues led SPO to extend implementation beyond the vendors associated with the strategic deals alone to those involved in helping SPO design, implement and manage them

BC – Vendor Scorecard (2014) Two new tools: one for “strategic deals”, one for the consultants that support them Driver – recognition that improving performance requires clear and ongoing communication concerning performance expectations Rating carried out by both parties and aims to foster discussion about areas needing improvement or differences of opinion For consultants: If more than 2 of 8 questions unsatisfactory = 1 demerit 3 demerits = potential removal from RFQ SPO has developed two Vendor Scorecard tools. The first is for use in assessing the relationship and performance of with the vendors involved in the strategic deals themselves. The current portfolio of these strategic deals consists of 13 major projects with a total contract value of $5.9 billion and annual contract expenditures of around $550 million. The second tool is designed to assess the performance of consultants involved in supporting SPO’s mandate, (including negotiations, procurement support, training, development of tools and analysis of deals, etc.) which involves several dozen individuals and firms, initially selected through a RFQ process, and expenditures of $2-4 million annually.   At the conclusion of each contract, the contract manager and the direct client contact (i.e. the person most closely linked to the work performed by the consultant) will jointly complete a consultant Rating Sheet. A copy of the completed Rating Sheet will be provided to the consultant and a debriefing meeting held if requested. A Rating Sheet will receive a “Fail” if two or more of the eight questions achieve the following marks: “Sometimes / Rarely / Never / No / Unsatisfactory”. Each failed Rating Sheet is a “demerit” and recorded in the Vendor’s SPO Vendor Scorecard. Three demerits recorded may result in the consultant (individual or firm) being removed from the RFQ List of Qualified Suppliers. Companies can reapply to subsequent RFQ’s issued by the SPO once the current RFQ has expired (in 2016). Removal from the RFQ list does not preclude vendors from bidding on other procurements issued by government. Each SPO Vendor Scorecard will be retained and held by the SPO and may be used for future reference checking purposes.

Questions? Note – TEB piloting initiative of survey for consultants designed to increase communication about performance feedback for both parties re: managing costs maintain project schedule Write clear/comply with contract specifications manage support staff/third party contractor(s) communicate effectively Overall management abilities (e.g. problem solve, document and minimize risk) Areas where most effective or needing improvement Overall satisfaction Additional Q for Project Manager: To what extent did the final product or service satisfy the terms of the contract?