Compliance Performance Measure Proposals and Preliminary Trends Greg Pierce – Chair, Performance Measures Task Force Compliance and Certification Committee Meeting March 10-11, 2010

2Topics  PMTF Tasks and Approach  Performance Measure Proposals  Preliminary Trends  Outreach Effort  Next Steps

3 PMTF Tasks  Establish an open, consistent and repeatable metric development process to identify and evaluate metrics  Develop compliance improvement measures at the highest levels to provide insight in terms of a set of compliance principles  Coordinate and work with Regional Entities, RMWG, OC, PC and CIPC to ensure agreement  Recommend how to benefit from the compliance improvement measures

4Approach  Adopt RMWG’s open metric development process  Review data source on compliance and other key input factors  Propose a set of performance measures at high levels  Gather data and validate trends  Solicit feedback and other ideas from stakeholders  Prepare a draft report starting April 2010

5 Proposed Performance Measures  Compliance Culture  Timeliness of CVI/CIQ  Average Duration of Mitigation Plan Completion  Violation Risk Index  Average Duration of Audit Report Completion w/ No Violations  Average Duration of Audit Report Completion w/ Potential Violations  Availability/timeliness of 'lessons-learned’ guidance from audits, operating experience, violations and disturbances  Feedback to standards development  The detailed individual metric definitions and specifications are available at

6 Compliance Culture Measure  Metric – Compare Self-Disclosed Violations to the Total Number of Discovered Violations  Self-Disclosed = Self-report + Self-certification  Indicates the level of trust and cooperation within the “NERC Enterprise” as well as the culture of compliance of the stakeholder community

7 Trend By Discovery Method

8 Compliance Culture Trend

9 Timeliness of CVI/CIQ  Average Duration – for CVI/CIQ with Open status  Total number of CVIs/CIQs for Open and Closed status  Number of lessons learned and trends issued to industry

10 CVI/CIQ Timeliness Measure Duration (Days)CVICIQ OldestAverageOldestAverage NERC Region

11 Triggers for NERC/Regional CVIs/CIQs

12 Average Duration of Mitigation Plan Completion  Duration – from Date plan received to Mitigation close date  Sum of all durations divided by total number of violations discovered in a particular quarter Remove zero duration days

13 Average Number of Days for Completing Mitigation Plans

14 Violation Risk Index (VRI)  Violation Risk Index is defined as a risk factor and severity level weighted average based on unmitigated violations Assess potential consequences of a particular unmitigated violation. Unmitigated violations of higher risk factor requirements have more weighting value than of lower risk factors. Determine the change in reliability levels due to Standard requirement violations.  Weighting factors and an example are included in Appendix A.

15 NERC Unmitigated Violations (6/18/07 to 12/31/2009)

16 Violation Risk Trend

17 Average Duration of Audit Report Completion w/ No Violations  Measure audit process efficiency and ability to promptly advise industry of audit results  These reports do not have potential violations  Starting point: audit ending point  Ending point: report finalized with entity and received by NERC

18 Average Duration of Audit Report Completion w/ No Violations

19 Average Duration of Audit Report Completion w/ Potential Violations  Measure enforcement process efficiency and ability to promptly advise industry of audit results  These reports have potential violations  Starting point: audit ending point  Ending point: report finalized with entity and received by NERC  Data is not available until second quarter of 2010

20 Timeliness of Lessons-Learned  Establish a common and central platform to share lessons-learned  Timeliness of actionable lesson-learned guidance from operating experience, violations and disturbance events Lessons-Learned guidance Recommendations from event analysis Compliance analysis Advisories

21 Feedback to Standards  Timeliness of compliance results/feedback to standards # of SARs (e.g., refine a particular requirement) # of suggestion and comment submittals The Standards input form is available at al_2008June30.doc al_2008June30.doc  Direct input to “results-based” standards initiative Focus on risk-based standards, using first assessment triage sheet

22 Outreach Effort  Chair met with and presented to the RMWG on February 18, 2010  Chair presented to the NERC/Regional Entity Compliance Group at its conference call on February 26, 2010  Chair presented to the OC/PC/CIPC at their joint WebEx conference call on March 9, 2010

23 Next Steps  Continue reaching out to seek guidance Members will meet with OC and PC on March 16-17, 2010  Recommend the proposals in June, desiring CCC approval in September 2010  Comments and suggestions are due April 2, 2010 Send to PMTF Chair, Greg Pierce at and NERC at

24 Question & Answer

25 Appendix A  Violation Weighting Factor Table Note: 1. The weighting values are derived by applying similar ratios developed in the Base Penalty Amount Table described in section 4 of the ERO Sanction Guidelines, Appendix 4B to the NERC Rules of Procedure. 2. Reference materials are available in a NERC white paper “Toward Ensuring Reliability: Reliability Performance Metrics”. It can be viewed at:

26 VRI Example