1. Notice of Compliance Audit
Phil O’Donnell
Manager, Compliance, Operations and Planning
Audits and Investigations
My Name is Phil O’Donnell, Manager of Operations and Planning Audits
I am going to be discussing the
Compliance Audit Notice, Information on formats for submitting your documentation and later discuss types of Audits we conduct and some expectations and advice on audit documentation

2. Audit Frequency 3 Year Cycle Balancing Authority Transmission Operator
Reliability Coordinator
All other registered functions
Subject to flexibility in the future as part of NERC’s Reliability Assurance Initiative
Previously 6 years,

3. Compliance Audit (on-site vs. off-site)
Documentation sent to WECC before audit for preliminary review
The audit team reviews evidence during off-site week or the first week of the audit and completes its review during the second week or on-site week
Data Requests or DRs
Tours to observe facilities
In-person interviews for clarification
In addition to it being longer Primary difference is the face to face contact at your facilities and tours/inspections

4. Compliance Audit (on-site vs. off-site)
Documentation sent to WECC before audit for preliminary review
Data Requests or DRs
Entity may be present at audit if desired
Telephone interviews for clarification
In addition to it being longer Primary difference is the face to face contact at your facilities and tours/inspections

5. Compliance Audit (on-site vs. off-site)
Primary difference is:
Location of audit conduct
Scope is typically smaller for off site
On Site – Required for RC, BA, TOP functions
Per NERC Rules of Procedure
Primary reason for on site visits is to observe your control centers and other facilities and conduct interviews

6. Audit Timeline 145 days 90 days 60 days 30 days 15 days AUDIT CIP v5
Request for
Information
Notice of
Audit
Pre-Audit
Survey
Due
Objections
to Team
Members
Evidence
Due
15 days to return the CIP RFI
Inherent Risk Assessment is taking place 180 days for ICE (Internal Controls Evaluation) 145 days for IRA – you’ll receive a survey in that timeline.

7. Notice of Audit Packet Notice of Audit Letter
ATT D: Audit Scope and WECC RSAWs
ATT A: Compliance Monitoring Authority
ATT E: Certification Letter
Letter
ATT F: Pre-Audit Survey
ATT B: Audit Team Biographies
ATT G: Pre-Audit Data Requests
ATT C: Confidentiality Agreements
ATT H: Post Audit Feedback Form
MORE DETAIL LATER
If you were to get an audit notice to day this is what you would get.
This information will always be available but we are working on changing the process so some of the static information is made available on our website or on request.

8. Notice of Audit Letter 90-Day Notice of Audit Letter
Details of your specific audit
Audit Engagement Dates
Audit Period
Registered Functions within Audit Scope
Audit Team Composition Observers (if applicable)
May include FERC/NERC
Date/time of proposed Pre-Audit Conference Call
Links to reference documents
Body of Notice
ALSO INCLUDES CONTACT INFORMATION

9. Attachments A, B and C Attachment A Attachment B Attachment C
Explanation of Compliance Monitoring Authority
Attachment B
Short Biographies of the WECC Audit Staff
Attachment C
Signed Confidentiality Agreements of the WECC Audit Staff

10. Attachments D and E Attachment D Attachment E Audit Scope
Reliability Standard Audit Worksheets (RSAWs)
Attachment E
Certification Letter
Must be printed on your company letterhead and signed by an Authorized Officer
Scope is determined by the Inherent Risk Assessment between the Enforcement and Audit Team Leads. And may be modified by the ICE if applicable
RSAW’s have been customized for your audit. Please complete the highlighted areas of the worksheets labeled “(Registered Entity Response Required).” Please reference any outstanding self-reports or mitigation plans in each RSAW, as applicable. Please use only the Compliance RSAWs included with YOUR notice package to prepare for the Audit.
Some are WECC Regional Standards
Some have Regional Variance sections (Not in NERC RSAW)
Some enhanced audit approaches.
Keep in Editable Word format!
ATT E: Certifies that the information being provided for the Audit is accurate.

11. Attachment F Attachment F Pre-Audit Survey Verify Registered Functions
Audit Logistics
Signed by Authorized Officer
Please complete all applicable fields
May be different if on site audit had previous questionnaire sent.
Describe company’s organization and structure
Description of your system, provide your RC, BA, TOP, PA, TP and RP. If you are a GO, provide your GOP.
When listing your PA, TP and RP please confirm with them or state that you have not confirmed this.

12. Attachment G Attachment G
Pre-Audit Data Requests – Clarifications for Data Submittal
One Line Diagram
Delegation agreements (if applicable)
CCA and non-CCA lists
Public Key Encryption
This is an evidence checklist to provide assistance when filling out your RSAW’s. It has been customized based on registered functions and the audit scope.
Due with your evidence submittal – 30 days
Some evidence may apply to more than one Standard
One copy is sufficient, but document inventories or “roadmaps” are appreciated
WECC strongly recommend using PKE for your Cyber Security Documents. This further increase the security process and adds that extra layer of protection.
If you choose to utilize PKE please your certificate or public key to the CPC. Our and direct lines are located within
Att G

13. Feedback is encouraged for all phases of audit!
Attachment H
Attachment H
Audit Feedback
Sent with initial package
Feedback is encouraged for all phases of audit!

14. Evidence Submittal WECC Enhanced File Transfer (EFT)
Any questions regarding log in or user credentials please contact or call
Audit Data Folder
We will upload to the wecc notifications folder

15. Evidence Submittal File Folder COM COM-001-1
Master folder name is the Reliability Standard
Sub-folders for all related standards
Additional sub-folders for requirements

16. Evidence Submittal Adobe Portfolios COM
Master folder name is the Reliability Standard
Portfolio files for related standards in sub-folders with specific standard name
Requirement folders within the PDF portfolio

17. Audit Approaches We audit to the Requirements of the Standards.
General Approaches included in RSAW
RSAW may ask specific questions
Always includes the section:
“Describe, in narrative form, how you meet compliance with this requirement.”

18. Audit Approaches
“Describe, in narrative form, how you meet compliance with this requirement.”
Describe here how your company knows it is compliant with this requirement and how you know you have been compliant for the entire period of the audit
Your place to describe your internal controls
Your evidence should support your narrative
No need to duplicate information provided through the ICE

19. List the evidence provided in the RSAW
Audit Approaches
List the evidence provided in the RSAW
This road map is important
Compliance Assessment Approach in RSAW is used as a checklist
Data Request (DR) for gaps or samples
Document and records review are primary
Interviews and observations are usually for corroborating

20. Sufficient Audit Evidence
Sufficiency of Evidence
The measure of the quantity of evidence
Quantity of evidence is dependent on the scope of the audit
Extra quantity does not make up for poor quality
Ensure you provide enough evidence to demonstrate compliance for the entire audit period
HOW MUCH IS ENOUGH

21. Sufficient Audit Evidence
Sampling is used to limit the amount of detailed evidence provided
Normally used in conjunction with summary of a full set of data
Sampling used to assess details
Reduces the burden on the Audit Team but not really on the Entity
Audit Team must select the samples
SAMPLING

22. Appropriate Audit Evidence
Appropriateness
The measure of the quality of evidence
Relevance
Validity
Reliability
WHAT IS GOOD EVIDENCE?

23. Appropriate Audit Evidence
Quality of Evidence
Good Internal Controls point to reliable evidence
Direct observation is more reliable than indirect observation
Examination of original documents is more reliable than examination of copies
Testimonial evidence from system experts is more reliable than from personnel with indirect or partial knowledge
RELATIVE QUALITY

24. Types of Evidence Physical Evidence Documentary Evidence
Testimonial Evidence
Compliance Audits may use all three types but Documentary Evidence is by far the most frequent type of evidence assessed and relied on.
FORMS OF EVIDENCE

25. Testimonial Evidence
Attestations of Compliance or Statements of Compliance are generally not accepted as the only available evidence.
Attestations may be used to explain minor gaps in documentation or to state if no conditions occurred which are subject to a requirement.
Attestor must be knowledgeable and qualified.
SIMPLE VERBAL ATTESTATIONS NOT VERY GOOD
FORMAL WRITTEN ATTESTATIONS A LITTLE BETTER
INTERVIEWS ARE VERY VALUABLE TO COOBORATE AND SUPPORT OTHER EVIDENCE.

26. Evidence for Procedural Documents
The characteristics of a valid procedural or policy document include:
Document title
Definition or Purpose
Revision level
Effective dates
Authorizing signatures
DOCUMENTS - PROCEDURES

27. Non Applicable Requirements
Three instances are acceptable for use of term “Not Applicable”
Entity is not registered for the applicable function (only TOP responsible for TOP requirements)
Entity does not own, operate or maintain the equipment addressed by the requirement (UVLS, UFLS, SPS etc.)
Entity does not use the program or process specified by the requirement (and is not required to… ATC, CBM, etc.)
WHAT DOES NOT APPLICABLE MEAN

28. Evidence for Tasks Performed
When the standard calls for a task to be performed it must be documented.
Records
Logs
Reports
Work Orders
Phone recordings
Transcripts of phone recordings
Shift Schedules
Dates & Times are critical
PERFORMANCE EVIDENCE

29. Evidence of “Coordination” with other entities
Typical evidence provided initially is a single .
“…If you have any comments please contact ______”
This alone is neither sufficient or appropriate to demonstrate coordination between two or more parties.
If s or correspondence are used
Two way communications are needed
Better are:
Meeting Agendas
Meeting Minutes
Attendance Lists
DOCUMENTS FOR COORDINATION

30. Evidence of “Distribution” of information
Typical evidence provided initially is a single with a large distribution list.
“…please see attached”
This alone is typically neither sufficient or appropriate to demonstrate distribution to others.
If s or correspondence are used
Need clear identification of the personnel on the distribution list.
Even better is corroboration by receipt acknowledgement
DISTRIBUTION/POSTING