1. 2011 CIP Compliance Monitoring – On-site Audits for Entities with Critical Cyber Assets
Lew Folkerth
January 31, 2011 1

2. Agenda CIP On-site vs. Off-site CIP Audit Scope CIP Compliance Period
CIP Evidence Requests
Applicable Versions of the CIP Standards
Scheduling of CIP Audits
Guidance
Lessons Learned
What if…?
2011 CIP Audit Process
Questions 2

3. CIP Off-site vs. On-site
If an entity has identified no Critical Cyber Assets (per CIP R3), then that entity is exempt from CIP R1, R3, R4, R5, R6 and CIP through CIP In other words, the scope for that entity is:
CIP R1 (Risk-based Assessment Methodology)
CIP R2 (Identify List of Critical Assets, may be null)
CIP R3 (Identify List of Critical Cyber Assets, will be null)
CIP R4 (Management Review and Approval)
CIP R2 (Designation of Senior Manager)
These audits will be conducted at the ReliabilityFirst offices. 3

4. CIP Off-site vs. On-site
If an entity has identified Critical Cyber Assets, that entity must be compliant with all requirements of CIP through CIP
These audits will be conducted at the entity’s location
Note that some entities on the 6-year audit cycle will have an on-site CIP audit. This is a departure from the practice for 693 audits. 4

5. CIP Audit Scope Actively Monitored Standards
The NERC 2011 Actively Monitored Standards list has identified 36 of the 43 CIP requirements to be actively monitored. The requirements not actively monitored are:
CIP R1 (Cyber Security Policy)
CIP R3 (Exceptions)
CIP R4 (Information Protection)
CIP R1 (Awareness)
CIP R2 (Training)
CIP R3 (Change Control)
CIP R5 (Testing Backup Media) 5

6. CIP Audit Scope Actively Monitored Standards
Note that all entities that have identified Critical Cyber Assets must continue to comply with these requirements.
These requirements will not be actively audited in This means that for 2011 CIP audits:
They will not be on the audit agenda.
They will not be on the evidence request list.
They will not be reviewed by the audit team unless the audit team has a reason to believe that an entity is not compliant with one or more of these requirements.
Example: The audit team observes a Critical Cyber Asset (CCA) list that does not appear to be protected. The audit team may request a copy of the entity’s information protection program to determine how such information is to be protected. If the CCA list has not been protected as required by this program, the audit team may write a Possible Violation for CIP R4. 6

7. CIP Audit Scope Risk-based and Performance-based Assessment
This is not the risk-based assessment required by CIP-002
Documented in the NERC CMEP 2011 Implementation Plan
NERC and the Regional Entities have developed risk-based and performance-based criteria for determining the scope of compliance audits in 2011
These criteria were used to determine the initial scope of 2011 compliance audits throughout all of North America
NERC and Regional Entity criteria may be used to modify the scope of audits in the Regional Entity’s footprint
Audit scope may be expanded to other applicable requirements
Audit scope may not be reduced 7

8. CIP Audit Scope Risk-based and Performance-based Assessment
Risk-based Criteria
Most violated standards across North America
Most violated standards in Regional Entity footprint
Regional standards most violated
Registered Entity specific issues, including, but not limited to:
Operational issues
Operational footprint changes
Corporate restructuring
Other trends, etc.
Random determination
Registered function trends and concerns
Standards rising in prominence through trend analysis
Compliance culture, including overall strength of compliance 8

9. CIP Audit Scope Risk-based and Performance-based Assessment
Performance-based Criteria
Past performance of Registered Entity
As it relates to the operation of the BES
Relative strength of compliance controls
More detailed review and testing of the Registered Entity’s programs and procedures to ensure the programs are actually being implemented, rather than relying only on documentation 9

10. CIP Audit Scope Applicable Versions of the CIP Standards
The audit team will use the appropriate version of the standard for the time period under review:
Version 1 (CIP thru CIP-009-1)
In effect from 7/1/2008 through 3/31/2010
Version 2 (CIP thru CIP-009-2)
In effect from 4/1/2010 through 9/30/2010
Version 3 (CIP thru CIP-009-3)
In effect beginning 10/1/2010
This is the primary version under review. Prior versions may be used as needed to review compliance prior to 10/1/2010 10

11. CIP Compliance Period
CIP through CIP specify document retention as “previous full calendar year”
CIP audits performed in 2011 will examine compliance at the time of the 90 day audit notification
At the audit team’s discretion, the audit team may examine compliance for the period of January 1, 2010 to the end date of the audit, for the applicable version of the CIP Standards. 11

12. CIP Evidence Requests
Provided with the audit notification 90 days before the audit
Pre-audit Evidence Request
Requests evidence of compliance to all active requirements
To be delivered 40 days before the scheduled start of the audit
CIP Data List for Sampling
Requests lists of certain items in order that the audit team may select a sample of the items for further review
To be delivered to ReliabilityFirst 30 days after the audit notification 12

13. CIP Evidence Requests Evidence requests during pre-audit review
Used to fill any identified gaps in the compliance evidence
On-site evidence requests
Generally reserved for the most sensitive information
May be needed to complete the compliance picture 13

14. Scheduling of CIP Audits
36 Requirement CIP Audit
Scheduled separately from 693 audit
1:00 PM Monday thru Noon Friday (usually)
One or more teams as needed
May extend into an additional week (or more) if needed to complete the audit 14

15. Guidance ReliabilityFirst Compliance Staff:
CANNOT tell you what to present as evidence.
CAN request what the standards are seeking for compliance to a requirement.
CANNOT tell you what to show us to be compliant. The language of the standards, the measures, and the Questionnaire/RSAWs are what is needed to comply.
CAN give you help and guidance to get you organized and headed in the right direction. 15

16. Guidance Audit Preparation
This process is on-going throughout the year.
Items to consider:
Develop a Compliance Culture within your organization
Create internal compliance processes to provide for comprehensive internal review
Be organized
Utilize consultants to help if needed
Remain connected to NERC and the Regions 16

17. Guidance A Compliance Culture is a mind set and should:
Be maintained within the whole organization
Have Senior management involved and available
Involve all employees
Utilize training to assist in development
Have policies and procedures available to all personnel
Not be something you do when the regions come knocking on your door 17

18. Guidance
Create Internal Compliance Processes to provide for continual review by:
Having a corporate compliance program
Consider assigning a FTE to monitor and be responsible for compliance
Discuss with other entities how they have set up an internal compliance program
Perform internal audits
Consider a process, procedures, etc. tracking or updating system to maintain compliance to the standards 18

19. Guidance Good organization is crucial. It’s recommended that you:
Organize by Standard.
Include the completed QRSAW and copies of the evidence with each standard.
Know your evidence! (And have the right person/people in the room.)
Use simple, descriptive file names for your electronic documents.
Have a means to provide additional information at the audit reviews
“Help us help you!”
The more organized you are, the easier it is for the auditors to see that you’re compliant, and the easier (and quicker) the process will go. 19

20. Guidance Remain connected to NERC and the Regions:
Review their websites
Attend workshops, webinars, conferences, etc
Know who to contact and utilize appropriate personnel
Be a Standard Drafting Team member
Comment on standards and other documents
Knowledge is the key! 20

21. Guidance
Remember, audit preparation should be on-going throughout the year to ease the burden upon your entity when audit review time comes about. Be aware that:
The NERC Implementation Plan is posted in November of each year.
ReliabilityFirst will send audit notifications 90 days in advance of the scheduled audit review date
The latest version of the Questionnaire/RSAWs are posted on the NERC website.
They will help you get organized and get your evidence ready.
Regional Entity sections must be completely filled out and returned to the region 30 days before the audit review date.
All sections that say (Registered Entity Response Required) must be filled out.
Leave blank any requirements that are not in scope. 21

22. Guidance It’s an open book test! It’s all about the evidence!
Make sure you have covered everything in the standard (remember “shall” and “must”).
It’s all about the evidence!
Reminder –the Regions have the authority to look at and take evidence via 18 CFR 39.2, NERC Rules of Procedure, Federal Power Act, FERC Order 672, etc.
Evidence is normally thought of as: procedures, test results, s, spreadsheets, screenshots, voice recordings, attestations and reports.
But it can also include: Statements, interviews, observations, logbooks. 22

23. Guidance Quality of Evidence Too old (2007), or too new (“wet ink”).
Consolidated vs. multiple documents.
A hand-written Post-it vs. an official-looking corporate document.
Sections of procedures rather than the entire procedure.
Documents with appropriate identifying marks
Company name or logo
Date
Revision information
Title
Page numbers 23

24. Guidance
Auditors look for entity identification, revision dates, histories, effective dates, and authorizing signatures with dates.
You must have evidence to show compliance during the entire applicable period (no gaps).
In this case, January 2010 until the date of the audit review
It is the entity’s responsibility to ensure that there are proper markings placed on all evidence given to the Regions
Ex: Confidential, Non-Public, Contains CEII Information, etc. 24

25. Lessons Learned Not required, but we’ve seen these things work well:
Having specific portions of the evidence highlighted (Word or Adobe).
Bookmarks or hyperlinks to aid in reviewing the evidence.
Arguably the two most important things to make yours and the auditors job easier!
Please don’t feel like you have to be “high tech” to impress the auditors or pass the audit ---the content is much more important!
Make sure your evidence proves you do what your policies and procedures say you will do. Review the evidence you are supplying for errors and areas of possible non-compliance!
It is difficult to overstate the importance to the audit process of having clear, well-organized, readable evidence. 25

26. Lessons Learned
A large portion of the entities that we audit could improve their document management program by:
Providing a narrative explanation of how your evidence shows you are compliant to the standard. This is asked for in the QRSAWs
Ensuring that your evidence is identifiable as your entity’s documents and have proper authorization signatures.
Providing revision histories, page numbers,
Ensure that all documentation is updated according to the standard
Specifying time periods (ex: does “annually” mean one calendar year, or 365 days from the last event?) for your procedures, etc.
Providing deadline tolerances (ex: deadlines are +/-one month for periodic reviews).
Explaining your compliance or intent to comply with the requirement 26

27. What if…?
You consider your information to be sensitive? (ex: Homeland Security) Discuss it with the ATL. Auditors must still see at least the portions that show compliance.
During your audit preparation you discover a possible violation? Self-report it immediately.
You know you have evidence (data, report, document, etc.), but you can’t find it to show the auditors before the end of the audit? It may be deemed a “possible violation”, discuss with the ATL openly. 27

28. 2011 CIP Audit Process
The CIP standards (CIP-002 through CIP-009) are NERC Reliability Standards, and CIP audits are conducted as Reliability Standard audits. This means:
The audits are conducted in accordance with the ReliabilityFirst Compliance Monitoring and Enforcement Program (CMEP), and, therefore, also in accordance with Generally Accepted Government Auditing Standards (GAGAS)
Compliance with a requirement is confirmed by examination of evidence
Subject Matter Experts (SMEs) are requested to explain to the audit team how an entity’s evidence demonstrates compliance 28

29. 2011 CIP Audit Process The 2011 CIP Audits consist of:
90 day Notification
85 day Conference Call
Entity Submittal of Lists for Sampling
Audit Team Requests Additional Information Based on Sample Selection
Entity Submittal of Information 40 days Before On-Site Review
Audit Team Pre-Audit Review
On-Site Review
Audit Report Completion 29

30. 2011 CIP Audit Process
Ninety Days prior to the on-site audit review ReliabilityFirst will:
Provide notification of your up-coming audit and a request for information. The notification will contain:
General Instructions
Scope
Audit Team
Audit Team Work Histories, Conflict of Interests and Confidentiality Agreements
Preliminary Audit Agenda
Attachment “C” CIP Data Sampling Spreadsheet
CIP Evidence List
Compliance Pre-Audit Survey
Applicable QRSAWs
Attachment “B” Pre-Audit Questionnaire and QRSAW Certification Letter
Audit Preparation Guidelines
Attachment “A1” Audit CIP Worksheet 30

31. 2011 CIP Audit Process The Audit Team consists of:
Audit Team Lead (ATL) (RFC Compliance Staff)
One or more teams consisting of:
Team Lead
Scribe
Coordinator
NERC Observers or Participants (at NERC’s discretion)
FERC Observers or Participants (at FERC’s discretion) 31

32. 2011 CIP Audit Process
A Registered Entity can object to an audit team member
On the grounds of conflict of interest, or the existence of other circumstances that could interfere with the team’s impartial performance of its duties
Objection must be in writing to the Compliance Enforcement Authority no later than 15 days prior to the start of the on-site audit
The Compliance Enforcement Authority will make the final determination concerning the team member’s participation in the audit
NERC and FERC staff can not be limited in their participation on an audit 32

33. 2011 CIP Audit Process Audit Team members are:
Bound by their Code of Conducts or applicable Confidentiality Agreements
Provided to the Audited Entity
NERC staff falls under the statement of NERC's obligation on the ROP (Section 1500) and code of conduct
Regional staff fall under their Code of Conduct and confidentiality statement per their delegation agreement
Contractors will sign NERC confidentiality agreements
Regional staff will not sign an entity’s confidentiality agreement
COI statements ensuring that no COI exists by the team are provided 33

34. 2011 CIP Audit Process ReliabilityFirst will provide an agenda which:
Identifies the requirements to be audited
Covers the expected days to complete the audit
Identifies audit sub-teams as appropriate
Contains a schedule for standards to be audited and time allotted for presentations
Contains interview, off-site visits and group meeting schedules 34

35. 2011 CIP Audit Process
We have developed an Evidence Spreadsheet which:
Is a guidance tool for the entities in compilation of evidence. Having these documents does not ensure compliance but assists and simplifies the process.
Is not an all inclusive listing.
Is a listing of evidence as per the requirements in the standards needed for review by the auditors.
May request some substantiating evidence to assist the auditor in a determination of compliance. 35

36. 2011 CIP Audit Process
The audited entity is asked to provide a Pre-audit Survey which will include:
General information on the entity’s organization, registration and general structure
Physical Security Requirements
Identification requirements
Restrictions
Escorts
Corporate security requirements
Logistical Information
Hotel, airport and travel information 36

37. 2011 CIP Audit Process
Questionnaires/Reliability Standard Auditor Worksheets (QRSAWs):
Auditor worksheet for the reliability standards and requirements covered in the audit
Entity sections must be fully completed and returned 40 days before your audit
Entities must review the QRSAW Certification Letter and complete the Attachment “B” Pre-Audit Questionnaire
Provides guidelines concerning the requirements (Compliance Assessment Approach)
Does not add additional requirements
Posted on NERC Website 37

38. 2011 CIP Audit Process
“Audit Preparation Guidelines” is a document identifying the needs of the audit team in performance of the on-site review. This document covers:
Equipment needs
Room needs and layout
Lunch provisions 38

39. 2011 CIP Audit Process
Eighty-five days prior to the On-Site audit ReliabilityFirst will:
Contact the audited entity to discuss:
Scope
Entity questions on the audit process or notification documentation
On-Site Agendas
Any other questions or concerns 39

40. Forty days prior to the On-Site review, entities are to submit:
2011 CIP Audit Process
Forty days prior to the On-Site review, entities are to submit:
Completed QRSAWs, surveys, questionnaires
Evidence pertinent to compliance 40

41. 2011 CIP Audit Process
After receipt of the evidence of compliance the audit team will conduct a pre-audit team review to:
Discuss the entity submittal
Develop additional Requests for Information (RFIs)
Make determinations of “No Finding” where possible
Identify potential problem areas to further evaluate with the entity
Develop an On-Site review approach 41

42. 2011 CIP Audit Process
Sample populations to be developed for on site review:
Will use the evidence listings provided to develop consistent sample sizes to maintain a prescribed confidence level
Will require entities to send listing of entire populations (substations, relays, etc.) so the auditors can develop an objective random sample to be tested. 42

43. 2011 CIP Audit Process The On-Site review consists of:
One, possibly two (or more, if needed) weeks for completion
Second week reserved for On-Site continuation (ATL decision)
Opening Briefing
Review of all requirements in the scope of the audit
Daily Status Report from audit team
Control Room and Data Center visits
Off-site visits
Exit Briefing 43

44. 2011 CIP Audit Process
Opening Briefing of management and participants of the audit:
Allows audit team to:
Introduce team members
State Objective and Scope
Explain process of the audit
Discuss Confidentiality of Information and data handling
Set the tone for the audit
Provide the roles of the audit team and audited entity
Use as opportunity to seek clarification on issues from RSAWs and any other preliminary information submitted.
Allows registered entity to:
Provide overview of the their system and operations
Provide logistic and security information
Seek clarifications on scope of the audit or other matters 44

45. 2011 CIP Audit Process
Control Center and Data Center tours conducted to seek validation:
Operator training
Processes, procedures, etc. have reached the necessary personnel
View the security environment
Observe and gain an understanding of the entity operations
Other observations as determined by the audit team 45

46. 2011 CIP Audit Process Visits to other facilities may be:
Conducted as part of the CIP monitoring processes
Selected using a sampling methodology or random method
Conducted in performance of monitoring processes, and where it is believed to be necessary to actually visit a facility:
1st) The audit team will seek evidence such as documentation, pictures, logs, work orders, etc to determine if compliance is met;
2nd) The audit team will seek to view an entity's internal compliance documentation and internal controls to confirm if compliance to a requirement is met;
3rd) If neither of these approaches can produce sufficient evidence to substantiate that an entity meets compliance, the audit team may, at its discretion, do actual field visits to substantiate that compliance has been met for a requirement. 46

47. 2011 CIP Audit Process
Exit Briefing with entity management and all participants of the audit to:
Review the scope of the audit
Provide the findings
Explain the basis for the team findings on possible violations
Discuss confidentiality and data retention
Discuss the report process and timeline
Request completion of feedback forms 47

48. 2011 Post-audit Timeline
Send final report to RFC VP and Director of Compliance Manager, NERC and Registered Entity
The Audit Team Lead transmits the final report for audit team review
The Audit Team Lead receives all comments from the Audit team
20 business days
10 business
days
5 business days
20 Business days
5 business days
5 business days
5 business days
The Audit Team will work to deliver a draft report to the Registered Entity
Audit Team to provide comments
Revision of the draft report
Registered Entity to review and provide comments
Revision of the draft report
Audit Team to provide comments
Audit Team Lead to revise draft report
Edit the final report upon receipt of Audit Team’s comments
Audit Team Lead sends the draft report to the Audit Team for their review and comments
Edit the draft report upon receipt of Registered Entity’ s comments
The Audit Team Lead sends the draft report to the Registered Entity for their review and comments 48

49. 2011 CIP Compliance Monitoring
Questions? 49