1. Boot Camp - Conformity
March 7, 2011
SF Bay Area

2. Boot Camp - Conformity Overview Org Chart
<month year>
doc.: IEEE <doc#>
Boot Camp - Conformity
Overview
Org Chart
What does this Working Group do?
Previous meetings – Knoxville, San Francisco, DC, Detroit, Fort Lauderdale
Terms we use
NIST Activities
Our Activities
Slide 2
Bruce Muschlitz, EnerNex
Page 2
<author>, <company>

3. UCAIug Org Chart (simplified)
Technical Oversight Committee
IEC61850
Testing
CIM
OSG
Security
Comms
Conformity
Edge
Enterprise
System
SGIP TCC

4. Boot Camp – Conformity Goals and Activities of the working group
<month year>
doc.: IEEE <doc#>
Boot Camp – Conformity Goals and Activities of the working group
Coordinate and create Conformity Task Forces based on needs.
Edge Conformity
Enterprise Conformity
Security Conformity
Provide overall guidance
Propose/Review task force deliverables
Slide 4
Bruce Muschlitz, EnerNex
Page 4
<author>, <company>

5. Boot Camp – Conformity Previous Meeting (Knoxville)
<month year>
doc.: IEEE <doc#>
Boot Camp – Conformity Previous Meeting (Knoxville)
Organized Group
Introduced as one models
Stressed conformance != interoperabilty
Explained abstract vs. detailed tests
Introduced “virtual” test environments
Slide 5
Bruce Muschlitz, EnerNex
Page 5
<author>, <company>

6. Boot Camp – Conformity Previous Meeting (San Francisco)
<month year>
doc.: IEEE <doc#>
Boot Camp – Conformity Previous Meeting (San Francisco)
Continued Group Organization
Discussed “plug-fest” – won’t do this
Discussed how others do this:
ISO Guide 65
IEC and 17025
(superseded by ANSI-ASQ Accreditation Board
Discussed Product Mark (logo)
Recognized: Testing, SGIP TCC
Slide 6
Bruce Muschlitz, EnerNex
Page 6
<author>, <company>

7. Boot Camp – Conformity Previous Meeting (McLean)
<month year>
doc.: IEEE <doc#>
Boot Camp – Conformity Previous Meeting (McLean)
Organized Security Conformity
Re-organized by Horizontal teams
Discussed interaction with SGIP TCC
Recognized Edge/ENT might work better as (Physical) Device/ (Middleware) Interface
Slide 7
Bruce Muschlitz, EnerNex
Page 7
<author>, <company>

8. Boot Camp – Conformity Previous Meeting (Detroit)
<month year>
doc.: IEEE <doc#>
Boot Camp – Conformity Previous Meeting (Detroit)
NIST TCC IPRM and CPRM coordination
Abstract Test Case (ATC) template defined
Slide 8
Bruce Muschlitz, EnerNex
Page 8
<author>, <company>

9. Boot Camp – Conformity Previous Meeting (Ft.Lauderdale)
<month year>
doc.: IEEE <doc#>
Boot Camp – Conformity Previous Meeting (Ft.Lauderdale)
Worked with SGIP SGTCC to complete IPRM
(IPRM=Interoperability Process Reference Manual)
Agreed to align CPRM with NIST SGIP TCC IPRM
Collaborating with EPRI to establish requirements-based testing activity
Agreed Security Conformity Group to work collaboratively with SGIP CSWG
Slide 9
Bruce Muschlitz, EnerNex
Page 9
<author>, <company>

10. Boot Camp – Conformity Common Terms
<month year>
doc.: IEEE <doc#>
Boot Camp – Conformity Common Terms
Conformance – meets spec?
Interop – plays well with others?
Positive tests – does it work right?
Negative tests – recovers gracefully?
Black Box tests – no inside knowledge
White Box tests – view algorithms
Slide 10
Bruce Muschlitz, EnerNex
Page 10
<author>, <company>

11. Boot Camp – Conformity NIST SGIP Collaboration
<month year>
doc.: IEEE <doc#>
Boot Camp – Conformity NIST SGIP Collaboration
SGIP Test and Certification Committee
Interoperability Process Reference Manual
Published Q4 2010
Promotion during 2011 through WG8
End to End Testing group
Work closely with Edge and Enterprise Conformity groups
Slide 11
Phil Beecher, PG&E
Page 11
<author>, <company>

12. Boot Camp – Conformity Background Material
<month year>
doc.: IEEE <doc#>
Boot Camp – Conformity Background Material
Slide 12
Bruce Muschlitz, EnerNex
Page 12
<author>, <company>

13. Boot Camp – Conformity Questions?
<month year>
doc.: IEEE <doc#>
Boot Camp – Conformity Questions?
Slide 13
Phil Beecher, PG&E
Page 13
<author>, <company>

14. Edge / Enterprise Conformity
Boot Camp

15. Edge / Enterprise Conformity Activity
Certification Process Reference Manual
Test Methodology and Abstract Test Cases
29 December 2018
Conformity Boot Camp 15 15

16. What is the CPRM? Overview of device and system requirements
Identifies best practice for product (device and system) protocol design
Describes testing of specific technologies identified by OpenSG membership
Describes the process used to define and maintain the quality of a Certification Program
Companion document to SGTCC IPRM
29 December 2018
Conformity Boot Camp 16 16

17. CPRM Guiding Principles
Open standards based
Clean, layered architecture
Robust certification program
Focussed on application programming interface, not specific applications
Layered conformance testing
Performance testing considerations
Economically viable
29 December 2018
Conformity Boot Camp 17 17

18. CPRM Organisational Requirements
Identifies organisational structure to support a robust certification and interoperability testing program for products
Addresses “devices” (incorporating hardware) and “system applications” (software only)
Considerations:
Program management
Test laboratory qualification
Logo management
Change control
Dispute resolution
29 December 2018
Conformity Boot Camp 18 18

19. Program Overview 29 December 2018 Conformity Boot Camp 19 19
Interoperability Test and Certification Authority P R O G A M
Certification Program Manager
Device / Product Certification Body
Lead Laboratory
Testing Organizations E Q S / L I C Y
SSO / SDO
Periodic Normalization
Industry Implementers
29 December 2018
Conformity Boot Camp 19 19

20. SG System Components
29 December 2018
Conformity Boot Camp 20 20

21. Context of Individual Test Suites
29 December 2018
Conformity Boot Camp 21 21

22. Abstract Test Documents
Contents / methods based on
X291 OSI Conformance Testing Methodology and Framework for Protocol Recommendations for ITU-T Applications – Abstract Test Suite Specification
Each document summarises requirements for:
Test Methodology or Methodologies
PICS proformas
Test suites
Abstract Test Cases
Other information?
29 December 2018
Conformity Boot Camp 22 22

23. General Model for Multi-Party Testing
Lower Tester Control Function
TCP
LT3
UT3
LT2
TCP
UT2
LT1
TCP
UT1
PCO
PCO
PCO
Y-ASPs
IUT
(P) PDUs
PCO
(P) PDUs
PCO
PCO
(P) PDUs
X-ASPs
X-Service-Provider
7 September 2010
Abstract Test Cases

24. Current Status CPRM Test Case Documents Completed recirculation ballot
Available as companion document to SGTCC IPRM describing organisational and testing requirements to help achieve interoperable products
Test Case Documents
Enterprise Conformity project underway
OpenHAN – working on test methodologies, test harness functionality
Need to start OpenADE and OpenADR
29 December 2018
Conformity Boot Camp 24 24

25. EPRI (CIM) Test Methodology to OpenSG Systems
Test Methodology designed to compatible with:
IEC CIM 61968
SGSystems (AMI-Ent, OpenHAN, OpenADR, OpenADE, etc.)
Test process defined, documented and repeatable
Interop test using Virtual Lab Environment
29 December 2018
Conformity Boot Camp 26 26

26. Agenda for Thursday, March 10th
Overview Security Conformance & Charter
Overview TCC and CSWG Testing & Certification Subgroup
Review Security Testing Methodology

27. Review Security Conformity TF Charter
Establish security conformance requirements for laboratories desiring to certify smart grid components and systems and;
Establish clear scoping boundaries, perform research to identify existing models, and propose a high-level philosophy of approach.
Chair: Bobby Brown, EnerNex
Vice-chair: needed (Sandy Bacik)

28. Review CSWG Testing & Certification
Is NISTIR 7628 Testable / Actionable?
Is AMI Security Profile 2.0 Testable / Actionable?
SGIP TCC Coordination Tasks
Miscellaneous Tasks

29. Testing & Metrics
GAO Report – “no metrics for evaluating cyber security”
Open Source Security Testing Methodology Manual (OSSTMM) by Institute for Security and Open Methodologies

30. OSSTMM Purpose Test conducted thoroughly
Test included all necessary channels
Posture for test complied with laws and regulations
Results are measurable
Results are consistent and repeatable
Results contain only facts derived from tests themselves?

31. Security Test Audit Report
Serves as proof of a factual test
Holds Analyst responsible for test
Provides clear result to client
Provides comprehensive overview
Provides understandable metrics

32. Outward Support CSWG Testing & Certification Sub-group
SG Security CyberSec-Interop