1. Conformity Assessment: activities & systems
Lisa Carnahan
NIST Standards Coordination Office
Standards Services

2. Topic Map Background Actors, activities and relationships
Definition of Conformity Assessment
Conformity Assessment: Needs and Confidence
Discussion of conformance confidence and its relationship to risk, and cost
General factors to consider in designing a conformity assessment system
Actors, activities and relationships
Actors in conformity assessment
Activities in conformity assessment
Example models for conformity assessment

3. Conformity Assessment
“demonstration that specified requirements relating to a product, process, system, person or body are fulfilled”
ISO/IEC 17000

4. So you want confidence that your purchased product or service conforms…how much confidence?
The need for conformity assessment is primarily driven by risk
The perception of risk associated w/ non-conformity drives the need for regulatory and market confidence
A successful CA system provides that amount of confidence at minimal cost

5. Factors in CA System Design
The risks associated with non-compliance should be proportional to the rigor and independence of the CA system.
System over-design will add too much cost.
System under-design will result in too little confidence of compliance.
Penalties associated with non-compliance may reduce the needed rigor and independence of the conformity assessment system.
Timely mechanisms that effectively remove non-compliant products from the market may also reduce the needed rigor and independence of the system.

6. Risk and Conformity Assessment-- How Much Confidence is Needed?
Perceived Risk
Independence and Rigor of Conformity Assessment
Supplier’s Declaration
1st party conformity assessment
Certification
3rd party conformity assessment

7. Relationship of CA Types and Confidence
Supplier Declaration of Conformity
Listed Products
Certification
Certification Body
Accredited to
ISO Guide 65
Time, $$, Resources
Confidence
Qualified Products List based on ISO Guide 65 Certification
Qualified Products List based on ISO Guide 65 Certification
List of Declared Products
Testing by Accredited Labs
Testing by Accredited Labs
Testing by Accredited Labs
Testing by Accredited Labs
Supplier Declaration
of Conformity
Supplier Declaration
of Conformity
Supplier Declaration
of Conformity

8. Conformity Assesment Actors
Consumers
Manufacturers (resellers, integrators, etc.)
Accreditation Bodies (ABs)
Accreditation bodies for testing laboratories
Accreditation bodies for certification bodies
Testing Laboratories
Certification Bodies
Scheme owner
ISO 9000 Registrars
Inspection Bodies
Definition: certification scheme owner: person or organization that is responsible for developing and maintaining a specific certification scheme (3.2) NOTE The certification scheme owner can be the certification body itself, a governmental authority, trade association, group of certification bodies or other. ISO/IEC CD 17067

9. The Parties – Who Done it?
Conformity Assessment can be conducted by:
first party – seller or manufacturer
second party – purchaser or user
third party – an independent entity that has no interest in transactions between the 1st and 2nd parties

10. Components of Conformity Assessment
Testing
Supplier’s Declaration of Conformity
Certification
Accreditation
Surveillance

11. Testing
Use
When critical characteristics can be evaluated via measurement under specified conditions
Activities
Testing
Who does it
1st, 2nd or 3rd parties
Relationship to other components of CA
Test report may be used for evidence of conformance in supplier’s declaration
Test report may be used for evidence of conformance in a certification system
Test report may be used in surveillance
Related Standards
ISO/IEC (testing laboratories)

12. Supplier’s Declaration of Conformity
Use
Risk associated with nonconformity is low
Adequate penalities (consequences) exist for placing nonconformant product in the market
Adequate mechanisms for removing nonnconformant product
Activities
May use testing
May use quality system approach
Supplier attests to conformity
Who does it
1st party
Relationship to other components of CA
May use test report as evidence of conformity
Related Standards
ISO/IEC Parts 1 and 2

13. Certification
Use
Risks associated with non-conformity are moderate to high
Activities
Evaluation of evidence of conformity
Compliance decision
Attestation of conformity
Surveillance
Who does it
Conducted by only a 3rd party
Relationship to other components of CA
Certifer may be accredited
Test report used as evidence of conformance
May require accredited testing laboratories
Related Standards
ISO/IEC Guide 65 (certification bodies)

14. Accreditation
Use
Higher confidence for conformity assessment bodies (testing or certification)
Activities
Evaluation of competence to perform testing or certification activities within scope
Evaluation of conformity to management & technical requirements
Attestation of conformity and comptence
Surveillance of conformity assessment bodies
Who does it
3rd party
Relationship to other components of CA
May be required by scheme owner for testing and/or certification bodies
May be required by certification body for testing laboratories
May be required by regulator for testing and/or certification bodies
Related Standards
ISO/IEC 17011

15. Surveillance Use To enhance confidence in ongoing conformity
The frequency and rigor should be balanced with the cost and confidence needs. (This is typically resource intensive.)
Activities
May be performed through inspection
May be performed through testing
May be performed through audit
May be performed pre-market or post-market
These activities may be announced or unannouced
These activities may be done in conjunction with each other
Who does it
3rd party
Relationship to other components of CA
This is a key part of a certification program or a registration system (e.g., ISO 9000 series).
Related Standards
Required in ISO/IEC 17011
Required in ISO/IEC Guide 65

16. Conformity Assessment - ISO Guides and Standards1 2 3 4 A B C D

17. Conformity Assessment Hierarchy
Who Watches the Watchers?
Scheme owner
sets overall requirements of the CA system.
Accreditor(s)
Certifier(s)/Inspection Body(ies)/Laboratory(ies)
Manufacturers

18. Supplier’s Declaration Example - IPV6 Conformity Assessment
IPV6 Technical Specifications
Equipment
Accredited
IPV6
Testing Labs
Procurement
Agency
IPV6
Vendor $
SDoC *
Results $ + $
Lab Accreditor
* Supplier’s Declaration of Conformity per ISO/IEC parts 1 and 2
+ Assessment and accreditation

19. Accredited Testing Lab examples: NIST Cryptographic Module Validation Program

20. Third Party Testing & Certification Example: HHS EHR Certification Program
ONC-ACB Authorized Certification Body*
NVLAP-Accredited Test Lab*
ANSI (as an AB) Authorized Accreditor
ONC
accredits
performs testing against Criteria
certifies tested products
approves
NIST NVLAP National Voluntary Laboratory Accreditation Program
accredits
authorizes
ONC reviews and posts certified product to CHPL
Authorized Testing Body*
ACB Authorized Certification Body*
Authorized Testing Body*
ACB Authorized Certification Body*
Self developer/
Vendor
Product successfully passes testing
Product successfully achieves certification
*ONC-ACB and NVLAP Accredited testing bodies may be part of the same organization provided a firewall exists between the testing and certification operations
Source: Carol Bean, HHS EHR Certification Director, NVLAP Health IT Program Workshop

21. is specified in the rules for the particular type of device
Mullti-model Approach Example: FCC Participation Mutual Recognition Agreement Equipment Authorization Program
Maximum
Certification
(Approved by FCC
or TCB)
DoC
(Self-approval using
an accredited testing lab)
The type of approval
is specified in the rules
for the particular
type of device
SDoC
(Self-approval
Database by ACTA)
Telecommunications Certification Body (TCB) = accredited third-party
certification body
Verification
(Self-approval)
Minimum
Source: William Hurst, P.E. Federal Communications Commission
Office of Engineering and Technology
Laboratory Division

22. Equipment Authorization Types
The FCC Lab no longer certifies this equipment. However, this equipment may be certified by an accredited third-party certification body (TCB).
For several products the manufacturer is given the option to use either DoC or Certification.
Source: William Hurst, P.E. Federal Communications Commission
Office of Engineering and Technology
Laboratory Division

23. NIST Conformity Assessment Guidance for Agencies
reducing overlap and duplication and increasing efficiency
working with private sector
NTTAA directs NIST to coordinate Federal agencies in:
Federal agencies maintain their authority and responsibility to make regulatory, procurement and federal assistance decisions
NTTAA does not indicate a preference for any specific approach in conformity assessment
NIST advises Federal agencies on development of appropriate conformity assessment systems including the use of international CA standards
In conjunction with OMB deliberation on supplementary guidance to the Circular, NIST has begin the process of updating the guidance we issued in 2000 to agencies on conformity assessment.
NTTAA focus is on reducing overlap and duplication and increasing efficiency – within the government and between govt and private sector

24. NIST Recommends a Risk-Based Approach to Conformity Assessment System Design
Consider risks associated with non-compliance when determining the necessary rigor of a system
Over-design can be costly; may delay products to market
Under-design reduces confidence; may prevent market acceptance of the product
Marketplace consequences, regulatory penalties and effective recall processes may be considered in determining needed level of rigor in conformity assessment systems

25. FedRAMP Program Built on International Standards
Cloud Service Providers
Third Body Assessment Organization
(Inspection Body/ies)
Accreditor(s)
ISO/IEC 17011
+ technical requirements
FedRAMP PMO
Oversight &
Communication
ISO/IEC 17020
+ FedRAMP competency requirements
FedRAMP
requirements for Provisional Authorization
ISO/IEC 17011; Conformity assessment -- General requirements for accreditation bodies accrediting conformity assessment bodies
ISO/IEC 17020; General criteria for the operation of various types of bodies performing inspection

26. What is FedRAMP?
FedRAMP is a government-wide program that provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services.
This approach uses a “do once, use many times” framework that will save cost, time, and staff required to conduct redundant agency security assessments.

27. Policy on Security Authorization of Information Systems in Cloud Computing Environments
December 8, 2011 OMB Policy Memo The Office of Citizen Services and Innovative Technology (OCSIT), within the General Services Administration (GSA), is responsible for managing FedRAMP, to provide a unified and government-wide risk management framework that addresses these problems.

28. FedRAMP’s Purpose Problem:
A duplicative, inconsistent, time consuming, costly, and inefficient cloud security risk management approach with little incentive to leverage existing Authorizations to Operate (ATOs) among agencies.
Solution: FedRAMP
Uniform risk management approach
Standard set of approved, minimum security controls (FISMA Low and Moderate Impact)
Consistent assessment process
Provisional Authorizatoin

29. FedRAMP Executive Sponsors

30. NIST Role
NIST Cloud Computing Program: build a U.S. Government Cloud Computing Roadmap
Technical Advisor on FedRAMP
Collaborated with Federal CIO Council Security Working Group to develop FedRAMP concept
Collaborate with GSA to develop and implement a formal conformity assessment program
consistent independent, third-party assessments of security controls implemented by Cloud Service Providers
Technical Experts regarding FISMA compliance
Special Publications (SP) and
Federal Information Processing Standards (FIPS) 199 and 200
Advise Joint Authorization Board on compliance requirements

31. FedRAMP Goals The goals of FedRAMP are to:
Accelerate the adoption of cloud solutions through reuse of assessments and authorizations
Increase confidence in security of cloud solutions
Achieve consistent security authorizations using a baseline set of agreed upon standards and accredited independent third party assessment organizations
Ensure consistent application of existing security practices Increase confidence in security assessments
Increase confidence in security assessments
Increase automation and near real-time data for continuous monitoring

32. FedRAMP Stakeholder Roles and Interaction

33. FedRAMP and the Security Assessment and Authorization Process
Maintains Security Baseline including Controls & Continuous Monitoring Requirements
Maintains Assessment Criteria
Maintains Active Inventory of Approved Systems
Ongoing A&A
(Continuous Monitoring)
Continuous Review of Risk
Oversight of the Cloud Service Provider’s ongoing assessment and authorization activities with a focus on automation and near real time data feeds. 3
Provisional Authorization
Grant Provisional Authorization
Joint Authorization Board reviews assessment packages and grants provisional authorizations
Agencies issue ATOs using a risk-based framework 2
Assessment
Independent Assessment
Before granting a provisional authorizations, Cloud Service Provider systems must be assessed by an approved, Independent Third Party Assessment Organization 1
Ongoing A&A Activities Will Be Coordinated Through:
DHS – CyberScope Data Feeds
DHS – US CERT Incident Response and Threat Notifications
FedRAMP PMO – POA&Ms
Authorizations:
Provisional ATO - Joint Authorization Board
ATO – Individual Agencies
Independent Assessors to be retained from FedRAMP approved list of 3PAOs
Consistency and Quality
Trustworthy & Re-useable
Near Real -Time Assurance

34. Benefits of leveraging a formal 3PAO approval process:
FedRAMP Third Party Assessment Organization (3PAO) Conformity Assessment Process
FedRAMP requires CSPs to use Third Party Assessment Organizations (3PAOs) to independently validate and verify that they meet FedRAMP security requirements.
FedRAMP worked with NIST to develop a conformity assessment process to qualify 3PAOs.
This conformity assessment process will qualify 3PAOs according to two requirements:
Independence and quality management in accordance with ISO standards; and
Technical competence through FISMA knowledge testing.
Benefits of leveraging a formal 3PAO approval process:
Creates consistency in performing security assessments among 3PAOs in accordance with FISMA and NIST standards
Ensures 3PAO independence from Cloud Service Providers in accordance with international standards
Establishes an approved list of 3PAOs for CSPs and agencies to choose when satisfying FedRAMP requirements.
NIST

35. Overview of 3PAO Role
Performs Initial and Periodic Assessments of CSP Security and Privacy Controls
Independent, Cannot Help CSP Prepare Documents!
Reviews CSP Documents for Accuracy
Develops Security Assessment Plan (SAP)
Conducts Security Testing
Use Test Case Workbooks
Manual Tests
Automated Tests
Develops Security Assessment Report (SAR)

36. FedRAMP Phases and Timeline
A phased evolution towards sustainable operations allows for the management of risks, capture of lessons learned, and incremental rollout of capabilities
FY12
FY13 Q2
FY14
Pre-Launch Activities
Initial Operational Capabilities (IOC)
Full Operations
Sustaining Operations
FedRAMP Finalizes Requirements and Documentation in Preparation of Launch
Launch IOC with Limited Scope and Cloud Service Provider (CSP)s
Execute Full Operational Capabilities with Manual Processes
Move to Full Implementation with On-Demand Scalability
Key Activities
Publish FedRAMP Requirements (Security Controls, Templates, Guidance)
Publish FedRAMP Compliance Guidance for Agencies
Accredit 3PAOs
Establish Priority Queue
Authorize CSPs
Update CONOPS, Continuous Monitoring Requirements and CSP Guidance
Conduct Assessments & Authorizations
Identify Scale Operations to Authorize More CSPs
Implement Electronic Authorization Repository
Scale to Steady State Operations
Outcomes
Initial List of Accredited 3PAOs
Launch FedRAMP in to Initial Operating Capabilities
Initial CSP Authorizations
Established Performance Benchmark
Multiple CSP Authorizations
Define Business Model
Measure Benchmarks
Authorizations Scale by Demand
Implement Business Model
Self-Sustaining Funding Model Covering Operations
Privatized Accreditation Board
FY12
Gather Feedback and Incorporate Lessons Learned
Matt starts here
IOC Launch: June 6, 2012

37. Questions & Discussion
Lisa Carnahan NIST Standards Coordination Office Standards Services

38. Additional information