1. Monitoring the Long-Term Effectiveness of Integrated Safety Management System (ISMS) Implementation Through Use of a Performance Dash Board Process Mike D. Kinney, CPF, CSP National Security Technologies, LLC ISM Best Practices Workshop September 12, 2006

2. Getting Started Good news! Opportunity to win free “stuff ” [OK, OK, beads] After receiving initial approval of their ISMS programs, some contractors have lacked an effective method to monitor long-term effectiveness of their ISMS program DEAR Clause 48 CFR 970.5223-1, “Integration of Environment, Safety and Health into Work Planning and Execution”, requires the overall integrity of the contractor ISMS program to be maintained

3. Background DOE has issued guidance to support annual contractor reviews DOE G 450.4-1B, “Integrated Safety Management System Guide for use with Safety Management System Policies (DOE P 450.4, DOE P 450.5, and DOE 450.6); The Functions, Responsibilities and Authorities Manual; and the Department of Energy Acquisition Regulation” Contains Continuing Core Expectations (CCEs) to guide ISMS long-term effectiveness reviews

4. Background The Nevada Site Office (NSO) issued a local Directive addressing long-term ISMS maintenance NV O 450.4, “Safety Management System Maintenance” In January of 2006, The Defense Nuclear Facilities Safety Board (DNFSB) has issued a technical report that examines current implementation of ISMS at DP sites DNFSB/TECH-36, “Integrated Safety Management: The Foundation For a Successful Safety Culture”

5. Challenges In some instances, DOE/NNSA contractors do not have a consistent method to effectively review long-term implementation of their ISMS program Some contractors attempt to rely on traditional inspections or assessments in lieu of a more programmatic approach Results don’t always accurately reflect actual level of implementation

6. Challenges (continued) Comparison of performance with previous evaluations (e.g., trending) can also prove difficult This series of collective challenges do not support being able to demonstrate to DOE/NNSA/DNFSB that ISMS is being effectively maintained Lack of a clear method to monitor long-term implementation of ISMS can also lead to reduced buy-in at the task level as well as lack of commitment from senior management

7. Method To assist with meeting Headquarters and Local Site Office commitments regarding ISMS long-term maintenance, NSO established an Integrated Safety Management Council (ISMC): Includes Contractors, User Organizations, and Federal staff Co-chaired by NSO and User Organization representatives Monthly meetings are utilized to address new initiatives or programmatic challenges Coordinates performance of annual ISMS reviews

8. Method To assist with the ISMS annual review, the NSO ISMC established a performance dash board process: Utilizes Continuing Core Expectations (CCE) contained in DOE G 450.4-1B Performance dash board also contains individual evaluation criteria for each CCE Completed performance dash board identifies current implementation status for each participating organization

9. CCECategoryCCECategory 1Annual ISMS updates7Review and update of List A/List B 2ISMS effectiveness (POCs)8Contractor and DOE assessments 3ISMS implementation at the work level9Improving ISMS 4 Roles and responsibilities, management responsibility for safety 10Approval of work by DOE 5Balancing of priorities11 DOE review of hazard analysis and feedback processes 6Feedback and improvement processes DOE Continuing Core Expectations (CCEs)

10. ISMS/CCE Performance Dash Board Appropriate worker involvement is a priority. Work is accomplished within controls. Work is properly authorized. Controls are developed and implemented. Work is defined, hazards are identified, and actions to prevent or eliminate the hazards are taken. DOEContractor C Contractor B Contractor A CCE -3: Work activities reflect effective implementation of the functions of ISMS.

11. Method NSO ISMC Performance Dash Board: Contains color gradients, passed on the following rankings: Blue: Significantly exceeds expectations Green: Meets expectations Yellow: Expectations not fully implemented Red: Limited implementation of expectations, management attention required

12. ISMS/CCE Performance Dash Board Appropriate worker involvement is a priority. Work is accomplished within controls. Work is properly authorized. Controls are developed and implemented. Work is defined, hazards are identified, and actions to prevent or eliminate the hazards are taken. DOEContractor C Contractor B Contractor A CCE -3: Work activities reflect effective implementation of the functions of ISMS.

13. ISMS/CCE Performance Dash Board Analysis Contractors A & C may chose to examine other contractors processes for authorization of work and worker involvement Contractor B may be able to assist other contractors regarding methods to enhance identification and control of hazards

14. Method In addition to identifying implementation for CCE criteria, implementation to the CCE title level is also identified To assist with tracking trending and support evaluation results, performance is compared with pervious years Five year base line Performance “arrows” utilized to identity annual performance for given color gradient/CCE title Annual results, including potential site wide challenges, are identified in collaborative manner with ISMC members

15. ISMS/CCE Performance Dash Board Analysis System Performance Indicator Key: Improving ▲; No Change ◄►; Declining ▼ Discussion: Processes for identification of hazards and controls (including selection of standards) have begun to improve. Y, ▲ Y, ▼ FY06FY05FY04FY03FY02 Y, ◄►

16. ISMS/CCE Performance Dash Board Analysis Implementation of the CCE title level also supports identification of programmatic challenges Conveys management level information in a timely manner while readily focusing attention of potential challenges

17. Conclusions Use of a DOE ISMS CCE Performance Dash Board has proven to provide the following benefits: Enhanced visibility of ISMS implementing processes Open communication of potential challenges as well as successes Ability to track and trend performance Enhanced participation throughout Contractor, User Organizations, and Federal personnel

18. Conclusions (continued) Other benefits from application of this process include: Continued support from ISMC participating organizations Value added method to address contractual requirements Ongoing support from management Perhaps most importantly, use of a DOE ISMS CCE Performance Dash Board is ensuring that the importance of ISMS implementation remains on the forefront versus being viewed as just another regulatory requirement