Electrical Fire in Terminal Room at Collider-Accelerator Department Relay-Fire Investigation Committee

Electrical Fire in Terminal Room  On November 9, 2010, in building 911, an electrical fire in a utility room containing stacks of electromagnetic relays near MCR  Activated the fire alarm system and a Halon fire suppression system  About 50 relays and associated wiring were damaged by fire; relays and wiring were associated with the AGS access controls system (ACS)  A safe configuration of the AGS ACS was developed and is in place for the FY11 running period, and the configuration includes features that reduce the risk of fire  There were no personnel injuries  The investigation revealed an ignition source was proximate to rapidly burning materials 2

3

Fire Damage to 1950s First Floor Terminal Room, November 2010  The root cause of this fire was the failure to identify the presence of non-flame retardant insulation on wiring proximate to CR120 relays that were known to overheat  Multi-conductor wiring installed in the 1950s had an outer sheath that was not flame retardant and inner insulation on individual wires that was found to be flammable in recent tests  Several circuit breakers (CB) powering relays tripped open preventing further damage; one breaker (CB19) did not trip although it should have  The 1978 recall of CR120 flammable contact-arm retainer was not known to Department employees 4

5

6

7

8

Fire Damage to 1970s Second Floor Terminal Room in 2001  The root cause of the 2001 fire was the failure to identify the presence of non- flame retardant covers proximate to CR120 relays that were known to overheat  Plexiglas covers were installed for electrical safety  No significant damage from this fire  Records show about 30 relay coils overheated between 1990 and

Main Causal Factors  Surveillance and Inspection of Relays LTA  After the 2001 fire, the thermal scanning practice adopted for relay temperature monitoring was inadequate, inconsistent and poorly documented  Tracking and Trending of Low Level Issues from Critiques LTA  The written guidance for Critiques is in a C-AD procedure and it does not require trending of specific problems or problem classifications over many years  Inaccurate Risk Perception  Managers, engineers and workers accepted relay-coil overheating failures and this was a latent organization weakness  Coil overheating failures are failsafe for radiation safety but they created an ignition source  Past overheating failures did not result in personnel injury, programmatic loss or in wiring fires  Recognition of Non-Flame Retardant Insulation on Wiring Was Inadequate  Obvious flammables such as Plexiglas covers were the focus of the flammable hazards reviews for Terminal Rooms, not the adjacent wiring that had insulation dating back to the 1950s 10

Other Causal Factors  3K105 Relay Coil Overheated and Initiated Fire  Response to Legacy Fire Barrier Deficiencies LTA  Unaware of 1978 Recall on GE Flammable Contact Arms  Previous Critique Corrective Action Closure Statement LTA  Critique Corrective Action Implementation LTA  Unused Cables in Terminal Room Were Not Removed  Circuit Breaker Inspection, Testing And Maintenance Were LTA 11

Corrective Actions  Remove re-called plastic contact-arm retainers from CR120 relays at C-AD and replace with non- flammable retainers on active relays or replace active relays with new relays  Target Completion Date: 10/01/2011  Disseminate GE 120V Relay Contact Arm Retainer recall notice lab-wide  Actual Completion Date: 11/24/

Corrective Actions  Seal penetrations between first floor and second floor Terminal Rooms  Target Completion Date: 10/01/2011  Install new relays and fuses on relay coils for AGS Access Control System relays in Building 911 first and second floor Terminal Rooms  Actual Completion Date: 12/20/

Corrective Actions  Examine Linac ACS Relay cabinet 4880 for unneeded penetrations and seal  Actual Completion Date: 12/20/2010  Develop a C-AD OPM procedure on critiques and include instructions for the Action Owners of Critique Corrective Actions, instructions for the QA Group on evaluating the adequacy of closure statements for critiques (e.g., provide objective evidence for closure statements), and require Critique Corrective Action Closure verification using the C-AD assurance programs  Target Completion Date: 04/01/

Corrective Actions  Add thermal indicating tape to Linac ACS CR120 relays, and add thermal indicating tape to AGS ACS CR120 relays  Actual Completion Date: 12/20/2010  Develop an improved OPM procedure for CR120 relays for SIM; include thermal scanning (e.g., include distance of measurement, timing of scans, thermal tape readings, acceptance criteria and the maintenance of records), trending, trending criteria, visual inspection, and maintenance  Target Completion Date: 04/01/

Corrective Actions  Review the available failure history and maintenance records for similar failures on other relays at the C-AD for access control systems, and write a report recommending corrective actions  Target Completion Date: 10/01/2011  Establish a project with milestones to remove unused cable and all flammable wiring proximate to ignition sources in Building 911 first and second floor Terminal Rooms  Target Completion Date: 07/01/

Corrective Actions  Establish a program to control and minimize combustible loading in Building 911 first and second floor Terminal Rooms  Target Completion Date: 04/01/2011  Establish a Laboratory Electrical Safety Committee approved program at C-AD for inspection, testing and maintenance for circuit breakers  Target Completion Date: 01/01/2012: 17

Corrective Actions  Complete the replacement of CR120 ACS relays with the PLC replacement program by FY13 for AGS, and by FY14 for Linac  Target Completion Date: 09/30/2014  Submit a Lessons Learned on pre-1970s wiring and potential for flammability to the BNL and DOE Lesson Learned system  Target Completion Date: 04/01/

Corrective Actions  Add questions to the C-AD Design Review Questionnaire that address the flame retardant characteristics of wiring when making modifications to the accelerator or experiments. Add questions that address use of relays as potential ignition sources  Target Completion Date: 04/01/2011  Evaluate non-flame retardant wiring in Building 911 relay rooms and at the Target Desk in Building 912, determine their proximity to ignition sources and replace wiring or disposition ignition sources where necessary  Target Completion Date: 10/01/

Corrective Actions  Establish a site wide program via the BNL Fire Protection Strategic Plan to identify and assess fire risk of non-flame retardant wiring in pre-1970 facilities or in facilities constructed with excess wiring from older construction projects  Target Completion Date: 09/01/2011  Establish a C-AD Legacy Hazard Committee and review all facilities at C-AD; use an established hazard identification tool and add non-flame retardant wiring and overheating from possible ignition sources such as relay coils  Target Completion Date: 01/01/

Lessons Learned  BNL should implement a site-wide process to receive and disseminate recall notices  Multi-conductor wires of this age (pre-1970s) in relay stacks must be considered as flammable fuel loading near an ignition source  Improved surveillance, inspections and maintenance will reduce the frequency of relay coil overheating  Converting to a PLC based system or replacing this relay with a newer version will reduce the frequency of relay coil overheating  Eliminating non-flame retardant wiring will reduce the severity of fires 21