Technical Review Organization, Strategy and Report Klaus Leismüller 20.01.2017 Catania

Technical Review Organization Site Manager N. Randazzo TRCoordinator K. Leismuller Marine K. Leismuller Mechanics Electronic F. Ameli Optics A. D’Amico Power R. Cocimano Add Equipment G. Riccobene T E A M - Carmelo D’Amato - A. Grmek - M. Sedita - A. Miraglia EXTERNAL - Gene Massion - C. Nicolau - F. Simeone - A. Lonardo - G. Riccobene - F. Amelli - C.A. Nicolau - A. Orlando EXTERNAL: - R. Masullo - J-W Scmelling - G Kieft - S. Pulvirenti - F. Simeone - S. Viola - C.A. Nicolau - Elio Poma Team Leaders nominate team members Choose by - Expertise (also outside the project) - Experience - Working Relationship

Capo Passero Site

Technical Review Strategy Review focus: - Junction Boxes - Subsea Cables - Inventory and Management - Project Organization - Quality Management System - Towers - Onshore Interface Phase I - July Establish teams and get all team leaders and team members approved Set primary goals for analysis Decide on format of report Phase II - November Have first review ready and merge the areas and their recommendations Start interdependence analysis Phase III – December/January Merge all findings and provide Technical Review Document to Management with strategy and recommendation for way forward

Technical Review - Outcome KM3NeT - High level observations   Overall the core areas of the project are in good shape. The technical requirements to perform the outlined task by the science community according to the Technical Design Report (TDR) are in place (REFERENCE to LOI, J. Phys. G: Nucl. Part. Phys. 43 (2016) 084001 (130pp) Letter of intent for KM3NeT 2.0 ) Good solutions have been found and are incorporated in the technical implementation of the project. The basic infrastructure is in place. Over and above the project infrastructure can provide support to additional applications for the science community. The main lesson learned is that: due to the extreme complexity of KM3NeT, challenging ambient conditions and an expected 20 year-long operation time, it is fundamental that none of the steps in the design, qualification, integration and test phases that ensure the high reliability of all detector components and of the overall system, are bypassed. The stringent time frame imposed by some funding agents and stakeholders had a strong impact regarding reliability, and overall quality control and quality assurance of the project.

Technical Review - Outcome Change of philosophy required - Project Management and Set-up   KM3NeT-IT lacks a clear and transparent coordination structure with set milestones for the various sub segments and the overall project. The project coordination must be harmonized with the KM3NeT Collaboration and with the shore/seabed infrastructure management The reviewed documentation and initiatives regarding those aspects must be strengthened. Human resources management must be optimized to achieve an effective project planning Essential management aspects of the project (e.g. in-storage management, testing plans, TDS, QA) are not addressed. The inventory management is disordered and insufficient. Traceability of components, assemblies and equipment is suboptimal. Assembly lines should be upgraded keeping industrial standards for similar activities in mind and as a reference.

Technical Review - Outcome Change of philosophy required - Qualification and Testing   The project lacks a rigid testing and qualification plan of components, sub systems and the entire system to ensure a 20-year reliability. There are no standards in place (or a set of standards used as a general guideline) to test against. With the current philosophy the goal of a 20-year lifetime detector cannot be achieved. It came to light that mechanics got the majority of attention for testing and qualification, whereas the other sub segments have been neglected or no testing was in place. In particular factory acceptance tests, sub system integration tests and final tests of the entire system are missing. Some equipment went subsea untested and some essential tests of subsystems were only performed during this TR. A rigid test (against a set standard) of the entire assembly has not been performed.

Technical Review - Outcome Change of philosophy required - Subsea Engineering   Many best practice aspects of subsea engineering have not been taken in consideration. The mechanical aspect of the design has been the driver for all other segments. A subsea project should be handled exactly the other way round. Manufacturing Engineering and System Engineering has been neglected or not been incorporated due to lack of in-house experience. The design is too complicated and complex. This is reflected in using several pressure vessels and a manifold to manage the system in the Junction Boxes. This results in an abundance of connectors and connections, as well as having more parts than necessary and different technologies. The issue of components in oil under pressure over a 20-year lifetime has not been fully assessed (in particular optical connectors internal to the manifold structure). This complexity and design philosophy in combination with the insufficient testing and qualification process cannot result in a system with 20 years lifetime.

Technical Review - Recommendations Restructure the project and adopt a similar organizational chart as for the TR. Hire an external Project Manager who has substantial knowledge in manufacturing engineering, system engineering and qualification and testing standards. Mechanics, also due to organizational changes, should have a new team leader in an adjusted role. The position should cover the gap between scientific necessities and best practice engineering, through a liaison function with industrial and external scientific partners.   A clear standard for all testing activities and a corresponding testing and qualification plan has to be established, implemented and executed A quality workshop should be held in Catania to highlight the relevance of qualification in the subsea environment to all persons working on the project. The manufacturing and assembly of basic subsea components should be outsourced to an industrial company with the relevant experience. The same applies to the electronic boards. The production of JB3 (current design) should be put on halt and a new JB design based on the new CTF philosophy should be developed by a third party. The goal of the new design must be: - Introduce ‘best practice’ in subsea engineering - Reduce complexity - Adaptation to subsea campaign requirements at 3500m - Longevity of all components and materials

Core Area – Power Rosanna Cocimano The main goals of the Power Review Verification of the status of the sea-reference connection scheme Verification of the status of the control system, monitoring, data storage and status of the calibration of the sensors - Review of the automatic switch-on procedures Review of system performances during transients under normal, over-current, and short-circuit conditions   For the last point a test bench was set up, it includes the electronics boards of two JBs, one tower-DU, one string-DU, some dummy loads, the cables, and two different power supply units (one bench power supply unit and an Alcatel MVC).

Core Area – Power Rosanna Cocimano The main actions based on the Power Review   Must: The BPS and the PCS at tower- DU base must be modified in order to fix problems arising during power on transients The fuse in the BPS and in the PCS of the tower-DU must be properly chosen The firmware of the BPS must be modified to reduce the trip time The PCS at the JB must be redesigned Current limiting must be implemented, unless the new design of the PCS at JB makes it not necessary A qualification and test process must be defined An expert must be consulted regarding the effects of current loops in seawater (this has an impact on mechanics)

Core Area – Power Rosanna Cocimano Actions continued Should: - Current sensing on both sides (positive and negative) of the 375V rails - Modify the firmware of the BPS and the relative on-shore control software to solve the issue on the 12V current sensor at short circuit - The firmware of the BPS and PCS should be modified to allow the switching of the positive and negative rails independently, the relative on-shore control software must implement this possibility - Have a tool to easily follow from on-shore the status of the off-shore system in case of operations in emergency mode should be implemented - A permanent test bench should be available to test the power network behaviour onshore

Core Area – Optics/Electronics Antonio D’Amico, Fabrizio Ameli The main results of the Optics/Electronics Review Non-nominal optical levels in JB1 and JB2 after deployment. It has been shown that the JB1 and JB2 systems are optically not stable. Regarding the non-nominal optical levels measured in the Tower system, it was not possible to perform an in-depth investigation of the root causes, since the occurred faults were fatal for the operation of the Tower.   A point of concern is the current architecture for communication and control of the amplifiers (EDFA) in JB2 and JB3. The recommendation is to adopt a simplified architecture (based on the JBE). Further point of concern was the control of optical connector cleanliness in manufacturing and in production. A suitable probe tip has been found and has been recommended to all production sites and the reference guidelines for inspection and cleaning of optical connectors prepared. EDFAs heat dissipation and their role in the JB1 control loop. There is no thermal conductive path between the EDFAs and the surrounding metal vessel, poor heat dissipation strongly worsens the EDFA reliability and since EDFAs are a mission critical component their heat sink must be thermally coupled to the metal mass of the JOP, preferably through a metal frame supporting the four devices.

Core Area – Optics/Electronics Antonio D’Amico, Fabrizio Ameli The main actions based on the Optics/Electronics Review Must: - String JB, EDFA’s automatic boot procedure: reconfigurable from shore - JBs and Towers, optical connectors: capability to inspect each optical pin with microscope - JB, optical splices under high pressure: use recoating - Tower JB, heat dissipation of the EDFAs: to be strongly improved - JBs and Towers, non-nominal losses in the optical path inside JBs: identify and correct the root cause of excess loss - Tower JB, EDFAs out of JB control loop: implement the same configuration of String JB - Connection of Seacon optical connectors: introduce a clear procedure for connection, cleaning and maintenance (including the test connectors)

Core Area – Optics/Electronics Antonio D’Amico, Fabrizio Ameli Actions continued Should: - String JB, communication with the EDFAs: transparent from shore - String JB, EDFA’s power supply: minimize the voltage drop - Both JBs and Tower base, optical components: optimize fibre routing

Core Area – Marine Operations Klaus Leismüller The main results of the Marine Review Suboptimal communication flow and decision making process (in particular input to Marine Operations Manual and standardizing the process) Missing equipment recovery strategy Lack of Technical Data Sheets (TDS), equipment on/off board procedures, in storage maintenance plans etc.

Core Area – Marine Operations Klaus Leismüller The main actions based on the Marine Review - Improve pro active communication and exchange of feedback - Review design of all structures with respect to recoverability - Introduce a in storage maintenance plan for equipment in Malta - Lessons learned meetings at LNS after sea campaign - Apply ‘best practice’ for subsea engineering and subsea design - Prepare a proper TDS (equipment documentation) for every assembly to be deployed

Core Area – Mechanics Mario Sedita /Andrea Miraglia (Documentation) The main goals of the Mechanics Review - Inventory of equipment - Analysis of qualification process (relevant equipment) - Definition of an in storage maintenance plan - Analysis of equipment that can be used for Phase II - Analysis of equipment components that failed - Realize JB3 Test Bench/Prototype - Analysis of current design - External

Core Area – Mechanics ext. Gene Massion The main conclusion based on the Mechanics Review The most significant risk to the project identified during the review can be summarized as the lack of a high reliability engineering methodology in the development of KM3NeT consistent with the stated 20 year mission of the telescope. Specifically, the following 3 critical elements required for a high reliability engineering approach are missing from the project. - System Engineering - Manufacturing Engineering - a rigorous, systematic verification and validation process

Core Area – Additional Equipment Giorgio Riccobene The main conclusions based on the AE Review - Additional equipment for the KM3NeT-IT infrastructure consists of three main systems: - time calibration system - positioning system - water column properties monitoring system. - The correct functioning of the three systems is necessary to exploit the full capability of the submarine infrastructure composed by Junction Boxes and “Tower” detection units to work as a neutrino telescope; although none of these subsystem is critical -in terms of electronic and mechanical robustness of the infrastructure and detector- their failure significantly reduces the scientific capability of the neutrino telescope. - Criticalities for the three systems have been identified: partial failure and data quality problems of some components (hydrophones, compass and tilt boards); lack of qualification of almost all components and sub-assemblies; lack of documentation and procedures. The electronics interfaces from the detector electronics to the auxiliary instrumentation is designed to protect the detector against short circuits and over-currents; a failure in any of the auxiliary sensors will not induce any damage in other parts of the detector.

Core Area – Additional Equipment Giorgio Riccobene The main actions based on the AE Review Must: - Fides analysis on Laser Beacon and LED beacons is needed (on-going) - Laser Beacon must be qualified - the DAQ system hardware (TerAsic board installed on DAQ PC) and software to properly read-out the PMT data during time calibration is not present and must be built - OM with nanobeacon already integrated to be tested before integration - Define upgraded version of hydrophone mechanics - Qualify the electronics and the hydrophone assembly in compliance with ISO 13628-6 standard - Qualify cable/connector interface of the hydrophone and LBL beacon and, in case of negative results, change them - Test all hydrophones in the (final) upgraded version - Ask for Fides analysis of LBL acoustic beacon electronics - Qualify the LBL Acoustic beacon in compliance with ISO 13628-6 standard - Run Fides analysis on AHRS and failure mode analysis for the assembly with FCM - Upgrade and test AHRS firmware - Define and apply calibration procedure for AHRSs

Core Area – Additional Equipment Giorgio Riccobene Actions continued Must: - Test and recalibrate all sensors before deployment since instruments were acquired more than 2 years ago and stored in not proper conditions - Check cable/connector interfaces and change in case of problems - Save oceanographic properties data on the relational database Should: - Final document clearly describing the procedures for time calibration on-shore and off- shore has to be written - Database interfaces have to be set-up to save time calibration coefficients - Redefine laser Beacon position: from one tower to the Junction Box - Define proper tables to save calibration parameters in the relational Database - Set up proper storage rooms or racks if deployment happens several months later than re- calibration

Annex A

Annex B