1. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 1 ECAL Detector Control System G. Dissertori ECAL Cooling Review, Nov 4, 2004 Status report Trigger/DAQ project  DCS CMS DCSECAL DCS

2. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 2 Outline u Overview u Precision Temperature Monitoring (PTM) u ECAL Safety System (ESS) u PVSS Monitoring (and related issues) u Note : For details on the PTM and ESS hardware (sensors, readout) I refer to the DCS ESR, May 04!

3. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 3 Overview ECAL module crystal PCBs for VFE and FE cooling LV, HV Laser system Offline, DAQ Data Bases Data Bases PVSS prec. temp. sensor Monitoring ESS temp. sensor + WLD Monitoring Safety, PLC interlock cooling, faults humidity sensor

4. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 4 PTM Precision Temperature Monitoring - PTM S. Zelepoukine, IHEP Protvino and ETHZ

5. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 5 PTM n Requirements: u very high precision temp. monitoring : check stability of water cooling system, to stay within 0.02 - 0.05 deg C. l no hardwired feedback to cooling u relative precision : 0.01 deg C u one sensor/module on thermal screen and grid u main water IN, main water OUT

6. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 6 Patch panel Patch panel Thermal screen Grid Sensor probes location inside SM: Probes of type 1 (N = 4 + 4)Probes of type 2 (N = 2) Number of probes (per SM): 4 + 4 + 2 = 10 Each probe has its own signal cable – twisted pair M1M2M3M4 PTM : Sensor location

7. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 7 PTM : Sensor location (SM0) M3 probe M3 cable from M3 probe cables from M1 and M2 enter M3 via different windows Patch panel location

8. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 8 55 mm St.steel; outer / inner dia = 4 / 3 mm by F.Mossiere PTM : immersion probes

9. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 9 Location of immersion probes Turned out during electronics integration for SM10: Location/mounting of these probes is critical, in the sense that operations on the PP can lead to damaging them. Alternatives?

10. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 10 SM – top view PTM immersion probes Cooling manifolds Current probe location is proven for probe to be easily damaged while people are working at the patch panel… PP Better location would be: Now… Location of immersion probes

11. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 11 PTM/HM: intermediate connection SM – top view PP SM10 in H4: 2 sensors from 14 (1 PTM and 1 HM ) are lost due to the poor intermediate connection quality… SM production (from now on): All PTM/HM sensors are to be installed in B27 without an intermediate connection – sensor cables are directly connected to the PP connector. SM production (from now on): All PTM/HM sensors are to be installed in B27 without an intermediate connection – sensor cables are directly connected to the PP connector. Intermediate Connections

12. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 12 PTM : Readout - Layout ELMB Cur.src PC / PVSS ECAL UXC55 balcony: Counting room 20x2 STP cables CANbus cables ~ 500 sensors (NTC thermistor Betatherm; 100K @ 25C; indiv.calibrated; better than 0.01C rel.accuracy) Readout electronics: ELMB based; ELMB reads voltages and converts to digital data, then sends data via CANbus. DC-DC converters DC Power supplies DC Power supplies Monitoring application: PVSS based; data storage/archiving, visualization/trending, warnings/alarms. Network access Galvanic isolation

13. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 13 Recent Results n First results from SM10 at H4: u Sensors, sensor probes, read-out practically final u Reading 4 sensors on grid, 3 sensors on thermal screen, and Water IN/OUT l One sensor lost, most likely due to bad intermediate connection inside SM before PP u Since about one week : installed 2 sensors on backplate l Above M1 and M4

14. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 14 Cooling Stability Zoom

15. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 15 M4 M3, M2 M1 Grid Thermal screen Water - IN 0.01C M1 M4, M3 Cooling stability = 0.01C (incl. day/night oscillations) Zoom :

16. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 16 Just to show uncompressed (in time) data… 10 min Grid Thermal screen Water - IN Ambient Div = 1.0C Div = 0.01C

17. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 17 M2 M1, M3 M4 M1 M2 M3 M4 28.5 30.6 28.2 25.7 M1 M2 M3 M4 28.5 30.6 28.2 25.7 Inside the electronics compartment From ESS, see later…

18. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 18 Grid side Water – out (blue) Th.screen side Water – in (red) Grid side (zoom) Th.screen side (zoom) Water – in (black) Ambient 4x sensors are located on the SM10 external surfaces: On Backplate (Grid side): M4, M1 Thermal screen side: M4, M1 4x sensors are located on the SM10 external surfaces: On Backplate (Grid side): M4, M1 Thermal screen side: M4, M1 Th.screen side – roughly equal to ambient (18.2C); Grid side – 23.4C (and much heat is dissipated by the massive metallic support!). Th.screen side – roughly equal to ambient (18.2C); Grid side – 23.4C (and much heat is dissipated by the massive metallic support!).

19. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 19 Just to show uncompressed (in time) data… Grid side Water – out (blue) Water – in (red) Th.screen side Grid side (metallic surface) Th.screen side (plastic surface) Water – in (black) ambient Div = 1.0C Div = 0.2C

20. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 20 18.01 C 23.4 C 18.01 C Air conditioner ~18 C SM10 in H4 18.2 C heat ~30 C Heat flow out of ECAL?

21. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 21 heat ? ? ? ? Heat flow out of ECAL?

22. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 22 Conclusions on PTM n Cooling stability: u Achieved 0.01 C in the long-term, including day/night variations u Have a monitoring application, Serguei is working on a more “final” version… n Temperatures on outer surfaces u Recent measurements performed by Serguei u Is heating towards outside (or inside, depending on orientation) an issue? For other detectors? For other SMs ? u As next step, Serguei will move external sensors to lateral surfaces… n Hardware u Resolve issue of immersion probes u Intermediate connections will not be there any more

23. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 23 ESS ECAL Safety System - ESS formerly : TSS (Temperature Safety System) Belgrade group, ETHZ See also CMS NOTE 2004-013

24. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 24 Full system autonomy in all aspects; Independent, continuous temp. monitoring of the ECAL VFE + FE environment in both ECAL SM + EE; Precision : 0.5 deg C Archiving of temp. data and system status information for analysis of system and detector performance; Reliable hardwired interlocks with ECAL HV and LV Power Supply systems; External Alarm Interfaces with ECAL Cooling system, as well as interface with general CMS DSS; Also with the newly proposed Water leakage detection system! Prompt reaction on any external alarm or critical change of temperature inside the ECAL by issuing, in a proper time sequence, Warnings and Alarms to: 1. HV System Crates (hardwired interlocks), 2. LV System Crates (hardwired interlocks), 3. System Operator (soft PVSS Warnings and Alarms); Maximum possible level of robustness, reliability, safety and maintainability; ESS : Requirements

25. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 25 Three interconnected system layers: ESS FE Layer Temperature conversion and channel multiplexing - ESS FE Layer, ESS PLC Layer Data acquisition, data processing and interlock generating - ESS PLC Layer, ESS Soft Layer System monitoring and system control - ESS Soft Layer; Schematic layout RS485 ?

26. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 26 288+64 NTC 470 Ohms There are in total 288+64 NTC 470 Ohms thermistors (EPCOS) positioned in pairs at each measurement point ( “Twin” sensors ). 2x twisted pair cable SCEM = 04.21.51.704.4 SCEM = 04.21.51.704.4, 2x2x0.05mm2, PETP insulation outside diameter 2.9mm, PETP insulation ESS : Sensor Location 40-90m long 9xSTP cable sections SCEM = 04.21.51.405.2 SCEM = 04.21.51.405.2 – 9x2x0.008 mm2 outside diameter 7.2 mm, MA 18 P

27. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 27  We got 2 complete setups needed for a redundant system Siemens S7- 400 Family PLC System for ESS: ESS : PLC

28. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 28

29. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 29 Principle Controls Actions n See also Xavier’s alarm/action matrix… n Critical Reading : Temperatures (TSS) u Software warning. If persistent: u Switch OFF relevant HV/LV power supplies via ECAL-PLC n Critical Reading : Water Leak u Software warning. If persistent: u Switch OFF relevant HV/LV power supplies via ECAL-PLC u Message to Cooling PLC : close SM n safety lines n Critical Reading : Sniffers (on CMS-DSS PLC) u Switch OFF power on ECAL racks (to be implemented by CMS-DSS) ? u See also Ch. Schaefer’s talk…

30. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 30 Principle Controls Actions n DSS Watch Dog : ECAL-PLC not running u Switch OFF power of the ECAL racks u Note : ECAL-PLC checks that Cooling-PLC is running! n Critical Reading on General Cooling PLC (TS) : u Message to the CMS-DSS PLC and the Cooling PLC u Actions (?) : Stop water, switch OFF power of ECAL racks n Critical Reading : DCU temperatures (on PCBs/Capsules) u Software warning. If persistent? u I would say : no automatic action n Critical Reading : PTM temperatures u Software warning. u I would say : no automatic action

31. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 31 Principle Controls Actions n In general, for ESS - DSS interactions: u ESS should take principle actions u Only when DSS sees that ESS is “dead”, DSS should take action! u Exceptions : l Sniffers (go directly to DSS) l Others?

32. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 32 Experience up to now n Excellent performance of ESS u Sent relevant interlocks during SM0 tests u No problem during SM10 running up to now u See ESR May04 review talk for details on SM0

33. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 33 PVSS Monitoring and Control S. Zelepoukine, P. Milenovic, R. Ofierzynski, F. Beaudette, A. Lister, R. Gomez-Reino Also : Thanks to S. Schmeling / JCOP

34. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 34 Standard Controls n Continuous Monitoring (display/storage via ECAL-PVSS) u Temperatures from ESS u Temperatures from PTM u Temperatures from DCUs (not implemented yet at H4) u Water Leakage Detection System (via ESS) u Humidity u Cooling System (temperatures, flow, pressures) u HV and LV u Status of Laser System

35. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 35 PVSS at H4 : Prototype for ECAL n First complete set-up of PVSS DCS system u At H4, includes: u PTM - HM - ESS - LV - HV - Cooling - Laser u + Supervisory system u Every application implements a Finite State Machine u Based on PVSS II, v.3, and the JCOP framework u Distributed system (Windows XP) on 3 PCs : l 1 : Supervisor, Laser, Cooling l 2 : HV, LV l 3 : PTM, HM, ESS u Should be similar at P5 l I see no reason to have dedicated PC just for Cooling Monitoring

36. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 36 PVSS at H4 : Data Logging u For the moment, all data go to the PVSS archive (proprietary) u Not easy to get data out (apart trending, which is ok) l not foreseen as final solution (!) u Done because of pressure to get the FSM at H4 etc running u On the long term : ORACLE u Some first tests last year with HV, but did not learn too much u A new expert is joining the efforts (Rick Egeland, Minnesota) l He will try to help us setting up the commun. with ORACLE and to develop scripts to easily retrieve data u We will not write all DCS data to conditions DB at high frequency. Rather at low freq. ( 1/ (1 - 10) min ?) or when major changes occur l Could write all DCS data to some dedicated DCS DB, which is archived at some stage (after a couple of years?). TBD u But not more to say for the moment…..

37. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 37 Supervisory System : 1

38. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 38 Supervisory System : 2

39. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 39 Supervisory System : 3

40. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 40 Cooling Application See also Xavier Pons’ talk: For ways of changing limits, OPC connection, etc. Note: This application only monitors user can not operate the cooling system from here !

41. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 41 ESS Application

42. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 42 Conclusions n Hardware u PTM : u performs according to specs u Able to monitor temperature stability to 0.01 C u To be resolved : issue of immersion probe location u ESS: u Have purchased redundant Siemens PLC system u All tests up to now very satisfactory u To be finalized : l Water leak detection l Hardware links with Cooling l Exact protocol of further software communication with Cooling

43. G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 43 Conclusions n Software u Have developed first (in CMS) complete distributed monitoring system based on PVSS u Was rather difficult and painful endeavour, have learned important lessons, also for other subdetectors u To be implemented : Monitoring of PCB/capsule temperatures from DCUs u Monitoring of cooling : ONLY monitoring, no action/operation by shifter u Have to identify man-power for the development of the P5 set-up, based on what we developed for H4