1. CMS-ECAL DCS Prototype
Trigger/DAQ project  DCS
CMS DCS
Status report
ECAL DCS
G. Dissertori / ETHZ JCOP Meeting, Dec 2, 2004
ECAL - DCS

2. Outline Overview of ECAL DCS, short description of PVSS Monitoring
Precision Temperature Monitoring (PTM)
Humidity Measurement (HM)
ECAL Safety System (ESS)
PVSS Monitoring
Experience gained with a distributed system during a testbeam campaign (Oct-Nov 04, H4 area, Prevessin)
ECAL - DCS

3. CMS ECAL 61’200 Barrel Crystals 14’648 Endcap Crystals ECAL - DCS

4. Supermodule SM10
Supermodule
1700 Crystals
ECAL - DCS

5. Overview PVSS Offline, DAQ prec. temp. sensor ESS temp. sensor + WLD
Monitoring
ESS temp. sensor + WLD
Monitoring
Safety, PLC
interlock
Laser system
cooling
humidity sensor
Data
Bases
ECAL module
Offline, DAQ
crystal
PCBs for VFE and FE
PVSS
LV, HV
cooling, faults
ECAL - DCS

6. Precision Temperature Monitoring - PTM
S. Zelepoukine, IHEP Protvino and ETHZ
ECAL - DCS

7. PTM
Requirements:
very high precision temp. monitoring : check stability of water cooling system, to stay within deg C.
no hardwired feedback to cooling
relative precision : 0.01 deg C
one sensor/module on thermal screen and grid
main water IN, main water OUT
ECAL - DCS

8. PTM : Readout - Layout Network access Counting room UXC55 balcony:
PC / PVSS
Counting room
CANbus cables
DC Power
supplies
Monitoring application:
PVSS based;
data storage/archiving,
visualization/trending,
warnings/alarms.
Network access
Galvanic
isolation
ELMB
Cur.src
UXC55 balcony:
20x2 STP cables
Readout electronics:
ELMB based;
ELMB reads voltages and
converts to digital data, then
sends data via CANbus.
DC-DC converters
ECAL
~ 500 sensors
(NTC thermistor Betatherm;
25C; indiv.calibrated;
better than 0.01C rel.accuracy)
ECAL - DCS

9. H4
ECAL - DCS

10. Cooling Stability
Zoom
ECAL - DCS

11. Zoom : Grid M4 0.01C M3, M2 M1 Thermal screen M4, M3 M1 0.01C
Water - IN
0.01C
Cooling stability = 0.01C (incl. day/night oscillations)
ECAL - DCS

12. Humidity Monitoring - HM
S. Zelepoukine, IHEP Protvino and ETHZ
ECAL - DCS

13. Requirements For HM we want to Sensors Readout
Measure humidity level inside modules, precision about 5%
Sensors
One per module
UPS600 from Ohmic Instruments
resistive type, cheap, radiation hard
Transducers also from Ohmic Instr.
Readout
Same principles as for PTM : ELMB + Canbus to PC
ECAL - DCS

14. M1, M3, M4 M2 (conn.broken) RH – inside SM10 RH -- ambient
RH cable shield grounded to SM10
M1, M3, M4
M2 (conn.broken)
RH – inside SM10
RH -- ambient
Amb1/2 (SM10)
Amb_3 (DCS rack)
water out
Water temperatures
water in
Ambient
temperatures
ECAL - DCS

15. ECAL Safety System - ESS
See also CMS NOTE
Belgrade group, ETHZ
ECAL - DCS

16. Schematic layout Three interconnected system layers:
Temperature conversion and channel multiplexing - ESS FE Layer,
Data acquisition, data processing and interlock generating - ESS PLC Layer,
System monitoring and system control - ESS Soft Layer;
RS485 ?
RS485
ECAL - DCS

17. ESS : Sensors Temperature sensors : NTC 470 Ohms thermistors (EPCOS)
positioned in pairs at each measurement point ( “Twin” sensors ).
Water Leak Detection: TraceTek
ECAL - DCS

18. ESS : PLC Siemens S7- 400 Family PLC System for ESS: ECAL - DCS

19. ECAL - DCS

20. M1 M2 M3 M4 28.5 30.6 28.2 25.7 ESS Panel: Temperatures inside the
electronics compartment
M M2 M3 M4 M2
M1, M3 M4
ECAL - DCS

21. Experience up to now Excellent performance of ESS
No problem during SM10 running
Interlock tests at the startup were successful
Afterwards, no interlock
ECAL - DCS

22. Other Hardware High Voltage Low Voltage Laser monitoring
CAEN SY1527
Low Voltage
CAEN EASY + CAEN SY2527
Laser monitoring
Just received (via DIM interface) basic status parameters
Cooling system
Complete prototype of cooling system at H4 testbeam area, communication with cooling PLC via OPC
ECAL - DCS

23. PVSS Monitoring and Control
S. Zelepoukine, P. Milenovic, R. Ofierzynski, F. Beaudette, A. Lister, R. Gomez-Reino, F. Glege
ECAL - DCS

24. PVSS @H4 Have implemented the monitoring of the following subsystems:
Temperatures from ESS
Water Leakage Detection System (via ESS)
Temperatures from PTM
Humidity (HM)
Cooling System (temperatures, flow, pressures)
Status of Laser System (via DIM interface)
HV and LV
Not implemented/used
Link with Run Control was there, but only partly used (reading of status)
Temperatures from DCUs
Storage to ORACLE DB
ECAL - DCS

25. PVSS @H4 : Prototype for ECAL
First complete set-up of PVSS DCS system
includes:
PTM - HM - ESS - LV - HV - Cooling - Laser
+ Supervisory system
Every application implements a Finite State Machine
Based on PVSS II, v.3, and the JCOP framework (v2.0.9)
Distributed system (Windows XP) on 3 PCs :
1 : Supervisor, Laser, Cooling
2 : HV, LV
3 : PTM, HM, ESS
ECAL - DCS

26. ECAL FSM Structure CMS_ECAL Channel06 Channel07 Channel08 Channel09
ECAL_LV
ECAL_HV
ECAL_ESS
ECAL_LASER
ECAL_HM
ECAL_PTM
ECAL_COOLING
ECAL_BARREL-
ECAL_ENDCAP+
ECAL_ENDCAP-
ECAL_BARREL+
ECAL_SM01
ECAL_SM18
ECAL_SM10
ECAL_COOLING_SM10
ECAL_PTM/HM_SM10
ECAL_ESS_SM10
ECAL_HV_SM10
ECAL_LV_SM10
ECAL_LV_SM10M1
ECAL_LV_SM10M2
ECAL_LV_SM10M3
ECAL_LV_SM10M4
ECAL_HV_SM10M1
ECAL_HV_SM10M2
ECAL_HV_M10M3
ECAL_HV_SM10M4
Channel06
Channel07
Channel08
Channel09
Channel11
Channel12
Channel13
Channel14
Channel15
Channel16
Channel17
Channel18
ECAL - DCS

27. ECAL FSM Structure pcethdcs1 pcethdcs2 pcethdcs3 CMS_ECAL
ECAL_ESS
ECAL_HM
ECAL_PTM
ECAL_PTM/HM_SM10
ECAL_ESS_SM10
pcethdcs3
CMS_ECAL_PTM_HM
CMS_ECAL_ESS
ECAL_LV
ECAL_HV
ECAL_HV_SM10
ECAL_LV_SM10
pcethdcs2
CMS_ECAL_HV
CMS_ECAL_LV
ECAL_BARREL+
ECAL_LASER
pcethdcs1
ECAL_SM10
CMS_ECAL_COOLING
ECAL_COOLING
ECAL_COOLING_SM10
ECAL - DCS

28. Supervisory System : 1
ECAL - DCS

29. Supervisory System : 2
ECAL - DCS

30. Supervisory System : 3
ECAL - DCS

31. HV Application
ECAL - DCS

32. LV Application
ECAL - DCS

33. Cooling Application Note: Communication With cooling System via OPC
This application
only monitors
user can not
operate the
cooling system
from here !
ECAL - DCS

34. ESS Application
Comm.
with PLC
via OPC
ECAL - DCS

35. : Data Logging
For the moment, all data go to the PVSS archive (proprietary)
Not easy to get data out (apart trending, which is ok)
not foreseen as final solution (!)
Done because of pressure to get the FSM at H4 etc running
Faced some problem of heavy writing to archive because of difficulty to create alarms out of logical connections of states
On the long term : ORACLE
A new expert is joining the efforts (Rick Egeland, Minnesota)
He will try to help us setting up the commun. with ORACLE and to develop scripts to easily retrieve data
Detailed discussions on ConditionsDB and ConfigurationDB just started
ECAL - DCS

36. Major Issues During development :
Already preparing for the final ECAL application
Some debugging of Framework tools, in particular FSM and DIM, was necessary. Thanks to C. Gaspar for her feedback.
Suffered from missing documentation of Framework
Note : The continuous and friendly help by S. Schmeling was absolutely essential !! Thanks!!
HV - LV:
Major problems with the connection to CAEN crates
See later
ECAL - DCS

37. Experience with distr. PVSS system
The different parts of the tree should be created as installable applications and not as complete projects
Reduce the overload
No PVSS port conflicts
Easy to move or recover an application
PVSS as a service
Naming agreements
On the system names
On the component/applications names
On the folder names
On the FSM node names and FSM type names
On the library names
ECAL - DCS

38. Experience with distr. PVSS system
System names in all the dp references of the tree components (panels, libs…) should be a dollar/reference parameter that would depend on where the tree part is installed
Network in the testbeam area can create problems….
VNC is the best friend for (shared and non-screen-blocking) remote connections
It is very difficult to overestimate the needed resources in terms of man power … d;-)
ECAL - DCS

39. Experience with CAEN crates
HV : A1520PE
PC in 32 and CAEN 1527 in 867 (cmsechv05) never worked consistently
longest period with stable connection about 20 hours
average time otherwise: 30 mins
both firmware versions (1.14, ) were tried
PC in 32 and CAEN 1527 (1.14) in 27 (cmsechv00) worked better
longest period with stable connection about 5 days
average time otherwise: 2-3 days
testbeam setup (CAEN 1527 (2.00.2) cmsechv04[->05->03], PC in H4)
1. connected to central infrastructure hub -> no connection, also PCs did     not work on this initially
2. connected to LANtronix switch -> worked in the beginning and then     showed intermittent connection
3. connected to hub (10Mbit old type) in CR -> since then it works without     problem
ECAL - DCS

40. Experience with CAEN crates
LV: (A3100 EASY)
PC in 867 and CAEN 1527 in 867
average time: 30 mins (July)
end July: little activity otherwise in 867 -> worked well
PC in 867 and CAEN 2527 in 157
average stable time: 1 day
PC in H4 and CAEN 2527 in H4 (lvecal2)
1. connected to local network (BNC) in lower area -> worked fine for one week suddenly stopped
2. connected with UTP cable to LANtronix switch -> worked shortly but then failed completely
3. connected to central infrastructure hub -> worked shortly but then failed completely
4. connected with UTP cable to LANtronix switch -> worked shortly but then failed completely
5. connected with UTP cable to hub (10Mbit old type) -> since then intermittent connection
October 25th:
OPC server stopped, LV crate rebooted (hard and power cycle), OPC server restarted -> no change
When LV crate not reachable via OPC, also ping not possible (from linux pc), after PC rebooting, ping worked again for 5min, but then OPC and ping showed again intermittent behaviour.
October 29th:
upgrade of OPC server (pcethdcs2) to 2.8 -> no change
ECAL - DCS

41. Conclusions Hardware Software
PTM / HM / ESS / Cooling: perform according to specs
HV + LV (CAEN crates) : problems with network connections, now investigated by experts (from JCOP and CAEN)
Software
Major efforts gone into development of DCS prototype
Mostly achieved our goals
Note : A major effort in terms of man power was needed (here : approx. 1y FTE). This should be a warning for similar projects…..
ECAL - DCS