1. Peter Chochula, DCS Workshop Geneva, February 28, 2005 DCS Test campaign (Talk based on presentation to TB in December 2004) Peter Chochula

2. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Purpose of the tests DCS is requested to provide information on system scale Test of hardware compatibility (e.g. PCI controllers installed in PCI risers), components verification before the mass purchase Test procedure for the components delivered by sub- detectors Test procedures for hardware components before the installation

3. Peter Chochula, DCS Workshop Geneva, February 28, 2005 The System Scale Two main activity areas of DCS tests: –Performance and stability tests –Resource consumption (implications on system scale and hardware requirements) The tests cover also the DCS core computing (database, domain controllers, RAS…), system management and security

4. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Performance tests with impact on DCS scale planning PVSS II is a system which can be distributed and/or scattered to many CPUs Two extreme approaches are possible: –group all processes on one machine Even if this configuration runs stable for some systems, there could be a problem if peak load occurs –Dedicate one machine per task (LV, HV…) Surely the computer resources would be wasted Definition of optimal balance between the performance and the size of the system requires tests with realistic hardware and data

5. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Who is doing what? –ACC follows tests performed by other groups and provides feedback to sub-detectors –ACC performs test which complement the work of JCOP and other groups. This includes: Tests not performed by external groups Tests for which the external planning is incompatible with our schedule (e.g. OS management) Alice-specific tests (FERO)

6. Peter Chochula, DCS Workshop Geneva, February 28, 2005 PVSS Performance Tests Tests performed in the frames of the SUP –Communication between 130 PVSS systems. The influence of heavy load on PVSS has been studied Alarms absorption, display, cancellation and acknowledgment Data archival Trending performance Influence of heavy network traffic SUP test hierarchy Data generated by leaf nodes was transported to the top level machines

7. Peter Chochula, DCS Workshop Geneva, February 28, 2005 (Some) SUP results 130 systems (with ~5 million DPEs defined) interconnected successfully Connection of 100 UIs to a project generating 1000 changes/s has been demonstrated –These tests were later repeated by our team in order to understand the remote access mechanism Performance tests on realistic systems –40000 DPE/machine equivalent to 5 CAEN crates –1000 alerts generated on a leaf machine in a burst lasting 0.18 sec and repeated after 1s delay

8. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Alert absorption by PVSS system The PVSS system was able to absorb all alarms generated on a leaf node –Display of 5000 came alerts: 26s –Cancellation of 5000 alerts: 45 s –Acknowledgment of 10000 alerts: 2 min 20 s ETM is implementing a new alarm handling mechanism which includes alarm filtering, summary alarms and will provide higher performance

9. Peter Chochula, DCS Workshop Geneva, February 28, 2005 The archiving was fully efficient during these tests (no data loss) –We would like to see the performance with the RDB archive Trending performance depends on queue settings. Evasive PVSS action (protecting the PVSS system from overloading) can disturb the trend, but data can be recovered from archive once the avalanche is gone. Alert avalanche is memory hungry (>200B/simple DP) The ACC participated in additional tests (December 2004), where the network has been flooded –No performance drop in the abovementioned tests has been observed –Report was published (Paul)

10. Peter Chochula, DCS Workshop Geneva, February 28, 2005 DCS Test Setup Test setup installed in DCS lab –Prototypes of DCS core computers –Worker nodes (rental) Domain Controller Terminal Server Router Database Server CERN Worker Node

11. Peter Chochula, DCS Workshop Geneva, February 28, 2005 DCS core computers In order to operate the DCS following core computers will be needed: –Windows DC –Application Gateway for remote access –Database server(s) with mass storage DCS infrastructure node (prototype available) –Central operators computer Prototypes for all components available. Database servers need further testing

12. Peter Chochula, DCS Workshop Geneva, February 28, 2005 The DCS Lab Backend Prototype Pre-installation Servers Fronted prototype

13. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Backend Systems Backend Prototype Pre-installation Servers

14. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Complementary tests performed by the ACC – Remote access Tests performed by SUP indicated that a large number of UI can be connected to a running project –no interference observed up to 100 UIs – tests did not go further Our tests tried to simulate an external access using W2k3 Terminal Services to a heavy loaded system and observe the effects on –Terminal server –Running project Remark: The Terminal Server is technology recommended by CERNs security. Tests performed by ALICE were presented to JCOP

15. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Windows Server 2003 Windows Server 2003 Windows XP Pro Windows XP Pro Windows XP Pro Computer Infrastructure for Remote Access Tests DCS Private Net. 192.168.39.0 CERN Net. Terminal serverRouter PVSS Master Project Remote User Windows XP Pro

16. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Computer loads for large number of remote clients Terminal Server #clients Average CPU load [%] Mem [kB] 6011.2 2781282 5511.0 2788719 4513.8 2790181 3512.0 2672998 259.7 2067242 157.2 1448779 54.2 934763 04.9 666914 Workstation running the DCS project #clients Average CPU load [%] Mem [kB] 6085.1579666 5586.6 579829 4584.9579690 3581.3579405 2580.9 579384 1581.4 579463 583.0580003 083.7579691 Master project generated 50000 datapoints and updated 3000 /s. Remote client displayed 50 values at a time

17. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Conclusions on remote access tests Prototype performed well Memory consumption ~35 MB per heavy session CPU usage reasonable, one Xeon CPU running at 3GHz can handle the load Stability tested over weeks

18. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Additional possible bottlenecks to be tested SUP tests were focused only on performance of the PVSS system What is different in real DCS configuration? Please remember the discussion which we had this morning EM UI CM VA OPC c OPC s HW OPC c OPC c OPC s OPC s HW Peak load ? Data queuing ?

19. Peter Chochula, DCS Workshop Geneva, February 28, 2005 OPC tests Results from CAEN OPC tests performed by HMPID and JCOP made available in December 2004 Test setup covered the full controls hierarchy Time to switch 200 channels : 8s Recommendations: –Maximum number of 4 fully equipped crates (~800 channels) per computer Tests provided very useful comparison between real hardware and software simulators See Giacintos talk for details

20. Peter Chochula, DCS Workshop Geneva, February 28, 2005 What happens next We are preparing inventory of the hardware (number of channels, crates, etc.) –data available on DCS page, detectors are regularly requested to update the information More (not only) performance tests scheduled for early 2005

21. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Additional activities in the lab Component compatibility tests –Will the PCI cards work together ? –Are they compatible with the PCI riser cards and rack- mounted chassis ? –How do they perform in a rack ? Obvious but non-trivial questions –How do we arrange components in the racks? –What about cables, air-flow, … And of course – preparing for your software

22. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Component Compatibility Tests PCI Can Controller NI-MXI2 VME Master 2U PCI Riser

23. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Trying to arrange the components in the racks…

24. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Test schedule (as presented to the TB) 12/04 1/05 2/05 3/05 4/05 5/05 6/05 ConfDB tests………….. ArchDB tests………………….. Full system configuration…………… OPC stability………………..….. Alarms……………………………………….. ArchDB – connection to IT…………………………….. FERO – SPD prototype……………………. Mixed system environment…... Patch deployment……………………….. Network security tests………………… Additional delay with respect to this planning (~2 months) accumulated due to late delivery of computers, but we need to catch up

25. Peter Chochula, DCS Workshop Geneva, February 28, 2005 Input from sub-detectors is essential Most unexpected problems are typically discovered only during the operation –This experience cannot be obtained in lab –Pre-installation is a very important period for DCS Efficient tests can be performed only with realistic hardware –Components are missing –We do not have enough manpower to perform tests for all possible combinations of hardware at CERN