1. Clara Gaspar, July 2005 RTTC Control System Status and Plans

2. Clara Gaspar, July 2005 2 Architecture & Components ❙ Farm Infrastructure ❙ Gaudi Jobs ❙ Event Builder ❙ MEP Producer ❙ Run Control Event Builder Switch..................... Control PC PVSS SFC CPU SFC CPU SFC CPU ❚ Monitoring & Control of:

3. Clara Gaspar, July 2005 3 Farm Infrastructure ❚ Very Complete Monitoring of: ❙ Each CPU: ❘ Processes running, CPU usage, Memory usage, etc. ❘ Network traffic ❘ Temperature & Fan speeds, etc. ❚ Control: ❙ Task Manager: Start/Stop Jobs on nodes ❙ IPMI Manager: Switch on/off, reboot nodes ❙ FSM Automated Monitoring (& Control): ❘ Set CPU in "ERROR" when monit. quantities bad 〡 Could/will take automatic actions

4. Clara Gaspar, July 2005 4 Farm Monitoring

5. Clara Gaspar, July 2005 5 Gaudi Job Monitoring ❚ Gaucho Tasks: ❙ Start/Stop Gaudi Jobs on each node: ❘ 2 L1 (Euler) Processes ❘ 2 HLT (Moore) Processes ❙ Receive Monitoring Info from jobs: ❘ Counters (seen/rejected/accepted events) ❘ histograms, etc ❙ Accumulate and Save Statistics ❘ per job type/per node/per sub-farm/full farm ❙ Automate operations (start/stop run) ❘ According to configuration

6. Clara Gaspar, July 2005 6 Gaucho Example ❚ Accumulated Statistics from 2 nodes (4 L1/Euler Jobs)

7. Clara Gaspar, July 2005 7 RTTC Run Control ❚ Event Builder(s) & MEP Producer: ❙ Receive monitoring information ❙ FSM control: ❘ Deduce a state and start/stop RUN ❚ RUN Control ❙ Coordinate all components ❘ Distribute Configuration and RUN commands to: Evt_builder/ MEP Producer / Gaucho 〡 Depending on the state of the FARM infrastructure ❘ Show overall status and summary information

8. Clara Gaspar, July 2005 8 Control System Status ❚ Farm Monitoring ❙ First version ready ❙ Permanently running, acquiring: ❘ ~1000 quantities per node every 20 seconds ❚ Farm Control ❙ Power Control working ❙ Task Manager heavily used ❙ FSM rules currently disabled ❘ But still around 200 quantities per node being checked ❚ But: PVSS perf. degrades after some days...

9. Clara Gaspar, July 2005 9 Control System Status ❚ Gaucho ❙ Prototype Ready but: ❘ Can not run it for more than a few nodes currently ❚ Problem: FSM ❙ Current FSM implementation: ❘ A Control Unit (a sub-system) spawns one FSM process and one PVSS process ❘ A Device Unit (a device) is light weight ❙ The way most users designed their system is: ❘ One control unit per device (or few devices) (in Gaucho: 7 CU per node * 44 nodes) ❙ Uses too much memory

10. Clara Gaspar, July 2005 10 Solution: ❚ New FSM version ❙ A Control Unit (a sub-system) spawns an FSM process ❙ A Logical Unit (grouping devices) light weight ❙ A Device Unit (a device) light weight ❙ Only one PVSS process for all CUs ❚ Almost ready ❙ Will be released this week or next week

11. Clara Gaspar, July 2005 11 Plan ❚ Farm monitoring & Control ❙ Run alone on PC until PVSS "degradation" is understood ❙ Optimize project in general ❚ Gaucho ❙ Install new version of FSM ❙ Test Gaucho on large scale ❚ Run Control & others ❙ Integrate when above ready

12. Clara Gaspar, July 2005 12 Conclusions ❚ All control developments can be done in parallel with DAQ studies and measurements ❚ The Farm test bed will be used until all problems are solved. Our aim is still to have an integrated Run Control with all components in one Control PC. ❚ We will report regularly on progress