1. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 1 Shot by Shot Logging Status Report of Ti8 Tests AB/CO Technical Committee - 2nd Dec 2004 M.Pace

2. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 2 Outline System description System description Results Results Evolution Evolution

3. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 3 Outline System description System description Results Results Evolution Evolution

4. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 4 Architecture Evolution of TT40 version (L.Mestre) TGM lib Oracle Application Server OHS (HTTP Server) Enterprise Manager OC4J (J2EE Container) Data Extraction App Data Loading App Oracle 9i DB Timber SBS Logging Agent sps2001cmwrda JAPC DEVICE SPS MEAS Oracle DB STAMP SPS SC LASER/CAS GW CAS display XML

5. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 5 12 device groups 12 device groups –BDI (6) –MUGEF, MPS –BT –OASIS (New –FESA2) –BIC (New- FESA2) –COLLIM (New) Out of scope Out of scope –CV JAPC as unique API JAPC as unique API –various impl. behind Various logging rates Various logging rates –On cycle occurrence –At specific frequency Device access SBS Logging Agent sps2001 cmwrda JAPC DEVICE

6. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 6 Logging Data Input API – Java Logging Data Input API – Java PB with API if 1 single logging process PB with API if 1 single logging process => 12 logging processes  12 JVM on same machine Meta data hierarchy Meta data hierarchy –ROOT Ti8Ti8 –Beam Profiles –Beam Intensities –Magnet Currents –Etc … Extraction GUI on Timber Extraction GUI on Timber Data Loading & Extraction Oracle AS OHS (HTTP Server) Enterprise Manager OC4J (J\2EE Container) Data Extraction Ap Data Loading App Oracle 9i DB Timber SBS Logging Agent XML

7. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 7 Laser source API – Java Laser source API – Java For each process (12) For each process (12) –Surveillance alarm (process not responding) –Fault alarm (device access pb) –Fault alarm (loader access pb) Status sent every SPS SC Status sent every SPS SC –Subscription to SPS SC Start event through TGM lib Alarm display on CAS system Alarm display on CAS system Dedicated console for tests Dedicated console for tests Alarms handling SBS Logging Agent LASER/CAS GW CAS display

8. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 8 Current implementation Stamping source Stamping source sunslps.cern.ch:1521:sps", "spsmeas", "proton“ Stamp Stamp –Common to all devices –UTC time of SPS cycle start –Precision: HH: MM: SS Alternative implementation (not retained) CMW subscription to the SPS cycle descriptor (new Timing server) CMW subscription to the SPS cycle descriptor (new Timing server) Not in depth investigated : Not in depth investigated : –Support w.r.t. TGM LIB ? –Does not solve data coherency issue Data Stamping SBS Logging Agent SPS MEAS Oracle DB STAMP

9. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 9 Outline System description System description Results Results Evolution Evolution

10. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 10 Results (1) Logging proven to be useful From OP viewpoint From OP viewpoint –Surveillance of the device behaviour –PM analysis of TT40 beam incident on 25 oct 04 ReconstructionReconstructionReconstruction From CO viewpoint : validation of : From CO viewpoint : validation of : –Flexibility + Modularity of sbs logging –FESA2 device access scheme (OASIS, BIC) –LASER source Java API (logging = 1 st user) + complete chain (LASER/CAS GW) –Increased load testing of the LHC Logging Service (Multiple processes with high request rate: Max 6 requests/second)

11. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 11 Results (2) Logging highlighted problems Constraints Constraints –From Alarms : 1 single machine authorized (no backup) –From BT access (DIM impl) : Execution on Windows onlyExecution on Windows only Local installation of specific dllLocal installation of specific dll Technical issues Technical issues –Data Loader –Device access –Data stamping –Cycle selector –OASIS

12. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 12 Data Loader : Technical Issues ADDRESSED (Chris) Unclear exception msg in case of unacceptable value 2E-313 Unclear exception msg in case of unacceptable value 2E-313 Physical DB connections not closed after Logical connections closed following the previous exception Physical DB connections not closed after Logical connections closed following the previous exception Timber : issues or missing/unsuitable features Timber : issues or missing/unsuitable features Missing monitoring of AS via xcluc Missing monitoring of AS via xcluc OC4J restart after crash (ABJAS4, 25 Oct) due to JVM / OS bug OC4J restart after crash (ABJAS4, 25 Oct) due to JVM / OS bug TO BE FOLLOWED UP (Chris + Marine) AS machine down (human mistake, 6 Nov) => NO exception handled => To be investigated. AS machine down (human mistake, 6 Nov) => NO exception handled => To be investigated. New version of Data Input API developed (to allow 1 single logging process) => To be tested. New version of Data Input API developed (to allow 1 single logging process) => To be tested.

13. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 13 Device Access : Technical Issues TO BE FOLLOWED UP MAINPWS MAINPWS –23-25 oct: sporadic exceptions –05-07 nov: stopped working (timeout reached on SL-EQUIP call from PCRSRV9 to MRSBA3) –Spaghetti access (courtesy of Pierre) JAPC–CMW–SLEQUIP–RPC => dedicated WG CO - PO MUGEF MUGEF –23-25 oct: short disturbances ( CMW/SL-EQUIP GW machine stop, …) => To be replaced by FESA ? BT (SEPTA + KICKER) – Access via DIM (Contracts) BT (SEPTA + KICKER) – Access via DIM (Contracts) –24 oct + 6 nov: stopped receiving data without exception. OK after logging restart => PB not understood. To be replaced by FESA ?

14. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 14 Data Stamping : Technical Issues Present mechanism is WEAK NOT reliable enough NOT reliable enough –Meaningless “0” [1970, Jan 1 st ] value sporadically read => hole in logging –Possible unavailability of DB does NOT guarantee data coherency does NOT guarantee data coherency –Stamp possibly wrong w.r.t. data –1 cycle delay observed among data from # devices => Difficult correlation of logged data Improvement: STAMP should be available Improvement: STAMP should be available –Along with data from CMW /JAPC –Available and Common for ALL device groups

15. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 15 Cycle Selector : Technical Issues Present scheme is not scalable CS specific to device + to access interface CS specific to device + to access interface –SPS.USER.LHC546  BLMI (FESA1) –21890301  MPS + MUGEF,.. –21000301  BCT –211C0301  BLM –no CS  OASIS, BIC, BT CS hard coded in configuration file, not derivable from current cycle CS hard coded in configuration file, not derivable from current cycle

16. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 16 OASIS : Technical Issues ADDRESSED (OASIS team) operational AS not OK (23-25 nov) => use of backup dev AS operational AS not OK (23-25 nov) => use of backup dev AS GM to FESA migration GM to FESA migration 2000 points wave form logged (500 before) 2000 points wave form logged (500 before) xcluck monitoring of AS machine xcluck monitoring of AS machine TO BE FOLLOWED UP Scope/Channel settings for sbs logging : specialist + manual procedure => specific GUI AP required Scope/Channel settings for sbs logging : specialist + manual procedure => specific GUI AP required Handling of data resulting from rounding errors (scope delay - 2E-313) rejected by Oracle Handling of data resulting from rounding errors (scope delay - 2E-313) rejected by Oracle => responsibility to be clarified: OASIS or sbs logging or Data Loader?

17. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 17 Evolution : in view of LEIR Clarify UR : is present sbs logging suitable to LEIR? Clarify UR : is present sbs logging suitable to LEIR? –Continuous / on request logging ? –Predefined / dynamical devices ? –On line monitoring or Logging ? If sbs logging required : Deal with new constraint Deal with new constraint –UR : log each cycle  2.4 - 3.6 s –Is not compliant with present design => data buffering Have previous issues solved Have previous issues solved Provide new features Provide new features –Data persistency –Detection mechanism if no data received by HW ? –xcluc monitoring [ in view of LHC HW Com. : UR unknown yet ]

18. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 18 QUESTIONS ?

19. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 19 Beam incident timing (courtesy of J.Wenninger) The BLUE curve is obtained from a PC simulation (PC off) by AB/PO. The timing of PC current survey (0.1 % tolerance) and of the precise extraction time is obtained from the Beam Interlock System logging. This reconstrction is consistent (within ~ 0.5 ms) with the beam impact point. (reconstructed) (logging) Magnetic septum current change time within SPS super-cycle

20. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 20 History – MSE current (courtesy of J.Uthoven) Conclusion : MSE current appears to be ~2.5% low at extraction (Note: ~8 MSE trips previous to the accident) BACK

21. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 21 25 oct 04: beam incident where a nominal LHC batch impacted on a quadrupole vacuum chamber following a magnet interlock on the extraction septum MSE Pb: Interlock on the MSE power converter due to a SPURIOUS magnet fault that fell inside the time interval between the last current surveillance and the extraction. Cure : – –Additional interlock between the PLC that surveys the state of the MSE magnet (temperature, water…) and the TT40 BIC. – –New interlock logic for the MSE magnet : First an interlock is send to the BIC  inhibits extraction. 10 ms later the PC is switched off. New UR for logging on interlock references + settings: – –SW to set the parameters. – –Logging and tracing of changes + ref/tolerances

22. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 22 (1) The Java MMI running on PCOPCR05 is using the JAPC interface, with a simple pair (Name, Action) where name can be QD or QF1 and the Action is PCCurrent (2) The JAPC talks with CMW via RDA and there the translation between the name (QD or QF1) is made to find the business server machine, In our case it is ABCMW1 (was PCSLUX8) (3) In ABCMW1 there is a CMW/SL-EQUIP gateway named mugefSrv that translates these calls into an SL-EQUIP call to MUGEF_101 (or 102, 103,...), action ACQRST and mode RBI. (4) From ABCMW1, the SL-EQUIP call goes to HPSLZ22 in an other SL- EQUIP equipment server called clientRPC. (5) This clientRPC calls the (old) RPC server on MRSBA3 called MUPCSERV. (6) On MRSBA3, the process mupcserv calls the (old) RPC server also on MRSBA3 called ACQSERV which does the actual access to a MOPS data structure that reflects the state of the equipment This is the description of a READ command. The problem that is currently happening is that the ACQSRV in the very end of the chain takes too long to reply, therefore the top clients gets messages like ("SVM-Msg : Nobody responding..." or "Timeout in the MH...")