 Software Tools for MPS Kajetan Fuchsberger MPP Workshop, 2013, Annecy Many thanks for input from M. Zerlauth, J. Wenninger, R. Schmidt, G. Kruk, V. Baggiolini, G. Papotti, D. Jacquet and the TE-MPE-MS Software Team

 Software Tools for MPS The solution for everything ;-)

3 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Content Tests & ProceduresEarly detection of failuresAperture Meter

4 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Content Tests & ProceduresEarly detection of failuresAperture Meter

5 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Tests and Procedures  Currently: Sharepoint Sites  Disadvantages:  Order not enforced (scheduling done ‘on the fly’).  Nothing enforces the tests to be really done.  No real overview, what was done, what still has to be done?

6 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS The AccTesting Framework  Designed from the experience of Hardware Commissioning.  Successfully Used in HWC Campaigns 2012, 2013  Enforces Correct Order of Tests  Many additional features:  Automatic Scheduling  Constraints  Statistics

7 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS AccTesting Concept  Migrate of MPS Commissioning Procedures:  1) Transform all the MPS comissioning steps to AccTesting (simple sign-only tests).  2) Replace one by one with automated tests. Test-Phases. Tests

8 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Barriers  When the commissioning of a system reaches a Barrier Point, it has to wait until all other systems (which have the same barrier) reach it also. QPSIST1 BLMIST2IST3IST4INJ1INJ2 BISIST5IST6IST7INJ3 LBDSIST8INJ4 RQTL…IST10IST11POW1POW2 Ready for Powering Ready for Injection  Knowledge of System-Dependencies required!

9 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Interlocks based on Test Plan?  Examples:  Prohibit, that circuit can be powered, before all the tests are done.  Do not allow injection of beam before all necessary MPS tests are done.  Possibly: Enforce Tests after changes:  E.g. Which tests have to be executed, after a change of a QPS – card?  Pre-defined test plans or (partially) auto-generated?   Knowledge about System-Dependencies required!  What to interlock?  E.g. Additional Circuit lock on PIC level which allows tests?  Injection interlock?

10 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Automation?  Automate whatever can be automated! (Avoid Errors, Reproducibility)  First steps: Concentrate on systems, where many interlocks of same type:  BIC/PIC: Already dedicated tool, in the pipeline to be integrated with AccTesting  Collimators  Integrate BIS connection Test?  Do we really have to look ‘by eye’ on lossmaps?  Could there be a Hierarchy-check without a reference?  Vacuum  BLMs:  E.g.: Test Of Latency: Close collimators: Done in pt3, pt7: Could be done for more, if automatic.  Others?

11 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS System Dependencies & Information  Key Concept: Use existing Sources!  Prototype in place  Possible Cooperation between TE-MPE-MS software team and BE-CO-DO (S. Jensen, D. Csikos). System Information Server System Providers: Lsa Layout Db … Dependency Providers: Layout Db Runtime info (JMS) Java code analysis … Information Providers: Status Issues (Jira) Deployments Faults? … Clients (e.g. AccTesting, Diamon, ….)

12 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Test Analysis  Vision:  Simple description of test-expectations (assertions) on signals resulting from tests (Java).  Universal GUI components to visualize problems.  First Implementation:  Cooperation between TE-MPE-MS software team and BE-CO-DA (R. Gorbonosov, A. Jalal)  Used in AccTesting  could be used e.g. in future PM modules. (Maybe also in sequencer tasks and other checks?)

13 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Data Analysis  TE-MPE-MS Vision: More general Data Analysis Framework  Key Concept: Perform analysis as close as possible to the data!  (Potential) Collaboration with BE-CO-DA to optimize resources, avoid duplication of efforts  First Implementation in place. Analysis Framework (Cluster?) Logging DB Hadoop Cluster? Post Mortem Clients (e.g. AccTesting, Timber, Custom Analysis….) Others?

14 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Content Tests & ProceduresEarly detection of failuresAperture Meter

15 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Early detection of Failures YASP LSA Server trim PcInterlock GUI PC trim subscribe SIS monitor Dump! ? ? Not Allowed! PcInterlock Server

16 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS LSA: The natural place to check LSA Server  Applications:  YASP  Lumi-Scan application  Others…  Checks:  Power Converter Currents  Collimator Settings  BPM readings (actual + predictions)  online modeling?  Others?

17 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS LSA checks - proposal  Vision:  Have an API like ‚isAllowed(TrimRequest request)‘  First implementation:  Use ‚TrimPostProcessor‘ mechanism of LSA.  Executed after trim saved in DB, on error  LSA will do a rollback.  Incorporation = Trim  Same mechanism to avoid propagating trims, which would otherwise e.g. fail during ramp.  Override for MDs.

18 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Further securing LSA  LSA Method protection  Currently: Only basic protection (Sending of timing events, Trim, Drive)  Every method in LSA should be reviewed, if it can do harm or not and protected appropriately  Protect reference Cycles for PcInterlock/SIS:  First approach: TrimPostProcessor which evaluates RBAC roles

19 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Content Tests & ProceduresEarly detection of failuresAperture Meter

20 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Aperture Meter

21 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Aperture Meter - Status  Prototype Operational  Proved to be very useful during MDs  Follows the operational cycle (optics, BP, time in BP)  Listens to (some) LSA trims  What is missing?  Improvements of User Interface  Performance Improvements  Tighter integration with Operational changes (Collimator offsets after alignment, BPM usage from Yasp)  Could be used as another source for LSA trim-verification (e.g. large collimator movements, Collimator Hierarchy)

22 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Summary  MPS Commissioning:  Tracking in Acctesting Framework.  Automate! (Step by Step) Candidates?  Preview: Two new frameworks:  Tracking of system dependencies  (Test-) Data Analysis  Early detection of potential problems could be done in by additional trim checks in LSA.  Aperture Meter: Useful prototype, but needs improvements.

23 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Tools, Tools, Tools, Tools, Tools, …?  …Tools do not solve everything!  Culture & communication  (High Quality) Software is not for free!  Challenges:  Large part of Software manpower goes already into maintenance.  A lot of ‘grown’ projects, partly written by un-experienced programmers (E.g. Students).  How to improve?  Reliability = Maintainability = Quality (Self explaining code + Automated Testing, Testing, Tes…. )  Supervision of any SW projects from two sides: System Expert + SW expert (‘Mentoring’)  Who has the overview and can judge what is already out there and into which framework a certain requirement would fit best?  Communication!  Who is authorized to define priorities? Resources?

24 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Thank you for your Attention! Questions? Small Steps - Big Visions!

25 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Fail Fast!  Actual BP:  Would a certain trim exceed some limits?  Check before functional BP:  Collimator-limits prediction  Corrector Interlocks vs. Lsa settings   Check at incorporation time

26 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Single Source for „Operational State“  For certain applications BeamMode is not enough!  Combination of: Beam Mode, Beam Process, Time within BP, (Optics)  Currently (at least) 2 implementations:  Combination of Beam Modes and PC state changes (Aperture Meter)  Dedicated timing events (PcInterlock)   Both not optimal. Reference Implementation required.

27 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Fail Fast!  „A fail-fast system is designed to immediately report […] any failure or condition that is likely to lead to failure. ” [Wikipedia]

28 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Data Analysis  Mandate in BE-CO-DA (C. Roderick) to provide Data Analysis Solutions  Currently in Use Case Analysis Phase  Aim to be as easy as possible for users to specify:  What to produce for an analysis case (e.g. signal names, units, description)  What are the inputs (signals and sources)  Which algorithms should be applied  When analysis should be performed (event driven e.g. fill / beam mode, time interval driven e.g. every 15 minutes)  Without user needing to:  Configure a specific development environment  Care about compilation and deployment  Look elsewhere for signal names etc. i.e. provide directory of signals and sources  Key concept: Perform data analysis as close as possible to the data!

29 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS MPS Commissioning  What is missing?  Dependencies between Systems (e.g. ELQA for a circuit is ready, when the test are done for all magnets within the circuit)  Global Phases? QPSIST1 BLMIST2IST3IST4INJ1INJ2 BISIST5IST6IST7INJ3 LBDSIST8INJ4 RQTL…IST10IST11POW1POW2 IST Powering Injection

30 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS System Dependencies  Vision: Central Instance (Server?) to answer questions like:  Which magnets are connected to a certain power converter? (AccTesting – ELQA)  What is the common point for a set of crates? (Diamon)  What systems have to be potentially re-tested if e.g. a QPS card is changed? (AccTesting)  What systems are potentially affected by a OS upgrade on a certain host? (SUWG)

31 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Interlocks during functional BPs  In place for orbit correctors. Ready to be extended to other circuits.  Potentially useful for other systems, e.g. Collimator movements?