1. XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Matthias Clausen, DESY ICFA Seminar, 29.October 2008 Global Collaboration on Accelerator Operations and Experiments Matthias Clausen, DESY ICFA Seminar, 29.October 2008

2. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 2 Globalization … … in the Financial World –Has a bad taste. Socializing risk? Privatizing win? … in the HEP Community –Is key to build the next big machines … in the Controls Community –Within the existing (homogeneous) collaborations working and thinking globally is by default no problem. –Collaborating globally between heterogeneous partners can be a difficult task. We must make sure that we reach a Win/Win situation for all participating parties and that nobody is left behind.

3. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 3 Overview Globalization issues Technical Realization –Existing Implementations –Future Scenario Collaborations –Control System Collaborations Outlook

4. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 4 Existing Implementations Technical Realizations in the Past SNS TTF/ FLASH GAN/MVL Work in Progress LHC/CMS at Fermilab

5. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 5 Messages Alarms Control Systems – general Layout FEC BPM Magnets Master Timing LLRF Cavities Process Controller PLC Utilities Process Controller PLC Process Controller PLC Cryo Experiment Slow Control P T T Process Controller PLC Vacuum Camera Application Display Network Layer DAQ Remote Local Gateway Network Layer

6. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 6 Messages Alarms Examples for remote Operation: TTF FEC BPM Magnets Master Timing LLRF Cavities Process Controller PLC Vacuum Camera DAQ Remote Local Router X-Window Protocol APP ?

7. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 7 Examples for remote Operation: TTF Implementation: –X-Window Protocol Advantage: –Existing Technology Disadvantage: –Security Risk (Protocol) –Access only to local applications –Remote diagnostics with custom application not possible Lessons learned: –Remote Operation is feasible –Experts can participate remotely

8. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 8 Examples for remote Operation: GAN-MVL Camera Remote Local Instrument MVL Server Router Access Control

9. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 9 Examples for remote Operation: GAN-MVL Implementation: –GAN-MVL Projekt Group Advantage: –Full Access to Local Instrument –Interactive Video and Audio Support –(Limited) Access to local control system Disadvantage: –No transparent access to local control system Lessons learned: –Setting up a project group dealing with control system issues

10. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 10 Messages Alarms Examples for remote Operation: SNS Magnets Master Timing Cavities PLC Cryo P PLC Vacuum DAQ Remote Local Gateway Router FECLLRF Process Controller Process Controller commissioning Commissioning remotely Read access Copying ApplicationsRemote Read Access

11. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 11 Examples for remote Operation: SNS Implementation: –EPICS toolkit Advantage: –Full Access to local Input Output Controllers (IOCs) –Running the identical applications local and remote for commissioning Disadvantage: –Remote access using the native control system protocol opens a security whole which requires additional security mechanisms in addition to access control locally on the IOC –Only applicable in homogeneous control system environments Lessons learned: –Remote access to the local control system protocol eases commissioning and remote diagnostics

12. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 12 Messages Alarms Examples for remote Operation: LHC/CMS at Fermilab FEC BPM Magnets Master Timing LLRF Cavities Process Controller PLC Utilities Process Controller PLC Process Controller PLC Cryo Experiment Slow Control P T T Process Controller PLC Vacuum Camera DAQ Remote Local Router Authentication Server User-ID/ Role Database Equipment-Access/ Role- Database Introducing Common Authentication and Authorization Based on Common Protocol Authentication Server User-ID/ Role Database Equipment-Access/ Role- Database

13. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 13 LHC@FNAL Web Page

14. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 14 Unfortunate: CERN Cryo Status

15. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 15 If the contribution to the project from remote collaborators is exceeding a certain level, the commitment of the collaborating institutions beyond the construction phase in commissioning, and operation is mandatory, because of the host laboratory will not be able to handle the whole facility with its own staff. On the other hand, this commitment cannot be made by relocating the technical staff on the site of the accelerator Far remote operating ( operating in the widest sense, that is including running the accelerator, performing maintenance, trouble shooting and repairs, tuning-up the hardware systems, maintaining and managing spare inventory, pushing performance, ) is required The need for Far Remote Operating © F. Willeke (underscore by M.Clausen)

16. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 16 Collaboration of Equal Partners (no “host” laboratory but “near-by” laboratory) Facility far away from most Collaborating institutions Each collaborator responsible for major section of the machine incl. subsystems design, construction, commissioning, maintenance, trouble shooting, development Collaborators remain responsible for the part they contributed after construction Experts remain based at the home institution Most of the activities via remote operating and remote access Central Management responsible for the over-all issues, performance goals, design, interface, schedule, quality control, standards, infra structure, safety Operation centrally organized: planning & coordination, commissioning, operation, maintenance, machine development Operation performed by decentralized operations crews Model for a Remote Facility © F. Willeke

17. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 17 Messages Alarms Proposed Layout of a Global Collaborative Control System FEC BPM Magnets Master Timing LLRF Cavities Process Controller PLC Utilities Process Controller PLC Process Controller PLC Cryo Experiment Slow Control P T T Process Controller PLC Vacuum Camera DAQ Remote Local Gateway Router Replica No local consoles No direct internet access Authentication and Authorization Databases Remote Replica Heterogeneous Front End Controller Local Infrastructure Extendable Console Applications Common Protocol Router Authentication Server User-ID/ Role Database Equipment-Access/ Role- Database FECLLRF Process Controller Process Controller FEC Process Controller Process Controller Messages Alarms Master DAQ Gateway Router Authentication Server User-ID/ Role Database Equipment-Access/ Role- Database C

18. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 18 The Proposed Layout provides: Transparent access from local and remote sites –No ‘primus interparis’ Access control down to the equipment –Access roles and policies will be defined by all participating partners Heterogeneous contributions are (maybe not desirable) but possible In-kind contributions are fully supported

19. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 19 Basic requirements for individual components in the GCCS Client Applications –Running on all major operating systems -> Java –Reference Implementations for Authentication/ Authorization –Pluggable Applications –Support for –at least- the common control system protocol –Multi language support Authentication/ Authorization –Role based Authorization Scheme throughout the whole system down to the front end controller –Clean Interfaces to these services allow implementations on all platforms Network Protocol –Serving as the common ‘glue’ Alarm-/ Message Systems –Using well define Message Based Implementations Archiving/ DAQ –Supporting the common protocol Front End Systems –(Common) Generic configuration Tools –Support for –at least- the common control system protocol

20. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 20 The Control System Protocol (TCSP) How realistic is the development of an new common Control System Protocol? Status and Plans for Control System Protocols: System/ InstitutionProtocolPlansComment EPICSChannel AccessNew protocol for version 4.0 Depends on available funding TANGOCorbaInterested in new protocol TINE/ DOOCSMerged their protocols (TINE’) TCSP candidate? CERNCorbaNo known plans to change ITERTo be definedPartner for a new development?

21. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 21 Global Collaborative Control System Components ComponentStatusCandidate Client Application(s)Requirements are already fulfilled by current developments Starting from CSS(DESY) with an existing TINE’ plug Authentication/ AuthorizationExamples for role based authorization are available from several sites (CERN, DESY…) Check existing implementations. Define common interfaces and implementation for TCSP. Network ProtocolCollaborations are open for a change. A candidate is available. Develop TCSP by starting from TINE’. Alarm-/ Message SystemsMany sites are already using Java Message Systems (JMS). CERN, DESY(2x), SNS … JMS is a stable foundation. Agree on message structure Archiving/ DAQMany custom solutions are available… Pick one (new?) Front End SystemsThe ‘heart’ of each control system architecture. Preserve investment by adapting to the new protocol.

22. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 22 Setting up a Collaboration on The Control System Protocol (TCSP) The HEP community has a long tradition to set up collaborations. (e.g. HEP experiments or the next large machine) But How to set up a collaboration on control system developments?

23. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 23 Brief History of Control System Collaborations and Sharing Control System Software In 1993 Berend Kuiper presented his ideas to share control system software. –The ever lasting question how a magnet is controlled and how to model a power supply is symptomatic for the fundamental questions – how control system software can be shared. The discussion on a ‘software bus’ lead to an implementation by Chip Watson (cdev) in 1999. –The acceptance in the community was not high enough for a break through. It was just too easy to use straight channel access. Since then the EPICS collaboration was ever growing and became this way the biggest community sharing control system software. –The endless list of available drivers and the list of generic and specialized applications are two of the success factors for EPICS.

24. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 24 Factors to form Control System Collaborations Specialized control objects require device specific class libraries. This perfectly works in a single instance but does not match to many other installations. Record or device oriented implementations leave enough freedom to the software engineers to adapt it to their individual installation. Object orientation Collaborative lg 100 1

25. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 25 Collaborative Development Process Once a mutual agreement (between the HEP partners) on the basic functionality of common Control System Components has been reached… How can this be promoted in the collaborations? –There’s nobody to write a check to –There’s no way to implant a new functionalities without taking care of the interest of the whole collaboration. (Nobody is left behind) –It is a long process with one final goal: A Win/ Win situation for the promoting bodies as well as the collaboration members How would this work in the EPICS collaboration? (next page)

26. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 26 Collaborative Development Process – Example of a success story Development of a software package at DESY to support redundancy on front end controllers Preparation phase ( middle of 2006): –Define required redundancy functionality –New functionalities my not break existing code Start implementation based on existing EPICS version –Testing of first prototype –Parallel discussion with EPICS core developers how to add new features to the core code base Final tests of new version –Final discussion with core developers on integration Integration of redundancy features into EPICS core during an EPICS codeathon (EPICS core developers) in spring 2008 New EPICS version includes redundancy support  Clearly a Win/ Win situation:  EPICS collaboration: free access to redundancy support  DESY: can stay in synch with current and any new EPICS version Last EPICS meeting (Oct. 2008): –Presentation by KEK on a redundancy support for ATCA based on the DESY redundancy implementation  Additional payoff for the collaboration and for DESY The ideal case

27. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 27 How to start the collaborative development? The probability to agree on a common protocol has never been as good as these days (See discussion at last ICALEPCS) It is very likely to agree on common Control System Components since many labs already share the same technologies Client applications supporting multiple protocols are already available Get the key players/ collaborations involved Provide funding from major labs

28. Matthias Clausen, DESY ICFA Seminar, 29.October 2008 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 28 It is a good time for a …It is a good time for an initiative To form a Collaboration for a common Control System Protocol and other basic control system functions … … funded by current Projects (And Projects in the near Future) Outlook