1. DAQ control system for multi-beamline simultaneous experiments at SACLA Takashi SUGIMOTO Japan Synchrotron Radiation Research Institute (JASRI/SPring-8) 1

2. SPring-8 and SACLA 2

3. SPring-8: Light source complex 3 Electron Injector (1-GeV Linac and 8-GeV booster synchrotron) New Subaru (1.5-GeV Storage Ring) SPring-8 (8-GeV Storage Ring) SACLA (X-ray Free Electron Laser Facility)

4. SACLA Beamlines 4 XFEL MOM305, HOSODA et al., “A Control System for a Dedicated Accelerator for SACLA Wide-band Beam Line” FEL BL3 and BL2 are in operation. BL3/BL2 fast-switched XFEL distribution has just started in FY2015. BL1 is under commissioning. Operation time is 7,000 hour/year.

5. Overview of SPring-8 Campus Network 5

6. Machine Control Research (Development and Experiment) Research (Development and Experiment) General Purpose DMZ SPring-8 Campus Network 6 Data Analysis Internet

7. Machine Control Research (Development and Experiment) Research (Development and Experiment) General Purpose DMZ SPring-8 Campus Network 7 SPring-8 Control SPring-8 Control SACLA Control SACLA Control Software Developme nt SPring-8 Experiment SPring-8 Experiment SACLA Experiment SACLA Experiment SACLA Data Acquisition SACLA Data Acquisition Data Analysis Office Network Public Servers HPC System Defense-in-depth oriented network system (2010-) VPN IPS Internet Router w/ ACL

8. Machine Control Research (Development and Experiment) Research (Development and Experiment) General Purpose DMZ SPring-8 Campus Network 8 SPring-8 Control SPring-8 Control SACLA Control SACLA Control Software Developme nt SPring-8 Experiment SPring-8 Experiment SACLA Experiment SACLA Experiment SACLA Data Acquisition SACLA Data Acquisition Data Analysis Office Network Public Servers HPC System Defense-in-depth oriented network system (2010-) VPN 3rd (CS) 2 /HEP, 2011, Grenoble IPS Router w/ ACL Internet

9. SPring-8 Experiment Network 9 Presented on 3rd (CS)2/HEP Workshop, 2011

10. Office Network History of SPring-8 Experiment Network (1of3) 10 SPring-8 Control SPring-8 Control SPring-8 Experiment SPring-8 Experiment Internet In early days of SPring-8 (1997-1998), the experimental network is a part of office network without any access control. Public Servers

11. Office Network History of SPring-8 Experiment Network (1of3) 11 SPring-8 Control SPring-8 Control SPring-8 Experiment SPring-8 Experiment Internet In early days of SPring-8 (1997-1998), the experimental network is a part of office network without any access control. In 1990s, number of network-connected PCs increased. Anyone (with/without evil intention) in office can control experimental instruments. Public Servers

12. History of SPring-8 Experiment Network (2of3) 12 SPring-8 Control SPring-8 Control SPring-8 Experiment SPring-8 Experiment Office Network Internet In 1998, the experimental network is segregated using firewall(s). Public Servers

13. History of SPring-8 Experiment Network (2of3) 13 SPring-8 Control SPring-8 Control SPring-8 Experiment SPring-8 Experiment Office Network Internet In 1998, the experimental network is segregated using firewall(s). Public Servers However, because of such historical reason, the experiment network have internet connectivity.

14. History of SPring-8 Experiment Network (2of3) 14 SPring-8 Control SPring-8 Control SPring-8 Experiment SPring-8 Experiment Office Network Internet In 1998, the experimental network is segregated using firewall(s). Public Servers However, because of such historical reason, the experiment network have internet connectivity. With internet connectivity, recent VPN technologies can pass through the firewalls. (Unmanaged VPNs: e.g. TeamViewer, splashtop, …)

15. History of SPring-8 Experiment Network (3of3) 15 SPring-8 Control SPring-8 Control SPring-8 Experiment SPring-8 Experiment Office Network Public Servers Internet We installed “next-generation firewall” in 2010, to block such unmanaged VPNs.

16. History of SPring-8 Experiment Network (3of3) 16 SPring-8 Control SPring-8 Control SPring-8 Experiment SPring-8 Experiment Office Network Public Servers Internet We installed “next-generation firewall” in 2010, to block such unmanaged VPNs. When malware infected PC is connected to the experiment network, the NG-FW can block the C & C traffic. C & C Server

17. 17

18. History of SPring-8 Experiment Network (3of3) 18 SPring-8 Control SPring-8 Control SPring-8 Experiment SPring-8 Experiment Office Network Public Servers Internet We installed “next-generation firewall” in 2010, to block such unmanaged VPNs. When malware infected PC is connected to the experiment network, the NG-FW can block the C & C traffic. Why do we install such a expensive firewalls? Experimental network must have internet connectivity to transfer data to office and/or other institutes. (Historical reason)

19. History of SPring-8 Experiment Network (3of3) 19 SPring-8 Control SPring-8 Control SPring-8 Experiment SPring-8 Experiment Office Network Public Servers Internet We installed “next-generation firewall” in 2010, to block such unmanaged VPNs. When malware infected PC is connected to the experiment network, the NG-FW can block the C & C traffic. Why do we install such a expensive firewalls? Experimental network must have internet connectivity to transfer data to office and/or other institutes. (Historical reason) It is difficult to catch up the newest technologies and vulnerabilities.  We made tight security policy at the SACLA.

20. SACLA Network System 20

21. Machine Control Research (Development and Experiment) Research (Development and Experiment) General Purpose DMZ SPring-8 Campus Network 21 SPring-8 Control SPring-8 Control SACLA Control SACLA Control Software Developme nt SPring-8 Experiment SPring-8 Experiment SACLA Experiment SACLA Experiment SACLA Data Acquisition SACLA Data Acquisition Data Analysis Office Network Public Servers HPC System VPN Today’s Topic IPS Router w/ ACL Internet

22. Summary SACLA Experimental Network Policy 1.No internet connectivity 2.Segregate LANs based on purpose 3.Logically segmented by experimental area/unit, to perform multi-beamline experiments 4.Physically segmented by beamlines to guaranty DAQ performance 22

23. 1. No Internet Connectivity 23 SACLA Control SACLA Control SACLA Experiment SACLA Experiment SACLA Data Acquisition SACLA Data Acquisition HPC System Router w/ ACL Data transfer servers At first, we decided to have no internet connectivity at the SACLA network systems. Dedicate servers take charge of data transfer to other institutes. Internet By segregating network systems from the Internet, experimental systems are prevented from recent vulnerabilities which utilize the internet.

24. 1. No Internet Connectivity 24 SACLA Control SACLA Control SACLA Experiment SACLA Experiment SACLA Data Acquisition SACLA Data Acquisition HPC System Router w/ ACL Data transfer servers

25. SACLA Control SACLA Experiment SACLA Data Acquisition 2. Segregate LANs based on purpose 25 Camera DAQ ServersStorage User Workstation (User-WS) Undulator Shutter Focusing Shutter Sample Changer Beamline Master Server (BL-Master) Beamline Workstation (BL-WS)

26. 2. Segregate LANs based on purpose 26 Camera DAQ ServersStorage Undulator Shutter Focusing Shutter Sample Changer SACLA Control SACLA Experiment SACLA Data Acquisition User Workstation (User-WS) Beamline Master Server (BL-Master) Beamline Workstation (BL-WS) BL-WS is accountable for X-ray optics control.

27. 2. Segregate LANs based on purpose 27 Camera DAQ ServersStorage Undulator Shutter Focusing Shutter Sample Changer SACLA Control SACLA Experiment SACLA Data Acquisition User Workstation (User-WS) Beamline Master Server (BL-Master) Beamline Workstation (BL-WS) BL-WS is accountable for X-ray optics control. BL-Master is accountable for DAQ system and movers. (e.g. sample changer)

28. 2. Segregate LANs based on purpose 28 Camera DAQ ServersStorage Undulator Shutter Focusing Shutter Sample Changer SACLA Control SACLA Experiment SACLA Data Acquisition BL-Master is accountable for DAQ system and movers. (e.g. sample changer) User Workstation (User-WS) Beamline Master Server (BL-Master) Beamline Workstation (BL-WS) BL-WS is accountable for X-ray optics control. Experimental users send control message to BL-Master. (e.g. start DAQ, sample change, open shutter…)

29. 2. Segregate LANs based on purpose 29 Camera DAQ ServersStorage Undulator Shutter Focusing Shutter Sample Changer SACLA Control SACLA Experiment SACLA Data Acquisition BL-Master is accountable for DAQ system and movers. (e.g. sample changer) User Workstation (User-WS) Beamline Master Server (BL-Master) Beamline Workstation (BL-WS) BL-WS is accountable for X-ray optics control. If instrument belongs to SACLA Control, BL- Master forward control message to BL- WS.(e.g. open shutter) Experimental users send control message to BL-Master. (e.g. start DAQ, sample change, open shutter…)

30. 2. Segregate LANs based on purpose 30 Camera DAQ ServersStorage Undulator Shutter Focusing Shutter Sample Changer SACLA Control SACLA Experiment SACLA Data Acquisition BL-Master is accountable for DAQ system and movers. (e.g. sample changer) User Workstation (User-WS) Beamline Master Server (BL-Master) Beamline Workstation (BL-WS) BL-WS is accountable for X-ray optics control. If instrument belongs to SACLA Control, BL- Master forward control message to BL- WS.(e.g. open shutter) Experimental users send control message to BL-Master. (e.g. start DAQ, sample change, open shutter…)

31. 2. Segregate LANs based on purpose 31 Camera DAQ ServersStorage Undulator Shutter Focusing Shutter Sample Changer SACLA Control SACLA Experiment SACLA Data Acquisition BL-Master is accountable for DAQ system and movers. (e.g. sample changer) User Workstation (User-WS) Beamline Master Server (BL-Master) Beamline Workstation (BL-WS) BL-WS is accountable for X-ray optics control. If instrument belongs to SACLA Control, BL- Master forward control message to BL- WS.(e.g. open shutter) Direct access from User-WS to BL-WS is forbidden. The reason are 1. To ensure control system consistency, such as status. 2. Single UI provided by BL-Master for controlling all experimental instruments. Experimental users send control message to BL-Master. (e.g. start DAQ, sample change, open shutter…)

32. SACLA Experimental Hall 32 XFEL FEL Each beamline passing several experimental hutches (EHs). By dividing EHs, experiments and preparation will be performed simultaneously.

33. EH5EH4 EH2 EH1 EH3 History and Current Status of SACLA Beamlines 33 BL3 is the first beamline, which delivers XFEL to users. (2011) BL3 EH: Experimental Hutch

34. EH5 EH6 EH4c EH2 EH1 EH4b EH3 History and Current Status of SACLA Beamlines 34 Fast-switched XFEL distribution BL2 is in operation. (2014) Fast-switched XFEL distribution to BL2/BL3 is started. (2015) BL3 BL2 BL3 is the first beamline, which delivers XFEL to users. (2011) EH: Experimental Hutch

35. EH5 EH6 EH4c EH2 EH1 EH4b EH3 EH4a History and Current Status of SACLA Beamlines 35 Fast-switched XFEL distribution Dedicated Accelerator BL3 BL2 BL1 BL2 is in operation. (2014) Fast-switched XFEL distribution to BL2/BL3 is started. (2015) BL3 is the first beamline, which delivers XFEL to users. (2011) BL1 offers more experimental opportunity to users, especially from EUV to soft X-ray FEL. (2016) EH: Experimental Hutch commissioning

36. EH5 EH6 EH4c EH2 EH1 EH4b EH3 EH4a History and Current Status of SACLA Beamlines 36 BL3 BL2 BL1 BL2 is in operation. (2014) Fast-switched XFEL distribution to BL2/BL3 is started. (2015) BL3 is the first beamline, which delivers XFEL to users. (2011) BL1 offers more experimental opportunity to users, especially from EUV to soft X-ray FEL. (2016) EH: Experimental Hutch Under such circumstance, multi groups perform experiments and preparations, simultaneously. Preparation Experiment Preparation

37. EH5 EH6 EH4c EH2 EH1 EH4b EH3 EH4a History and Current Status of SACLA Beamlines 37 BL3 BL2 BL1 BL2 is in operation. (2014) Fast-switched XFEL distribution to BL2/BL3 is started. (2015) BL3 is the first beamline, which delivers XFEL to users. (2011) BL1 offers more experimental opportunity to users, especially from EUV to soft X-ray FEL. (2016) EH: Experimental Hutch Under such circumstance, multi groups perform experiments and preparations, simultaneously.  Security issue occurred Preparation Experiment Preparation

38. EH5 EH6 EH4c EH2 EH1 EH4b EH3 Security Issue: wrong shutter operation 38 BL3 BL2 Preparation Experiment Group-A Group-B During machine time of group-A, Shutter CLOSED ??? shutter closed suddenly.

39. EH5 EH6 EH4c EH2 EH1 EH4b EH3 Security Issue: wrong shutter operation 39 BL3 BL2 Preparation Experiment Group-A Group-B During machine time of group-A, shutter closed suddenly. Shutter ??? Trouble source is control message from EH5 to BL-Master. BL-Master (BL2) BL-Master (BL3) CLOSED

40. EH5 EH6 EH4c EH2 EH1 EH4b EH3 Security Issue: wrong shutter operation 40 BL3 BL2 Preparation Experiment Group-A Group-B During machine time of group-A, shutter closed suddenly. Shutter OPEN Trouble source is control message from EH5 to BL-Master. BL-Master (BL2) BL-Master (BL3) Another situation: People are in EH2 to check instruments. It might be a radiation accident. !!! (This situation does not occurred actually. People are protected by interlock system.)

41. Problem: Network Segment 41 EH5 EH6 EH4c EH2 EH1 EH4b EH3 BL3 BL2 BL-Master (BL3) BL-Master (BL2) All of EHs shared one network segment. (This design is same as SPring-8 experiment network) Since IP address of User-WS/carry-in PCs was arbitrary, the BL-Master can not distinguish message authority: “where this message come from?” User WS ???

42. 3. Logically segmented by experimental area/unit 42 EH5 EH6 EH4c EH2 EH1 EH4b EH3 BL3 BL2 BL-Master (BL3) BL-Master (BL2) In 2014, we divided network segments corresponding to each EHs. User WS BL3-EH1 BL3-EH2BL3-EH4BL3-EH5 BL2-EH3 BL2-EH4 BL2-EH6 172.30.96.0/23 172.30.98.0/23 172.30.102.0/23 172.30.112.0/23 172.30.68.0/23 172.30.70.0/23 172.30.82.0/23

43. MADOCA2 Message Routing with ACL 43 object namemanagement method account@hostname sr_ms_serveMS*@* sr_ms_manageMScontrol@localhost object_fwd1MS:host-1*@* object_fwd2MS:host-1,host-2*@* object_ip1MS*@172.24.12.15 object_ip2MS*@172.24.12.0/24 WEPGF107, MATSUMOTO et al., “Multi-host Message Routing in MADOCA II” We can use IP/subnet as ACL keys

44. 3. Logically segmented by experimental area/unit 44 EH5 EH6 EH4c EH2 EH1 EH4b EH3 BL3 BL2 BL-Master (BL3) BL-Master (BL2) User WS BL3-EH1 BL3-EH2BL3-EH4BL3-EH5 In 2014, we divided network segments corresponding to each EHs. Using MADOCA2, the BL-Master distinguish message authority, like these: BL3-Master: Accept message from BL3-EH2 “drives motor in BL3-EH2”

45. 3. Logically segmented by experimental area/unit 45 EH5 EH6 EH4c EH2 EH1 EH4b EH3 BL3 BL2 BL-Master (BL3) BL-Master (BL2) User WS BL3-EH1 BL3-EH2BL3-EH4BL3-EH5 In 2014, we divided network segments corresponding to each EHs. Using MADOCA2, the BL-Master distinguish message authority, like these: BL3-Master: Accept message from BL3-EH2 “drives motor in BL3-EH2” BL3-Master: Discard message from BL3-EH5 “open shutter of BL3-EH1”

46. 3. Logically segmented by experimental area/unit 46 EH5 EH6 EH4c EH2 EH1 EH4b EH3 BL3 BL2 BL-Master (BL3) BL-Master (BL2) User WS BL3-EH1 BL3-EH2BL3-EH4BL3-EH5 In 2014, we divided network segments corresponding to each EHs. Using MADOCA2, the BL-Master distinguish message authority, like these: BL3-Master: Accept message from BL3-EH2 “drives motor in BL3-EH2” BL3-Master: Discard message from BL3-EH5 “open shutter of BL3-EH1” BL3-Master: Accept message from BL3-EH5 “drives motor in BL3-EH5”

47. 3. Logically segmented by experimental area/unit 47 EH5 EH6 EH4c EH2 EH1 EH4b EH3 BL3 BL2 BL-Master (BL3) BL-Master (BL2) In 2014, we divided network segments corresponding to each EHs. User WS BL3-EH1 BL3-EH2BL3-EH4BL3-EH5 Firewall offers more essential filters. “Drop packets from BL3-EH5 to BL2-Master” Using MADOCA2, the BL-Master distinguish message authority, like these: BL3-Master: Accept message from BL3-EH2 “drives motor in BL3-EH2” BL3-Master: Discard message from BL3-EH5 “open shutter of BL3” BL3-Master: Accept message from BL3-EH5 “drives motor in BL3-EH5”

48. 4. Physically segmented by beamlines 48 MPCCD SACLA BL2 Data handling Servers Tier-1 Storage Experimental Hall Computer Room SACLA BL3 Data Flow Each beamline occupies dedicate physical network from detector frontend to Tier- 1 storage. The data-handling servers are used for buffering (several seconds). In addition, on-the-fly low-level filtering are performed using the data-handling servers. ICALEPCS2013 TUPPC015, San Francisco

49. EH5 EH6 EH4c EH2 EH1 EH4b EH3 EH4a Summary 49 Fast-switched XFEL distribution Dedicated Accelerator BL3 BL2 BL1 commissioning SACLA Experimental Network Policy 1.No internet connectivity 2.Segregate LANs based on purpose 3.Logically segmented by experimental area/unit, to perform multi- beamline experiments 4.Physically segmented by beamlines to guaranty DAQ performance We are ready to perform multi-beamline simultaneous experiments at BL1/BL2/BL3 (and BL4/BL5 in the future).