1. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 20081 Computing Resources for DAQ & Online Amedeo Perazzo Photon Controls and Data Systems Online Manager SLAC, November 12 th 2008 LCLS FAC - Controls Breakout Session

2. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 20082 Contents Network organization Services Machines which provide services to user workstations, DAQ & control nodes Users Development, internet access DAQ Machines used for moving data from FEE to online data cache Controls Machines used for controlling all slow devices Data Cache Temporary storage for online data analysis and as buffer between DAQ and offline

3. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 20083 Network Zones Photon Controls & Data Systems (PCDS) network organized in 2 zones Back-end: provides networking services to the PCDS enclave Front-end: control and data acquisition traffic Network organization driven by new DOE security rules Back End Zone: provides the infrastructure for all service traffic Divided into six subnets: dmz : limited access from SLAC machines usr : user development and Internet access nodes service : NFS, DNS, NTP and AAA server nodes mgmt : utility nodes (terminal servers, shelf managers, etc...)‏ cds : service subnet for Control & DAQ nodes dss : service subnet for the Data Storage machines Must allow Control & DAQ to be operational, for limited amount of time, when connection to SLAC domain is down

4. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 20084 SLAC Domain Back End Zone Diagram DSS Service DMZ User Science bulk data CDS Service Traffic Data cache machines Control & DAQ nodes NFS, DNS, NTP, AAA Front End Zone Accelerator Domain MGMT

5. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 20085 Front End Zone Front End Zone: provides the infrastructure for the control traffic and the data acquisition traffic Divided into three networks: daq : science data, partition management, run monitoring and telemetry traffic DAQ operator consoles (L0), readout nodes (L1), processing nodes (L2) and data cache machines (L3)‏ epics : control traffic control operator consoles (E0), IOCs (E1), EPICS archiver (E3) and channel access gateway bld : low latency 120 Hz beam-line data traffic DAQ network further subdivided into experiment specific subnets: daq_amo, daq_xpp, daq_cxi, etc EPICS network further subdivided into: epics_amo, epics_xpp, epics_mcc, epics_xtod, etc Selected EPICS traffic may be exchanged between the different subnets via the channel access gateway

6. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 20086 Network Devices for Services‏ CISCO Catalyst 6509 720 Gbps switch fabric backplane 9 slots (currently: 1 supervisor, 1 x 48 1Gb RJ45, 1 x 24 SFPs)‏ (+)‏

7. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 20087 Controls‏ CISCO 3560 48 1Gb RJ45 + 4 SFPs (+)‏ Motorola MVME6100 MPC7457 PPC 1.2 GHz (+)‏ DELL R200 1U server 2 cores, 1 PCIe, 1 PCI-X (+)‏ ELMA VME chassis Redundant PWS, shelf manager 4, 8 and 21 slots (+)‏

8. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 20088 Network Devices for DAQ Cluster Interconnect Module (CIM) SLAC custom made ATCA switch Based on two 24-port 10Gb Ethernet switch ASICs from Fulcrum Up to 480 Gb/s total bandwidth Fully managed layer-2, cut-through switch (+)‏ ATCA chassis 5 slots, shelf manager (+)‏ 14 slots, shelf manager

9. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 20089 DAQ Nodes SLAC ATCA RCE boards 2 cores, 8 PGP lanes, 2x10Gb/s ethernet fabric interface, 2x1Gb/s ethernet base interface (+)‏ ATCA blades 8 cores, 16GB, dual 10Gb/s ethernet fabric interface, dual 1Gb/s ethernet base interface 2 AMC slots cPCI Concurrent PP512 4 cores, 4GB, 3x1Gb/s ethernet interfaces 2 PMC/XMC slots (+)‏

10. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 200810 Servers Services NFS, DNS, LDAP, NTP, AAA, logging, etc Supermicro 2U (8 cores, 32GB, 1TB SATA, redundant PWS)‏ Hot spare with automatic failover EPICS CAG, EPICS DB, EPICS archiver, etc Supermicro 2U (8 cores, 32GB, 1TB SATA, 8 NICs, redundant PWS)‏ Hot spare with automatic failover

11. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 200811 Data Cache RAID Server Supermicro 4U (8 cores, 32GB, 1TB SATA, 24TB SAS through hardware RAID SAS controller, redundant PWS)‏ In the beginning 2 servers are envisioned (+)‏ When one server is writing, the other is reading

12. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 200812 Users Consoles DELL OptiPlex with dual 20” monitor Development DELL Precision with 20” monitor

13. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 200813 Public Networks Not part of the PCDS enclave Desktops Part of SLAC network Two CISCO 3750 in NEH telecom room (+)‏ Dedicated redundant 1Gb/s link between telecom room and SCCS Network devices maintained by SCCS Wireless Part of SLAC visitor network 3 access points in basement floor, 3 in sub-basement, 2 in FEE (+)‏ Connected to switch in NEH telecom room Maintained by SCCS

14. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 200814 Status Built prototype for NEH online computing in bldg 84 lab Use same routing and firewall rules as for NEH Created/tested set of tools which will be used in NEH for: Maintaining service machines (NFS, NTP, DNS, AAA servers etc)‏ Synchronization with SCCS, backup, patching, logging Maintaining utility nodes (shelf managers, terminal servers, etc)‏ Manage connections inside and outside PCDS enclave Test-stand setup allows small scale testing of feasibility of network plan Will speed up commissioning in NEH Man power for online computing effort during 2008 1.5 FTE Shared among 2 PCDS people and 2 SCCS people Similar effort expected for 2009 Expenses for 2008 along expectations Enough resources allocated for 2009 Big ticket item L2 event building and processing farm

15. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 200815 Computational Alignment  k in k out q “The number currently used to obtain high-resolution structures of specimens prepared as 2D crystals, is estimated to require at least 10 17 floating-point operations” R. M. Glaeser, J. Struct. Bio. 128, (1999) ‏ Experimental Data (ALS) ‏ Difference of pyramid diffraction patterns 10º apart, Gösta Huldt, U. Uppsala “Computational Alignment” requires large computational power that might only be provided by performing offline analysis?  Save first, and Analyze later? To be investigated

16. Amedeo Perazzo Online Computing Resourcesperazzo@slac.stanford.edu November 12 th, 200816 Real-time Processing – Sorting in CXI Diffraction from a single molecule: single LCLS pulse noisy diffraction pattern of unknown orientation Combine 10 5 to 10 7 measurements into 3D dataset: Classify/sortAverageAlignment Reconstruct by Oversampling phase retrieval Miao, Hodgson, Sayre, PNAS 98 (2001) ‏ unknown orientation Gösta Huldt, Abraham Szöke, Janos Hajdu (J.Struct Biol, 2003 02-ERD-047) ‏ The highest achievable resolution is limited by the ability to group patterns of similar orientation Real-time?