1. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues1 Gamma-ray Large Area Space Telescope GLAST Large Area Telescope ISOC Peer Review Section 4.3 Pipeline, Data Storage and Networking Issues for the ISOC Richard Dubois SAS System Manager

2. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues2 Outline SAS Summary Requirements Pipeline Requirements Processing database Prototype status Networking Proposed Network Topology Network Monitoring File exchange Security Data Storage and Archive

3. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues3 Level III Requirements Summary Ref: LAT-SS-00020 Basic requirements are to routinely handle ~10 GB/day in multiple passes coming from the MOC @ 150 Mb/s – 2 GB should take < 2 mins Outgoing volume to SSC is << 1 GB/day NASA 2810 Security Regs – normal security levels for IOC’s; as practiced by computing centers already

4. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues4 Pipeline Spec Function –The Pipeline facility has five major functions automatically process Level 0 data through reconstruction (Level 1) provide near real-time feedback to IOC facilitate the verification and generation of new calibration constants produce bulk Monte Carlo simulations backup all data that passes through Must be able to perform these functions in parallel Fully configurable, parallel task chains allow great flexibility for use online as well as offline –Will test the online capabilities during Flight Integration The pipeline database and server, and diagnostics database have been specified (will need revision after prototype experience!) –database: LAT-TD-00553LAT-TD-00553 –server: LAT-TD-00773LAT-TD-00773 –diagnostics: LAT-TD-00876LAT-TD-00876

5. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues5 Expected Capacity We routinely made use of 100-300 processors on the SLAC farm for repeated Monte Carlo simulations, lasting weeks –Expanding farm net to France and Italy –Unknown yet what our MC needs will be –We are very small compared to our SLAC neighbour BABAR – computing center sized for them 2000-3000 CPUS; 300 TB of disk; 6 robotic silos holding ~30000 200 GB tapes total –SLAC computing center has guaranteed our needs for CPU and disk, including maintenance for the life of the mission. –Data rate small compared to already demonstrated MC capability ~75 of today’s CPUs to handle 5 hrs of data in 1 hour @ 0.15 sec/event Onboard compression may make it 75 of tomorrow’s CPUs too

6. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues6 Pipeline in Pictures State machine + complete processing record Configurable linked list of applications to run Expandable and configurable set of processing nodes

7. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues7 Processing Dataset Catalogue Datasets grouped by task Datasets info is here Processing records

8. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues8 First Prototype - OPUS Open source project from STScI In use by several missions Now outfitted to run DC1 dataset OPUS Java mangers for pipelines

9. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues9 Disk and Archives We expect ~10 GB raw data per day and assume comparable volume of events for MC –Leads to ~50 TB over 5 years for all data types No longer frightening – keep it all on disk –Use SLAC’s mstore archiving system to keep a copy in the silo Already practicing with it and will hook it up to OPUS –Archive all data we touch; track in dataset catalogue –Not an issue

10. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues10 Network Path: SLAC-Goddard ٭ ٭ ٭ ٭ ٭ SLAC Stanford Oakland (CENIC) LA UC-AID (Abilene) Houston Atlanta Washington GSFC (77 ms ping)

11. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues11 ISOC Stanford/SLAC Network SLAC Computing Center –OC48 connection to outside world –provides data connections to MOC and SSC –hosts the data and processing pipeline –Transfers MUCH larger datasets around the world for BABAR –World renowned for network monitoring expertise Will leverage this to understand our open internet model –Sadly, a great deal of expertise with enterprise security as well Part of ISOC expected to be in new Kavli Institute building on campus –Connected by fiber (~2 ms ping) –Mostly monitoring and communicating with processes/data at SLAC

12. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues12 Network Monitoring Need to understand failover reliability, capacity and latency

13. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues13 LAT Monitoring Monitoring nodes at most LAT collaborating institutions LAT Monitoring Keep track of connections to collaboration sites Alerts if they go down Fodder for complaints if poor connectivity

14. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues14 File Exchange Secure –No passwords in plain text etc Reliable –Has to work > 99% of the time (say) handle the (small) data volume –order 10 GB/day from Goddard (MOC); 0.3 GB/day back to Goddard (SSC) –keep records of transfers –database records of files sent and received –handshakes –both ends agree on what happened –some kind of clean error recovery –Notification sent out on failures Web interface to track performance Are investigating DTS now –Not super happy with how it’s built as far as reliability and security go –Installing it now –Starting to work with author on issues To be captured in an ICD with the SSC prior to DC2

15. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues15 Security Network security – application vs network –ssh/vpn among all sites – MOC, SSC and internal ISOC –A possible avenue is to make all applications secure (ie encrypted), using SSL. File and Database security –Controlled membership in disk ACLs –Controlled access to databases –Depend on SLAC security otherwise

16. GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR-03213-01 Section 4.3 Pipeline, Data Storage and Networking Issues16 Summary We are testing out the OPUS pipeline as our first prototype –Already outfitted to run DC1 dataset –Interfaces to processing database and SLAC batch done –Practice with DCs and Flight Integration We expect to need O(50 TB) of disk and ~2-3x that in tape archive –Not an issue We expect to use Internet2 connectivity for reliable and fast transfer of data between SLAC and Goddard –Transfer rates of > 150 Mb/s already demonstrated –< 2 min transfer for standard downlink. More than adequate. –Starting a program of routine network monitoring to practice Network security is an ongoing, but largely solved, problem –There are well-known mechanisms to protect sites –We will leverage considerable expertise from the SLAC and Stanford networking/security folks