2. SAM: Past, Present, and Future Lee Lueking All Dzero Meeting November 2, 2001

3. Lee Lueking - All Dzero Meeting2 SAM: Past, Present, and Future Part I: Past and Present 1.Stats: users,groups,datasets,projects,files. How is the system being utilized? 2.Cache and job management: How do the caching and fair share mechanisms work? 3.Central analysis groups and queues. 4.Tape access: What are encp stats for last month? Tapes, good, bad and recoverable. 5.Remote sites: data forwarding from remote MC processing centers Part II: Future (post shutdown) 1.New tape facilities 2.SAM on Farm and ClueD0 3.Storing user/group data into sam 4.Delivering data to remote sites 5.Problems and concerns

4. Part I: Past and Present

5. November 2, 2001Lee Lueking - All Dzero Meeting4 SAM Usage Statistics 428 registered SAM users in production 428 registered SAM users in production  283 of them have at some time run at least one SAM project  267 of them have run a SAM project at some time in the past year  181 of them have run a SAM project in the past 2 months 222 registered nodes 222 registered nodes 150,847 cached files on disk somewhere 150,847 cached files on disk somewhere  146,908 of them on d0mino  1299 on d0lxac1  2301 on a clued0 node  337 on imperial college test machine in the UK  503 on linux build machine 281,066 data files known to SAM 281,066 data files known to SAM  43,534 raw files (all stored on tape)  78,463 reconstructed files (76,305 of them actually stored)  19,700 root-tuple files

6. November 2, 2001Lee Lueking - All Dzero Meeting5 Active Stations Station Name Description protofarm Heidi’s protofarm Imperial Test Imperial College LancsLancaster Ccin2p3-analysisLyon Central-analysis FNAL D0 Analysis hoeve Nikhef farm Fnal-farm Real FNAL farm Clued0 Distributed analysis luhep Langston Oklahoma datalogger Online logger msu East Lansing Uta-hepArlington linux-analysis-cluster-1 Linux analysis cluster d0nevisColumbia D0small-01 Small linux station Prague-test-stationPrague

7. November 2, 2001Lee Lueking - All Dzero Meeting6 Central-analysis Cache All groups currently use Least Recently Used replacement algorithm All groups currently use Least Recently Used replacement algorithm Files can migrate from one group’s cache to another if used frequently by other group. Files can migrate from one group’s cache to another if used frequently by other group. Currently, caches are large and there is little turn over. Currently, caches are large and there is little turn over. Group Cache Allocation algo 100 GB cal 78 GB dzero 8 TB emid 10 GB thumbnail 50 GB trigsim 20 GB ttk1 2 TB

8. November 2, 2001Lee Lueking - All Dzero Meeting7 Central-analysis Cache Turn over

9. November 2, 2001Lee Lueking - All Dzero Meeting8 Resource Management Approaches Fair Sharing (policies) Fair Sharing (policies)  Allocation of resources and scheduling of jobs  The goal is to ensure that, in a busy environment, each group gets a fixed share of “resources” or gets a fixed share of “work” done Co-allocation and reservation (optimization) Co-allocation and reservation (optimization)

10. November 2, 2001Lee Lueking - All Dzero Meeting9 Fair Share and Computational Economy Jobs, when executed, incur costs (through resource utilization) and realize benefits (through getting work done) Jobs, when executed, incur costs (through resource utilization) and realize benefits (through getting work done) Maintain a tuple (vector) of cumulative costs/benefits for each group and compare them to its allocated fair share to set priority higher/lower Maintain a tuple (vector) of cumulative costs/benefits for each group and compare them to its allocated fair share to set priority higher/lower Incorporate all known resource types and benefit metrics, totally flexible. Examples:tape mounts, tape reads, network, cache, CPU, and memory. Incorporate all known resource types and benefit metrics, totally flexible. Examples:tape mounts, tape reads, network, cache, CPU, and memory.

11. November 2, 2001Lee Lueking - All Dzero Meeting10 Job Control: Station Integration with the Abstract Batch System Client Local RM (Station Master) Batch System Process Manager (SAM wrapper script) User Task Job Manager (Project Master) 2.submit to SM 4.submit To BS 6.dispatch8.invoke 5.Sam condition satisfied 10.resubmit 9.setJobCount/stop 3.invoke jobEnd Fair Share Job Scheduling Resource Co-allocation 1.user sam submit 7.Started

12. November 2, 2001Lee Lueking - All Dzero Meeting11 Replica Site WAN Data flow Station Mass Storage System User (producer) Forwarding + Caching = Global Replication NIKHEF (Amsterdam) 155 Mbps Sara Fermilab D0robot

13. November 2, 2001Lee Lueking - All Dzero Meeting12 Enstore Statistics: Delivery Start Date: "10/22/01 00:00:00" End Date: "10/29/01 00:00:00" Start Date: "10/22/01 00:00:00" End Date: "10/29/01 00:00:00" Delivered Files: 938 Total Delivered Files: 938 Total Delivered Bytes: 268.82 GB Delivered Bytes: 268.82 GB Average File Size: 293.47 +/- 107.66 MB Average File Size: 293.47 +/- 107.66 MB Average Delivery Time: 718.20 +/- 1017.20 s Average Delivery Time: 718.20 +/- 1017.20 s Average Queue Wait Time: 611.32 +/- 947.45 s Average Queue Wait Time: 611.32 +/- 947.45 s Average Mount Time: 3.25 +/- 13.45 s Average Mount Time: 3.25 +/- 13.45 s Average Seek Time: 24.07 +/- 42.30 s Average Seek Time: 24.07 +/- 42.30 s Average Transfer Time: 42.78 +/- 84.48 s Average Transfer Time: 42.78 +/- 84.48 s Average Transfer Rate: 9.10 +/- 2.24 MB/s Average Transfer Rate: 9.10 +/- 2.24 MB/s File Delivery Error Statistics Total Errors: 856 File Delivery Error Statistics Total Errors: 856 "USERERROR" Errors: 72 (8.41% of Total Errors) "USERERROR" Errors: 72 (8.41% of Total Errors) "NOACCESS" Errors: 675 (78.86% of Total Errors) "NOACCESS" Errors: 675 (78.86% of Total Errors) "NOTALLOWED" Errors: 109 (12.73% of Total Errors) "NOTALLOWED" Errors: 109 (12.73% of Total Errors)

14. November 2, 2001Lee Lueking - All Dzero Meeting13 Enstore Statistics: Store Start Date: "10/22/01 00:00:00" End Date: "10/29/01 00:00:00" Start Date: "10/22/01 00:00:00" End Date: "10/29/01 00:00:00" File Store Success Statistics Stored Files: 1622 File Store Success Statistics Stored Files: 1622 Total Stored Bytes: 514.27 GB Total Stored Bytes: 514.27 GB Average File Size: 324.67 +/- 231.03 MB Average File Size: 324.67 +/- 231.03 MB Average Delivery Time: 208.71 +/- 273.28 s Average Delivery Time: 208.71 +/- 273.28 s Average Queue Wait Time: 53.89 +/- 174.69 s Average Queue Wait Time: 53.89 +/- 174.69 s Average Mount Time: 8.34 +/- 18.85 s Average Mount Time: 8.34 +/- 18.85 s Average Seek Time: 34.98 +/- 52.26 s Average Seek Time: 34.98 +/- 52.26 s Average Transfer Time: 82.50 +/- 154.56 s Average Transfer Time: 82.50 +/- 154.56 s Average Transfer Rate: 4.37 +/- 2.11 MB/s Average Transfer Rate: 4.37 +/- 2.11 MB/s File Store Error Statistics Total Errors: 4 File Store Error Statistics Total Errors: 4 "USERERROR" Errors: 3 (75.00% of Total Errors) "USERERROR" Errors: 3 (75.00% of Total Errors) "EEXIST" Errors: 1 (25.00% of Total Errors) "EEXIST" Errors: 1 (25.00% of Total Errors)

15. November 2, 2001Lee Lueking - All Dzero Meeting14 Current Tape Storage Summary 45 TB on tape 45 TB on tape Total of 1362 volumes altogether Total of 1362 volumes altogether Currently there are 18 noaccess volumes Currently there are 18 noaccess volumes 80 notallowed 80 notallowed

16. Part II: The Future (post shutdown)

17. November 2, 2001Lee Lueking - All Dzero Meeting16 New Tape Facilities STK 9940 Drives STK 9940 Drives  Very reliable (no problems in 30 TB)  60 GB cartridge Share STK PowderHorn silo with other lab customers Share STK PowderHorn silo with other lab customers  have 6-7 x 9940 drives for us.  1000 tape slots In ~March, Move to our own PowderHorn In ~March, Move to our own PowderHorn  Space in FCC now being prepared  Robot already here  Deploy and test starting Jan- Feb. Dzero STK PowderHorn silo Dzero STK PowderHorn silo  have 9 x 9940 drives now, up to 20 drives.  5500 tape slots total.

18. November 2, 2001Lee Lueking - All Dzero Meeting17 Use Existing AML/2 for MC Replacing M2 drives with LTO. Replacing M2 drives with LTO. 100 GB cartridge 100 GB cartridge Have 6 drives, expand to 10 later. Have 6 drives, expand to 10 later. Very Reliable in tests so far (1 problem in 30 TB) Very Reliable in tests so far (1 problem in 30 TB) Plan to use for all MC and some Group data Plan to use for all MC and some Group data

19. November 2, 2001Lee Lueking - All Dzero Meeting18 SAM Distributed Cache Fnal-farm Fnal-farm ClueD0 ClueD0

20. November 2, 2001Lee Lueking - All Dzero Meeting19 Enstore Mass Storage Case Study:Distributed Reconstruction Farm Worker 1 Worker 2 Worker 3 Worker N D0bbin Farm Server No disks are cross mounted. Worker nodes get files directly from MSS via encp. Data is moved by SAM using rcp from where it is cached to where it is needed. 90 dual processor Linux nodes (growing) 30 GB disk each 100 Mbit ethernet NICs on workers D0bbin is 4 processor SGI O2000, Gigabit NIC LAN

21. November 2, 2001Lee Lueking - All Dzero Meeting20 Case Study:Distributed Analysis Cluster ClueD0 Desktop 1 Desktop 2 Desktop 3 Desktop 100+ Mass Storage Clued0-ripon (file server node) ClueD0-ripon node has 640 GB SAM cache disk 100+ linux desktop nodes have 4-5TB distributed SAM cache 5 nodes in SAM mode now All (tape) data enters the ClueD0 station through the main file server node ClueD0-ripon. The station migrates data as needed and manages the cache distributed among the many desktop constituents.

22. November 2, 2001Lee Lueking - All Dzero Meeting21 Storing Group Data in SAM Each group will have tapes allocated for specific tiers of data: gen, d0gstar, d0sim, reconstructed, root-tuples, others. Each group will have tapes allocated for specific tiers of data: gen, d0gstar, d0sim, reconstructed, root-tuples, others. Each group will have a tape allocation limit Each group will have a tape allocation limit Group data will be added with special tier designation “-bygroup” to distinguish it form farm and other production data. Group data will be added with special tier designation “-bygroup” to distinguish it form farm and other production data. Document describing details available under sam documentation “Storing Group Data into SAM”. Document describing details available under sam documentation “Storing Group Data into SAM”. Groups set up so far include top, higgs, and tauid. Groups set up so far include top, higgs, and tauid.

23. November 2, 2001Lee Lueking - All Dzero Meeting22 Replica Site WAN Data flow Station Mass Storage System User (producer) Routing + Caching = Global Replication

24. November 2, 2001Lee Lueking - All Dzero Meeting23 Issues Tape problems should be under control Tape problems should be under control CORBA naming server has caused problems in past. We are testing a new naming service with persistency that should resolve this. Plan to deploy this month. CORBA naming server has caused problems in past. We are testing a new naming service with persistency that should resolve this. Plan to deploy this month. Some queries have caused the system to jam. We have split user db server away from the dbserver for the stations. Looking into how to deal with long (usually event picking) queries. Some queries have caused the system to jam. We have split user db server away from the dbserver for the stations. Looking into how to deal with long (usually event picking) queries. User support is sometimes slower than people like: User support is sometimes slower than people like:  We are training many Dzero volunteers to help  Lauri is available at Dzero every Wednesday on DAB5 (my office). She has not been overwhelmed by walk-ins.

25. November 2, 2001Lee Lueking - All Dzero Meeting24 Conclusion Sam is heavily used by D0 Sam is heavily used by D0 The Cache management and Fair share resource allocations are designed to help control the use of resources in the system. The Cache management and Fair share resource allocations are designed to help control the use of resources in the system. SAM provides easy storage of data for on-site and off-site production customers. SAM provides easy storage of data for on-site and off-site production customers. In spite of many tape problems, the Ensore system has been storing and serving lots of data. In spite of many tape problems, the Ensore system has been storing and serving lots of data.

26. November 2, 2001Lee Lueking - All Dzero Meeting25 Conclusion (2) The new Tape and Robot technologies will make the tape- based data storage and access extremely reliable. The new Tape and Robot technologies will make the tape- based data storage and access extremely reliable. SAM provides a framework within which to operate distributed processing and analysis clusters. These will be very important in the future. SAM provides a framework within which to operate distributed processing and analysis clusters. These will be very important in the future. We are ready to store group data into the system on a regular basis. We are ready to store group data into the system on a regular basis. Delivery of data to remote stations from robot stores is coming. Delivery of data to remote stations from robot stores is coming. We have addressed, and continue to address many issues to make the system serve Dzero better than ever. We have addressed, and continue to address many issues to make the system serve Dzero better than ever.