1. The RAL PPD Tier 2/3 Current Status and Future Plans or “Are we ready for next year?” Chris Brew PPD Christmas Lectures 2007 17 th December 2007

2. 2 17/12/2007 PPD Christmas Lectures 2007 Contents What’s Tier 2 or Tier 3? Tier 2 and Tier 3 Hardware Staff Changes over the last year Some (new) details of the Tier 3 Service

3. 3 17/12/2007 PPD Christmas Lectures 2007 Tier 2 vs. Tier 3 The RAL PPD Tier 2 is the grid resources committed to GridPP/WLCG/EGEE as part of the SouthGrid Distributed Tier 2 –The Grid Batch Farm, the Storage Element The RAL PPD Tier 3 is the local user cluster available to the department –The User Interfaces, the local file servers

4. 4 17/12/2007 PPD Christmas Lectures 2007 The Tier 2 Hardware 384 Batch CPU cores –48 x Intel 2.8GHz PIV –208 x AMD 2.0GHz Opteron 270 –128 x Intel 2.0GHz Woodcrest 5130 158 TeraBytes of Disk Space in dCache –8 x 10TB servers –13 x 6TB servers Various middleware and infrastructure nodes 10Gb/s Link to Site Backbone (and so Tier 1) New

5. 5 17/12/2007 PPD Christmas Lectures 2007 The Tier 3 Hardware 3 Disk servers: –home and software servers Pair of 1GB RAID servers Home and software areas cross sync’d daily –Misc server 6.4TB RAID server Hosts Scratch, installation and other miscellaneous areas

6. 6 17/12/2007 PPD Christmas Lectures 2007 Tier 3 Hardware Continued 8 User Interfaces –heplnx101, 102 general SL4 –heplnx103 general SL3 –heplnx104 CMS SL3 PhEDEx –heplnx105, 106 Atlas SL3 –heplnx107 CMS SL4 –heplnx108 LHCb SL4 All * User interfaces upgraded to faster hardware with more memory

7. 7 17/12/2007 PPD Christmas Lectures 2007

8. 8 17/12/2007 PPD Christmas Lectures 2007 Support Staff Currently well below compliment –50% of Chris Brew In the process of recruiting 2 new system administrators With extra effort we will be able to do more things

9. Changes over the last year

10. 10 17/12/2007 PPD Christmas Lectures 2007 SL4 Migration All batch capacity, many backend servers plus half the front ends upgraded to SL4 Remaining front ends will be upgraded when SL3 is no longer required

11. 11 17/12/2007 PPD Christmas Lectures 2007 New Monitoring and Configuration Tools Nagios –Actively monitors hosts and services and sends alerts when things go wrong Cfengine –Configuration management tool –Change central config and all nodes automagically pick up the changes Pakiti –Monitors the patch status of nodes –May be pushed out to other linux boxes in the Department

12. 12 17/12/2007 PPD Christmas Lectures 2007 Tier 3 Integration with the Tier 2 We’ve taken a number of steps to integrate the Tier 3 with the Tier 2 –User account databases have been merged across both services –Disk mounts are shared between the services Local home, software and scratch areas are mounted on the Grid Workers The grid software and data areas are available on the frontends –SL4 Front end are pbs clients for the Grid batch system Allows direct submission of jobs from the SL4 front ends to the grid batch workers

13. Tier 3 Services

14. 14 17/12/2007 PPD Christmas Lectures 2007 RALPP and SouthGrid VOs For projects without a VO infrastructure –Either just getting going or too small or short to warrant setting one up RALPP VO is purely in the department SouthGrid will be supported at other SouthGrid sites if you need access to more resources

15. 15 17/12/2007 PPD Christmas Lectures 2007 5 Types of Disk Space Home Areas: –RAID Disks, backed up to tape daily and mirrored to backup server every 12 hours Experiment Areas: /opt/ppd/ –RAID Disks, mirrored to backup server every day, not backed up Data Areas: /pnfs/pp.rl.ac.uk/data/ –dCache, multiple RAID servers, single copy not backed up.

16. 16 17/12/2007 PPD Christmas Lectures 2007 5 Types of Disk Space NFS Scratch: /opt/ppd/scratch –RAID for speed/aggregation, no mirroring or backup Local Scratch: –Spare local disk on front ends and batch workers /scratch on frontends – no guarantees $WORKDIR on batch workers, cleaned up after job finishes

17. 17 17/12/2007 PPD Christmas Lectures 2007 Data Storage Will provide large data areas to Projects/Experiments via the dCache storage element –Write access Grid tools –Read access Grid tools dcap Xrootd Experiment areas for the Main supported VOs RALPP catchall for other users/projects

18. 18 17/12/2007 PPD Christmas Lectures 2007 Home areas quotas All home areas com with a quota quota -v Default quota is small, just 20MB –Many accounts never exceed this –Can exceed this up to 2GB for up to 90 days –If you need it increased, just ask

19. 19 17/12/2007 PPD Christmas Lectures 2007 Batch Submission for SL4 frontends Batch cluster uses PBS like CSF The local access queue is prod The default memory and walltime limits are low, if you need more, specify it: qsub –q prod \ –l mem=1024mb \ –l walltime=24:00:00 \ my-script.sh

20. 20 17/12/2007 PPD Christmas Lectures 2007 Printing Printing migrated from using the BTID print server to the new Departmental print server All departmental PostScript printers should be available lpr –P my-file.ps

21. 21 17/12/2007 PPD Christmas Lectures 2007 Access from Offsite Currently blocked to all nodes –Either use PPTP to tunnel in or ssh via the RAL Bastion host (http://www.itd.clrc.ac.uk/Activity/BastionServer) Currently looking at the possibility of running some sort of departmental bastion –Separate account database? –Restricted function? –Ssh keys only? –Gsissh only? –Scp/sftp only?

22. 22 17/12/2007 PPD Christmas Lectures 2007 Future Plans Upgrade UIs –Faster, 64bit nodes More batch and disk capacity Bastion Host Subversion Code Repository? Wiki? ?

23. 23 17/12/2007 PPD Christmas Lectures 2007 Christmas Shutdown (or lack of) All the Tier 2 and Tier 3 services will be running over the Christmas Break Service is “at risk”, I might log in occasionally to check on things, I might even read my email every now and again but don’t count on support.

24. 24 17/12/2007 PPD Christmas Lectures 2007 Conclusion Aim to provide scientific computing infrastructure for the department and the wider community Evolving in view of external changes Are we ready for the next year? –You tell us… –…but hopefully we are at least on the way