Cost-effective clustering with OpenPBS113/02/2003 Cost-effective clustering with OpenPBS Ben Webb WGR Research Group Physical and Theoretical Chemistry Lab. University of Oxford

213/02/2003Cost-effective clustering with OpenPBS Overview History of PBS Interests of the WGR group OpenPBS architecture: portability, security, scheduling Grid integration Alternatives

313/02/2003Cost-effective clustering with OpenPBS History of PBS PBS is the Portable Batch System Developed from 1993 to 1997 for NASA Intended to replace NQS Currently available as: –OpenPBS (open source) –PBSPro (commercial)

413/02/2003Cost-effective clustering with OpenPBS Interests of the WGR group High throughput –Virtual screening (cancer screensaver) –Met by loose grid of over 2 million PCs; United Devices/Intel High performance –Ab initio chemistry –Simulation of chemical reactions (free energy) –Met by OpenPBS at zero software cost

513/02/2003Cost-effective clustering with OpenPBS OpenPBS architecture Server: keeps track of all jobs Scheduler: tells the server when and where to run jobs MOM (Machine Oriented Miniserver): runs on each node to start, monitor, and terminate jobs, under instruction from the server POSIX compliant batch system Supports file staging for executables and data No need for shared filesystem (e.g. NFS) although this does simplify communication

613/02/2003Cost-effective clustering with OpenPBS An example OpenPBS setup

713/02/2003Cost-effective clustering with OpenPBS Advantages of PBSPro Pre-emptive job scheduling Scheduler backfilling Improved fault tolerance Desktop Cycle Harvesting Paid support (all OpenPBS support is via mailing lists) Largely compatible with OpenPBS

813/02/2003Cost-effective clustering with OpenPBS Portability Runs on most Unix-like systems: e.g. Linux/Irix/Unicos/HPUX/IA64 etc. MOMs for various architectures take advantage of system-specific features –e.g. checkpointing supported on certain architectures Full server/client/MOM support for heterogeneous networks

913/02/2003Cost-effective clustering with OpenPBS Queues and nodes Unlike NQS, PBS does not rely on queues for scheduling decisions Queues are not tied to nodes, but can specify resources Routing queues can pass jobs to execution queues, possibly on different PBS servers Nodes can have any number of virtual processors

1013/02/2003Cost-effective clustering with OpenPBS Resource definition Server-defined properties group nodes into classes - e.g. intel for all Intel architecture machines Additional resources (e.g. tape drives, software licences) can be specified by each MOM –Custom resources are not utilised by the default scheduler

1113/02/2003Cost-effective clustering with OpenPBS Resource usage Timeshared nodes: balanced by load Cluster nodes: jobs allocated to virtual processors, usually exclusively MOMs track jobs and kill any that exceed resource limits (e.g. CPU or wall time, memory) No unified mechanism for accounting of running and finished jobs –qstat for running jobs –Server accounting logs for finished jobs

1213/02/2003Cost-effective clustering with OpenPBS Scheduling Scheduler is just a privileged client Well-defined PBS scheduling API Facilities to write schedulers in C/BaSL/Tcl OpenPBS provides a simple FIFO scheduler, as well as custom schedulers to take advantage of system-specific features Maui scheduler (third party) also integrates with other batch systems, and provides powerful scheduling

1313/02/2003Cost-effective clustering with OpenPBS Security Uses rhosts mechanism for authentication of clients to the server (consistent user name space not required), but does not require rsh MOMs can use rsh, ssh or cp (via NFS) to stage files in and out Access Control Lists can also be used to provide extra security PBS daemons use non-random port numbers, and TCP for most communication, allowing straightforward firewalling All daemons run as root! (No reported vulnerabilities to date, however.)

1413/02/2003Cost-effective clustering with OpenPBS Parallel support Conventional MPI mechanisms rely on well- behaved users, and lack resource tracking OpenPBS provides a Task Manager (TM) API –Allows parallel PBS jobs to spawn processes on nodes other than the master –mpiexec (third party) allows start-up of MPI jobs via the TM mechanism (MPICH/EMP/LAM) –Current LAM CVS also has a PBS-TM boot SSI (system services interface) for job start-up

1513/02/2003Cost-effective clustering with OpenPBS Customisation Full source code available, for commercial or non-commercial use Site-specific modification routines allow easy customisation of likely targets Defined C API for job submission, query etc. Third-party projects and patches, e.g. mpiexec, Cplant (fault tolerance), PyPBS, scalability patches, AFS token management

1613/02/2003Cost-effective clustering with OpenPBS Grid integration Globus Resource Allocation Manager (GRAM) available for PBS Maui scheduler or PBSPro default scheduler support advance reservations Silver metascheduler is grid-aware, has full support for PBS, and can work with or without Globus

1713/02/2003Cost-effective clustering with OpenPBS Comparison with Sun Grid Engine Both systems perform balancing of jobs/load between managed nodes PBS server is a single point of failure; SGE supports shadow masters SGE seems to now be more actively developed than OpenPBS

1813/02/2003Cost-effective clustering with OpenPBS Summary and acknowledgements OpenPBS is a cheap solution for Linux clustering, conventional supercomputer management, and/or use of idle workstations Can upgrade easily to PBSPro if desired PBS includes software developed by NASA Ames Research Center, Lawrence Livermore National Laboratory, and Veridian Information Solutions, Inc. Visit for OpenPBS software support, products, and information. WGR group webpages: