The 2001 Tier-1 prototype for LHCb-Italy Vincenzo Vagnoni Genève, November 2000

The 2001 Tier-1 prototype for LHCb-Italy Vincenzo Vagnoni Outline 2001 LHCb Italian Tier-1 prototype architecture Linux diskless nodes howto Network attached storage Tests made in Bologna Conclusions

The 2001 Tier-1 prototype for LHCb-Italy Vincenzo Vagnoni 2001 LHCb Italian Tier-1 protoype 15 CPU 2001 (700 SI95) Tier-1 computer farm (at the beginning of the year, with further increase if we can show that more CPU power is needed for the italian groups) 15 single-processor motherboards, 256 MB RAM each or more, rack-mounted, diskless, with redundant power supply and cooling. 1 TB IDE disk array in RAID-5 configuration, hosted in a NAS (Network Attached Storage) unit (also to be increased if needed). 100 Mbps Ethernet Switch.

The 2001 Tier-1 prototype for LHCb-Italy Vincenzo Vagnoni 2001 Tier-1 Computer Farm Prototype

The 2001 Tier-1 prototype for LHCb-Italy Vincenzo Vagnoni Components (Motherboards, Racks, NAS, Switches)

The 2001 Tier-1 prototype for LHCb-Italy Vincenzo Vagnoni Advantage of diskless nodes Easy Operating System installation and maintainance Adding a machine to the farm requires to run a simple script on the disk server The Operating Systems are centralized and accessible through the file system of the disk server Enhanced disk fault tolerance The disk server can have large hard disk arrays with redundant information by using a RAID 5 No need of UPS battery for the client nodes Only the disk server needs UPS, no damage if a client is powered off System compactness The nodes are basically simple motherboards with integrated ethernet cards They can be arranged in racks, ~25 nodes in only 1 m 2.

The 2001 Tier-1 prototype for LHCb-Italy Vincenzo Vagnoni Working concept (using for example a PXE BootROM) At boot time the diskless client sends its ethernet card MAC address by making a bootp (or dhcp) broadcast request over the LAN The diskless client ethernet controller must be equipped with a BootROM (for example a PXE, Pre-eXecution Environment, compliant one) The bootp (or dhcp) server replies assigning the IP address to the client and passing the file name of the boot code on the server Using a PXE BootROM the boot code is a small loader (pxelinux.bin), provided with the syslinux package, also able to read a configuration file on the server with some additional information (i.e. the arguments to pass to the linux kernel) The client makes a TFTP connection to the server asking to download the PXE loader and executes it The loader tftp-downloads the linux kernel, and executes it

The 2001 Tier-1 prototype for LHCb-Italy Vincenzo Vagnoni Further details… The linux kernel needs to be compiled with root-over-NFS support to be able to mount the root filesystem through NFS The Ethernet Card driver must be compiled resident into the kernel (in principle could be also provided by an initial ramdisk) The clients can share some directories of the linux filesystem (/usr, /opt, …), but need some others to be private (at least /etc, /dev, /var, /tmp) Optionally some non-shareable directories (for example /dev, /var, /tmp) can be mounted on ramdisks

The 2001 Tier-1 prototype for LHCb-Italy Vincenzo Vagnoni Network Attached Storage OpenNAS RS15 RaidZone File Server supports NFS (also Windows SMB, but fortunately we don’t need this guy) Built-in Web Server and DNS Server R3 RAID5 (Redundancy with Rapid Recovery) Java-based remote interface for all OpenNAS functions, including disk array configuration 15 hot-swappable drives bays, with 15 Ultra ATA/100 drives (80 GB each) + 1 hot-swappable spare Dual PIII 800MHz CPUs and 256 MB ECC RAM Dual 100BaseT network connections (Gigabit optional) Dual redundant 300W hot-swappable power supplies Possibility of two RAIDZONE disk array expansions (up to 3 TB) Cost: $20850 (RS15) + optional $19595 (1 TB expansion)

The 2001 Tier-1 prototype for LHCb-Italy Vincenzo Vagnoni Tests made in Bologna Hardware setup 4 PC HP Vectra, single processor PIII 500 MHz, 256 MB RAM, one acting as root filesystem disk server and three as diskless clients 1 PC HP Kayak, dual processor PIII 733 MHz, 256 MB RAM, as an additional client with different architecture Software setup Linux RedHat 6.2, upgraded with kernel , or optionally (to be choosen at boot time) Linux SuSE 6.4 Tests made to evaluate: Complexity of installation and administration Performance of the system (especially interesting due to the absence of the system swap area, even if in principle a network based one could be arranged)

The 2001 Tier-1 prototype for LHCb-Italy Vincenzo Vagnoni Installation tests results Installation trivial Once the server system is configured, adding a new client takes 1 minute All the installations are by default identical No additional network activity observed (monitored directly into the switch) The OSes rarely need to access non cached information on the server (server disk almost completely inactive) Robust system Simulated network failures for few seconds do not interfere with the OS work Powering off the client machines (without a shutdown!) doesn’t create any problem at all (no need to check the disk data integrity at reboot)

The 2001 Tier-1 prototype for LHCb-Italy Vincenzo Vagnoni Performance test results Performance tested by running two HERA-B MonteCarlo simulation (ARTE) jobs on a dual processor 733 MHz, 256 MB RAM, HP Kayak with diskless installation (running Linux SuSE 6.4 in this case) Performance compared with an identical machine with non- diskless installation and local disk swap area Identical performances! The MC jobs were tuned to allocate ~100 MB RAM each Even with 256 MB RAM, two heavy jobs can run together without using a local disk swap area at all Conclusion: no problem for a single-processor motherboard equipped with 256 MB RAM running one single job

The 2001 Tier-1 prototype for LHCb-Italy Vincenzo Vagnoni Conclusions The 2001 LHCb Tier-1 prototype architecture is presented Based on diskless nodes configuration with a Network Attached Storage disk server and a 100 Mbps switched ethernet System proof of principle tested This kind of system can be an interesting contribution to the LHC computing community It saves time, room and money!