May 22, 2003VOLDEMORT-S. Timm--LSCCW 1 VOLDEMORT VOLatile Distribution of Electronic Media Over Rsync Transport May 22, 2003 LSCCW Steven Timm

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 2 Introduction: Disclaimer--Any resemblance to characters in Harry Potter books of J.K. Rowling is pure coincidence. Rsync is open-source package which allows to keep directories on remote machines synchronized with each other Common method at many installations of distributing volatile local files on machines that are already installed. Needs a set of supporting scripts to make it a useful tool.

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 3 VOLDEMORT overview Currently deployed on over 700 machines at Fermilab Works on RH Linux 6, 7, 9, Advanced Server, Sun, SGI Written in shell scripts and perl Two major uses: – Keeping production computing farms installations up to date without reinstalling – Partitioning of US/CMS computing dynamically

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 4 Major Goals of Voldemort Replace NIS with system to put passwd files locally on each node Have a unified structure to push new files out to existing nodes and install them on new nodes Have a single place where each volatile file is modified. Keep current capability to have special files for a single farm, subcluster, or node. Production Farms plus US/CMS have at least 13 different hardware configurations

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 5 Components of Voldemort 0.6 Voldemort-push, includes rsync_push binary and scripts to clone slave servers. Installed on all servers. Voldemort, installed on all clients, includes pullrsync and a number of auxiliary scripts that are called by pusher and puller. Tree file structure, set up in $VOLDEMORT_DIR/clusters Databases to describe the file structure Available as RPM or in Fermi ups/upd format.

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 6 Features Pullrsync included in 7.3 Farms workgroup post-install of Fermi Linux Scripts tested on all flavors of Linux Includes option to sync out changes in Fermi Linux comps file But—not tied to Fermi Linux..can and does work with other installation systems such as Rocks or System Imager

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 7 Why replace NIS? NIS was stable on CDF farms-169 nodes, no timeouts for months—BUT We had to have at least 64 NIS slave servers to accomplish this Pushing to all those slave servers is a network load in itself Yppush doesn’t gracefully handle when a node is down Initial configuration of ypinit –s is error prone and can’t be automated during install.

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 8 Why replace NIS cont’d Malformed map on one slave server can mess up several nodes NIS is small amount of network traffic but is very sensitive to bigger network flows and is disrupted by them. On our farms, we don’t store any real passwords in NIS, accounts change rarely. Ideal situation to distribute files

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 9 Installer vs on-line changes Whenever we made a change to the farm, we had to change in two places…on the nodes and in the installer. Often this has been forgotten Method of making installer changes is not straightforward Need to make a system where any file that goes on the system is only changed in ONE place.

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 10 Down nodes problem: Right now, if we put extra files on the system, we have to go back and fix nodes that were down later, manually. Need a system that will remember which nodes were down, and keep retrying until it gets them all.

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 11 Design goals of Voldemort Do not put any node-specific info into the Fermi Linux workgroup—we don’t want our whole structure available to world via anonymous FTP. (or our account names and groups or nfs servers). Replace /export/linux/Workgroups/Farms/nodes with a new structure that is used both by online activities and the installer. Keep our current capacity to have node specific, farm- specific, and subcluster-specific files

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 12 $VOLDEMORT_DIR/clusters/common/d b/nodes.conf Nodes.conf—database of nodes. Read by both rsync_push and pullrsync fncdf75:cdffarm1:Linux :i-acd:38400::N:2518 Reader Fields: – Node nameAPIC used in install – Cluster nameNode specific – FlavorSubclusters – Disk arrangement – Baud Rate

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 13 $VOLDEMORT_DIR/clusters/common/d b/files.conf Files.conf: Not fully populated yet. Three fields: – Full pathname to the file – (example: fnsfo/files/Linux /etc/passwd) – Files it depends on – common/templates/Linux /etc/passwd – fnsfo/templates/NULL/yppasswd – Command used to make it – (cat the above two files together).

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 14 $VOLDEMORT_DIR/ clusters/fnpce/ Prescripts—scripts that have to be executed before a rpm or file can be installed RPMS Files—single files that are pushed out to worker nodes Scripts—usually run only by the installer Tarballs—Mainly for pushing out /local/ups directory to worker nodes

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 15 $VOLDEMORT_DIR/clusters/fnpce/files Under each category, space for more than one flavor. Right now: Linux (731) Linux+2.4(711) Linux+2.2(612) IRIX+6.5 Can also define arbitrary flavor “foo” as long as database matches.

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 16 $VOLDEMORT_DIR/clusters/fnpce/files/ Linux Each subdirectory of files directory gets pushed out independently—governed by.pushdir files Four subdirectories (typ) /etc, /root, /usr/local, /var/adm Three types of files: – Passwd, group, netgroup, auto.*,.k5login – Non-standard config files for RPMS in redhat base – Hardware-specific or farm-specific files

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 17 $VOLDEMORT_DIR/fnpce/tarballs/Linux Currently only one tarball Structure same as files (.pushdir governs) /local/ups/localups.tar Tarball should be created to be untarred in the directory it’s pushed into. Had to add this option because pushing a ups/upd tree of 19K files (180 Mb) was too slow.

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 18 $VOLDEMORT_DIR/clusters/fnpce/RPM S/Linux RPMS that go here are either farm-specific or hardware-specific. Anything for whole farm should go into farms workgroup.

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 19 $VOLDEMORT_DIR/clusters/fnpce/[pre] scripts/Linux Scripts and prescripts are mainly executed during the install Installer calls /sbin/pullrsync –I which forces running of all scripts Scripts should be smart enough to detect if the action has already been done

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 20 Subclusters Subclusters can exist in any of five categories, files, tarballs, RPMS, scripts, prescripts Subcluster membership determined by the database Convention: All hardware specific files (ethernet, lm_sensors) go into a subcluster named after the motherboard type Node can be in more than one subcluster For files and tarballs, a.pushdir at the top level.

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 21 Node-specific files Can also have files specific to a single node Enabled by having field in database be “Y” instead of the default “N”

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 22 Rsync_push Rsync_push reads through the database and pushes to every node that matches the command-line options it was called with. *IMPORTANT* Default is to push to everything! There is an are-you-sure option now that warns you what you are pushing. Rsync_push –r allows you to retry nodes that didn’t push successfully the first time. Default transport is kerberized rsh. Others can be used as well. To push to a node, host principal of the server must be in /root/.k5login of the client node.

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 23 Rsync_push options 1 -c Push for a given cluster -f Push for a given flavor -b Push for a list of nodes -B Push for a range of nodes -l push for all the nodes in If more than one is specified, we take the AND Example: rsync_push –c cdffarm –f Linux –B fncdf “ ”

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 24 Rsync_push options 2 -R don’t push the RPMS -F don’t push the files -S don’t push the scripts -P don’t push the prescripts -T don’t push the tarballs -L don’t push the Linux /etc/workgroup Default is to push everything

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 25 Rsync_push options 3 -w specify the workgroup you are pushing (default is Farms) -e use an alternative rsh command besides /usr/krb5/bin/rsh -q quiet—minimum or no output -v verbose—the more v’s, the more verbose -i Install mode—run new scripts and prescripts when they are pushed out -I Install mode—run all scripts and prescripts when they are pushed out -C Clear out the RPMS, scripts, and prescripts directory on the worker nodes.

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 26 pullrsync Determines node ID and type either from local config file or from database read Runs only if machine wasn’t shut down clean, and during the install Options –h help –H –c -f –M -t -R –S –P –F –T –L –w –I –i –q –v (as in rsync_push)

May 22, 2003 VOLDEMORT-S. Timm--LSCCW 27 Future plans Version v0_6 current, no known bugs right now. Next version needs better and faster database Also need ability to automatically distribute the push across slave servers Big task, integrating more closely with ROCKS and RH oss.fnal.gov/scs/public/farms/doc/voldemort.html