1. EGEE is a project funded by the European Union under contract IST-2003-508833 Quattor Cookbook C. Loomis (LAL-Orsay) Clermont-Ferrand Quattor Tutorial May 17-18, 2004 www.eu-egee.org

2. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 2 Contents Overview Quattor Server OS Installation (RH7.3) Quattor Services Pan Templates Quattor Clients Manual Installation Automatic (PXE) Installation Writing NCM Components Summary

3. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 3 Overview Purpose Install basic quattor server. Understand & use pan templates. Install & manage clients with quattor. Write simple NCM component. Ground Rules Originally intended as informal one-on-one “tutorial”, i.e. expect glitches. Feel free to interrupt, correct, praise, complain, etc. I'm not a quattor developer! Only useful features (IMHO) are covered. Cookbook doesn't replace the complete quattor documentation.

4. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 4 Quattor Resources Website (http://quattor.web.cern.ch/quattor/)http://quattor.web.cern.ch/quattor/ Overview Design and architecture Current status Bug reports (via savannah) Documentation (http://quattor.web.cern.ch/quattor/documentation.htm)http://quattor.web.cern.ch/quattor/documentation.htm Installation guide Component writers' guide PAN language reference Tutorials Quattor Packages (http://quattor.web.cern.ch/quattor/software.htm)http://quattor.web.cern.ch/quattor/software.htm RedHat Linux 7.3 Others (FC1, RHES,...) not yet available

5. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 5 Quattor Server Requirements RedHat Linux 7.3 Daemons (Apache, DHCP, TFTP) Disk: 2.5 GB for system, 2.5 GB per client OS, 5 GB for LCG-2 “Basic” Quattor Server All quattor services on one machine. Some advanced services are not deployed. Should be OK for most sites. “Small” on order of 100 machines. Have not tested this myself! LCG-2 installation: Templates & components not complete. Community effort trying to make this happen.

6. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 6 RedHat 7.3 Installation Collect necessary information IP addresses (quattor server, DNS, gateway, NTP) MAC addresses, network mask, broadcast Standard “server” installation Edit partitions for >10 GB in /var. No firewall. Additional packages Web server group DHCP server (system environment/daemons/dhcp) TFTP server (system environment/daemons/tftp-server) NTP client (system environment/daemons/ntp) mtools (applications/system/mtools)

7. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 7 RedHat 7.3 Updates Quattor server itself not managed by quattor! Use standard tools for server system updates. RedHat no longer supports RH7.3. Fedora Legacy Project provides security patches. Follow the instructions at the link below to enable yum: http://www.fedoralegacy.org/docs/yum-rh7x.php Recommend: Enabling kernel updates. Enabling automatic updates. Using appropriate mirrors. Update system and reboot! yum update reboot

8. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 8 Downloads Complete OS for kickstart installs Available on web server for clients. Download RH7.3 images (valhalla*.iso). Copy image contents to /var/www/html/rh73/. Quattor packages Alpha-2 edg-cdb2sql package is not needed for this tutorial http://quattorsw.web.cern.ch/quattorsw/software/quattor/release/alpha_ 2/i386-rh7.3/ http://quattorsw.web.cern.ch/quattorsw/software/quattor/release/alpha_ 2/i386-rh7.3/ External Need to upgrade syslinux; others can be installed http://quattorsw.web.cern.ch/quattorsw/software/quattor/external/RPMS/ RH73/ http://quattorsw.web.cern.ch/quattorsw/software/quattor/external/RPMS/ RH73/ Download RPMs to temporary area.

9. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 9 Downloads (cont.) LCG-2 Use LCG-2 machine types as examples. Get all LCG-2 rpms for all machines: http://grid-deployment.web.cern.ch/grid- deployment/download/RpmDir/index_LCG-2_0_0.html http://grid-deployment.web.cern.ch/grid- deployment/download/RpmDir/index_LCG-2_0_0.html Put into temporary area with sufficient space.

10. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 10 Quattor Server Services Central Configuration Database (CDB) Contains configuration information for all quattor clients. Configuration done via Pan templates. Translated to low-level XML format for distribution to clients. Software Repository (SWRep) Contains all software packages for clients: OS, LCG-2, quattor,... Automated Installation Infrastructure (AII) Uses standard OS tool for initial installation. Quattor is used to update system after the first boot.

11. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 11 Quattor Client Services Configuration Cache Manager (CCM) Downloads & caches low-level XML profile. Node Configuration Manager (NCM) Runs necessary configuration components when profile changes. Software Package Management Agent (SPMA) Installs & updates software packages.

12. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 12 CDB Configuration Ensure that Apache is running: chkconfig httpd on service httpd start Check that the server responds to a browser. Verify configuration file: “top” is location of database (/var/lib/cdb) “pan” is location of executable (/opt/edg/bin/panc) “cake” is location of cake executable (/opt/edg/libexec/cake) Initialize CDB: /opt/edg/sbin/edg-cdb-setup Creates hld and lld directories in /var/lib/cdb. Links /var/www/html/profiles to /var/lib/cdb/lld/xml.

13. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 13 CDB Configuration (cont.) CDB notifications (for later) Add “server_module host.example.org” for synchronization. Or “server_module none” for no notifications. Two management tools: cdb-simple-cli (local management) cdbop (remote management)

14. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 14 CDB Client Configuration Simple command line (cdb-simple-cli) Set password for cdb account. Session-based tool (cdbop) Add “user::passwd” lines to /etc/httpd/conf/.passwd file. Crypt password: perl -e 'print crypt(“password”, “be”)' Replace /etc/httpd/conf/cdb.allow contents with line “cdb admin”. Copy /usr/share/doc/cdb-cli- /cdbop.cf to /etc/cdbop.cf. Add quattor server name to /etc/cdbop.cf. Test: cdb-simple-cli –list cdbop then user, password, “open”, “list”, “exit”. Commands should work but return nothing (as there are no templates yet).

15. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 15 SW Repository Configuration (Server) Copy example configuration file: /usr/share/doc/edg-swrep-server- /edg-swrep-server.cfg to /etc/swrep/. Edit parameters: name: “Cookbook Software Repository” owner: grid.support@example.orggrid.support@example.org url: http://quattor.example.org/swrephttp://quattor.example.org/swrep aclfile: /etc/swrep/edg-swrep.acl rootdir: /var/www/html/swrep enable-sync: no

16. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 16 SW Repository Configuration (Client) Copy example configuration file: /usr/share/doc/edg-swrep- /edg-swrep-client.cfg to /etc/swrep/. Edit parameters: repository: quattor_server@example.org runsync: no ssh-params: modification needed only if using kerberos

17. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 17 SW Repository Authorization Generate ssh keys if necessary: ssh-keygen -b 4096 -t rsa Copy id_rsa.pub to /var/swrep/.ssh/authorized_keys2 Add: 'environment=”SSH_USER=swrep” ' before “ssh-rsa” ACL (/etc/swrep/edg-swrep.acl) should already have: “swrep:/” as the first line

18. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 18 SW Repository Organization Organize as you want. Typically something like: edg-swrep-client addplatform i386_rh73 edg-swrep-client addarea i386_rh73 /base edg-swrep-client addarea i386_rh73 /updates But for this tutorial will use: edg-swrep-client addarea i386_rh73 /lcg2 edg-swrep-client addarea i386_rh73 /quattor Adding/removing packages: edg-swrep-client put i386_rh73 /tmp/eg.rpm /lcg2 edg-swrep-client list i386_rh73 edg-swrep-client remove i386_rh73 /lcg2 eg.rpm If all worked, then SW repository server and client are OK.

19. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 19 Fill SW Repository Load all LCG-2 rpms into repository: edg-swrep-client put i386_rh73.rpm /lcg2 Load all quattor rpms into repository: edg-swrep-client put i386_rh73.rpm /quattor Generate a repository template: edg-swrep-client template i386_rh73 Rename appropriately (see “structure template” line of generated file).

20. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 20 Pan Overview Pan Language Used to specify machine configuration. Flexible, hierarchical organization of information. Allows arbitrary validation of configuration. Organization (pro_declaration_structure_*.tpl) /hardware cpus... /software... /system network...

21. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 21 Pan Types Built-in Types long double string “Standard” types: pro_declaration_type_validation_function_*.tpl pro_declaration_type_*.tpl ip, ipv4, ipv6, fqdn, hostname, shorthostname URI, absoluteURI, hostURI asndate, isodate, date (deprecated) email, hwaddr (MAC)

22. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 22 Standard Templates Load standard types into CDB: cdb-simple-cli --add pro_declaration_type*.tpl Load LCG-2 templates: cdb-simple-cli --add pro_software_lcg2_*.tpl Load repository template: cdb-simple-cli --add repository*.tpl

23. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 23 Hardware Templates Create hardware templates (CPU, RAM, disk, NIC): e.g. pro_hardware_cpu_GenuineIntel_Xeon_2400.tpl e.g. pro_hardware_ram_2048.tpl e.g. pro_hardware_harddisk_scsi_32.tpl e.g. pro_hardware_card_nic_broadcom_NetXtreme.tpl Create machine type: Dual Xeon with 2GB RAM, 32GB SCSI disk, 3 network interfaces e.g. pro_hardware_dell_poweredge_1750_v00.tpl Load hardware templates: cdb-simple-cli --add *hardware*.tpl

24. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 24 Software Templates SW repository templates: Typically generated with edg-swrep-client command. e.g. repository_lal_repository_i386_rh73.tpl Can create templates for standard web servers. e.g. standard LCG-2 package repository Package lists Templates exist for LCG-2.0.0. Machine types: pro_software_lcg2_machine_*.tpl Base RH7.3 system: pro_software_lcg2_machine_base.tpl User interface: pro_software_lcg2_machine_ui.tpl Detailed service lists: pro_software_lcg2_service_*.tpl

25. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 25 System Templates Specify host information in object template: Hostname NIC parameters Kernel version Create common system template (pro_system_common.tpl). Cluster name Root mail address Disk partitions Components...

26. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 26 Object Templates Object template corresponds to client machines. e.g. profile_grid06.tpl Validation All object templates are validated for ALL template changes! Ensures confidence in generated machine profiles. Can be very painful to get right the first time. Must have working profile before trying client.

27. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 27 Manual Client Installation Prepare machine: Install RedHat Linux 7.3 “server” installation No firewall Make sure enough space (~10 GB) on “/” partition! Optionally update with yum (see previous directions). Don't need to setup for automatic updates. Download quattor rpms: Alpha-2 and external from previous links. Install all RPMs except syslinux and edg-cdb2sql.

28. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 28 Manual Client Configuration Configuration Cache Manager (CCM): Copy /usr/share/doc/edg-ccm- /eg/edg-ccm.conf to /etc. Edit “profile” parameter: change: http://cdb/$host.$domain.xmlhttp://cdb/$host.$domain.xml to: http://quattor.example.org/profiles/profile_$host.xmlhttp://quattor.example.org/profiles/profile_$host.xml Comment out “context”. Run: /usr/sbin/edg-ccm-initialise. Node Configuration Deployer (NCD) Defaults in /etc/ncm-ncd.conf should be OK. Software Package Management Agent (SPMA) Set “userpkgs” in /etc/spma.conf to “no”. Other defaults should be OK.

29. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 29 Manual Client Configuration (cont.) Configuration Distribution Protocol Daemon (CDP) Copy /usr/share/doc/edg-cdp-listend- /examples/edg-cdp- listend.conf to /etc. Change “fetch” parameter to /usr/sbin/edg-ccm-fetch. Other defaults should be OK. Configuration Dispatch Daemon (cdispd) Defaults in /etc/ncm-cdispd.conf should be OK.

30. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 30 Manual Client Update Fetch the XML profile from the server. edg-ccm-fetch Start configuration dispatch daemon. service ncm-cdispd start Check log files for changes: ncm-cdispd: /var/log/ncm-cdispd.log SPMA: /var/log/spma.log NCM components: /var/log/ncm/* On quattor server: Edit /opt/edg/etc/cdb.conf to add client “server_module”. Change package list and verify changes are propagated to client.

31. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 31 Automatic Client Installation AII works from quattor server. Uses native machine installation tools. For linux: kickstart. Bootstrap using PXE. DHCP server needed (IP address, kernel location) TFTP server needed (boot kernel) HTTP server needed (OS images, packages)

32. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 32 DHCP Setup Copy /usr/share/doc/aii- /eg/dhcpd.conf to /etc. Edit to ensure that common setting are appropriate for site. Individual hosts will be added & updated by AII scripts.

33. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 33 TFTP Setup Create directories /osinstall/nbp and /os/install/nbp/rh73. Copy the pxelinux.0 file to /osinstall/nbp. cp -f /usr/lib/syslinux/pxelinux.0 /osinstall/nbp Edit /etc/xinetd.d/tftp: “disable = no” “server_args = -s /osinstall/nbp Edit /etc/hosts.allow to permit clients to connect! Copy boot files to /osinstall/nbp/rh73: cp /mnt/cdrom/images/pxeboot/* /osinstall/nbp/rh73 Adjust location to where RH7.3 installation disk is located. Restart the xinetd service to activate changes!

34. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 34 Kickstart Setup Create directory /osinstall/nbp/ks. Make kickstart files available via web server. ln -s /osinstall/ks /var/www/html/ks Copy the pxelinux.0 file to /osinstall/nbp. Contents of installation disks must be available from web server. cp -r /mnt/cdrom/* /var/www/html/rh73 Copy acknowledgement script to cgi location. cp /usr/sbin/aii-installack.cgi /var/www/cgi-bin Allow apache to run AII commands without password. Add line “apache ALL=(ALL) NOPASSWD: ALL” to /etc/sudoers.

35. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 35 Edit AII Templates Edit pro_software_component_aii.tpl: File contains lots of local machine information. time zone keyboard type language Partitioning used for the local disks. Location of profiles: /software/components/aii/osinstall/ks/cdbserver This must be the URL where the machine profiles are located. Make sure that updated component is loaded into CDB. Make sure /usr/lib/aii/osinstall/rh73.kstpl file is OK. Can make modifications to post install script here if necessary. Contains names of additional RPMs which must be installed.

36. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 36 AII Client Installation Edit /etc/aii-shellfe.conf setting “cdburl” to the CDB URL on server! Signal AII to do configuration for new client. aii-shellfe --configure quattor_client.example.org Mark this machine for installation. aii-shellfe --install quattor_client.example.org Verify changes: /etc/dhcpd.conf contains correct host information Profile available from http://quattor.example.org/profiles.http://quattor.example.org/profiles Kickstart file in /osinstall/ks is correct. Install Client Reboot client using PXE. Initial installation via kickstart; reboots on completion. Quattor finishes installation then reboots.

37. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 37 Common Problems PXE can't get installation parameters from DHCP server. DHCP server not configured correctly. (Check IP numbers, etc.) Firewall is blocking access to DHCP server. Another DHCP server is responding. TFTP server can't be contacted. Check /etc/hosts.allow file. Verify all necessary files are available in /osinstall area. Firewall is blocking access to TFTP server. Kickstart fails Check information in pro_software_component_aii.tpl Change “hda” to “sda” for SCSI interface. Verify URL download location.

38. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 38 Easiest to start from an example. Check out quattor (and NCM components) from CVS export CVSROOT=:pserver:anonymous@isscvs.cern.ch:/local/reps/elfmsanonymous@isscvs.cern.ch cvs co elfms/quattor Copy ncm-edglcg to ncm-example: cd elfms/quattor/ncm-components/lcg-2/ mkdir ncm-example cp ncm-edglcg/* ncm-example/ Rename & edit files in ncm-example. Component writes params.--alpha, beta, gamma--to /etc/example.conf. See example files. Writing NCM Components

39. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 39 Build the necessary RPMs: make rpm Load RPMs into SW repository. edg-swrep-client put i386_rh73 \ ncm-example-1.0.0-1.i386.rpm /quattor Generate new repository template and load in CDB: edg-swrep-client template i386_rh73 > tmp.tpl mv tmp.tpl repository_lal_repository_i386_rh73.tpl cdb-simple-cli --update \ repository_lal_repository_i386_rh73.tpl Incorporating New Component

40. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 40 Create template to include component & configuration. make rpm Add template to quattor client's object template. edg-swrep-client put i386_rh73 \ ncm-example-1.0.0-1.i386.rpm /quattor Check that changes are propagated to client. ls -l /etc/example.conf cat /etc/example.conf Make changes to configuration; verify changes on client. Incorporating New Configuration

41. Quattor Cookbook (C. Loomis), May 17-18, 2004 - 41 Summary Installed quattor server appropriate for “small” site. Used pan language to define client configuration. Managed manually-installed machine via quattor. Installed from scratch machine using quattor. Wrote and added a simple NCM component to configuration. Comments on improving the tutorial are very welcome!