1. Linux Configuration using April 12 th 2010 L. Brarda / CERN (some slides & pictures taken from the Quattor website) ‏

2. Quattor Part of ELFms (Extremely Large Fabric management system) with Lemon (Monitoring) and Leaf (Hardware and State management) ‏ System administration toolkit Linux (& Solaris) ‏ Configuration Software management Installation

3. Template based Hierarchical templates Templates are text file Pan declarative language Each server has it's own template

4. Quattor CDB Description of each node in a hierarchical structure of configuration templates written in Pan language. 3 main parts : –Hardware description –List of installed software –Configuration By defaults, 3 configuration levels –Site –Cluster –Node

5. Example of information in CDB Hardware –CPU –Hard disk –Network card –Memory size –Node location Software –Repository definitions –Service definitions = groups of packages (RPMs) ‏ System –Partition table –Load balancing information Cluster informationNot used –Cluster name and type –Batch master Audit informationNot used –Contract type and number –Purchase date

6. Node template object template plus20; variable loadpath=list("pro"); include {“quattor/profile_base”}; include {“hardware/machines/dell_sc1950”}; “/system/network/hostname" = "plus20"; "/hardware/location" = "S1D01"; "/hardware/serialnumber" = "7NSNN3J"; "/hardware/cards/nic/eth0/hwaddr" = "00:1D:09:70:D3:24"; "/hardware/cards/nic/eth1/hwaddr" = "00:1D:09:70:D3:26"; "/system/network/interfaces/eth0/ip" = "10.128.21.20"; include {“hardware/partitions/server”}; include {“plus”}; include {“end_config”};

7. Pan language Produces an information tree –xml format (pan or xmldb variants) 3 branches at the root of the tree –Hardware –Software –System 3 main parts of the software branch –Components –Packages –repositories

8. Pan language “/system/network/hostname" = "plus20"; Path in the tree Value Values can be : –Strings: "plus20" –Booleans: true –Lists: list(“value1”, “value2”,...) ‏ –Named list : nlist(“name1”, “value1”, “name2”, “value2”,...);

9. Schemas Configuration schema defined in schema/tpl files # define the root type of the configuration data tree bind "/" = structure_profile; type structure_profile = { "hardware" : structure_hardware "software" ? structure_software "system" : structure_system };...

10. Components A Quattor 'plugin' designed to configure a specific part of the system : –cron, accounts, nfs,... A perl module A schema –Informations needed to configure that part –Validation logic

11. Cron component schema type structure_cron = { 'name' : string 'user' ? string 'group' ? string 'frequency' : string 'command' : string 'comment' ? string 'env' ? string{} }; type component_cron = { include structure_component 'entries' ? structure_cron[] }; bind '/software/components/cron' = component_cron; Separators: ':' Mandatory '?' Optional

12. Node hardware template template hardware/machines/dell_sc1425_raid; include {“hardware/machines/dell_sc1425”}; "/hardware/harddisks" = nlist("sda", create("hardware/disk/maxtor_250"), "sdb", create("hardware/disk/wdc_250")); -------------------------------------------------------------- template hardware/machines/dell_sc1425; "/hardware/vendor" = "dell" ; “/hardware/cpu" = list(create("hardware/cpu/GenuineIntel_Xeon_2800"), create("hardware/cpu/GenuineIntel_Xeon_2800")); "/hardware/harddisks" = nlist("sda", create("hardware/disk/st_40")); "/hardware/ram" = list(create("hardware/ram/1024"), create("hardware/ram/1024")); "/hardware/cards/nic" = nlist( "eth0", create("hardware/card/nic/Intel_82541GI"), "eth1", create("hardware/card/nic/Intel_82541GI")); "/hardware/cards/nic/eth0/boot" = true; "/hardware/model" = "Dell SC1425 raid"; "/hardware/num_of_cpu" = 4; "/hardware/ipmi/needs_username" = true;

13. System type template template plus; variable kernel_arch?="x86_64"; include {"software/lhcb_online_slc4_"+kernel_arch}; include {"software/packages_cern_slc4_release_compat_i386”}; include {"software/packages_cern_slc4_release_compat_i386_dev”}; include {"lhcb_ux”}; include {"system_FMC_controlNode”}; include {"ssh_host_keys”}; include {"components/nfs/config”}; "/software/components/nfs/mounts" = push( nlist("device","daqnfs:/daqarea", ”mountpoint","/daqarea", "fstype","nfs", "options","rw"), nlist("device","store04:/histograms", "mountpoint","/histograms", "fstype","nfs", "options","rw,hard,intr") ); System type template

14. Site template template lhcb_ux; include {“pan/functions ”} ; include {“quattor/functions/network ”} ; "/system/cluster/name" = "LHCb Online"; "/system/siterelease" = "Scientific Linux 4"; "/system/network/domainname" = "lbdaq.cern.ch" ; "/system/network/nameserver" = list("10.130.16.5", "10.128.16.5"); "/system/network/interfaces/eth0/netmask" ?= "255.252.0.0"; "/system/network/interfaces/eth0/broadcast" ?= "10.135.255.255"; "/system/network/interfaces/eth0/gateway" ?= "10.132.10.3"; "/system/rootmail" = "lhcb-online-admins@cern.ch"; variable kernel_ver?="2.6.9-67.0.15.EL.cern"; variable inst_arch ?= { if (kernel_arch == "i686") "i386_sl4" else kernel_arch+"_sl4"; }; "/hardware/arch" = kernel_arch; "/system/kernel/version" = { if (value("/hardware/num_of_cpu") == 1) kernel_ver else kernel_ver+"smp"; }; "/hardware/arch" = kernel_arch;

15. Site template # ncm components # -------------- # # SPMA: software package manager agent include {“components/spma/config”}; "/software/components/spma/userpkgs" = "yes"; include {“components/ccm/config”}; "/software/components/ccm/debug" = 1; "/software/components/ccm/profile" = "http:/"+"/cdb01/profiles/"+value("/system/network/hostname")+".xml"; # AII: automatic installation infrastructure include {“components/aii/pit”}; "/software/components/aii/osinstall/options/selinux" = "disabled"; # grub: configures the grub bootloader include {“components/grub/config”}; # ntp: configures the network time protocol daemon include {“components/ntpd/config”}; "/software/components/ntpd/servers" = list ("time01.lbdaq.cern.ch", "time02.lbdaq.cern.ch"); include {“components/sendmail/config”}; "/software/components/sendmail/smarthost" = "mail01";...

16. Software template template software/lhcb_online_slc4_x86_64; include {“software/packages_defaults_x86_64sl4”}; include {“software/packages_defaults_lhcb_x86_64sl4”}; include {“software/packages_cern_slc4_release_base”}; include {“software/packages_cern_slc4_kernels”}; include {“software/packages_cern_slc4_release_interactive”}; include {“software/packages_cern_slc4_release_compat_i386”}; include {“software/packages_cern_slc4_release_dev”}; include {“software/packages_cern_slc4_quattor”}; "/software/packages"=pkg_add("wireshark"); "/software/packages"=pkg_repl("gksu"); "/software/packages"=pkg_repl("libgksu");... "/software/packages"=pkg_repl("cyrus-sasl-gssapi"); ############################################################ # which repositories are in use # "/software/repositories/0" = create("repository/lhcbonline_x86_64_sl4base"); "/software/repositories/1" = create("repository/lhcbonline_x86_64_sl4extra"); "/software/repositories/2" = create("repository/lhcbonline_x86_64_sl4lhcb"); "/software/repositories/3" = create("repository/lhcbonline_x86_64_sl4onlycern"); "/software/repositories/4" = create("repository/lhcbonline_x86_64_sl4quattor"); "/software/repositories/5" = create("repository/lhcbonline_x86_64_sl4updates");

17. Quattor Structure Install server base OS dhcp pxe nfs/http Vendor System installer RH73, RHES, Fedora,… System services AFS,LSF,SSH,accounting.. Installed software kernel, system, applications.. CCM Node Configuration Manager (NCM) ‏ RPM, PKG nfs http ftp Software Servers packages (RPM, PKG) ‏ SWR ep packages CDB Managed nodes Node (re)install cache SW package Manager (SPMA) ‏ Install Manager

18. Quattor On quattor server(s) : –(S)CDB : Configuration DataBase –SwRep : Software repositories –Vendor system installer (kickstart) ‏ On each node : –CCM : Configuration Cache Manager –SPMA : Software Package Manager –NCM : Node configuration manager NCM Components for each configurable item

19. Software management SPMA : Software packages Manager Agent (NCM component on the nodes) ‏ Repository templates Package group templates –Generated from RedHat's comps.xml Defaults templates –Define default versions for all packages Ability to add/replace specific packages/versions

20. Installation Based on standard software –DHCP –PXElinux (NBP) ‏ –Kickstart aii-shellfe/aii-installfe tools : –Generate kickstart files –Configures dhcp & PXElinux –From (S)CDB data

21. NCM &NCM components NCM (Node Configuration Manager) is responsible for ensuring that reality on a node reflects the desired state in CDB. Framework system, where service specific plug-ins called Components make the necessary system changes –Regenerate local config files (eg. /etc/sshd/sshd_config) ‏ –Restart/reload services (SysV scripts) ‏ –configuration dependencies (eg. configure network before sendmail) ‏ Components invoked on boot, via cron or on CDB config changes (synchronisation subsystem) ‏

22. Available NCM components access_control Accounts Altlogrotate Authconfig Autofs Chkconfig Cron Dirperm Etcservices Filecopy Fmonagent Grub Interactivelimits Iptables Ldconf Lmsensors Logrotate Mailaliases Modprobe Named Netdriver Network Nfs Ntpd Portmap Profile Rproxy Serialclient Smartd Ssh State Symlink Sysctl yaim Internal quattor components Aii Ccm Cdp Spma

23. CDB tools Quattor CDB cli : cdbop –Transfer templates to/from CDB –Compiles templates with panc Panc : Pan templates compiler –Validates templates –Compiles templates  Creates one XML file per node

24. CDB tools Quattor CDB cli : cdbop –Transfer templates to/from CDB –Compiles templates with panc Panc : Pan templates compiler –Validates templates –Compiles templates  Creates one XML file per node Replaced by SCDB : Subversion Ant Makefile

25. Tools for repositories repos_gen/fill/fill.py : generates “Default” templates –We probably could add some function to this script to also generate repositories templates (in a lighter way than swrep-soap-client) : Done –uses yum functions Tool to generate template lines repos_gen/fill/gen_rpm_info.py Tool to test the software configuration checkdeps