1. Handling of Unix Application Software Stephan Wiesand DESY -DV - May 25, 2004

2. Motivation in early 2003, we looked for a replacement for our Unix software deployment scheme: installsoftware in /afs/ /products create symlinks in /afs/cell/@sys/usr/local all systems have these symlinks /products -> /afs/ /products /usr/local -> /afs/ /@sys/usr/local the replacement was designed, implemented and deployed since, as time allowed (no dedicated resources), and is presented here

3. The Problem provide additional software on all client systems sw not included with the distribution, or needed in different or multiple versions, or built with different options (kerberized...) targets: Linux, Solaris, & any other potentially relevant platform desktops, farm nodes, (workgroup) servers, controls, notebooks many have disks too small to install all software locally some must function properly without network or other infrastructure

4. Additional Requirements automatic (un)installation, up/downgrade configurable per client (tests...) keep systems in a well defined, working state reproducible without backup verifyable decent dependency handling (shared libraries allowed) within add-on software and against the host system for every client, even if some software isn't local lightweight solution, don't invent a new framework use standard tools, unmodified

5. Solution use a standard package format and a package database separate from the host system's split packages into a main package that can be replaced by a single symlink into a central “reference installation” an optional “default” package (always installed locally) populating directories in PATH, MANPATH etc. with symlinks populate the reference installation from same packages that would be installed locally on the client, maintain the package database as though all packages were installed locally

6. Packager Choice: RPM (all platforms) available and working on all potential targets has all required features versioned dependencies/conflicts, virtual packages automatic for shared library dependencies --justdb switch for updating the database only structured package building from specs is a plus build procedure is fully documented automatically powerful macro facility (one spec for all platforms) writing specs is becoming a common skill excellent documentation for beginners available

7. Closing the Dependency Chain using a separate RPM DB is possible and consistent across platforms doesn't interfere with any mechanism maintaining the client but leaves a dependency gap between system and add-ons solution: glue packages, providing virtual packages (ABI version of shared libraries,...) ghost files (/bin/sh,...) safeguards in pre[uninstall] script enforce --justdb verifying a glue package will check that the system installation keeps its promises

8. Glue Packages to build a glue package, define macros for the missing dependencies for virtual/real packages: add Provides: for files, add %ghost in files section provide a verify script checking existence of files (test -f ) or using /bin/rpm -q --whatprovides can have one or more (glue-devel, glue-legacy,...) possible enhancement: mirror package for system DB requiring everything the glue package provides & vice versa

9. Filesystem Layout & Packaging deliberately chose not to use /usr/local /opt/ products/ bin/ perl->/opt/products/perl/5.8.2/bin/perl pine->/opt/products/pine/4.58/bin/pine perl/ 5.8.2/ bin/ perl* pine/ 4.58/ bin/ pine* perl-5.8.2-2.i586.rpm pine-4.58-1.i586.rpm perl-default-5.8.2-2.i586.rpm pine-default-4.58-1.i586.rpm

10. Packages: Rules base packages contain /opt/products/ / + content nothing else can be replaced by a single symlink to / / default packages contain links to files from base packages anywhere in /opt/products except in any directory owned by a base package must require their base package

11. Refinements what if we want /opt/products/bin/perl5.8.0 ? -> alias packages (like default, but multiple versions ok) what if we want to add files in a base directory ?(additional perl modules, configuration files) -> daughter packages allowed to write anywhere below parent's directory must require their parent names prefixed by _ - what if we want to maintain certain files by other means? (again, configuration files) -> local packages (like default, but links point to local fs)

12. Shared Library Dependencies we do populate /opt/products/lib (and include) but use is discouraged and it's not searched by the dynamic loader by default instead, set runtime search path for shared libs at build time (link)... -Wl,-rpath,/opt/products/ / /lib... sometimes hard to do (improper use of libtool...) means we have to rebuild dependent packages more often also means rolling out new versions doesn't break anything needs a modified find-requires not processing shared libs in /opt/products (spec macro __find_requires)

13. Deployment manual/scripted installation is possible: mkdir -p /opt/products/RPMDB rpm -p --verify --verbose.../glue-2.0-2.i586.rpm rpm --dbpath... -i --justdb.../glue-2.0- 2.i586.rpm rpm --dbpath... -i.../pine-4.58-1.i586.rpm mkdir -p /opt/products/perl ln -s /perl/5.8.2 /opt/products/perl rpm -i --justdb.../perl-5.8.2-2.i586.rpm rpm -i.../perl-default-5.8.2.i586.rpm rpm -i.../pine-default-4.58-1.i586.rpm possible and straightforward, but tedious...

14. Tools instead, implemented ppm perl wrapper calling rpm finds packages in a repository on central filesystem locatedb for locating package files RPM DB (created with -i --justdb) for dependencies, content accepts input like this: [install] justdbglue2.02 localpine4.581default linkperl5.8.22 default,alias

15. ppm features accepts one or more flat file names as input processed sequentially conflicting/redundant input ok last one takes precedence -> overrides, delegation (also has an include mechanism) there's a [remove] tag as well set of input files defines absolute final state not an incremental change final state is verified & sanitized - if possible resolve dependencies automatically correct children of base packages installed as link...

16. Status so far, we mostly built packages for software that needed a rebuild or were unavailable before and a few essential ones and their dependencies in full production on DESY Linux 5 most important packages available, few missing in use on DL4 and Solaris 8 hosts to varying degrees DL4 now obsolete, Solaris actually started this year on both sites (independent of host management solutions) also in use on Linux Notebooks

17. Experience improved preservation and transfer of knowledge about building an application or library properly no diligence (installation notes...) required package builders have to go through a learning curve about one day for building a first package not yet built starting from a similar example once used to it, little overhead to other procedures at least any producing transparent, reproducible results rebuilds and small changes much faster and more reliable we're probably past break even