June 2011 David Front Weizmann Institute Frontier rpms June 2011 David Front Weizmann Institute 8 June 2011 frontier rpms

8 June 2011 frontier rpms

Why rpms? Once a site (takes the effort and) moves to use rpms upgrading SW is faster, and easier compared to using tarballs It should also be more safe … assuming that proper testing of the new rpm release has been done Yum package manager is supported by frontier rpms With yum, installing, upgrading and removing is easier that via rpm: - yum is aware of rpm dependencies - No need to specify rpm version, only package name Automatic upgrade possibility: - Periodically do ‘yum upgrade package’ (via crontab) Note, however, that relocation is not supported by yum 8 June 2011 frontier rpms

Frontier rpms – history and status Frontier has been written by CMS and distributed as tarballs Favia Donno did create initial frontier rpms for ATLAS: frontier-squid frontier-tomcat frontier-servlet frontier-awstats A new release of frontier rpms is available, is in use at CERN ATLAS product servers since today, and is planned to be defined as ‘production ready’ soon: In particular, there is a need for rpms to be tested out of CERN first Current use of frontier rpms: ATLAS (and CMS?) has squids installed as rpms The other frontier rpms are installed only at ATLAS servers at CERN: CMS, and non CERN ATLAS frontier servers are installed via tarballs. 8 June 2011 frontier rpms

Installing frontier rpms: Prerequisites The (configurable) user to own rpm files should exist prior to rpm installation: Default: ‘squid’ for squid ‘dbfrontier’ for the other rpms If you are installing frontier rpms, instead of existing non rpm (tarball) package/s: First uninstall the existing non rpm SW: Remove the existing directory in which the software was installed Take care to remove/undo changes at other files that have been edited manually or by scripts. In particular, undo crontab (manual) changes. Verify that squid/tomcat service actually stopped: Be aware that: Stopping squid/tomcat may take a long time, or: ‘service frontier-{squid or tomcat} status’ may say by mistake ‘stopped’, while ‘ps –efw|grep {squid or tomcat}’ is not empty. 8 June 2011 frontier rpms

Installing frontier rpms: configuration Manually take care of per rpm and per site and per site/machine files, if any. For details, consult later slides, and: http://frontier.cern.ch/dist/rpms/_README Configuration changes: As a rule of thumb, configuration changes are applied to files as follows: The directory is /etc/{squid|awstats(|tomcat)} Configuration changes of customize.sh are applied when the appropriate server restarts. Hence, there should be no need to reinstall an rpm in order to apply (such) configuration changes. In addition, for frontier-squid, squidconf is applied at install time 8 June 2011 frontier rpms

Installing frontier rpms (cont.) In order to install frontier rpms, as user root, import the gpg public key once (per machine): rpm --import http://frontier.cern.ch/dist/rpms/cernFrontierGpgPublicKey Using the command 'rpm': as user root: rpm -Uvh [--prefix <RPM_INSTALL_PREFIX>] http://frontier.cern.ch/dist/rpms/<rpm> Relocating an rpm: The defaults of RPM_INSTALL_PREFIX are: frontier-squid: / frontier-awstats: / frontier-tomcat: /data/dbfrontier frontier-servlet: /data/dbfrontier/tomcat/webapps/atlr (for servlet atlr) The frontier rpms are relocatable Exception: frontier-tomcat and frontier-servlet rpms can not be relocated to / The plan is that / will be the default for them as well. 8 June 2011 frontier rpms

The current rpm versions At: http://frontier.cern.ch/dist/rpms/: frontier-awstats-6.0-1.noarch.rpm frontier-servlet-3.29-1.noarch.rpm frontier-tomcat-6.0.32-1.noarch.rpm frontier-squid-2.7.STABLE9-5.3.sl5.x86_64.rpm 8 June 2011 frontier rpms

Installing via yum get cernFrontier repository file once per machine: wget -O /etc/yum.repos.d/cern-frontier.repo http://frontier.cern.ch/dist/rpms/cern-frontier.repo Now you may use yum as usual. for example: yum install package1 [package2] [...] yum upgrade package1 [package2] [...] yum remove package1 [package2] [...] 8 June 2011 frontier rpms

Testing the installation: tomcat, squid For a simple test of tomcat and squid servers, using fnget.py: you may adjust and run commands like the following, from bash: wget -O /tmp/fnget.py http://frontier.cern.ch/dist/fnget.py chmod +x /tmp/fnget.py server=atlasfrontier1.cern.ch # Adjust this to your server servlet=atlr # Adjust this to you servlet port=8000 # To test tomcat. Adjust if needed /tmp/fnget.py \ --url='http://'${server}':'${port}'/'${servlet}'/Frontier' \ --sql="select 'TOMCAT_IS_HAPPY' from dual" \ --refresh-cache port=8080 # To test squid. Adjust if needed /tmp/fnget.py \ --url='http://'${server}':'${port}'/'${servlet}'/Frontier‘ \ --sql="select 'SQUID_IS_HAPPY' from dual“ \ --refresh-cache If all went well, you should get records like the following: 'TOMCAT_IS_HAPPY' CHAR 'SQUID_IS_HAPPY' CHAR 8 June 2011 frontier rpms

Testing the installation: awstats, logrotate To see that awstats works, one may wait a while and see that the numbers or the upgraded squid do grow Logrotates: Logs should not grow too much: /data/tomcat_logs/catalina.out /data/tomcat_logs/modifiedqueries.out /data/squid_logs/access.log To see the allowed sizes: sudo –u <squid/tomcat user> crontab –l Find there when logrotate should work and sizes are allowed 8 June 2011 frontier rpms

Testing the installation: load (After a while,) monitor the upgraded server/s host load, verifying that it did not rise or drop considerably, as a result of the upgrade: CERN: use lemon https://lemonweb.cern.ch/ Else where: ganglia or whatever host monitoring tool is in use 8 June 2011 frontier rpms

Building a frontier rpm rpms should be built at voatlas137 The related sources should be checked out from svn, under: /data/$USER/rpms Build an rpm: cd /data/$USER/rpms/<rpm Name> pushd ./scripts; ../../scripts/rpmbuild_.sh;popd 2) Copy rpm to the web: pushd ./scripts; ../../scripts/rpm2web.sh ; popd 8 June 2011 frontier rpms

Plan Once the new rpms will be defined as production ready, it will be suggested that all ATLAS frontier sites will use these frontier rpm/s instead of existing rpm or tarball/s Enhance frontier rpms to support CMS Support more than one servlet at a site. ATLAS does not require this because it has only one servlet per site Add a /etc/tomcat/servlets.conf configuration file Add a /etc/tomcat/servlets.passwd configuration file (will reside at sindes at CERN) 3) Avoid file duplication at underlying (CMS) tarball and at the rpms code (mainly at frontier-squid). This will make the maintenance of frontier code simpler 4) TBD: Add more monitoring tools into the rpms 8 June 2011 frontier rpms

8 June 2011 frontier rpms

Configuration: frontier-squid Optional configuration files: 1) /etc/squid/squidconf: Used at rpm installation Consists of export commands Main usage: to specify user and group for frontier-squid, other than default user and group: squid, squid Example (as of CERN): export FRONTIER_USER=dbfrontier export FRONTIER_GROUP=users 2) /etc/squid/customize.sh Used per restart of the squid server. Consists of specific commands that control the content of squid.conf file. Examples: Appear at /etc/squid/customize.sh.example_* and at https://svnweb.cern.ch/trac/frontier/browser/rpms/frontier-squid/trunk/src/frontier-squid-utils/etc/squid -See also: Flavia Donno's documentation for previous version of frontier-squid rpms: https://twiki.cern.ch/twiki/bin/view/PDBService/SquidRPMsTier1andTier2 8 June 2011 frontier rpms

Configuration: frontier-servlet Per servlet configuration file should be edited manually. May be done after installation: <install dir>/tomcat/conf/Catalina/localhost/<servlet name>.xml Example of such a file name: /data/dbfrontier/tomcat/conf/Catalina/localhost/atlr.xml 8 June 2011 frontier rpms

Configuration: frontier-awstats Optional file to control environment variables at install time /etc/awstats/awstats.conf Syntax: export <variable>=<value> Example: export FRONTIER_USER=<something other than dbfrontier> Editing of the following files may be done (before or) after rpm installation: 1) A file for rsync with the password of user $SITE$PROJ (Example: cernatlas) /etc/awstats/password-file 2) An optional per awstats client site/machine config file. For CERN clients, this file is not needed, but for other sites it may be needed. File: /etc/awstats/customize.sh ## WARNING Care should be taken when changing variable in this file, ( and in particular PATH_LOG). Testing of frontier rpms has not been done with changing of these variables The rpm installs an example file: /etc/awstats/customize.sh.example_CERN: https://svnweb.cern.ch/trac/frontier/browser/rpms/frontier-awstats/trunk/src/etc/awstats/customize.sh.example_CERN 8 June 2011 frontier rpms

urls Documentation of new versions of frontier rpms: http://frontier.cern.ch/dist/rpms/_README The main url of frontier: http://frontier.cern.ch Current frontier rpms: http://frontier.cern.ch/dist/rpms/ To browse svn files of frontier rpms: https://svnweb.cern.ch/trac/frontier/browser/rpms To checkout a frontier rpms svn directory (example: frontier-tomcat): svn co svn+ssh://svn.cern.ch/reps/frontier/rpms/frontier-tomcat/trunc frontier-tomcat Atlas frontier documentation by John Destefano: https://www.racf.bnl.gov/docs/services/frontier/ Documentation of (previous) frontier server rpms, by Flavia Donno: https://twiki.cern.ch/twiki/bin/view/LCG/WLCGFrontier Documentation of (previous) versions of frontier-squid rpm, by Flavia Donno: https://twiki.cern.ch/twiki/bin/view/PDBService/SquidRPMsTier1andTier2 Previous versions of frontier rpms, created by Flavia Donno, are at: http://grid-deployment.web.cern.ch/grid-deployment/flavia/?C=M;O=D CMS has a lot of frontier documentation 8 June 2011 frontier rpms