1. Installation and configuration of a Computing Element
Alfonso Pardo
EPIKH School, System Admin Tutorial
Beijing, August 30th – 2010 September 3th

2. OUTLINE OVERVIEW INSTALLATION & CONFIGURATION TESTING FIREWALL SETUP
TROUBLESHOOTING

3. OVERVIEW The Computing Element is the central service of a site.
Its main functionally are:
manage the jobs (job submission, job control)‏
update to WMS the status of the jobs
publish all site informations (site location, queues, about the CPUs status, and so on) via ldap (site BDII service)‏
It can run several kinds of batch system:
Torque + MAUI
LSF
SGE
Condor

4. TORQUE + MAUI The Torque server is composed by a:
pbs_server which provides the basic batch services such as receiving/creating a batch job.
The Torque client is composed by a:
pbs_mom which places the job into execution. It is also responsible for returning the job’s output to the user
The MAUI system is composed by a:
job_scheduler which contains the site's policies in order to choose which job must be executed.

5. Site BDII** By default it is installed on the CE
It collects all site GRISes* (for example SE,RB,LFC,etc..)‏
The name of the service is bdii
The list of GRISes you want to publish is:
/opt/glite/etc/gip/site-urls.conf
Log file: /opt/bdii/var/bdii.log
*GRIS=Grid Resource Information Service
**BDII=Berkely Database Infomatin Index

6. Computing Element installation & configuration using YAIM

7. WHAT KIND OF CE? There are several kinds of metapackages to install:
glite_CE
LCG ComputingElement without batch system packages.
glite_CE_LSF
LCG ComputingElement with LSF. IMPORTANT: providedfor consistency, it does not install LSF but it apply some fixes via ig_configure_node.
glite_CE_torque
LCG ComputingElement with Torque+MAUI.

8. HOW TO GET AN HOST CERTIFICATE
Host certificate for CE.
Please, request it to your RA
Install host certificate (hostcert.pem and hostkey.pem) in /etc/grid-security.
mkdir /etc/grid-security
chmod 644 hostcert.pem
chmod 400 hostkey.pem
If you have a p12 file:
wget
sh pkcs12toglobus.sh –h xxxxx.p12 .

9. Repository settings
REPOS=”lcg-ca dag ig glite-lcg_cream_torque glite-bdii"
Download and store repo files:
for name in $REPOS; do wget it.cnaf.infn.it/mrepo/repos/sl5/x86_64/$name.repo -O /etc/yum.repos.d/$name.repo; done
wget -O /etc/yum.repos.d/gilda.repo
wget it.cnaf.infn.it/mrepo/repos/jpackage.repo -O /etc/yum.repos.d/jpackage.repo

10. INSTALLATION yum install jdk java-1.6.0-sun-compat yum install lcg-CA
yum install glite-CREAM
yum install glite-TORQUE_server
yum install glite-TORQUE_utils
If it's also the site bdii collector:
yum install glite-BDII
Gilda rpms:
yum install gilda_utils

11. Customize ig-site-info.def
Copy ig-site-info.def template file provided by ig_yaim in to gilda dir and customize it
cp /opt/glite/yaim/examples/siteinfo/ig-site-info.def /opt/glite/yaim/etc/gilda/<your_site-info.def>
Open /opt/glite/yaim/etc/gilda/<your_site-info.def> file using a text editor and set the following values according to your grid environment:
CE_HOST=<write the CE hostname you are installing>
BATCH_SERVER=$CE_HOST

12. Customize ig-site-info.def
WN_LIST=/opt/glite/yaim/etc/gilda/wn-list.conf The file specified in WN_LIST has to be set with the list of all your WNs hostname. WARNING: It’s important to setup it before to run the configure command

13. Customize ig-site-info.def
Copy users and groups example files to /opt/glite/yaim/etc/gilda/
cp /opt/glite/yaim/examples/ig-groups.conf /opt/glite/yaim/etc/gilda/
cp /opt/glite/yaim/examples/ig-users.conf /opt/glite/yaim/etc/gilda/
Append gilda users and groups definitions to /opt/glite/yaim/etc/gilda/ig-users.conf
cat /opt/glite/yaim/etc/gilda/gilda_ig-users.conf >> /opt/glite/yaim/etc/gilda/ig-users.conf
cat /opt/glite/yaim/etc/gilda/gilda_ig-groups.conf >> /opt/glite/yaim/etc/gilda/ig-groups.conf

14. Customize ig-site-info.def
GROUPS_CONF=/opt/glite/yaim/etc/gilda/ig-groups.conf USERS_CONF=/opt/glite/yaim/etc/gilda/ig-users.conf JAVA_LOCATION=”My Java path” SITE_NAME=GILDA SITE_LOC=“Beijing, CHINA" SITE_LAT=37.5 SITE_LONG= SITE_WEB=" SITE_TIER="GILDA Testbed"

15. Customize ig-site-info.def
JOB_MANAGER=lcgpbs CE_BATCH_SYS=pbs BATCH_BIN_DIR=/usr/bin BATCH_VERSION=torque CE_CPU_MODEL=Opteron CE_CPU_VENDOR=AMD CE_CPU_SPEED=3000 CE_OS="Scientific Linux“ CE_OS_RELEASE=5.3 CE_OS_VERSION="SL“ CE_MINPHYSMEM=2048 CE_MINVIRTMEM=4096 CE_SMPSIZE=2 CE_SI00=1000 CE_SF00=1200 CE_OUTBOUNDIP=TRUE CE_INBOUNDIP=TRUE

16. Customize ig-site-info.def
DPM_HOST=“dpm_hostname” SE_LIST="$DPM_HOST“ SITE_BDII_HOST=$CE_HOST BDII_REGIONS="CE SE“ BDII_CE_URL="ldap://$CE_HOST:2170/mds-vo- name=resource,o=grid“ BDII_SE_URL="ldap://$DPM_HOST:2170/mds-vo- name=resource,o=grid“ VOS=“gilda” ALL_VOMS=“gilda”

17. Customize ig-site-info.def
QUEUES="short long infinite“ SHORT_GROUP_ENABLE=$VOS LONG_GROUP_ENABLE=$VOS INFINITE_GROUP_ENABLE=$VOS In case of to configure a queue fo a single VO: QUEUES="short long infinite gilda“ GILDA_GROUP_ENABLE=“gilda”

18. CE Torque CONFIGURATION
Now we can configure the node:
/opt/glite/yaim/bin/ig_yaim -c -s /opt/glite/yaim/etc/gilda/<your_site-info.def> -n creamCE -n TORQUE_server -n TORQUE_utils -n BDII_site
/opt/glite/yaim/bin/yaim -r -s /opt/glite/yaim/etc/gilda/<your_site-info.def> -n creamCE -f config_cream_blparser
# service tomcat5 restart

19. Computing Element
testing

20. Testing
Check if the local GRIS and the site BDII are running on CE and are publishing the right informations (CPU, site name and so on)‏
ldapsearch -x -h <ce_hostname> -p b mds-vo- name=resource,o=grid
ldapsearch -x -h <ce_hostname> -p b mds-vo- name=<site_name>,o=grid

21. Testing Become a gilda user # su – gilda001 Edit a file and write:
#!/bin/sh
sleep 20 #(it's useful to see the job status)‏
hostname
Save it and set the permission of execution:
chmod 700 test.sh

22. Testing
gilda001]$ qsub -q short test.sh gilda001]$ qstat -a ce.localdomain: Req'd Req'd Elap Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time wn.localdo gilda001 short test.sh :15 R --

23. Testing [gilda001@ce gilda001]$ qstat -a [gilda001@ce gilda001]$
The job execution has finished and we have to list the output file:
gilda001]$ ls
test.sh.e3 test.sh.o3
And show them:
gilda001]$ cat test.sh.e3 (error file)‏
gilda001]$ cat test.sh.o3 (output file)‏
wn.localdomain

24. FIREWALL SETUP

25. /etc/sysconfig/iptables (1/2)‏
*filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :RH-Firewall-1-INPUT - [0:0] -A INPUT -j RH-Firewall-1-INPUT -A FORWARD -j RH-Firewall-1-INPUT -A RH-Firewall-1-INPUT -i lo -j ACCEPT -A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport maui -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport pbs_mom -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport pbs_resmom -j ACCEPT

26. /etc/sysconfig/iptables (2/2)‏
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport pbs -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 3878:3879 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 1020:1023 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 20000: j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 32768: j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 32768: j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m tcp --syn -j REJECT -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited COMMIT

27. IPTABLES STARTUP
/sbin/chkconfig iptables on /etc/init.d/iptables start

28. Troubleshooting

29. Troubleshooting
plt]$ globus-job-run <ce_hostname>:2119/jobmanager-lcgpbs -q short /bin/hostname GRAM Job submission failed because the connection to the server failed (check host and port) (error code 12)‏ solution: check if the globus-gatekeeper daemon is up and running on CE GRAM Job submission failed because authentication failed: GSS Major Status: Authentication Failed GSS Minor Status Error Chain: init.c:499: globus_gss_assist_init_sec_context_async: Error during context initialization init_sec_context.c:171: gss_init_sec_context: SSLv3 handshake problems globus_i_gsi_gss_utils.c:888: globus_i_gsi_gss_handshake: Unable to verify remote side's credentials globus_i_gsi_gss_utils.c:847: globus_i_gsi_gss_handshake: Unable to verify remote side's credentials: Couldn't verify the remote certificate OpenSSL Error: s3_pkt.c:1046: in library: SSL routines, function SSL3_READ_BYTES: sslv3 alert bad certificate (error code 7)‏ solution: probably there is no GILDA CA rpm installed on CE

30. Troubleshooting
plt]$ edg-gridftp-ls gsiftp://<ce_hostname>/ error the server sent an error response: LCMAPS credential mapping NOT successful solution: check on CE the VO mapping in /opt/edg/etc/lcmaps/gridmapfile /opt/edg/etc/lcmaps/groupmapfile

31. Troubleshooting
The CE is publishing wrong informations such as: GlueCEStateFreeCPUs: 0 GlueCEStateRunningJobs: 0 GlueCEStateStatus: Production GlueCEStateTotalJobs: 0 GlueCEStateWaitingJobs: 4444 Run the script: /opt/glite/etc/gip/plugin/glite-info-dynamic-scheduler-wrapper and check if it gives some errors. Often it doesn’t work because the batch system is down or in lock state. In this case restart torque service: /etc/init.d/pbs_server restart

32. Troubleshooting
If a query to the site BDII doesn’t show the information about a site, you have to look at the bdii log file /opt/bdii/var/bdii.log
For example:
GILDA: ldap_bind: Can't contact LDAP server
Check if:
bdii is up & running (ps aux |grep bdii)‏
That resource url is in the list file /opt/glite/etc/gip/site-urls.conf
Firewall setup