1. Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

2. EMI INFSO-RI-261611 Outline Brief introduction to IPv6 enabled server code Nagios dual stack service monitoring Nagios probes extension to support dual stack check EMI IPv6 Testbed

3. EMI INFSO-RI-261611 IPv6 enabled server

4. EMI INFSO-RI-261611 05/09/2012 4 Dual stack Server binds both 0.0.0.0, :: addresses

5. EMI INFSO-RI-261611 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4 setsockopt(4, SOL_SOCKET, SO_SNDBUF, [65536], 4) = 0 setsockopt(4, SOL_SOCKET, SO_RCVBUF, [65536], 4) = 0 setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0 fcntl(4, F_GETFL) = 0x2 (flags O_RDWR) fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0 setsockopt(4, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 bind(4, {sa_family=AF_INET6, sin6_port=htons(12366), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0 Listen(4, 100) = 0 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 5 setsockopt(5, SOL_SOCKET, SO_SNDBUF, [65536], 4) = 0 setsockopt(5, SOL_SOCKET, SO_RCVBUF, [65536], 4) = 0 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0 fcntl(5, F_GETFL) = 0x2 (flags O_RDWR) fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 bind(5, {sa_family=AF_INET, sin_port=htons(12366), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 listen(5, 100) = 0 select(6, [4 5], [4 5], [4 5], {30, 0}) = 0 (Timeout) select(6, [4 5], [4 5], [4 5], {30, 0}) = 1 (in [5], left {12, 929341}) fcntl(5, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK) fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 accept(5, {sa_family=AF_INET, sin_port=htons(34736), sin_addr=inet_addr("127.0.0.1")}, [16]) = 6 setsockopt(6, SOL_SOCKET, SO_SNDBUF, [65536], 4) = 0 setsockopt(6, SOL_SOCKET, SO_RCVBUF, [65536], 4) = 0 setsockopt(6, SOL_TCP, TCP_NODELAY, [1], 4) = 0 recvfrom(6, "POST / HTTP/1.1\r\nHost: 127.0.0.1"..., 65536, 0, NULL, NULL) = 606

6. EMI INFSO-RI-261611 05/09/2012 6 Dual stack Server binds :: address, using IPv4 mapped addresses (RFC 4038)...but: IPv4 mapped addresses – not supported by OpenBSD, Windows {2000, XP, 2003}

7. EMI INFSO-RI-261611 05/09/2012 7 Dual stack Server binds only 0.0.0.0 => Service is unavailable on IPv6 socket Problem if: Host has AAAA DNS record Client prefers IPv6 to IPv4 In case of fail client doesn't try IPv4 OK if: Clients are aware of this (MySQL) => no try for IPv6 connection

8. EMI INFSO-RI-261611 05/09/2012 8 Getaddrinfo() IP protocol version agnostic Used for DNS queries Used for acquiring addresses for bind() Returns linked list The sorting function used within getaddrinfo() is defined in RFC 3484 /etc/gai.conf The order can be tweaked for a particular system by editing /etc/gai.conf

9. EMI INFSO-RI-261611 Real world example 1 [root@gtv6-emi14 ~]# cat /etc/redhat-release Scientific Linux release 6.3 (Carbon) [root@gtv6-emi14 ~]# uname -a Linux gtv6-emi14 2.6.32-131.2.1.el6.x86_64 #1 SMP Thu Jun 2 09:49:26 \ CDT 2011 x86_64 x86_64 x86_64 GNU/Linux [root@gtv6-emi14 ~]#./getaddrinfo.app #00: IPv4 address: 0.0.0.0 (-) #01: IPv6 address: :: (-) [root@gtv6-emi14 ~]# netstat -tunap | grep srm Tcp 0 0 0.0.0.0:8446 0.0.0.0:* LISTEN 1092/srmv2.2

10. EMI INFSO-RI-261611 Consequence Client is unable to connect via IPv6 SYN: 14:02:57.041337 IP6 2001:1458:301:a873::100:140.53729 > 2001:1458:301:a87c::100:1cd.8446: S 117254423:117254423(0) win 5760 RESET: 14:02:57.041923 IP6 2001:1458:301:a87c::100:1cd.8446 > 2001:1458:301:a873::100:140.53729: R 0:0(0) ack 117254424 win 0

11. EMI INFSO-RI-261611 Consequence detail

12. EMI INFSO-RI-261611 Quick hack [root@gtv6-emi14 ~]# cat /etc/gai.conf label ::/0 0 label 0.0.0.0/0 1 precedence ::/0 40 precedence 0.0.0.0/0 10 [root@gtv6-emi14 ~]#./getaddrinfo.app #00: IPv6 address: :: (-) #01: IPv4 address: 0.0.0.0 (-) [root@gtv6-emi14 ~]# netstat -tunap | grep srm Tcp 0 0 :::8446 :::* LISTEN 4215/srmv2.2

13. EMI INFSO-RI-261611 Nagios dual stack monitoring

14. EMI INFSO-RI-261611 05/09/2012 14 Nagios checks dual stack Before using NCG patch After using NCG patch

15. EMI INFSO-RI-261611 05/09/2012 15 Nagios configuration

16. EMI INFSO-RI-261611 Metric configuration change to declare dual stack check support: $WLCG_SERVICE->{'org.sam.SRM-All'}->{native} = "Nagios"; $WLCG_SERVICE->{'org.sam.SRM-All'}->{config} = {%{$SERVICE_TEMPL->{60}}}; $WLCG_SERVICE->{'org.sam.SRM-All'}->{probe} = 'org.sam/SRM-probe'; $WLCG_SERVICE->{'org.sam.SRM-All'}->{metricset} = "org.sam.SRM"; $WLCG_SERVICE->{'org.sam.SRM-All'}->{dependency}->{"hr.srce.SRM2-CertLifetime"} = 1; $WLCG_SERVICE->{'org.sam.SRM-All'}->{dependency}->{"hr.srce.GridProxy-Valid"} = 0; # line declaring that this service supports --4 and --6 switches $WLCG_SERVICE->{'org.sam.SRM-All'}->{flags}->{'DEFAULTDUALSTACK'} = 1; 05/09/2012 16 NCG Hash.pm AAAA DNS record is required For host to use dual stack checks

17. EMI INFSO-RI-261611 Probe extension to support dual stack tests

18. EMI INFSO-RI-261611 Problem We want probes to support --4, --6 switches If no switch from {--4,--6} is provided: -> default behavior (let system resolver decide) If --4 is provided: -> probe MUST use SOME IPv4 address from DNS response for testing service If --6 is provided: -> probe MUST use SOME IPv6 address from DNS response for testing service

19. EMI INFSO-RI-261611 Probes Some probes uses clients with support for particular IP stack test => no need to hack in order to test particular stack. hr.srce.check_nmap_tcp uses nmap (network scanner) [root@gtv6-emi02 ~]# nmap gtv6-emi14.cern.ch -p 8446 -v Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2012-09-04 14:53 Initiating ARP Ping Scan against 128.142.136.156 [1 port] at 14:53 Initiating SYN Stealth Scan against gtv6-emi14.cern.ch (128.142.136.156) Interesting ports on gtv6-emi14.cern.ch (128.142.136.156): PORT STATE SERVICE 8446/tcp open unknown MAC Address: 00:15:5D:FF:53:79 (Microsoft)

20. EMI INFSO-RI-261611 Probes Some probes uses clients with support for particular IP stack test => no need to hack in order to test particular stack. hr.srce.check_nmap_tcp uses nmap (network scanner) [root@gtv6-emi02 ~]# nmap gtv6-emi14.cern.ch -p 8446 -v -6 Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2012-09-04 14:55 Machine 2001:1458:301:a87c::100:1cd is actually LISTENING on probe port 80 Initiating Connect() Scan against gtv6-emi14.cern.ch \ (2001:1458:301:a87c::100:1cd) [1 port] at 14:55 Discovered open port 8446/tcp on 2001:1458:301:a87c::100:1cd Interesting ports on gtv6-emi14.cern.ch (2001:1458:301:a87c::100:1cd): PORT STATE SERVICE 8446/tcp open unknown

21. EMI INFSO-RI-261611 05/09/2012 21 Wrapping Wrapping does not work with many probes (BDII lookup): 2012-09-04T12:44:35Z Querying BDII ldap://emiipv6bdiit.cern.ch:2170 2012-09-04T12:44:35Z No information for [base: o=grid; filter: (|(&(GlueChunkKey=GlueSEUniqueID=128.142.136.156) (|(GlueSAAccessControlBaseRule=dteam) (GlueSAAccessControlBaseRule=VO:dteam))) (&(GlueChunkKey=GlueSEUniqueID=128.142.136.156) (|(GlueVOInfoAccessControlBaseRule=dteam) (GlueVOInfoAccessControlBaseRule=VO:dteam))) (&(GlueServiceUniqueID=*://128.142.136.156*) (GlueServiceVersion=2.*) (GlueServiceType=srm*))); attribute(s): ['GlueServiceEndpoint', 'GlueSAPath', 'GlueVOInfoPath']] in [ldap://emiipv6bdiit.cern.ch:2170 [128.142.140.128]]. CRITICAL: METRIC FAILED [org.sam.SRM-GetSURLs]: CRITICAL: No information for [attribute(s): ['GlueServiceEndpoint', 'GlueSAPath', 'GlueVOInfoPath']] in [ldap://emiipv6bdiit.cern.ch:2170 [128.142.140.128]].

22. EMI INFSO-RI-261611 Resolver

23. EMI INFSO-RI-261611 Resolver

24. EMI INFSO-RI-261611 Resolver

25. EMI INFSO-RI-261611 Resolver

26. EMI INFSO-RI-261611 Resolver

27. EMI INFSO-RI-261611 Resolver

28. EMI INFSO-RI-261611 Resolver

29. EMI INFSO-RI-261611 Resolver

30. EMI INFSO-RI-261611 05/09/2012 30 Using resolver 2012-09-04T11:50:55Z Querying BDII ldap://gtv6-emi03.cern.ch:2170 2012-09-04T11:50:55Z GlueServiceEndpoint: httpg://gtv6-emi14.cern.ch:8446/srm/managerv2 Resolving gtv6-emi14.cern.ch to 128.142.136.156 SRM endpoint(s) to test: srm://128.142.136.156:8446/srm/managerv2?SFN= /dpm/cern.ch/home/dteam 2012-09-04T12:06:17Z Querying BDII ldap://gtv6-emi03.cern.ch:2170 2012-09-04T12:06:17Z GlueServiceEndpoint: httpg://gtv6-emi14.cern.ch:8446/srm/managerv2 GlueVOInfoPath: /dpm/cern.ch/home/dteam Resolving gtv6-emi14.cern.ch to [2001:1458:301:a87c::100:1cd] SRM endpoint(s) to test: srm://[2001:1458:301:a87c::100:1cd]:8446/srm/managerv2?SFN= /dpm/cern.ch/home/dteam With --4 switch With –-6 switch

31. EMI INFSO-RI-261611 Gridmon python probes resolver.setRecord('cern.ch', '127.0.0.1' ) resolver.resolve('cern.ch' ) # will return 127.0.0.1 resolver.unsetRecord('cern.ch' ) resolver.resolve('cern.ch' ) # will return cern.ch def setResolver (self, resolver): """Set another resolver to probe""" def resolveHost(self, host): """Resolve host with internal resolver, if none - use identity""" endpoint2=endpoint.replace(self.hostName, \ self.resolveHost(self.hostName)) Framework for Nagios probes Central metric invocation => suitable for extension Resolver object: Methods in base class: Usage in metric (base class takes care about resolving):

32. EMI INFSO-RI-261611 Gridmon perl probes Library for Nagios probes Lack of central metric invocation, only helper classes Usage: use GridMon::DualStackUtils qw( &getResolver ); use GridMon::DualStackResolver; use Socket qw( AF_INET AF_INET6 ); my $resolver = getResolver( $plugin->opts->hostname, Socket::AF_INET); $ENV{ DPNS_HOST} = $resolver->resolve( $plugin->opts->hostname);

33. EMI INFSO-RI-261611 Want to know more? See documentation: https://tomtools.cern.ch/jira/browse/GTSL-32 https://tomtools.cern.ch/jira/browse/GTSL-33

34. EMI IPv6 testbed

35. EMI INFSO-RI-261611 Testbed Site name: cert-tb6-cern VO: emiipv6

36. EMI INFSO-RI-261611 Testbed ServiceHost nameIPv6 Tested BDII Topemiipv6bdiit.cern. ch BDII Sitegtv6- emi03.cern.ch VOMSgtv6- emi06.cern.ch MyProxygtv6- emi08.cern.ch DPM head+diskgtv6- emi04.cern.ch DPM head+diskgtv6- emi14.cern.ch LFCgtv6- emi07.cern.ch FTS+FTAgtv6- emi11.cern.ch CREAM, Torqueemi-ipv6- ce.cern.ch WNgtv6- emi05.cern.ch ARGUSgtv6- emi13.cern.ch APELemiipv6apel.cern. ch Missing WMS/RMGA L&Bgtv6- emi12.cern.ch Missing WMS SAM Nagiosgtv6- emi02.cern.ch

37. EMI INFSO-RI-261611 Java IPv6 compliance “Using IPv6 in Java is easy; it is transparent and automatic. Unlike in many other languages, no porting is necessary. In fact, there is no need to even recompile the source files.” => java based services should work with high probability [ http://docs.oracle.com/javase/1.5.0/docs/guide/net/ipv6_guide/index.html ]http://docs.oracle.com/javase/1.5.0/docs/guide/net/ipv6_guide/index.html

38. EMI INFSO-RI-261611 Java based services ServiceHost nameIPv6 Tested BDII Topemiipv6bdiit.cern. ch BDII Sitegtv6- emi03.cern.ch VOMSgtv6- emi06.cern.ch MyProxygtv6- emi08.cern.ch DPM head+diskgtv6- emi04.cern.ch DPM head+diskgtv6- emi14.cern.ch LFCgtv6- emi07.cern.ch FTSgtv6- emi11.cern.ch CREAMemi-ipv6- ce.cern.ch WNgtv6- emi05.cern.ch ARGUSgtv6- emi13.cern.ch APELemiipv6apel.cern. ch L&Bgtv6- emi12.cern.ch SAM Nagiosgtv6- emi02.cern.ch

39. EMI INFSO-RI-261611 Does it really work?

40. EMI INFSO-RI-261611 Service testing All daemons are running, according to System administrator guide or service reference card. No critical errors or important warnings were found in log files SAM Nagios for all services -> no critical problem reported Tested by user interface (UI) client applications (if applicable -> service has all needed dependencies to run) no error found, everything works test of high-level services (FTS, CE) documented in testing protocol ({packet trace, log files, system call trace} available)

41. EMI INFSO-RI-261611 FTS test coverage ServiceHost nameIPv6 Tested BDII Topemiipv6bdiit.cern. ch BDII Sitegtv6- emi03.cern.ch VOMSgtv6- emi06.cern.ch MyProxygtv6- emi08.cern.ch DPM head+diskgtv6- emi04.cern.ch DPM head+diskgtv6- emi14.cern.ch LFCgtv6- emi07.cern.ch FTS+FTAgtv6- emi11.cern.ch CREAM, Torqueemi-ipv6- ce.cern.ch WNgtv6- emi05.cern.ch ARGUSgtv6- emi13.cern.ch APELemiipv6apel.cern. ch L&Bgtv6- emi12.cern.ch SAM Nagiosgtv6- emi02.cern.ch

42. EMI INFSO-RI-261611 CE Test coverage ServiceHost nameIPv6 Tested BDII Topemiipv6bdiit.cern. ch BDII Sitegtv6- emi03.cern.ch VOMSgtv6- emi06.cern.ch MyProxygtv6- emi08.cern.ch DPM head+diskgtv6- emi04.cern.ch DPM head+diskgtv6- emi14.cern.ch LFCgtv6- emi07.cern.ch FTS+FTAgtv6- emi11.cern.ch CREAM, Torqueemi-ipv6- ce.cern.ch WNgtv6- emi05.cern.ch ARGUSgtv6- emi13.cern.ch APELemiipv6apel.cern. ch L&Bgtv6- emi12.cern.ch SAM Nagiosgtv6- emi02.cern.ch

43. EMI INFSO-RI-261611 Testbed summary All deployed services are running Majority of services was tested by client programs Services are being tested by Nagios probes Grid testbed is usable - submit job, transmission job, dpm, voms, etc... (no problem was found) But it is still not perfect...

44. EMI INFSO-RI-261611 IPv4 only services ServiceHost nameIPv4-Only BDII Topemiipv6bdiit BDII Sitegtv6-emi03 VOMSgtv6-emi06VOMS entry 15000 MyProxygtv6-emi08MyProxy srvr 7512 DPM head+diskgtv6-emi04 DPM head+diskgtv6-emi14 LFCgtv6-emi07 FTS+FTAgtv6-emi11 CREAM, Torqueemi-ipv6-ceBnotifier, MAUI, PBS_server WNgtv6-emi05PBS_mom (u1023, 15002, 15003) ARGUSgtv6-emi13 APELemiipv6apel L&Bgtv6-emi12 SAM Nagiosgtv6-emi02

45. EMI INFSO-RI-261611 Strange binding svcs ARGUS: tcp 0 0 ::ffff:128.142.18.55:8150 :::* LISTEN 55256/java tcp 0 0 ::ffff:128.142.18.55:8152 :::* LISTEN 61197/java tcp 0 0 ::ffff:128.142.18.55:8154 :::* LISTEN 61102/java Cause: YAIM configuration uses host FQDN to specify socket to bind -> by DNS resolved to public IP address [root@gtv6-emi13 ~]# telnet 127.0.0.1 8150 Trying 127.0.0.1... telnet: connect to address 127.0.0.1: Connection refused [root@gtv6-emi13 ~]# telnet 128.142.18.55 8150 Trying 128.142.18.55... Connected to 128.142.18.55. Escape character is '^]'.

46. EMI INFSO-RI-261611 Tools for testing

47. EMI INFSO-RI-261611 Manipulated DNS server Hostname: emiipv6dns.cern.ch Recursive DNS server +... Uses Response Policy Zone (RPZ) mechanism in order to answer on DNS queries from foreign zone with defined answer – ability to tamper DNS responses. [root@emiipv6dns dklinec]# cat /var/named/rpz $TTL 60 @ IN SOA localhost. root.localhost. ( 100 ; serial 10m ; refresh 10m ; retry 10m ; expiry 10m) ; minimum IN NS localhost. ; just testing record non-existing-domain.comCNAME www.cern.ch. emi-ipv6-ce.cern.chINA 137.138.163.53 ; hide IPv6 record ;emi-ipv6-ce.cern.chINAAAA 2001:1458:201:b30a:215:5dff:feff:449b

48. EMI INFSO-RI-261611 Getaddrinfo() res. order [root@gtv6-emi14 ~]#./getaddrinfo.app #00: IPv6 address: :: (-) #01: IPv4 address: 0.0.0.0 (-) Returns result of getaddrinfo() suitable for binding Using PF_UNSPEC, AI_PASSIVE Usage:

49. EMI INFSO-RI-261611 Port binding check Helps to reveal IPv4-Only services, not properly configured services, firewall configuration problems ################################################################################ #Netstat analysis host: gtv6-emi13.cern.ch ################################################################################ All listening services: tcp 2170 (8076/slapd) |W: 0.0.0.0 1 :: 0 tcp 8150 (55256/java) |W: 0.0.0.0 0 :: 1 tcp 8152 (61197/java) |W: 0.0.0.0 0 :: 1 tcp 8154 (61102/java) |W: 0.0.0.0 0 :: 1 IPv6 Only services: tcp 8150 (55256/java) |W: 0.0.0.0 0 :: 1 tcp 8152 (61197/java) |W: 0.0.0.0 0 :: 1 tcp 8154 (61102/java) |W: 0.0.0.0 0 :: 1 IPv4 Only services!!! : tcp 2170 (8076/slapd) |W: 0.0.0.0 1 :: 0 Results: L4 Protocol: tcp ! Problem with tcp 8150 (55256/java) on :: IPv4: 0 IPv6: 1; Closed port on 2001:1458:301:a868::100:2a ! Problem with tcp 8152 (61197/java) on :: IPv4: 0 IPv6: 1; Closed port on 2001:1458:301:a868::100:2a ! Problem with tcp 8154 (61102/java) on :: IPv4: 0 IPv6: 1; Closed port on 2001:1458:301:a868::100:2a

50. EMI INFSO-RI-261611 $./wrapper.py --cmd './longtest.sh' --strace --tcpdump --cmdid longtest \ --destdir /tmp/longtest/ --prefix alfa --suffix t0 --wait 10 ## Starting TCPDump: /usr/bin/sudo /usr/sbin/tcpdump -w "/tmp/longtest//alfa-tcpdump-longtestt0. pcap" ## Starting work job:./longtest.sh ## Starting blocking operation: ['/usr/bin/strace', '-f', '-s', '512', '-v', '-o', '/tmp/longtest//alfa-strace-longtest-t0', '--', './longtest.sh'] ## Thread should be stopped now: ['/usr/bin/strace', '-f', '-s', '512', '-v', '-o', '/tmp/longtest//alfa-strace-longtest-t0', '--', './longtest.sh'] ## Work finished! ## Stdout+stderr (/tmp/longtest//alfa-stdout-longtest-t0): ================================================================================ PING 128.142.18.54 (128.142.18.54) 56(84) bytes of data. 64 bytes from 128.142.18.54: icmp_req=1 ttl=58 time=0.965 ms 64 bytes from 128.142.18.54: icmp_req=2 ttl=58 time=1.24 ms 64 bytes from 128.142.18.54: icmp_req=3 ttl=58 time=1.08 ms --- 128.142.18.54 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 0.965/1.097/1.243/0.117 ms OK Ending Now! ================================================================================ ## Going to sleep for 10 seconds. ^C## Exception reported, ending waiting, ## Stopping tcpdumps ## Stopping dumper ## Stopping tailers ## Thread should be stopped now: ['/usr/bin/sudo', '/usr/sbin/tcpdump', '-w', '/tmp/longtest//alfatcpdump- longtest-t0.pcap'] Artifact collection (wrapper)

51. EMI INFSO-RI-261611 Artifact collection (wrapper) Files produced: $ ll /tmp/longtest// total 372 -rw-r--r-- 1 root root 404Aug 31 12:40 alfa-stdout-longtest-t0 -rw-r--r-- 1 root root 29234Aug 31 12:40 alfa-strace-longtest-t0.29352 -rw-r--r-- 1 root root 6054Aug 31 12:40 alfa-strace-longtest-t0.29353 -rw-r--r-- 1 root root 17483Aug 31 12:40 alfa-strace-longtest-t0.29354 -rw-r--r-- 1 root root 6054Aug 31 12:40 alfa-strace-longtest-t0.29355 -rw-r--r-- 1 root root 164Aug 31 12:40 alfa-tcpdump-longtest-t0.out -rw-r--r-- 1 root root 236649Aug 31 12:40 alfa-tcpdump-longtest-t0.pcap

52. EMI INFSO-RI-261611 Artifact collection (server side) [root@gtv6-emi11 glite-transfer-submit]# ll total 8128 -rw-r--r--. 1 root root 0 Aug 30 17:17 logfile-fts-catalina.out-t0 -rw-r--r--. 1 root root 141654 Aug 30 17:20 logfile-fts-glite-transfer-channel-agent-urlcopy-CERN3-CERN3.log-t0 -rw-r--r--. 1 root root 138377 Aug 30 17:20 logfile-fts-glite-transfer-vo-agent-DTEAM.log-t0 -rw-r--r--. 1 root root 145599 Aug 30 17:20 logfile-fts-glite-transfer-vo-agent-EMIIPV6.log-t0 -rw-r--r--. 1 root root 2982 Aug 30 17:19 logfile-fts-org.glite.data-t0 -rw-r--r--. 1 root root 0 Aug 30 17:17 logfile-system-messages-t0 -rw-r--r--. 1 root root 0 Aug 30 17:17 logfile-system-secure-t0 -rw-r--r--. 1 root root 1784 Aug 30 17:17 strace-fts- glite_transfer_channel_agent_urlcopy_CERN3_CERN3.63194-t0.13001 -rw-r--r--. 1 root root 16392 Aug 30 17:17 strace-fts- glite_transfer_channel_agent_urlcopy_CERN3_CERN3.63194-t0.13002 -rw-r--r--. 1 root root 45 Aug 30 17:17 strace-fts-glite_transfer_vo_agent_DTEAM.59354-t0.59354 -rw-r--r--. 1 root root 788651 Aug 30 17:20 strace-fts-glite_transfer_vo_agent_DTEAM.59354-t0.59355 -rw-r--r--. 1 root root 45 Aug 30 17:17 strace-fts-glite_transfer_vo_agent_EMIIPV6.59373-t0.59373 -rw-r--r--. 1 root root 902175 Aug 30 17:20 strace-fts-glite_transfer_vo_agent_EMIIPV6.59373-t0.59374 -rw-r--r--. 1 root root 45 Aug 30 17:17 strace-fts- _usr_lib_jvm_jre_1_6_0_openjdk_x86_64_bin_java.57775-t0.57775 -rw-r--r--. 1 root root 51 Aug 30 17:17 strace-fts- _usr_lib_jvm_jre_1_6_0_openjdk_x86_64_bin_java.57775-t0.57780 -rw-r--r--. 1 root root 129817 Aug 30 17:19 strace-fts- _usr_lib_jvm_jre_1_6_0_openjdk_x86_64_bin_java.57775-t0.58739 -rw-r--r--. 1 root root 162 Aug 30 17:20 tcpdump-fts-t0 -rw-r--r--. 1 root root 371476 Aug 30 17:20 tcpdump-fts-t0.pcap -rw-r--r--. 1 root root 164 Aug 30 17:20 tcpdump-system-t0 -rw-r--r--. 1 root root 2772184 Aug 30 17:20 tcpdump-system-t0.pcap

53. EMI is partially funded by the European Commission under Grant Agreement RI-261611 Questions? 05/09/2012 53