R-GMA Producers and Archiver Paul Mealor

edg-pinger-timeping edg-netmon-rgma-info Producer API edg-gridftplog2rgma Producer API Log Here be servlets Overview A C DBP P P P edg-netmon2rgma Producer API edg-netmon-archiver Archiver API MySQL DB gsiwuftp FIFO

edg-netmon2rgma Maintains producers for the tools –Tools scripts are rerun for each measurement batch –Program running APIs must be persistent Files –/opt/edg/sbin/edg-netmon2rgma.pl –/opt/edg/var/edg-netmon2rgma/edg-netmon2rgma-fifo –/etc/rc.d/init.d/edg-netmon2rgmad Configuration –RGMA client configuration

Tool/netmon2rgma interaction addmetric pinger rtt 10 NMIdSource=pc18.hep.ucl.ac.uk tool=pinger PingER Normal log edg-netmon2rgma CircularBufferProducer_new(NetworkRTT, WHERE NMIdSource=pc18.hep.ucl.ac.uk AND tool=pinger); CircularBufferProducer_setRemoteBufferSize($producer,10*2); Web interface If producer doesnt exists If bufferSize < count * 4 / 3

Tool/netmon2rgma interaction measurement pinger rtt pc18.hep.ucl.ac.uk edgnm.gridpp.rl.ac.uk pinger PingER Normal log edg-netmon2rgma CircularBufferProducer_insert($producer, INSERT INTO NetworkRTT (NMIdSource, NMIdDestination, tool, packetSize, value) VALUES (pc18.hep.ucl.ac.uk, edgnm.gridpp.rl.ac.uk, pinger, 100, 57)); Web interface

edg-netmon2rgma Possibly should rewrite in Java –Problem in C (Perl) API results in memory leak and heavy performance hit –Current solution is to restart the daemon with a cron job /etc/cron.d/edg-netmon2rgma-restart –But java lacks the text-handling of Perl –Would have to have java on the NM

Tools modifications Generally 2 new functions initialise_rgma –Writes initialisation directives to the FIFO –i.e. fixed columns, metric names, expected number of entries Netmon2rgma creates producers, changes buffer sizes when necessary produce_it –Writes the results of the measurements No extra configuration –Assumes /opt/edg/var/edg-netmon2rgma/edg- netmon2rgma-fifo

edg-netmon-rgma-info Publishes NetworkSE, NetworkCE tables –Republishes every 12 hours (i.e. keepalive) Files –/opt/edg/sbin/edg-netmon-rgma-info.pl –/opt/edg/etc/edg-netmon-rgma-info.conf Local NM, associated SEs and CEs –/etc/rc.d/init.d/edg-netmon-rgma-infod RGMA client configuration

edg-ftlog2rgma Publishes file-transfer logs (i.e. GridFTP) Files –/opt/edg/sbin/edg-gridftplog2rgma.pl –/etc/rc.d/init.d/edg-gridftplog2rgmad Reads from GridFTP log –/var/log/gsiwuftpd.log –Publishes new entries when they are added RGMA client configuration

edg-netmon-archiver Program –/opt/edg/share/edg-netmon-archiver.jar –/opt/edg/sbin/edg-netmon-archiver runs the Java code –/opt/edg/bin/edg-netmon-archiver-setup Configures MySQL database; requires mysql root password What to do in LCFG? –/opt/edg/etc/edg-netmon-archiver.conf List of tables to archiver plus predicates for each –/etc/rc.d/init.d/edg-netmon-archiverd –Also RGMA client configuration

edg-netmon-archiver Ironically this might be the best package to write in C or Perl –Archiver API is all but unaffected by C API problem –But might run on R-GMA servlet machine Interaction with edg-netmon-info –Archives every row that edg-netmon-info produces –Perhaps need to automatically remove duplicates (via mysql)

Package progress Tools, edg-netmon2rgma, edg-netmon-rgma- info –99% –Latest tweaks still need to be uploaded edg-ftlog2rgma –75% –rc script, autobuild process edg-netmon-archiver –95% –need to upload, tag, check build process (ant)

Testing Tools on pc18.hep.ucl.ac.uk edg-netmon2rgmad on pc18.hep.ucl.ac.uk User servlets on pc23.hep.ucl.ac.uk Registry and Schema servlets on pc23.hep.ucl.ac.uk –Something always goes wrong when I try to connect to the WP3 Registry at infocat.gridpp.rl.ac.uk >1200 RTT/loss measurements archived by Tuesday >130 TCP throughput measurements to localhost :)

GridFTP logging in MDS Paul Mealor

Status Information provider –Will be in EDG 2 on SE Publishes the most recent information for each host pair MDS being phased out in EDG 2 –GIIS structure replaced by R-GMA –Will be all R-GMA later

Details Publishes achieved throughput –Filesize, time etc as parameters Publishes most recent transfer between each host pair Specifies source and host pair, plus, if known, source and host NM –NM names taken from config file as with other netmon info providers Uses same schema as other netmon –But doesnt use the full tree structure

Schema dn: NMMeasureId=$source/$dest,$opt{rootdn} objectClass: NetworkMeasurement NMMeasureId: $source/$dest SourceHost: $source SourceNE: $sourceNE DestHost: $dest DestNE: $destNE MetricName: gridftpthroughput MetricValue: $throughput MetricUnit: bit/s MonitorTool: GridFTP MetricTime: $timestamp Parameter: time: $etime Parameter: size: $size Parameter: buffersize: $buffer Parameter: streams: $streams Parameter: stripes: $stripes createTimestamp: $timestamp modifyTimestamp: $timestamp

END