Mark Leese Daresbury Laboratory GridMon EGEE JRA4 R-GMA workshop Thursday 22 nd July 2004 University College London Mark Leese

Presenter Name Facility Name Mark Leese Daresbury Laboratory Contents Very simple (like me ;-) GridMon: –where it has come from –how it works –what it does –where it is going Conclusion Lots of questions

Presenter Name Facility Name Mark Leese Daresbury Laboratory Project: brief This became… kit of monitoring tools to be deployed at each e- Science centre, focusing on: –publication to Grid middleware –visualisation for humans –end-to-end performance (what users see) “…design and deploy an infrastructure for network performance monitoring within the UK e-Science community.” UK e-Science approved project proposal to…

Presenter Name Facility Name Mark Leese Daresbury Laboratory GridMon: progress Work began in June Resulting toolkit called “GridMon” (everyone needs a good name ;-) Building on work of EDG WP7, initial presence of toolkit established at 12 original UK e-Science Centres (Cambridge, Daresbury, RAL…) As always, some problems – but being debugged Visualisation for humans

Presenter Name Facility Name Mark Leese Daresbury Laboratory GridMon: review Well received by people who use it Experience feeding into other UK projects Fed back into EDG Interest growing, e.g. UK HEP groups, and UK National Grid Service (NGS) In some form or another will likely be part of proposed UK GOC So, good foundations laid But how does it work and what’s next?

Presenter Name Facility Name Mark Leese Daresbury Laboratory Simple Architecture Grid middleware Publication Service Monitor Node IperfER PingER UDPmon 30 mins MiperfER bbcp/ftp Linux data stored locally.txt files Installed on dedicated & similar nodes at each centre MESH PingER(TCP) loss.oneWay (TCP) delay.roundTrip Errr, based on ping. Developed by SLAC et al. IperfER(TCP) bandwidth.achievable.singleStreamBased on PingER. Developed at UCL UDPmon(UDP) bandwidth.achievable.singleStream (UDP) loss.oneWay & delay.jitter Developed at Manchester HEP How does it work?

Presenter Name Facility Name Mark Leese Daresbury Laboratory Where next? New funding from UK e-Science until March 2005 You’ve seen visualisation for humans, so next is publication to middleware: –Back in June 2002, LDAP looked the way to go –Then in response to WP7, R-GMA looked the best way (middleware would use R-GMA queries to get data) –Now we (and fortunately most of the World) are looking at web and Grid services Will use GGF NM-WG schemas for data requests and publication of corresponding results: –In GridMon’s case you can request only historic data, no predictions or on- demand/future tests (at least not yet) –For simplicity, graph plotting and publication to middleware will have their own interfaces into the data But first, we must select a new data storage method, because the current.txt files would prove to a be problem: –.txt files are good for the human visualisation Writing into an ascii text file is relatively quick and easy Minimal programming required The performance graphs plot continuous data for a particular period, so no need to “jump around” within a dataset –.txt files bad for powerful web service Don’t easily support complex queries, like “give me all measurements from X  Y for the last week, where RTT ≥ 100mS” So investigation will have to be made into alternative choices, which are likely to be RRDTool (see next slide) or RDBMS, or both.

Presenter Name Facility Name Mark Leese Daresbury Laboratory RRD Tool? Final year undergraduate student (Chong Hei Cheng) produced prototype enhanced graph plotter for GridMon –Allows plots of data from all tools on the same graph –Uses “Round Robin Database Tool” to do this System to store and display time series data Produces plots like MRTG’s graphing functions Takes.txt for time period, inserts into RRD, then plots Better if data is stored in RRD when it is first collected. Don’t know if human visualisation is of interest to JRA4 – perhaps for GOC?

Presenter Name Facility Name Mark Leese Daresbury Laboratory Fast GGF advert GGF Network Measurements Working Group (NM-WG) are defining XML schemas for requesting tests and historic data, and publishing network measurements Aims: to standardise communication, and… …use XML, for web services and OGSI/WSRF model Simple use case… All request & result messages can be formatted using standardised schemas = truly powerful combination How data is stored and then retrieved to be sent as XML is irrelevant (could be LDAP, R-GMA,.txt….) it is obviously hidden from the requestor! Network Monitoring Service test request (request schema) tests results (publication schema)

Presenter Name Facility Name Mark Leese Daresbury Laboratory Another fast advert GGF GHPN-RG (Grid High Performance Networking) looking at network services. Diagram shows (numerous and varied) potential clients of network monitoring service: All this GGF work is funded by GridPP2 Network Monitoring Service Network domain Y Grid Middleware Grid Applications Automated Test Systems GOC/NOC Admin Software Grid/Net Operations Other Network Services Other Network Services Other Network Services Network Monitoring Service Network domain Z Network Monitoring Service Network domain X

Presenter Name Facility Name Mark Leese Daresbury Laboratory Web service: how? How is web service likely to be realised? –Apache Tomcat & Axis providing web service interface to Java programs that query database/datastore –Simple example shown at 2003 UK e-Science All Hands Meeting –“Proper” version will need to be much more complicated and robust In RelaxNG format from OASIS, not W3C’s WXS –More readable –More modular Will use Document/Literal SOAP binding –allows validation of messages against the schemas –RPC/Encoded binding (most common/default) makes validation difficult or impossible

Presenter Name Facility Name Mark Leese Daresbury Laboratory ? ? ? ? ? Conclusion GridMon: –UK infrastructure with existing graph plotting functionality –graph plotting will be enhanced –Web services interface for middleware etc. to be added, using GGF NM-WG schemas –Migration of datastore to more powerful technology to be investigated Questions