WP5: Cloud Correlation Arpad Szomoru Joint Institute for VLBI in Europe

NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium Outline  Objectives  Progress  Achievements

e-VLBI and NEXPReS  e-VLBI works great!  Stable, robust connectivity  Big telescopes, long baselines at full 1024 Mbps in real time  Enabling new science, increasing demand  But, limitations remain…  Some (important) telescopes will not (ever?) be able to join  No such thing as a perfect system/hardware, failures do occur  Re-correlation with different parameters currently impossible  Higher bandwidth/data rates becoming available, crucial for science  Need to accommodate 2, 4, 10? Gbps per telescope  4 Gbps dBBCs being rolled out in EVN  100 Gbps technology on its way NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

WP5 - objectives  Extend the benefits of real-time e-VLBI to all EVN observations  Transforming the VLBI operational chain  scheduling  setting up of observations and actual observing  buffering, transferring and correlation of the data  Prepare the EVN for the much higher data rates that are becoming available, both recorded and real-time  Accommodate bandwidth-on-demand techniques (topic of WP6) in a transparent and efficient way  Implement monitoring, authorization and remote control NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

WP5 - tasks  Task 1: Flexible buffering  JIVE, AALTO  Task 2: Implementation of VDIF standard  JIVE  Task 3: Continuous quality monitoring & station remote control  TUM, MPG, JIVE  Task 4: Automated network-dependent correlation  JIVE NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

WP5 – Progress at TUM/MPG  Software development with continuous integration and rapid prototyping  Improvements and additions to authentication and authorization software  System monitoring: standardization of interfaces  Development prioritized at the cost of remote control  No influence on deliverables NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

„e-RemoteCtrl“ software – the original test bed TIGO Concepción/Chile GARS O’Higgins/Antarctica RT Wettzell/ Germany TTW Wettzell/ Germany Remote Control on Site Communication Tele-working NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

FS Monitor Client GUI TCP/IP-Network FS Monitor Client Communication FS Monitor Server Communication FS Monitor Server Functionality FS Monitor Device … Server functionality Communication generated with the software generator idl2rpc.pl Graphical User Interface (GUI) Client (currently wxWidgets) NASA Field System e-RemoteCtrl RPC Interface- definition RPC RPC = (Sun) Remote Procedure Calls SSH = Secure Shell „e-RemoteCtrl“ software – the communication stack for a remote operator Wettzell Software Toolbox (well tested SW modules) NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

FS Monitor Client GUI TCP/IP-Network FS Monitor Client Communication FS Monitor Server Communication FS Monitor Server Functionality FS Monitor Device … Server functionality Communication generated with the software generator idl2rpc.pl Graphical User Interface (GUI) Client (currently wxWidgets) NASA Field System e-RemoteCtrl RPC Interface- definition RPC Establish& Control SSH tunnels sshbroker RPC = (Sun) Remote Procedure Calls SSH = Secure Shell „e-RemoteCtrl“ software – the communication stack for a remote operator Wettzell Software Toolbox (well tested SW modules) NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

Status monitor Logging User input Mark 5 capacity System temperature Connection state Connection information „e-RemoteCtrl“ software – some impressions Chat NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

Current status – a “e-RemoteCtrl” network parallel remote attendance test IVS Cont11 Test NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

SysMon – System Monitoring at Wettzell Local safety for people and systems in combination with reliability in operations Data for science and analysis Data for system operations Data for diagnosis Meteo, WVR, Clock offsets, … => low sampling rates => as scheduled Power supply, wind uploads, emergency stops, rack temp., … => medium sampling rates => permanently Servo currents, contouring errors, … => high sampling rates => on demand See also: Monitoring and Control Interface Collaboration Group NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

SysMon – System Monitoring at Wettzell NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

Windsensor display Safety-system display SysMon – System Monitoring at Wettzell NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

WP5 – Progress at JIVE  Further increase of functionality of JIVE-developed Mark5 control code  Recently: implementation of full Mark5 command set  New monitoring and alarm system at JIVE  Buffering control and administration of multiple copies  All based on completely re-designed database structure  Big effort towards live demonstration Ef, On, Yb, JIVE, recording/real-time at up to 4 Gbps  Involving DBBCs, Fila10G, Mark5Cs, Harroboxes  Many of which had never been used operationally before…  Provided focal point for several deliverables NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

Continuing development of Jive5AB  Mark5 units: PCs, Linux based, with proprietary hardware/libraries  Original control code woefully inadequate  Full re-write during EXPReS (reverse engineering)  Crucial in success of global e-VLBI  Re-write for NEXPReS:  As many threads as needed, as many CPUs as available  Much improved stability, performance, re-use of code  Simultaneous recording/transmitting  Partial and full cornerturning  Re-framing of packets  Packet spacing  Implementation of full Mark5 command set  Has expanded beyond Mark5 hardware  Will run on practically any Linux-based system NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

CAIM: Continuous Automated Intelligent Monitoring system  Based on Zabbix  Used for monitoring hardware health of Mark5s, cluster nodes, switches, etc  Monitors “quality” of fringes of selected calibrators, weights  Generates warnings via (or sms) NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

CAIM NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

WP5 – start of second period  Several deliverables depend on availability and roll-out of 4Gbps- capable equipment  Mark5C recorders  DBBCs (digital baseband converters)  Fila10G boards  VDIF support  Firm- and software for all of the above  Towards end of 2011: no clear indication any of this was happening  Mark5C: only 2Gbps, designers seemed to have lost interest  DBBC: had only been tested in 1 and 2Gbps mode  Only 1 DBBC operational within EVN  Fila10G: only Mark5B format, no VDIF support, untested  No plans, or available staff, to write additional firmware for Fila10G  Production of DBBC and Fila10G delayed  Things were looking bleak… NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

WP5 – actions undertaken  Several telecons with EVN TOG (Technical and Operations Group) officers, designer of DBBC, Haystack engineers  Re-affirmation of commitment to work towards higher bandwidth  Attempt to speed up development, fix timeline  Detached JIVE engineer to Haystack January 2012  December 2011: discussions with EC officer on deliverable 5.2  Demonstrate progress of WP5 through live demo  3 EVN stations, 4Gbps recording, simultaneous real-time 1Gbps correlation  Before end of second period...  Imposed a clear timeline, for all parties involved  Focus for the effort of several deliverables in WP5 NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

4 Gbps recording/1 Gbps real time  Effelsberg, Onsala, Yebes  DBBCs in PFB mode, untested  One DBBC broken  Fila10G cards  Only 3 available (4?) available in all of Europe  Of which one was sent to Chile…  Only ever tested at 2 Gbps  Production delayed  Mark5C recording at 4 Gbps  Never tested in EVN(?)  No Mark5C at Yebes  Yebes 10G connection not yet up  Very limited test time  What could possibly go wrong!!! NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

4 Gbps demo  Mark5C shipped from JIVE to Yb  Fila10G from Chile to Germany in hand luggage  Impounded by customs  Fila10G for Mh lent to Yb  Modules for switches, cables  Powerful 1U servers with multiple 10G interfaces purchased, shipped to stations, installed (HarroBoxes)  Software for “chunking” written  Modifications to SFXC correlator  *much* testing, bug fixing, new insights  Excellent way to get many techniques and tools developed in NEXPReS to actually work and work together  Fantastic support from networks NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

2nd NEXPReS Board Meeting– Copenhagen June 20, Gbps demo

4 Gbps demo: local networking setup NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

2nd NEXPReS Board Meeting– Copenhagen June 20, 2012  Some initial problems  Flooding  Mark5C configuration  Scan length recording issue  Some packet loss at Ef  Technically everything just worked from the start!!!  All equipment keeping up  Networks stable and performing flawlessly  But….. No fringes  DBBC configuration, Fila10G time synchronization?  Completely new equipment, need to learn 4 Gbps demo

WP5 – presentations and publications NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

WP5 – completed/delayed deliverables DescriptionDueStatus D5.01Mixed e-VLBI/recorded-VLBI at stations jive5ab34 D5.02Metrics, selection criteria and user feedback44 D5.03Definitions of specifications monitoring and remote control44 D5.05VDIF Mark5 conversion software154 D5.07Unattended correlation194 D5.04Automated job scheduling9-194 D5.06Control systems and scheduling mechanisms for 4-Gbps recording equipment 184 D5.08Monitoring, remote control and client software including the authorization and authentication layer 224 D5.09Control systems and scheduling mechanisms for 4-Gbps at correlator 233 D5.10Corner-turning platform244 NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

WP6: High Bandwidth on Demand Paul Boven Joint Institute for VLBI in Europe

Bandwidth on Demand: Rationale  e-VLBI requires dedicated network paths (For performance, and because switched networks are cheaper/greener than routed)  We now use static lightpaths (SDH/VLAN)  International BoD would allow for a more efficient and flexible network for e-VLBI  LOFAR LTA traffic patterns are similar, would benefit from BoD services NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

Tasks in WP6 1)Integration of e-VLBI with Bandwidth-on-Demand JIVE, SURFnet, NORDUnet, OSO, CSIRO 2)On-demand access for large archives ASTRON(LOFAR), SURFnet 3)Testing and validation of on-demand circuits UMAN, JIVE 4)Multi Gb/s on demand for e-VLBI (4Gb/s, 10Gb/s) JIVE, SURFnet, NORDUnet, OSO NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

 Networking Services Infrastructure (NSI) is a standard for BoD under development by the Open Grid Forum (OGF)  In WP6, we decided to focus on NSI, to not have to build a client for each national BoD system  Client was successfully demonstrated, report and source code available on the WP6 Wiki D6.01: Proof-of-concept system for reserving and provisioning On-Demand networking capacity NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

NEXPReS NSI Client block diagram NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

NEXPReS Second Board Meeting – 2012 June 20 – NORDUnet, Copenhagen Denmark NEXPReS NSI Client screen shot NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

 Due to possible lack of network resources, set up as two independent tracks:  OSO to JIVE (NSI)  Metsähovi to UMAN (AutoBahn)  10Gb/s equipment in Copenhagen was provided (temporarily) by SUNet  Both demonstrations completed on (revised) time, deliverable report on the Wiki D6.07: Demonstration of international BoD at 4Gb/s NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

D6.07: Demonstration of international BoD at 4Gb/s NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

D6.07: Demonstration of international BoD at 4Gb/s NEXPReS Second Board Meeting – 2012 June 20 – NORDUnet, Copenhagen Denmark NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

 Created 10G link between 'Target' in Groningen, and BigGrid storage at SARA, Amsterdam  Configure using NSI command line client  Tested at 10G, now in use for LTA transport  Delivered on time, report on the WP6 Wiki D6.03: Configuration of 10Gb/s BoD for the LOFAR Long-Term Archive NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

Other activities in WP6  5 Telecons so far in P2  Minutes on the WP6 Wiki  Presentations at TNC, NORDUnet conference, GLIF meeting, e-VLBI WS  Several partners are active in the NSI development / standardization effort NEXPReS Second Annual Review– September 21, 2012, Brussels, Belgium

WP6 – Completed deliverables DescriptionDueStatus D6.01Proof-of-concept system for reserving and provisioning on-demand networking capacity 204 D6.03Configuration of 10Gb/s BoD for LOFAR archive244 D6.07Demonstration of international BoD at 4Gb/s184