Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 The road to 980Mbps eVLBI Harro Verkouter Joint Institute for VLBI in Europe

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 How to get involved... eVLBI demo at APAN conference 27/28 Aug 2007, in Xi'An want to include Shanghai LP up but poor TCP/IP performance (2Mbps) lossy + out-of-order reception investigate possibility of UDP transport suffers from those too both TCP and UDP based on IP-datagrams no retransmits/ramp-up/etc (no feedback!)

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 hacking Mark5A.c Haystack codebase does TCP reliably should do UDP but didn't looked like coding was stopped halfway reasonably easy to get it going revealed coding and UDP issues race conditions lack of feedback packetloss

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 bytes blocks Mk5 I/O board while (1) { wait( n_empty>0 ); wait_for_blocksize_bytes( ); copy_block_to_mem(); n_filled++; n_empty--; } Read thread Network Write thread while (1) { wait( n_filled>0 ); send_block_chunks( ); n_empty++; n_filled--; } circular buffer in MEM (shared memory + semaphores 'n_empty' and 'n_filled') indicates settable parameter via net_protocol=... command

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 First UDP tests... unstable datatransfer to Sh LAN worked fine 10% loss at low datarate (256Mbps) not quite correlatable :( packets never get on the wire sender doesn't know or care (no feedback!) overwrites own sendbuffer 1Gbps standard microbursts not all equipment can handle this

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 Ratecontrol/packet spacing... known that it was necessary Simon Casey and Richard H. Jones APAN demo nearing (5 days to go)... needed to get something, soon! tried to be smart computing packet spacing t based on datagramsize + linkrate need few s (MTU 1500, 1Gbps 12 s)

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 t sending packets Current: Ideal: With formatter sync:

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 Ratecontrol?!? didn't work at all! Linux is not a RealTime OS (...) sleeps() quantized in units of 10ms threads (==processes) 100Hz now what?! how to synchronize sender? suddenly realized we have a near perfect synchronous device... the formatter + Mk5 I/O board

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 Ratecontrol? remember the inter-thread circular buffer? n_empty + n_filled = n_total now imagine... n_total = 1 buffer degenerates into a mutex... which means the block is either filled from the I/O board synchronously at current bitrate! or sent to the network

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 Ratecontrol! worked amazingly well APAN also solution for Ar 1Gbps NIC 155Mbps ATM n_total = 2 blocksize = 1500 bytes got 64Mbps (previously, with TCP, couldn't do eVLBI) only good for datarates up to ~256Mbps

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 t sending packets! Bad (how we began): Ideal (couldn't get this to work): With formatter sync:

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 Demo ok, UDP looks promising, but! we had a bigger problem UDP transfer could not be restarted required killing + restarting Mark5A not suitable for operations spent few weeks on the issue kept on getting (weird) streamstor failures DMA errors, memorylock error,... made no progress at all

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 A new beginning? write testprogram to investigate streamstor behaviour disk2net UDP multithreaded different thread synchronization queue + condition variable could not recreate streamstor failures transfers could be started/stopped at will

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 jive5a extending the testprogram want to do in2net rather than disk2net need to program the Mark5 I/O board mostly copied from Haystack's Mark5A fairly easy to get it to work primary use eVLBI needed net2out as well

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 Early october 2007 jive5a first used for eVLBI want to take it further 512Mbps still not operational switch 'collapsed' (Paul JIVE solved this) (1) receiving end has to deal with packet loss out-of-order reception (2) sending end has to space packets as evenly as possible

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 (1) need protocol for this... sender + receiver agree on datagramsize added command for this 64bit sequence number in each datagram suffices to detect + fix out-of-order reception detect + fix loss can do statistics

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 (2) Spacing the packets sleep()'ing between packets not an option whatever the solution prefer something portable if possible good enough = good enough for(volatile int i=0; i<ipd; ++i); does the job surprisingly well... gave us reliable 512Mbps allows per n-coreCPU/link finetuning

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08

Jun 07 Dec 07 Jun 08 Sep 07 Mar ? Check! Next stop: BaseX links effectively ~980Mbps of userdata (UDP) need to trow away a bit of data simplest approach do not send a datagram every once in a while implement packet-drop-rate if 0, drop every 1 in pdr packet too simplistic correlator had problems staying in sync

Jun 07 Dec 07 Jun 08 Sep 07 Mar (the pdr version) – Dec 2007 smartened up the sender find headers in datastream on-the-fly do not drop packet if it contains header drops packet at earliest convenience worked! correlator stayed in sync with pdr~20 (5%) does add CPU usage cannot do 1024 on 1CPU/PIII system anyway 2 core/cpu Mk5's can do this

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 Drawback of dropping randomly dropping not correlated between senders total loss of visibilities ~ twice the amount of loss per station on the order of 10%... start thinking of dropping coherently dropping the same channel at each station but that requires lot of extra CPU power needed at least 2 CPUs to begin with anyway

Jun 07 Dec 07 Jun 08 Sep 07 Mar by means of channeldropping? not for the weak at heart... specify 64bit bitmask ones for the tracks you want to keep jive5a builds list of shifts + masks to fill 'holes' in the mask generate c-source for (de)compressionfn invokes compiler to generate object file dynamically loads that back into itself

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 SRC: DST: SRC: DST: SRC: DST: SRC: DST: SRC: DST: SRC: DST: SRC: DST: SRC: DST: SRC: DST: SRC: DST: SRC: DST: SRC: DST:

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 channel dropping results Concept proven to work! requires extra, CPU intensive, thread 3 core/CPU system at minimum correlates dropping 2x 2bits (out of 32) at 1024Mbps correlator couldn't sync CPU and network keeping up Just Fine under investigation

Jun 07 Dec 07 Jun 08 Sep 07 Mar 08 Thank you for your patience!