1. iBOB Data Acquisition and Network Streaming Jan Wagner et al. TKK / Metsähovi Radio Observatory 7th International eVLBI Workshop Shanghai, 2008

2. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #2 Current e-VLBI Data Acquisition Introduction Telescope data generated by Data Acquisition (DAQ) hardware e-VLBI : telescope data is sent via networks to correlator a)capture DAQ data locally to RAID disk array, send afterwards b)send data in real-time, capture and store or real-time process remotely Current DAQ situation in the EVN: Receiver and RF front-end : »512 MHz RF bandwidth VLBA rack downconverters: VCs & BBCs, 8 MHz base-band / ch Mark4-style A/D digitizers: max 32 Ms/s/channel 2-bit 64 Mb/s / ch max theoretical rate : 16 x 2-ch BBCs 2 Gbit/s (512 MHz RF) max rate in practice : formatting, Mark5 disk&NIC 1 Gbit/s => Stuck at 256/512 MHz, 1 Gbit/s?

3. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #3 Towards More DAQ Bandwidth Todays networks and PC systems easily support over 1G rates => just a matter of providing antenna data samples faster => need a DAQ with more bandwidth More RF bandwidth is beneficial for astro VLBI Correlation/synthesis yields better sensitivity, noise: improved images Observation time per object could be reduced, more objects per session More bandwidth from bit reso allows AGC, band equalizer, RFI mitigation, … To improve on the current DAQ: Input bandwidth: off-the shelf gigasample A/D converter(s) Processing: keep as wideband for astro, fine channelization for geo(?) Output interface: copper Ethernet, UDP packet streams For Metsähovi DAQ development and tests, used iBOB platform Introduction C. Phillips, e-VLBI Scientific Benefits, ATNF CSIRO memo, 2004

4. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #4 iBOB DAQ Hardware Platform UC Berkeleys iBOB FPGA board Virtex II Pro 2vp50 with PPC CPUs two copper 10GbE ports (CX4) two expansion ports (Z-DOK) Berkeley iADC board Atmel AT84AD001 dual-channel, 8-bit, 1 Gs/s/ch => 1 GHz MHz BW, 16 Gbit/s / board Nat LMX2531 or ADI ADF4360 ~950..1080 MHz PLL for ADC f s Similar board ROACH (meerKAT) available soon, has 4 x 10GbE, iBOB designs should drop in For iBOB, created several DAQ firmware test designs iBOB DAQ Description

5. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #5 iBOB IF-to-Ethernet Design Example iBOB DAQ Description

6. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #6 iBOB DAQ and Protocol for 10GbE Real-time e-VLBI: data integrity less important than timely send data loss factors f10% tolerable, image degrades packet loss and reorder ok, payload integrity non-critical dont bother to retransmit lost packets UDP checksums unnecessary: frame CRC32->2·10 -9 *BER congestion control: none - or client-specified sample mode changes The only essential points – implemented in Test-DAQ UDP must know which RF band/channel a sample comes from: sample series from same channel placed into same UDP packet for <60k channels can just send channel X to IPv4/6 port# Y samples must be time stamped: trigger acquisition at 1PPS, give packets a 64-bit packet sequence# assume 4kB UDP => ambiguity after 4.5e15 samples : 245 years @ 10Ts/s rest is constant a priori info, doesnt belong into the stream (YMMV) UDP Packet Format

7. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #7 iBOB DAQ Tests Local 10G LAN tests Analog signal sampled with iBOB, streamed as UDP to 10G LAN UDP stream recording with udp2raid, raw2raid (~tcpdump, gulp) udp2raided to RAID-0 at up to 4 Gbps (current RAID ~4.3G, XFS) Data, spectrum checks with Matlab and swspectrometer iBOB Streaming Tests

8. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #8 iBOB Streaming Tests Mh-On Streaming/Network Tests First tests on 14/15th May Supplied Onsala with test firmware 100 MHz clock, 1024 Ms/s sampling I/Q: VLBI IF 90..540 MHz, -10dBm On streamed to Mh RAID-PC Mh PC busy with other computations… Recorded 1.6, rest with slow-PC loss Checked for and found PCAL Second test 11th June used adc2tsunami, proto/test design sent 2 channels (I/Q) to On IBOB % regwrite dest_ip_ich 0x82F20A05 # 130.242.10.5, Onsala IBOB % regwrite dest_port_ich 46330 # I-channel to port 46330 IBOB % regwrite dest_ip_qch 0x82F20A05 # … IBOB % regwrite dest_port_qch 46331 # … IBOB % regwrite teng_wordsperframe 512 # 512x8 : 4kB UDP IBOB % regwrite teng_decfactor 0 # sample rate reduction 1/(1+n) 3.2G, (6.4G), 1.6 Gbps; 2 channels © NORDUnet (http://www.nordu.net/stats/)http://www.nordu.net/stats/

9. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #9 iBOB Streaming Tests Mh-On Streaming Test - Route © NORDUnet (http://www.nordu.net/stats/)http://www.nordu.net/stats/ © OptoSUNET (http://stats.sunet.se/)http://stats.sunet.se/ FunetNordunetSunet Mh On

10. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #10 iBOB Streaming Tests Mh-On Streaming Test - Traffic 2G, 4G, 7.8G

11. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #11 iBOB Streaming Tests Mh-On Streaming Test - Errors

12. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #12 iBOB Streaming Tests Mh-On Streaming Test - Comments Borje Josefsson (Sunet chief tech): AFAIK the tests started at 10AM Swedish time, why can't I ever go and have a beer at that time :-) BJ made Internet Land Speed record in 2004, 4.3 Gbps over 42 hops of shared networks from commercial ISP http://proj.sunet.se/LSR3-s/ http://proj.sunet.se/LSR3-s/ Will upgrade Sunet-Nordunet to 40G this week Roy Molini (Funet/CSC): It is definitely exciting to see data streaming from Finland at such incredible rates across infrastructures built specifically for meeting just those demands. The idea of Research Networking is clearly shown.

13. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #13 Conclusion Conclusions Basic Mh iBOB designs allow to digitize antenna signal, different rates stream as UDP to local or remote IP On->Mh RAID recording was lossless only at 1.6 Gbps try again when PC really free - local tests give 4G to disk Mh->On network streaming No problems with 0 – 7.8 Gbps traffic Credit goes to Nordunet, Sunet, Funet: it is straightforward and welcome to stream multi-Gbps over Internet/RN Future test or demo: stream and record a maser source with 2-bit / 8-bit, 2 x 512 MHz BW, analyze in swspectrometer

14. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #14 Conclusion Questions/Answers Contact information Jan Wagner M.Sc./Researcher EXPReS (Metsähovi) jwagner@kurp.hut.fi Additional Information http://expres-eu.org/[note: only one s] http://www.metsahovi.fi/en/vlbi/ EXPReS is made possible through the support of the European Commission (DG-INFSO), Sixth Framework Programme, Contract #026642

15. 12 June 2008iBOB Data Acquisition and Network Streaming, 7th eVLBI Workshop, ShanghaiSlide #15 Conclusion Refs Comparison of two frequency synthesizers: LMX2531 and ADF4360 http://www.metsahovi.fi/en/vlbi/ibob/Comparing-ADI-NAT.pdf http://www.metsahovi.fi/en/vlbi/ibob/Comparing-ADI-NAT.pdf Multiplier-Free Band-Selectable Digital Filters Arnaud Santraine, Sébastien Leprince, Fred Taylor, ICASSP 2004 Programmable, multiplier free channelizer Arnaud Santraine, Sébastien Leprince