IVS General Meeting, Feb 2004, Ottawa eVLBI Research in the European VLBI Network Steve Parsley JIVE

IVS General Meeting, Feb 2004, Ottawa PoC Targets FOR JIVE and the EVN Evaluate feasibility of eVLBI Interfaces: network, correlator, field system Network installation cost and timescale Data rates Logistics Standards Network protocols Parameter tuning for high data rates Procedures at telescope and correlator New Capabilities Higher data rates Improved reliability Quick response to transient phenomena FOR GÉANT and the NRENs To see significant network usage with multiple Gbit streams converging on JIVE from diverse European sources. Minimum is three telescopes (not including Westerbork) Must be seen to enable new science and not just solve an existing data-transport problem Real-time operation is seen as the ultimate aim, buffered operation accepted only as a development stage.

IVS General Meeting, Feb 2004, Ottawa PoC Participants DANTE/GÉANT Pan-European Network GARR Italian NREN UKERNAUK NREN PSNC Polish NREN DFN German NREN SURFnetDutch NREN KTHNOC/NORDUnet Nordic NREN Manchester UniversityNetwork application software JIVEEVN Correlator Westerbork telescopeNetherlands Onsala Space Observatory Sweden MROFinland MPIfRGermany Jodrell Bank UK TCfAPoland CNR IRA Italy

IVS General Meeting, Feb 2004, Ottawa Project Timeline You are here Mk5 data routinely available 6 X 1Gb/s installed JIVE Cabled-up for multiple GE 10Gb connections into the Netherlands Gigabit connections to Wb & On Complete? RHJ, RS Good progress, More work to do. Awaits useable system  Hardware probably ready  Software and testing to do  Understood in theory  Prototype hardware in preparation  Currently “hand crafted”  Expected to be not difficult Done Demonstrated by Metsahovi/Jodrell using PCEVN Now routine using FTP Core developments for real-time can now be planned in more detail  Not viable for development

IVS General Meeting, Feb 2004, Ottawa eVLBI –Mark5s LAN connection Gigabit Ethernet fibres

IVS General Meeting, Feb 2004, Ottawa DWDM Equipment, Dwingeloo LX Optics “clip” GE lines to DPU row via patch panel Cisco ONS Fibre pair from Amsterdam Optical Splitters

IVS General Meeting, Feb 2004, Ottawa Recent tests: 1 Network memory-memory (network only) Gbit NIC PC Motherboard I/O Mk5 Stream- Store Gbit NIC PC Motherboard I/O Mk5 Stream- Store Mem UDPmon ~960Mb/s

IVS General Meeting, Feb 2004, Ottawa Recent tests: 2 Network Disk buffer X2 Formatter Gbit NIC PC Motherboard I/O Mk5 Stream- Store Gbit NIC PC Motherboard I/O Mk5 Stream- Store Data Processor (SU) in2disk disk2net net2disk

IVS General Meeting, Feb 2004, Ottawa Recent tests: 2 Network Disk buffer X2 Formatter Gbit NIC PC Motherboard I/O Mk5 Stream- Store Gbit NIC PC Motherboard I/O Mk5 Stream- Store Data Processor (SU) disk2out Bench: 540Mb/s Private network, JIVE-AMS loop, Wb-JIVE: ~400Mb/s Shared network (Bologna PoP – JIVE) ~300 Mb/s

IVS General Meeting, Feb 2004, Ottawa Recent tests: 3 Network Disk buffer X1 Formatter Gbit NIC PC Motherboard I/O Mk5 Stream- Store Gbit NIC PC Motherboard I/O Mk5 Stream- Store Data Processor (SU) net2disk in2net

IVS General Meeting, Feb 2004, Ottawa Two telescopes: On & Wb at 256Mb/s Recent tests: 3 Network Formatter Gbit NIC PC Motherboard I/O Mk5 Stream- Store Gbit NIC PC Motherboard I/O Mk5 Stream- Store Data Processor (SU) disk2out Disk buffer X1 Three Telescopes: On, Wb & Jb, at 128Mb/s for image

Westerbork Netherlands Gbit link Network Topology Onsala Sweden Gbit link Chalmers University of Technology, Gothenburg Network North-West 150Mbit link Cambridge UK Jodrell Bank MERLIN Microwave link Dwingeloo DWDM link

Onsala/Cambridge 256Mb/s Onsala/Cambridge/Westerbork 128Mb/s

IVS General Meeting, Feb 2004, Ottawa Recent tests: 4 Formatter Network Gbit NIC PC Motherboard I/O Mk5 Stream- Store Gbit NIC PC Motherboard I/O Mk5 Stream- Store Data Processor (SU) in2net net2out Real-time Valid data confirmed at JIVE from Jodrell Bank at 64Mb/s

IVS General Meeting, Feb 2004, Ottawa Recent tests: 5 MK5 -1- SU Test Programme Configure Mk5-1 & SU and run normal scan. Virtual ribbon-cable LAN MK5 -2- Patch cable GE MK5 -3- GE Mk5 Terminal Mk5-2: in2net Mk5-3: net2out

IVS General Meeting, Feb 2004, Ottawa Results Summary Why do we get a lower data rate using UDP? Why are these “private” links slower than via a patch cable? JJ 341Mb/s Why so much slower when accessing the disks? 456Mb/s JJ J J Is this typical variation on the NREN-GEANT network?

IVS General Meeting, Feb 2004, Ottawa Parameter settings (disk2net2disk, TCP) recv/send(B)socbuf(B)datarate(Mb/s)

IVS General Meeting, Feb 2004, Ottawa Intel(R) SE7501BR2 Intel(R) 7501 chip set Dual Intel(R) Xeon (single, 2.8G, 512K L2 cash installed) Dual channel DDR266 registered SDRAM (512MB) Dual peer PCIX 64/100, 2+2 slots 32/33 PCI, 2 slots On-board TX1000 and T10/100 NICs On-board Ultra320 SCSI On-board PATA Unit assembled and tested as Mk5, works smoothly. Working on the second unit to start data transfer tests. Mark 5 Motherboard Upgrade

IVS General Meeting, Feb 2004, Ottawa Correlator developments for real-time

IVS General Meeting, Feb 2004, Ottawa Data Processor Existing System DPU or MK5 Data with timecodes; Telescope Observe Time (TOT) Check timecode Timing control Servo commands to tape/disk unit Buffer 0-160ms Reconstituted Observe Time (ROT)  Data re-construction  Geometric Delay  Correlation

IVS General Meeting, Feb 2004, Ottawa UT standard 0-20ms delay across Europe Real-time System 1 MK5 NETWORKNETWORK Data Processor UT standard Station backend UT standard Station backend UT standard Station backend Fixed delay compensation Worries:  Requires ~10millisecond synchronisation of ROT  Don’t yet know how difficult this will be  Hardware delays may be variable “Fixed” delay + network

IVS General Meeting, Feb 2004, Ottawa Data Processor Real-time system 2 DPU or MK5 Check timecode Timing control Use the servo system to keep within the buffer Buffer 0-160ms Only coarse sync of ROT needed  Data re-construction  Geometric Delay  Correlation Gbit NIC PC Motherboard I/O Mk5 Stream- Store Mem Mark5 output board has a 500MByte FIFO =4 seconds at 1Gb/s Worries:  Is servoing available in net2out?  How much software rework to make it so?  Will the PC micro be able to handle servo and network workload?

IVS General Meeting, Feb 2004, Ottawa Real-Time Development Stages March: Correlator-ROT Synchronisation April: Real-time bench tests May, June: Two station fringe tests July onward: Multi-station eVLBI

IVS General Meeting, Feb 2004, Ottawa Other Activities Next three months: Understand/remove Mk5 data rate limitations Develop/understand use of UDP for higher rates Higher rate Bologna – JIVE using UDP Next six months Further tests between Wb, On & Jb as link speed improves More NREN PoP – JIVE tests to push larger data volumes across the European networks PoC runs to the end of the current GÉANT project in early 2005 but a further year of support is possible under GÉANT2

IVS General Meeting, Feb 2004, Ottawa Realities Mark 5 is unable to deliver 1 Gb/s to the network Gigabit Ethernet is also limited to something less VLBI is restricted to fixed “octaves”: - 64 Mb/s, 128 Mb/s, 256 Mb/s, 512 Mb/s, 1 Gb/s Many stations are unlikely to get more than one GE connection The lines into JIVE are GE

IVS General Meeting, Feb 2004, Ottawa Revised Goals Real-time (in2net2out) at up to 512Mb/s (max with one MK5 and 1Gbit Ethernet) Non real-time (disk2net2disk) for maximum network loading: up to limit of Mk5 and Gbit Ethernet