5 - 8 January 2009National Radio Science Meetings1 The ALMA Data Transmission System – Digital Portion Chris Langley ALMA Back End Integrated Product Team

5 - 8 January 2009National Radio Science Meetings2 The Challenge Transmit 4 – 12 GHz Astronomical Data from the Front End (FE) Band Cartridges to the Correlator using Commercial Off The Shelf equipment wherever possible.

5 - 8 January 2009National Radio Science Meetings3 The Challenge Transmit 4 – 12 GHz Astronomical Data from the Front End (FE) Band Cartridges to the Correlator using Commercial Off The Shelf equipment wherever possible. The Proposal Convert FE data Digitally and Optically prior to transmission from each of 66 antennas.

5 - 8 January 2009National Radio Science Meetings4 The Flaw COTS, or any other, D/A converters capable of 4 – 12 GHz inputs were not available during R&D.

5 - 8 January 2009National Radio Science Meetings5 The Flaw COTS, or any other, D/A converters capable of 4 – 12 GHz inputs were not available during R&D. The Solution Separate, or Down Convert, the 4 – 12 GHz two polarity band into eight 2 – 4 GHz basebands prior to data conversion and transmission.

5 - 8 January 2009National Radio Science Meetings6 Astronomical Data Down Conversion & Transmission

5 - 8 January 2009National Radio Science Meetings7 Data Transmission System

5 - 8 January 2009National Radio Science Meetings8 Design Considerations (1/2) Operate at OC-192 ( Gb/s) optical fiber signaling speed

5 - 8 January 2009National Radio Science Meetings9 Design Considerations (1/2) Operate at OC-192 ( Gb/s) optical fiber signaling speed Use of time division digital multiplexing to transform the input signaling rate to the channel signaling rate

5 - 8 January 2009National Radio Science Meetings10 Design Considerations (1/2) Operate at OC-192 ( Gb/s) optical fiber signaling speed Use of time division digital multiplexing to transform the input signaling rate to the channel signaling rate Insertion of fill bits to convert input rate to signaling rate

5 - 8 January 2009National Radio Science Meetings11 Design Considerations (1/2) Operate at OC-192 ( Gb/s) optical fiber signaling speed Use of time division digital multiplexing to transform the input signaling rate to the channel signaling rate Insertion of fill bits to convert input rate to signaling rate Use of time division digital de-multiplexing to transform the channel signaling rate to the output signaling rate

5 - 8 January 2009National Radio Science Meetings12 Design Considerations (1/2) Operate at OC-192 ( Gb/s) optical fiber signaling speed Use of time division digital multiplexing to transform the input signaling rate to the channel signaling rate Insertion of fill bits to convert input rate to signaling rate Use of time division digital de-multiplexing to transform the channel signaling rate to the output signaling rate Elimination of un-needed fill bits upon reception

5 - 8 January 2009National Radio Science Meetings13 Design Considerations (1/2) Operate at OC-192 ( Gb/s) optical fiber signaling speed Use of time division digital multiplexing to transform the input signaling rate to the channel signaling rate Insertion of fill bits to convert input rate to signaling rate Use of time division digital de-multiplexing to transform the channel signaling rate to the output signaling rate Elimination of un-needed fill bits upon reception Use of three OC-192 channels per 2 GHz baseband to achieve required capacity

5 - 8 January 2009National Radio Science Meetings14 Design Considerations (2/2) Low-voltage differential signaling (LVDS) –Fast rise/fall times –Noise resistant

5 - 8 January 2009National Radio Science Meetings15 Design Considerations (2/2) Low-voltage differential signaling (LVDS) –Fast rise/fall times –Noise resistant Multiple FPGA design per channel –More economical than single FPGA –Ball Grid Array package  Lots of IO –625+ MHz input signal capability

5 - 8 January 2009National Radio Science Meetings16 Design Considerations (2/2) Low-voltage differential signaling (LVDS) –Fast rise/fall times –Noise resistant Multiple FPGA design per channel –More economical than single FPGA –Ball Grid Array package –625+ MHz input signal capability Commercial Optical “Half” Transponders –Change from original design –Became economical –Built in mux /demux, clock recovery

5 - 8 January 2009National Radio Science Meetings17 Design Considerations (2/2) Low-voltage differential signaling (LVDS) –Fast rise/fall times –Noise resistant Multiple FPGA design per channel –More economical than single FPGA –Ball Grid Array package –625+ MHz input signal capability Commercial Optical “Half” Transponders –Economical –Built in mux /demux, clock recovery Air cooled (flow through) module, RFI shielded (-50 dBm)

5 - 8 January 2009National Radio Science Meetings18 Data Frame Organization

5 - 8 January 2009National Radio Science Meetings19 Data Transmission System Closer View

5 - 8 January 2009National Radio Science Meetings20 Data Transmitter Module (Digitizer and Formatter, 4 per Antenna)

5 - 8 January 2009National Radio Science Meetings21 Data Transmitter Module 4 / Antenna

5 - 8 January 2009National Radio Science Meetings22 Digitizer Assembly University of Bordeaux

5 - 8 January 2009National Radio Science Meetings23 Formatter with 3 Optical Transmitting Transponders

5 - 8 January 2009National Radio Science Meetings24 Data Receiver Module (De-Formatter with 3 Optical Receiving Transponders)

5 - 8 January 2009National Radio Science Meetings25 Data Receiver Module 4 / Antenna

5 - 8 January 2009National Radio Science Meetings26 DTS Modules for 1 Antenna Digitizer Clock IRAM, NRAO Data Transmitters U of Bordeaux, NRAO Fiber Optic Multiplexer Jodrell Bank Observatory Fiber Optic Amplifier / Demultiplexer Jodrell Bank Observatory Data Receivers NRAO

5 - 8 January 2009National Radio Science Meetings27 DTS Link Tests - ALMA Antenna to Lab Chile, 8/2008

5 - 8 January 2009National Radio Science Meetings28 DTS Link Tests - ALMA Antenna to Lab Chile, 8/2008

5 - 8 January 2009National Radio Science Meetings29 Things We’d Do Differently … Single FPGA per channel! –FPGA logic timing is difficult –Economics will likely catch up Closer interaction between hardware and firmware designers –Each should be the other’s backup Invite external expert’s opinions sooner during the design process Test Stand –Design and build once assembly form factors are determined Communication between remote team members was good, but could have been better –Specify an early DTS design review for the international partners

5 - 8 January 2009National Radio Science Meetings30 Acknowledgements Robert Freund, Principle Engineer, Arizona Radio Observatory Paula Metzner, DTS Product Engineer, Atacama Large Millimeter Array, National Radio Astronomy Observatory … and the entire DTS teams from North America, the University of Bordeaux, IRAM (Grenoble, FR), and Jodrell Bank Observatory (~Manchester, UK). R. W. Freund, ALMA Memo 420: Digital Transmission System Signaling Protocol, 2002 R. W. Freund and C. Langley, BE Critical Design Review, References

5 - 8 January 2009National Radio Science Meetings31 Auxiliary Slides

5 - 8 January 2009National Radio Science Meetings32 Data Transmission System Overview The Partners

5 - 8 January 2009National Radio Science Meetings33 System Requirements Repeatable latency with no loss of samples

5 - 8 January 2009National Radio Science Meetings34 System Requirements Repeatable latency with no loss of samples Bit error rate < (End of Life)

5 - 8 January 2009National Radio Science Meetings35 System Requirements Repeatable latency with no loss of samples Bit error rate < (End of Life) Multi-channel synchronization loss < s

5 - 8 January 2009National Radio Science Meetings36 System Requirements Repeatable latency with no loss of samples Bit error rate < (End of Life) Multi-channel synchronization loss < s 16 GHz analog bandwidth source

5 - 8 January 2009National Radio Science Meetings37 System Requirements Repeatable latency with no loss of samples Bit error rate < (End of Life) Multi-channel synchronization loss < s 16 GHz analog bandwidth source Nyquist sampled data

5 - 8 January 2009National Radio Science Meetings38 System Requirements Repeatable latency with no loss of samples Bit error rate < (End of Life) Multi-channel synchronization loss < s 16 GHz analog bandwidth source Nyquist sampled data 3-bit data word

5 - 8 January 2009National Radio Science Meetings39 System Requirements Repeatable latency with no loss of samples Bit error rate < (End of Life) Multi-channel synchronization loss < s 16 GHz analog bandwidth source Nyquist sampled data 3-bit data word Data transmission synchronized with ALMA timing

5 - 8 January 2009National Radio Science Meetings40 System Overview Explicit requirements 4 GSa/s per 2 GHz bandwidth IF channel

5 - 8 January 2009National Radio Science Meetings41 System Overview Explicit requirements 4 GSa/s per 2 GHz bandwidth IF channel 3 bits per sample

5 - 8 January 2009National Radio Science Meetings42 System Overview Explicit requirements 4 GSa/s per 2 GHz bandwidth IF channel 3 bits per sample 2 Polarizations x 4 IF channels

5 - 8 January 2009National Radio Science Meetings43 System Overview Explicit requirements 4 GSa/s per 2 GHz bandwidth IF channel 3 bits per sample 2 Polarizations x 4 IF channels 96 Gb/s per antenna (120 Gb/s encoded data)

5 - 8 January 2009National Radio Science Meetings44 System Overview Explicit requirements 4 GSa/s per 2 GHz bandwidth IF channel 3 bits per sample 2 Polarizations x 4 IF channels 96 Gb/s per antenna (120 Gb/s encoded data) 250 MHz input word rate (96-bit wide parallel word)

5 - 8 January 2009National Radio Science Meetings45 System Overview Explicit requirements 4 GSa/s per 2 GHz bandwidth IF channel 3 bits per sample 2 Polarizations x 4 IF channels 96 Gb/s per antenna (120 Gb/s encoded data) 250 MHz input word rate (96-bit wide parallel word) 125 MHz output word rate (192-bit wide parallel word)

5 - 8 January 2009National Radio Science Meetings46 System Overview Explicit requirements 4 GSa/s per 2 GHz bandwidth IF channel 3 bits per sample 2 Polarizations x 4 IF channels 96 Gb/s per antenna (120 Gb/s encoded data) 250 MHz input word rate (96-bit wide parallel word) 125 MHz output word rate (192-bit wide parallel word) Grouping of a polarization pair: 24 Gb/s per pair

5 - 8 January 2009National Radio Science Meetings47 System Overview Explicit requirements 4 GSa/s per 2 GHz bandwidth IF channel 3 bits per sample 2 Polarizations x 4 IF channels 96 Gb/s per antenna (120 Gb/s encoded data) 250 MHz input word rate (96-bit wide parallel word) 125 MHz output word rate (192-bit wide parallel word) Grouping of a polarization pair: 24 Gb/s per pair Walsh function 180° switching

5 - 8 January 2009National Radio Science Meetings48 System Overview Explicit requirements 4 GSa/s per 2 GHz bandwidth IF channel 3 bits per sample 2 Polarizations x 4 IF channels 96 Gb/s per antenna (120 Gb/s encoded data) 250 MHz input word rate (96-bit wide parallel word) 125 MHz output word rate (192-bit wide parallel word) Grouping of a polarization pair: 24 Gb/s per pair Walsh function 180° switching 15 Km (maximum) distance

5 - 8 January 2009National Radio Science Meetings49 System Overview Implied requirements Configurable if not deterministic timing (repeatable latency)

5 - 8 January 2009National Radio Science Meetings50 System Overview Implied requirements Configurable if not deterministic timing (repeatable latency) Fast frame synchronization

5 - 8 January 2009National Radio Science Meetings51 System Overview Implied requirements Configurable if not deterministic timing (repeatable latency) Fast frame synchronization Continuous transmission of data

5 - 8 January 2009National Radio Science Meetings52 System Overview Implied requirements Configurable if not deterministic timing (repeatable latency) Fast frame synchronization Continuous transmission of data Low error rates throughout operational life

5 - 8 January 2009National Radio Science Meetings53 System Overview Implied requirements Configurable if not deterministic timing (repeatable latency) Fast frame synchronization Continuous transmission of data Low error rates throughout operational life Operation independent of payload content

5 - 8 January 2009National Radio Science Meetings54 System Overview Implied requirements Configurable if not deterministic timing (repeatable latency) Fast frame synchronization Continuous transmission of data Low error rates throughout operational life Operation independent of payload content Testing strategies

5 - 8 January 2009National Radio Science Meetings55 System Overview Implied requirements Configurable if not deterministic timing (repeatable latency) Fast frame synchronization Continuous transmission of data Low error rates throughout operational life Operation independent of payload content Testing strategies Economical implementation

5 - 8 January 2009National Radio Science Meetings56 Auxilary Slides DTS Single Bit Data Path (VHDL Top Level)

5 - 8 January 2009National Radio Science Meetings57 Frame Implementation 160 bit frame 128-bit payload 10-bit synchronization word 5-bit meta-frame sequence word 1-bit meta-frame index 16-bit odd parity check word

5 - 8 January 2009National Radio Science Meetings58 Design Decisions Use of a data block (frame) to facilitate multiplexing

5 - 8 January 2009National Radio Science Meetings59 Design Decisions Use of a data block (frame) to facilitate multiplexing Use of a synchronization (framing) word to facilitate frame detection

5 - 8 January 2009National Radio Science Meetings60 Design Decisions Use of a data block (frame) to facilitate multiplexing Use of a synchronization (framing) word to facilitate frame detection Use of scrambling techniques to minimize bit sequence effects, low frequency content, and to maintain signal balance

5 - 8 January 2009National Radio Science Meetings61 Design Decisions Use of a data block (frame) to facilitate multiplexing Use of a synchronization (framing) word to facilitate frame detection Use of scrambling techniques to minimize bit sequence effects, low frequency content, and to maintain signal balance Use of a meta-frame to synchronize the reception of multiple channels

5 - 8 January 2009National Radio Science Meetings62 Design Decisions Use of a data block (frame) to facilitate multiplexing Use of a synchronization (framing) word to facilitate frame detection Use of scrambling techniques to minimize bit sequence effects, low frequency content, and to maintain signal balance Use of a meta-frame to synchronize the reception of multiple channels Use of a meta-frame index to synchronize reception to ALMA timing under varying propagation delays

5 - 8 January 2009National Radio Science Meetings63 Design Decisions Use of a data block (frame) to facilitate multiplexing Use of a synchronization (framing) word to facilitate frame detection Use of scrambling techniques to minimize bit sequence effects, low frequency content, and to maintain signal balance Use of a meta-frame to synchronize the reception of multiple channels Use of a meta-frame index to synchronize reception to ALMA timing under varying propagation delays Use of a checksum word to facilitate continuous monitoring of received data integrity

5 - 8 January 2009National Radio Science Meetings64 Auxilary Slides DTX Formatter FPGAs and Transponders

5 - 8 January 2009National Radio Science Meetings65 Auxilary Slides DRX De-Formatter FPGAs and Transponders

5 - 8 January 2009National Radio Science Meetings66 Auxiliary Slides Frame Synchronizations ►Frame synchronization (10-bit synchronization word) Multi-channel synchronization (5-bit meta-frame sequence word) Determination of propagation delay (1-bit meta- frame index)

5 - 8 January 2009National Radio Science Meetings67 Auxilary Slides Frame Synchronizations Frame synchronization (10-bit synchronization word) Multi-channel synchronization (5-bit meta-frame sequence word) Determination of propagation delay (1-bit meta-frame index)

5 - 8 January 2009National Radio Science Meetings68 Auxiliary Slides Frame synchronization (10-bit synchronization word) Unique or “unique enough” pattern to minimize acceptance of erroneous patterns in random data Long enough pattern to eliminate the acceptance of erroneous pattern in static data Partitioned pattern to eliminate the acceptance of a correct pattern in an incorrectly configured system Three acceptance stages required to qualify a 10-bit quantity as the synchronization pattern

5 - 8 January 2009National Radio Science Meetings69 Auxiliary Slides Stages for Frame Synchronization Search: selection of an initial location within the serial bit- stream followed by the shifting of the location until a candidate synchronization word is located Check: continued observations in subsequent frames until unsuccessful criterion (failure) Monitor: once confirmed, continuous monitoring of all frames to ensure proper operation

5 - 8 January 2009National Radio Science Meetings70 Auxiliary Slides Frame Synchronizations Frame synchronization (10-bit synchronization word) ►Multi-channel synchronization (5-bit meta-frame sequence word) Determination of propagation delay (1-bit meta- frame index)

5 - 8 January 2009National Radio Science Meetings71 Auxiliary Slides Multi-channel synchronization (5-bit meta-frame sequence word) Sequence word large enough to accommodate worst case relative variation in propagation delay across the three channels Integer number of meta-frames contained within one ms timing period Transmitter simultaneously writes the identical incrementing sequence number in frames of all three channels Receiver compares and re-times the frames from the three channels thus synchronizing the meta- frames

5 - 8 January 2009National Radio Science Meetings72 Auxiliary Slides Frame Synchronizations Frame synchronization (10-bit synchronization word) Multi-channel synchronization (5-bit meta-frame sequence word) ►Determination of propagation delay (1-bit meta- frame index)

5 - 8 January 2009National Radio Science Meetings73 Auxiliary Slides Determination of propagation delay (1-bit meta- frame index) Transmitter uniquely identifies the first meta- frame following a ms timing event Monitor and Control system obtains the count of frames received following the local ms timing event and the detection of the meta-frame index bit Monitor and Control system command the receiver to adjust its internal frame delay to a specific relative value

5 - 8 January 2009National Radio Science Meetings74 Auxilary Slides Scrambling Modification of source data to accommodate specific characteristics of the communication channel Provides adequate timing for the clock and data recovery electronics Provides a signal balance for the AC coupled circuits which minimizes threshold errors Pattern is easily produced by a maximally length shift register generator Sync word is exempt from scrambling For 149 -> 0, Result <= output pattern (shifted +1) XOR input pattern

5 - 8 January 2009National Radio Science Meetings75 Auxilary Slides Data Integrity Parity computation is easier than CRC 16-bit parity word over 144 bits Each parity bit monitors 9 other bits Permits continuous monitoring of transmission quality

5 - 8 January 2009National Radio Science Meetings76 Auxiliary Slides Self Test Methods 10 GHz clock recovery Frame detection Multiple channel synchronization Scrambled pattern exercises Random Number Generation Checksum (parity) checks FFT of pseudo Front End data (gain flatness, CW beacon)

5 - 8 January 2009National Radio Science Meetings77 Formatter Block Diagram

5 - 8 January 2009National Radio Science Meetings78 Auxilary Slides Digitizer Clock Module

5 - 8 January 2009National Radio Science Meetings79 Auxilary Slides Digitizer Clock Assembly IRAM, Grenoble

5 - 8 January 2009National Radio Science Meetings80 Auxilary Slides Fiber Optic Multiplexer Jodrell Bank Observatory

5 - 8 January 2009National Radio Science Meetings81 Auxilary Slides Fiber Optic Amplifier / Demultiplexer

5 - 8 January 2009National Radio Science Meetings82 Auxilary Slides Transmitter Module – Internal View Formatter Digitizers Monitor Control & Power Supply Backplane

5 - 8 January 2009National Radio Science Meetings83 Auxilary Slides DTS Test Stand “Golden” Modules –2 Data Transmitters –1 Digitizer Clock –2 Data Receivers Support Electronics –PC with LabView Interface –System timing –Data Receiver backplane

5 - 8 January 2009National Radio Science Meetings84 Things We’d Do Differently … Single FPGA per channel! –FPGA logic timing is difficult –Economics will likely catch up

5 - 8 January 2009National Radio Science Meetings85 Things We’d Do Differently … Single FPGA per channel! –FPGA logic timing is difficult –Economics will likely catch up Closer interaction between hardware and firmware designers –Each should be the other’s backup

5 - 8 January 2009National Radio Science Meetings86 Things We’d Do Differently … Single FPGA per channel! –FPGA logic timing is difficult –Economics will likely catch up Closer interaction between hardware and firmware designers –Each should be the other’s backup Invite external expert’s opinions sooner during the design process

5 - 8 January 2009National Radio Science Meetings87 Things We’d Do Differently … Single FPGA per channel! –FPGA logic timing is difficult –Economics will likely catch up Closer interaction between hardware and firmware designers –Each should be the other’s backup Invite external expert’s opinions sooner during the design process Test Stand –Design and build once assembly form factors are determined