Physics Division The South Pole Telescope Readout System March 4, 2004 SPT EAB Meeting, U. Chicago OUTLINE Overview of System Overview of System Technical Challenges Technical Challenges Current Status Current Status

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 2 SPT Readout Team Matt Dobbs + Helmuth Spieler Matt Dobbs + Helmuth Spieler  overall coordination, design, integration, hands on implementation LBNL Engineers John Joseph and Chinh Vu LBNL Engineers John Joseph and Chinh Vu  Design, Layout and Commissioning of Osc/Demod and SQUID Controller boards John Clarke (UCB) and Sherry Cho (postdoc) John Clarke (UCB) and Sherry Cho (postdoc)  SQUID Expertise UCB Grad Student Trevor Lanting (A. Lee) UCB Grad Student Trevor Lanting (A. Lee)  Cold Frequency Domain SQUID Multiplexer commissioning and performance UCB Grad Student Martin Lueker (W. Holzapfel) UCB Grad Student Martin Lueker (W. Holzapfel)  SQUID shielding, SQUID Controller commissioning, software Technician Dennis Seitz Technician Dennis Seitz  SQUID Quality control and testing (experience from CDMS) much useful help and discussions from T. Crawford (Chicago) much useful help and discussions from T. Crawford (Chicago) extended visits to LBNL

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 3 Readout System Requirements TES Bolometer Readout TES Bolometer Readout  resolve bolometer noise floor (~10 pA/√Hz)  SQUID as 1st amplifier stage  maintain voltage bias for low (0.5 Ω) impedance sensor  low amplifier input impedance  shunt feedback  1/f knee low enough for drift scanning (~100 mHz)  AC bias Diagnostics Diagnostics  operate SQUID open loop  map SQUID output voltage vs. input current  monitor SQUID DC level during data taking  full spectral distribution for each readout channel Full computer control Full computer control

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 4 APEX-SZ is SPT Prototype SPT and APEX-SZ use same readout SPT and APEX-SZ use same readout sharing of resources between APEX-SZ and SPT readout systems provides SPT with early experience sharing of resources between APEX-SZ and SPT readout systems provides SPT with early experience  APEX-SZ is 320 channels  APEX-SZ baseline: no multiplexing  Readout includes all functionality req’d for multiplexing APEX-SZ timeline (deployment end of this year) means electronics will be available early for SPT APEX-SZ timeline (deployment end of this year) means electronics will be available early for SPT  serves as proof-of-concept for SPT  time for modifications / upgrades post-APEX

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 5Implementation 8-channel SQUID controller 8-channel SQUID controller  mounted directly on cryostat  full computer control + diagnostics  one cable per 8-ch controller connects to demodulator board Demodulator + digitizer for each bolometer channel Demodulator + digitizer for each bolometer channel  rack-mounted in receiver cabin  3 VME crates (9U) for 940 bolometers  16 demodulator channels per 9U board  each board includes digitization + computer interface  connections to computer optically isolated Use standard commercial ICs throughout Use standard commercial ICs throughout  low cost per channel

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 6Multiplexing reduce heat transfer through wiring to cold stage reduce heat transfer through wiring to cold stage scalability of system to larger arrays scalability of system to larger arrays minimize complexity of cold wiring minimize complexity of cold wiring reduce number of SQUIDs reduce number of SQUIDs  reduce cost  reduce testing time  minimize Prozac usage increased dynamic range requirements relative to single bolometer readout increased dynamic range requirements relative to single bolometer readout

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 7 Frequency Domain Multiplexing complementary to time domain multiplexing complementary to time domain multiplexing bolometers are AC biased in superconducting transition bolometers are AC biased in superconducting transition  f carrier well above bolometer bandwidth bolometers grouped in readout “modules”, 8-32 channels bolometers grouped in readout “modules”, 8-32 channels  each bolometer in module biased at unique f carrrier = MHz  each channel has stand-alone capability (no shared oscillators) incident radiation (sky signal) causes variation in R BOLO incident radiation (sky signal) causes variation in R BOLO  amplitude modulates carrier  transfers signal power to sidebands (~1kHz bandwidth)  bolometer signals at unique frequencies  can be summed in one wire bias currents are applied as “comb” of carriers bias currents are applied as “comb” of carriers  one wire for bias  module requires 2 wires frequency-selective demodulation to separate bolometer signals frequency-selective demodulation to separate bolometer signals

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 8 Frequency Domain Multiplexer f 500 kHz f 200 Hz 80 dBc Amp-Modulated by Bolo DDS Amplified 600 kHz f 100 Hz Demodulated 500 kHz 600 kHz f100 Hz f 500 kHz 200 Hz 80 dBc 600 kHz Shunt FB SQUID T=4 KT=0.3 K only 2 wires from cold stage

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 9 Frequency Domain Multiplexer Bolometers are AC biased Bolometers are AC biased  not sensitive to low frequency noise downstream of bolometer Shunt feedback Shunt feedback  provides low input impedance to maintain bolo voltage bias for AC carriers Use of Digital Direct Synthesizers Use of Digital Direct Synthesizers  programmable, cheap, and scalable. System is bolo noise limited.

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 10 Frequency Domain Multiplexer test chip of 8- channel Multiplexer (LBNL, UCB, NGC)

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 11 Osc/Demod Readout Board 16 Channels / board (9u x 400mm) 16 Channels / board (9u x 400mm) Separate oscillators for bolo bias + nulling Separate oscillators for bolo bias + nulling  independent amplitude + phase control  use DDS with common clock (both synchronous) harmonics, spurious signals < 80 dBc (typ < 100 dBc) harmonics, spurious signals < 80 dBc (typ < 100 dBc) noise sidebands low to 10 mHz. noise sidebands low to 10 mHz. Sampling Demodulator (+35 dBm IP3) Sampling Demodulator (+35 dBm IP3) Parallel channel to monitor SQUID DC output independent of carrier amplitude (flag flux jumping) Parallel channel to monitor SQUID DC output independent of carrier amplitude (flag flux jumping) 14 Bit ADC’s (4 per 16 channels) 14 Bit ADC’s (4 per 16 channels)  oversampling to obtain dynamic range  one sample-and-hold per channel Opto-isolated RS485 to PC for Control Opto-isolated RS485 to PC for Control  SQUID Controller commands passed through Opto-isolated LVDS to Opto-isolated LVDS to  Custom digital PCI I/O board in PC (FNAL design)  data push at 7 Mbits /s (about 1000 channels x 2.5 kHz = 6 Mbits/s) Power ~80W per board. Power ~80W per board.

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 12 Tayloe Mixer Sample input synchronously at f carrier and integrate with 1 KHz RC low-pass filter. This gives the baseband modulation signal. MOSFET switch is clocked by Local Oscillator running at the carrier frequency, e.g. 1 MHz. Signal Input We use only this branch. Balanced output provides add’l carrier suppression Very low noise: 1 nV/√Hz from diff amps 1.4 nV/rtHz from integration resistor 1/f noise is as good as post amp. Sampling demodulator aliases HF sidebands to baseband. No conversion loss, no non-linear elements required.

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 13 Warm Cabling and Mechanics 9U Osc/Demod boards housed in VME 64x Crates with custom backplanes 9U Osc/Demod boards housed in VME 64x Crates with custom backplanes  20 boards (16 ch ea.) / crate, 3 crates  1 clock distribution board per crate Warm Cabling Warm Cabling  commercial 10’ SCSI differential cables (Cryostat  Readout boards), twisted pairs in common shield  LVDS twisted pair to PC Power: Power:  5 kW for 60 boards = 920 Channels  commercial 250 kHz switching power supplies, 1% V p-p Ripple

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 14 8ch SQUID Controller 8 channel SQUID controller shunt feedback shunt feedback switch open loop/closed loop w. switchable gain, switch open loop/closed loop w. switchable gain, on board FPGA / DAC for programming SQUID on board FPGA / DAC for programming SQUID mates directly to cryostat mates directly to cryostat analog data out to demodulator boards. analog data out to demodulator boards. M in R FB 10 4 VΦVΦ e noise amp L in Signal IN MOSFET ColdWarm Not Shown: DC Squid bias DC Flux bias DC Amp Voltage Offset SQUID Voltage measure lead 19mm pitch

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 15 SQUID pre-amplifers Measure current in SC loop of LCR bolo Measure current in SC loop of LCR bolo 100 element Series array SQUIDs from NIST 100 element Series array SQUIDs from NIST  AC biased  1/f noise unimportant  Squid Noise: 0.1 μΦ 0 /√Hz ≈ 2 pA/√Hz, adequate Expect to receive first two wafers from NIST next week (destined for APEX-SZ) Expect to receive first two wafers from NIST next week (destined for APEX-SZ) Quality control and testing at Berkeley by Dennis Seitz (experienced with NIST arrays for CDMS) Quality control and testing at Berkeley by Dennis Seitz (experienced with NIST arrays for CDMS) Z Trans = 400 Ω Flux Bias PUT NIST DRAWING HERE.

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 16 SQUID Magnetic Shielding

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 17 SQUID Magnetic Shielding Two part shielding Two part shielding  0.056’ Cryoperm chimney style shield (attenuates B, by few 10 3  Nb film (pins B field lines, by >15) Measured performance (prototype) Measured performance (prototype)  DC attenuation of few 10 5  improves to >10 6 for AC B fields Production Drawings near-ready Production Drawings near-ready Finalizing Mechanical support/connetors Finalizing Mechanical support/connetors Calculated Attenuation from Nb Foil (Martin Lueker)

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 18 Measure Shield Performance

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 19 Nb 2 O 5 capacitors lossy, spoils voltage bias. Nb 2 O 5 capacitors lossy, spoils voltage bias. Solution: use external NP0 ceramic chip caps. Solution: use external NP0 ceramic chip caps.

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 20 MUX LC Chip 2 nd Generation MUX Chip New Mask for MUX chip, inductors only 1/3 Q Fabricated by A.Smith, Northrop Grumman Space Tech 1 st Generation MUX Chip Integrated LC’s. 8 channels

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 21 OLD LC Chip New L chip, with Surf mount caps Loss in LC

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 22 LC Chip Cross Talk

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 23 NOISE OF SENSOR JUST ABOVE ITS TRANSITION (HIGH VOLTAGE BIAS) WITH CMB ELECTRONICS (f bias =1.5 MHz) 100 Hz Sky signal 1/f dominated by active filter in demodulator  fixed in new Tayloe mixer measured noise as calculated

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 24 NOISE OF SENSOR IN TRANSITION WITH 100Hz SIMULATED SKY SIGNAL WITH CMB ELECTRONICS (f bias = 1.5 MHz)

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 25 Readout Challenges dynamic range: >80 dB for single bolometer dynamic range: >80 dB for single bolometer high Q LC resonator high Q LC resonator  solved- custom Nb inductors + NP0 chip capacitors intermodulation distortion intermodulation distortion  new Tayloe Mixer (35 dBm IP3)  requires (passive) nulling of bolometer carriers at SQUID input low frequency response low frequency response  measured DDS stability, good to 10 mHz  low noise Tayloe Mixer cold SQUID + warm amplifier feedback loop cold SQUID + warm amplifier feedback loop  requires short cold-warm wiring lengths

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 26 Intermodulation Distortion and Nulling Carrier inter-modulation products troublesome. Carrier inter-modulation products troublesome. Can avoid them or NULL the carriers to minimize their effect Can avoid them or NULL the carriers to minimize their effect Carrier nulling at 60 dB obtainable, accurate to 10 mHz. Carrier nulling at 60 dB obtainable, accurate to 10 mHz. New TAYLOE Mixer on demodulator board has excellent IP3=35 dBm. New TAYLOE Mixer on demodulator board has excellent IP3=35 dBm.

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 27 Cold  Warm Wiring Lengths SQUID locked in feedback loop w/ warm amp SQUID locked in feedback loop w/ warm amp Loop gain determines dynamic range. Loop gain determines dynamic range.  original goal: A LOOP =100 at 1 MHz  A LOOP =1 at 100 MHz need to maintain phase margin at freq. where A LOOP >1 need to maintain phase margin at freq. where A LOOP >1  limits wiring length between cold and warm  depends on max bias freq. x loop gain  Example: A LOOP =100 at 1 MHz round trip wiring length <20 cm for 45 deg phase margin  SQUID arrays relax dynamic range requirements  Loop gain requirement set by SQUID non-linearity (intermodulation products) -- t.b.d. for NIST arrays achieved wire length is 2x 15 cm achieved wire length is 2x 15 cm  can achieve A LOOP = 50 with some margin  carrier nulling reduces max. signal -- essential COLD | WARM I1I1 R fb SQUID Controller Series Array SQUIDs

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 28 SPT Readout Status SQUIDs SQUIDs  sample 8-turn NIST arrays tested with our system  first production SQUIDs arriving at Berkeley this month SQUID Shields SQUID Shields  prototypes fabricated at Amuneal  attenuation measured – adequate  designing production cryoperm shields now Cold wiring harnesses Cold wiring harnesses  prototypes ordered from TechData SQUID Controllers SQUID Controllers  4-channel analog and 8-channel digitally controlled prototypes tested (testing of 8-ch board continues)  Layout revisions of production boards begun Warm cabling Warm cabling  off the shelf SCSI cables, samples procured

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 29 SPT Readout Status Oscillator/Demodulator Readout Boards Oscillator/Demodulator Readout Boards  Prototype single channel board fabricated and tested Tayloe Mixer tested as add-on board Tayloe Mixer tested as add-on board  Prototype 16 channel board fabricated functionality verified, need to measure performance, test with system functionality verified, need to measure performance, test with system  VME Crates selected (Wiener), off-the-shelf power supplies selected Custom backplanes: power plane finished, signal/clock plane underway. Custom backplanes: power plane finished, signal/clock plane underway.  Firmware slave board firmware written/tested slave board firmware written/tested SQUID Controller / Master board firmware will start soon SQUID Controller / Master board firmware will start soon  Clock Distribution Board prototype fabricated and tested prototype fabricated and tested 9U prototype in the pipeline (very simple board) 9U prototype in the pipeline (very simple board) Digital IO Board Digital IO Board  “borrowed” Custom IO boards from Fermilab, 2 boards in hand Readout Software Readout Software  OO software in C++ on Linux OS, interfaced with experiment Software via TCP/IP  High Level Design Sketches/Brainstorming  Many low level Hardware interfaces and tuning algorithms written and in use.

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 30 Readout Status – System individual elements of readout system prototyped and tested individual elements of readout system prototyped and tested early prototype integrated readout system tested with bolometer early prototype integrated readout system tested with bolometer  analog SQUID controller, single channel demodulator… noise properties and two channel fMUX tested noise properties and two channel fMUX tested  commissioning 8 channel system with bolometers now testing of “production ready” integrated system forthcoming testing of “production ready” integrated system forthcoming (this spring/early summer)

Physics Division The South Pole Telescope Readout System Software March 4, 2004 SPT EAB Meeting, U. Chicago

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 32 Framework / Infrastructure Programming in C++ on Linux OS Programming in C++ on Linux OS Communication w/ Outside world via TCP/IP only. Communication w/ Outside world via TCP/IP only. Dedicated PC for Data Acquisition, receives digital data from readout boards, and makes it available on sockets for SPT control PC to archive (rates up to Hz, TBD) Dedicated PC for Data Acquisition, receives digital data from readout boards, and makes it available on sockets for SPT control PC to archive (rates up to Hz, TBD)  also supports streaming data directly to disk at rates up to 2.5 kHz, for debugging, etc. Readout Tuning and Monitoring PC Readout Tuning and Monitoring PC  algorithms to setup and tune SQUIDs/Bolometers  monitors noise and DC levels (flux jumping) in all channels knows algorithms to repair channels knows algorithms to repair channels  provides quality control data on sockets for SPT archive

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 33 Sampling Rate of Bolo Timestream sampling pulse can be generated externally or internally sampling pulse can be generated externally or internally the maximum bandwidth is determined by the bolometer time constant the maximum bandwidth is determined by the bolometer time constant (hardware allows us to reduce bandwidth below 1 kHz)  Bolometer time constant ~ 1 ms (not well known yet!)  bandwidth 1/(2πτ) = 160 Hz by low pass filtering the bolometer timestream at 160 Hz, and sampling at the Nyquist frequency 400 Hz, all the timestream information is encoded by low pass filtering the bolometer timestream at 160 Hz, and sampling at the Nyquist frequency 400 Hz, all the timestream information is encoded we sample with a 14 bit ADC, providing a LSB accuracy of ~ we sample with a 14 bit ADC, providing a LSB accuracy of ~ Why use the full bandwidth when the sky signal will be modulated at slower rates? Why use the full bandwidth when the sky signal will be modulated at slower rates?  we can gain further accuracy by over-sampling  Sampling beyond bolometer 3 dB bandwidth allows full reconstruction of signals within MUX bandwidth for diagnostics.  Data storage rate can be reduced by real time averaging during measurement  can be done in FPGA on demodulator board  can keep high sample rate and reduce readout rate.

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 34 Data Rate between ADCs and Readout PC Each data frame consists of: Each data frame consists of:  header information identifying each crate, board, and channel  960 channels x 14 bit data samples  GPS time encoded from IRIG-B signal. (exact format can be discussed in more detail another time – system alternatively allows for a sampling global sampling pulse) system alternatively allows for a sampling global sampling pulse) size of frame for one sample: 2.9 kBytes (of which about 40% is header information– see cartoon of frame on next slide) 400 Hz readout rate  1.1 MBytes / second 400 Hz readout rate  1.1 MBytes / second 2.5 kHz readout rate  7.1 MBytes / second 2.5 kHz readout rate  7.1 MBytes / second

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 35 Data Rate written to Disk Much of the header information can be removed before writing the data to disk Much of the header information can be removed before writing the data to disk  headers structured to allow recognition of corrupted data For each data frame, we’ll need For each data frame, we’ll need  10 Bytes GPS time stamp  960 channels x 2 Bytes data sample = 1920 Bytes we can average several data points and write to disk at a slower data rate we can average several data points and write to disk at a slower data rate 400 Hz  0.74 MBytes/s (62 GBytes / 24 hrs) 400 Hz  0.74 MBytes/s (62 GBytes / 24 hrs) 100 Hz  0.18 MBytes /s (15.5 GBytes / 24 hrs) 100 Hz  0.18 MBytes /s (15.5 GBytes / 24 hrs) 3 DVD-R’s 1/6 of an HP Ultrim Tape

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 36 Data Flow Cartoon Oscillator/Demodulator Boards (optically isolated from PC / GPS) DAQ Computer Control and Monitoring Computer (might be same PC) read/write controls & time stamp Data pushed at few MBytes / s subset of data for online monitoring & SQUID / Bolo tuning receiver housekeeping data RS 485 Serial To SPT Archive PC

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 37 Data Rate Between readout boards and PC Between readout boards and PC size of frame for one sample: 2.9 kBytes (of which about 40% is header information– see cartoon of frame on next slide)  400 Hz sampling rate  1.1 MBytes / second  2.5 kHz sampling rate  7.1 MBytes / second Data rate written to disk (bolo data only) Data rate written to disk (bolo data only)  400 Hz  0.74 MBytes/s (62 GBytes / 24 hrs)  100 Hz  0.18 MBytes /s (15.5 GBytes / 24 hrs)

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 38 Size of Data Frame: 2.9 kBytes Data Frame from Osc/Demodulator Data from one Crate 1 Byte CRATE ID 9 Bytes GPS Time 1 Byte BOARD ID 1 Byte CHANNEL ID 2 Bytes 14 bit data sample Data from oneOsc/Demod Board 3 crates 16 channels/board 20 boards/crate

Matt Dobbs & Helmuth Spieler, SPT Readout System Mar 4, 2004 EAB 39 Block Diagram: Bolo Data Flow SQUID Controller 8 MUX Modules / board 16 DDS Oscillators for Carriers 16 DDS Oscillators for Nulling 16 Demodulators ADC’s FPGA Opto-Isolators Osc/Demodulator Boards IRIG-B GPS Signal Digital IO Board With FPGA 2 MByte Buffer Persistent Storage on Hard Disk & Ultrim Tape Data Acquisition PC PCI Interface PC Memory Software Processing Data Reduction Bolometer time stream 1 MHz BW Comb of Carriers Comb for Nulling Bolometer time stream 1 kHz BW Data Cable: Diff. Twist Pair ≤ few MBytes/s time interval of encoded data need only be larger than signal bandwidth, ~ 100 Hz. ≈0.2 MB/s we are free to average some number of data samples here, while still maintaining benefits of oversampling. 60 x 16ch Boards in 3 9u Crates 15 x (8 Mod x 8ch) Boards RS 485 IO Differential & Isolated Control (and Monitoring?) PC PC Memory Software Processing External SPT Sampling Clock Control & Bolo / SQUID Tuning Provides GPSTimestamp Control Subset of Data for Monitoring Readout Hardware accommodates both of these timing Possibilities. Comment Flow of Timing Info Flow of Control Commands Flow of Bolo Data Legend Housekeeping Data