1. Tracker Strip and Pixel FEDs John Coughlan Tracker Readout Upgrade Meeting September 12 th 2007

2. Strip FED 9U VME64x 96 optical fibres Analogue ORx 96 ADC channels Limited by I/O Component density 1 nsec timing Board mounted Digital FPGAs Xilinx Cluster Algorithms Raw input data rate >3 GB/s. Processed Output rate < 200 MB/s VME FPGA Front-End data processing FPGA Power DC-DCs on board Output to DAQ Event Builder FPGA Modular FE Unit 96 ADC channels Double Sided Board Opto Receivers TTC rx 450 x boards + Slinks Synch APV Cost Driven

3. Strip FED Board Manufacture Board parameters: - 9U x 440 mm VME64x form factor - Optical/Analogue/Digital logic ; 96 ADC channels - Double-sided (secondary side with half of analogue channels) - 6,000 components (majority of passives 0402) (finest pitch < 20 thou) - 25,000 tracks - 37 BGAs (typical FPGA 676 pins on 1mm pitch). All BGAs located on primary side. - 14 layer PCB (incl. 6 power & gnd) - controlled impedance High density components. Close up of analogue section on primary side Almost all components on board are Surface Mount. VALUE is in the COMPONENTS Design for TEST Must have HIGH YIELD

4. Strip FED Quality Control 1. Custom CMS Tests At Assembly Plant Boundary Scan Analogue 3. Tests at CERN Prevessin 904 B186 Tracker Integration 2. Tests at RAL Optical, SLINK, Full crate 4. Installation at CMS USC55 0. Quality Controls during Assembly process AOI, X-ray RAL System Rig Assembly Plant FED Optical Test System From Imperial College RAL Test Rig CERN

5. Pixel FED 9U VME64x 36 optical fibres Analogue ORx 36 ADC channels Limited by SLINK output rate 1 nsec timing Digital FPGAs Altera Inputs are ZS Unsynchronised Events on fibres ROC 16 events Build DAQ Events Timestamp TTC Encoded header

6. Pixel FED 9U VME64x boards + Slink Mezzanine cards 9 Analogue, 5 FPGA Altera cards (not ORx) Common Electrical SLINK P3 64 bits @ 80 MHz + TTS throttle FMMs 40 x 9U boards + Slinks Reduces Motherboard complexity 10 layer (Micro vias for Alteras on mezz) Less risk. Testing, repairing easier. Spares Takes more space Connector reliability Signal integrity. Terminations Power

7. Strips and Pixel FEDs –Same Form Factor 9U VME. 450 vs 32 boards –Both Analogue & Digital designs –Same Interfaces ORx, ADCs –Different Constraints on Channel Count ; I/O vs Output rate –Same TTC, SLINK –Same fine clock skew 1 nsec –One board type in each system. –No trigger functions. Minor differences, power supply, slink connectors, fpga config, vme monitor bus Different PCB implementations - Motherboard vs Motherboard + Mezzanines Main Difference Digital Processing. (Altera vs Xilinx FPGAs) - Data Inputs. Raw & Synch vs ZS & Unsynch

8. FED Lessons “Good Design and Robust Manufacturing”. –Design evolves in response to technology (ASICs in TDR) –Invested a lot to get design right first time (2 board versions.) –Protoype to production was a long process. –Started at cutting edge finished with v. mature technology FPGAs (support) –Cost estimates evolve ADCs, FPGAs, Manufacture. Got FPGAs right. –Support of detector development v. underestimated, PMCs cf 9U FEDs –Prototype support underestimated ~ 50 proto + pre-prodn 9U FEDs –Tender process takes a long time. (not needed for pxFED) –Both ended up with “local” manufacturers. Good relationship essential. –Quality Control critical. Early BGA failures. V. High production yield. (also pxFED) No problems ORx assembly –Doing board testing at Assembly Plant took effort but paid off - Need for Custom Test equipment Optical Testers

9. FED Lessons –System interfaces came late. Electrical SLINK cards, FMMs. More elegant solutions. –Choice of industry standard VME for mechanics, power was good one –VME bus as monitoring is poor, better Ethernet (late design change?) –VME64x features not all used. (plug & play cost pxFED one unnecessary board iteration). Keep it simple. –Think about practical issues cf prototypes with final system, FPGA configuration. –Design issues with cooling. Tests in final configurations necessary. –(Un)expected effects (temperature variation of optical inputs for pxFED encoding) -Boards are built, debugged and delivered. But Firmware is never finished. –Custom protocols, SLINK, TTC, -> custom Firmware. Duplication of Firmware blocks (across CMS), SLINK, TTC, I2C, VME. – Firmware Libraries? Commercial/Vendor –Duplication of test software? Software effort ? (costings)

10. Future FED Possible Implementation FPGA Switch Off Detector sFED SDRAM Buffer SNAP 12 Rear Transition Module STTC 10G Serial Backplane DAQ Crate Event Builder 12 x 3 Gbps FE ATCA Crate 8U Cntrl/Mon Power ORx FPGAs MGBTs GBT PHY MAC Sparsification 1 per ORx ORx and FPGA on Mezzanine? ORx Mezzanine Prototype GBT Industry Standards ATCA crates Industry Protocols Data Transmission Serial PCIe … More use of Commercial Firmware cores. Data protocols, memory COTS Carrier Motherboards + CMS Mezzanines, Transition cards

11. Future FED-FEC Integrate FEC functions on FED FPGA Switch Off Detector sFED SDRAM Buffer MPT 12 Rear Transition Module STTC Trigger Throttle 10G Serial Backplane DAQ Crate Event Builder 12 x 3 Gbps sTTC FPGAs MGBTs GBT Recv MAC ZS 1 per ORx sAPV ORx OTx ATCA Crate 8U Cntrl/Mon Ethernet Power ORx ORx and FPGA on Mezzanine? Digital Inputs Zero Suppressed FE Data Inputs? Constraint Data Volumes on output

12. Final System Ideas n New sDAQ (sFEDs connected direct to Filter via Super Event Builder Network) n New sTTC (Broadcasting Filter Addresses to FEDs) n Crates u Just Mechanics, Power, Cooling. -> Control/Monitoring via Ethernet. u Serial Backplane based crates (Telecom ATCA, VME46?). F Less Slots (but wider) F Better Power & Cooling ? F Better control & monitoring ? F FED Event Builder Crate

13. Future FEDs –Common Tracker FED h/w probably technically feasible. Practical? –Common CMS FED ? Common SLINK, Common FEC –Go to Digital Input Data –Zero Suppressed at FE tbd –Large systems channel counts. Large form factors (cost driven). –Constraints on Channel count? –Not considered Tracker Trigger –Use of Emerging Industry Crate Standards e.g. Telecomms ATCA (VME) –Exploit Industry Data Protocols e.g. Serial PCIe (SLINK) –More use COTS Vendor Firmware cores, Industry standards –Use common COTS ATCA Carrier boards – with custom CMS Mezzanines (cost effective)

14. Spare Slides