1. Presentation for the NMI January 23rd 2009 John Coughlan Data Acquisition Systems for Big Science Dr John Coughlan STFC Rutherford Appleton Laboratory

2. Presentation for the NMI January 23rd 2009 John Coughlan Talk Outline STFC Technology, Who we are Data Acquisition for Big Science, CERN LHC LHC-CMS Data Acquisition boards. Next generation systems

3. Presentation for the NMI January 23rd 2009 John Coughlan STFC Science and Technology Facilities Council, STFC Largest of 7 UK Scientific Research Councils reporting to DIUS 2,000 scientists & engineers at main site Rutherford Appleton Laboratory STFC, Rutherford Appleton Laboratory, Oxfordshire UK

4. Presentation for the NMI January 23rd 2009 John Coughlan STFC Space Science, Lasers, X-Ray and Neutron sources, Particle Physics +… Operate several world-class large-scale research facilities on site, e.g. VULCAN Laser, ISIS Neutron source, Diamond X-Ray Light Source. UK Hub supporting activities of University researchers on other international facilities, e.g. ESA, CERN Geneva STFC, Rutherford Appleton Laboratory, Oxfordshire UK

5. Presentation for the NMI January 23rd 2009 John Coughlan Major Site Developments Harwell Science and Innovation Campus… New Detector Systems centre 2010/11 Agreement for new ESA research centre STFC, Rutherford Appleton Laboratory, Oxfordshire UK

6. Presentation for the NMI January 23rd 2009 John Coughlan STFC Technology STFC Technology Department 300 Electronic & Mechanical Engineers with strong Scientific Expertise Developing Instrumentation for Large Scale Science Experiments Telescopes to Medical Imaging, Micro MEMS to Major engineering structures, Novel sensors to Cryogenic Superconducting Magnets, Electronics and Microelectronics… Europe wide Microelectronics design support. EUROPRACTICE.

7. Presentation for the NMI January 23rd 2009 John Coughlan STFC Technology Electronics PCB Board and System Design FPGA Design. (Xilinx and Altera) Embedded systems. Board and System Test and System Integration Microelectronics, ASICs, MAPS sensors Current generation of Large Scale Projects for CERN completed, LHC New projects starting R&D phase, European Free Electron Laser XFEL

8. Presentation for the NMI January 23rd 2009 John Coughlan CERN European Laboratory for Particle Physics. Geneva Large Hadron Collider LHC Understanding the Origin of the Universe Started Operation (briefly!) in 2008. No Black Holes yet… Big Science

9. Presentation for the NMI January 23rd 2009 John Coughlan CMS Experiment at LHC 12,000 tons apparatus ~ 50 million electronic readout channels DAQ rate ~ 100 kHz 2,500 Scientists & Engineers in collaboration

10. Presentation for the NMI January 23rd 2009 John Coughlan CMS Installation 200 metres Underground STFC Microelectronics 80,000 x Analogue Pipeline ASICs Extremely Radiation Hard VME Racks Water Cooled

11. Presentation for the NMI January 23rd 2009 John Coughlan One Collision 40 Million Collisions every second. Keep images from 100,000 collisions every second. Each image built up from 10 million analogue sensors. Rarest new particles HIGGs ~ few dozen per year? Need for Massively Parallel (FPGA) based Processing ~ 1 TERA-Bytes / sec LHC Data Acquisition

12. Presentation for the NMI January 23rd 2009 John Coughlan LHC-CMS Data Acquisition Custom COTS

13. Presentation for the NMI January 23rd 2009 John Coughlan LHC DAQ Requirements Performance driven by science & experiments. Large channel counts. Large Form Factor PCBs High bandwidths. Custom data protocols. Specific Functions with Flexibility unknown science. FPGAs Electronics developed in collaboration with scientists. Large communities. In parallel with detector development. Loose specifications. Fully custom systems where performance and cost justify design investment Long development cycles. Long operating times. 10+10 Years Larger systems. Designs frozen early. Minimise risk.

14. Presentation for the NMI January 23rd 2009 John Coughlan LHC-CMS DAQ Board Board Features –9U x 440 mm (VME mechanics) –Optical Inputs, Analogue –96 ADC Channels 10 bit @ 40 MHz –10 x XC2V2000 –24 x XC2V40 –14 Layers –Double sided PCB System has –500 boards (large nr for PP) –24 Crates –8 Racks –50 KW Delivered to CERN 2006 Data In ~ 3 GByte/s. Data Out ~ 200 MByte/s

15. Presentation for the NMI January 23rd 2009 John Coughlan LHC-CMS DAQ Board Development Large boards. 9U VME Mechanics. High I/O count Analogue & Digital Large nr FPGA BGAs. 676 pins 1 mm pitch Concerns for BGA assembly Design for Test. Boundary Scan Design started in 2001. Prototypes (25 off) in 2003/4 before production (500 off) in 2005/6. FPGA Virtex II. Fix technology early. DCMs for channel synchronisation DCI useful but power hungry Double Data Rate I/O

16. Presentation for the NMI January 23rd 2009 John Coughlan Board parameters: - 9U x 440 mm VME64x form factor - Optical/Analogue/Digital logic ; 96 ADC @ 40 MHz channels - 14 layers (incl. 6 power). FR4. - Double-sided (secondary side with half of analogue components) - 6,500 components (most passives are 0402). Surface mount - ENIG metal finish, for BGAs. Pb/Sn - 20,000 connections ; 14,000 vias - 100 micron tracking, some 75 micron diff imp, min gap 100 micron, - 37 BGAs (larger FPGA 676 pins on 1mm pitch). All BGAs located on primary side. - Controlled impedance - Boundary Scan JTAG all Digital devices 440 FED boards required for full CMS Tracker readout Analogue components repeated on 2 nd side LHC-CMS DAQ Board Specs

17. Presentation for the NMI January 23rd 2009 John Coughlan FPGA Assembly on PCBs  3 rd Batch Prototypes 2003 ALL 6 failed Boundary Scan on several BGAs. Shorts under BGAs. Rework failed too. 17,000 BGAs in system EU Tender process for PCB & Assembly. Exception Ltd 1 out of 500 production boards failed Tests 2006

18. Presentation for the NMI January 23rd 2009 John Coughlan LHC CMS CERN Industry Awards Gold Award : Exception Ltd New Electronics Cover Story

19. Presentation for the NMI January 23rd 2009 John Coughlan Other LHC-DAQ Boards FPGA 130-90 nm generation 1 Gbps serial data. DDR2 VME Form Factors Conventional PCB manufacture. Step change design required for Next Generation of Projects

20. Presentation for the NMI January 23rd 2009 John Coughlan Operating in 2013 European X-Ray Free Electron Laser XFEL DESY Laboratory, Hamburg Next Generation Systems

21. Presentation for the NMI January 23rd 2009 John Coughlan Single Protein Molecules X-ray Imaging Just before XRay pulse During the pulse After pulse X-ray Diffraction Pattern 30,000 images per second each up to 16 MPixels

22. Presentation for the NMI January 23rd 2009 John Coughlan X-RAY Pixel Detectors DAQ For XFEL 16M + tiled pixel detector at 30K frames/s -> 1 – 6 TByte/s 128 x FPGA 40nm + 10G links Off detector DAQ next gen Advanced Telecoms ATCA crates 2008 - 2013… Scientific requirements ? 1 8 SFP+ FPGA Detector Pixel Sensors 10G Fibre 30 m 10 GBytes/sec x N cards Image Builders

23. Presentation for the NMI January 23rd 2009 John Coughlan Image Builder for XFEL Demonstrator Advanced Mezzanine Card 180 mm RAM CONNECTORCONNECTOR CONNECTORCONNECTOR CONNECTORCONNECTOR CONNECTORCONNECTOR KEEP OUT FPGA FMC FPGA KEEP OUT MGTMGT MGTMGT MGTMGT MGTMGT MGTMGT MGTMGT MGTMGT MGTMGT KEEPOUTKEEPOUT KEEPOUTKEEPOUT CONNECTORCONNECTOR CONNECTORCONNECTOR X- point TXTX TXTX TXTX TXTX RXRX RXRX RXRX RXRX FMC AMC Form Factor. Migrating to 8 FPGAs on 8U ATCA? FPGA ~ 16 x 3-6 Gbps serial links Analogue Cross Point for Image Building. 72x72 @ 3-6 Gbps DDR2/3 ~ 2-4 GBytes B/W 1-2 GBytes/sec In & Out VITA57 FMC Mezzanine I/O 2 x SFP+ opto TRx 10 Gbps (XAUI or RXAUI PHY) mTCA serial backplane

24. Presentation for the NMI January 23rd 2009 John Coughlan Next Generation Board Issues High speed diff pairs 3-6 Gbps. BGA pitch < 1 mm? Reduce Fabrication Risk. Advanced PCB design and construction techniques. Vias in pad, micro vias, Laser drill. Incremental build up layer PCB. FPGA 40nm generation FPGA to Memory interface. SO-DIMMs WASSO Memory controllers Hard/Soft IP from FPGA vendors 10 Gbps optical interfaces 3-6 Gbps Serial Backplanes. Power. Multiple POL. Analogue. Decoupling caps. next gen FPGA packages. Pb Free manufacture? Tools Signal Integrity analysis, how to measure eye diagrams on board?

25. Presentation for the NMI January 23rd 2009 John Coughlan Design Tools PCB Design Tool Flow based on CADENCE. Fast serial design Signal Integrity analysis, HyperLynx FPGA Design Flow Mentor Graphics How to integrate PCB and FPGA design flow? Pin outs. Need realistic FPGA designs to guide PCB layout, e.g. memory interfaces Our expert (Paul Hardy) is here today

26. Presentation for the NMI January 23rd 2009 John Coughlan Summary STFC Technology, Who we are Data Acquisition for Big Science, CERN LHC LHC Data Acquisition boards. Next generation systems, XFEL

27. Presentation for the NMI January 23rd 2009 John Coughlan http://www.scitech.ac.uk/ STFC, Rutherford Appleton Laboratory, Oxfordshire UK

28. Presentation for the NMI January 23rd 2009 John Coughlan Spare Slides