1. THE BTeV/CMS PIXEL TESTBEAM AT FERMILAB Lorenzo Uplegger

2. Introduction to BTeV The BTeV experiment was designed to investigate one of the most fundamental problems of elementary particle physic, the CP violation. The Standard Model of Particle Physics shows matter-antimatter asymmetry in K meson and B meson decays, but it fails to predict the magnitude of the baryon-antibaryon asymmetry observed in the universe. Looking among the B decays for new sources of matter-antimatter asymmetry was the goal of the experiment. In order to achieve this goal a highly sophisticated trigger system had to be used in order to collect the high statistics samples to make precision measurements. The challenge for the BTeV trigger and data acquisition system was to reconstruct particle tracks and interaction vertices for EVERY interaction and to select interactions with B decays.

3. In a B event, in addition to tracks at Interaction Vertex, there is a B that travels ~few mm and then decays. Only 1 event out of 500 is a B event and only a small fraction of them are interesting for CP violation The idea was to trigger at the first level selecting events with a vertex detached from the interaction vertex, thus triggering on the peculiar characteristic of events containing heavy quarks. THIS REQUIRES SOPHISTICATED TRACK AND VERTEX RECONSTRUCTION AT THE LOWEST LEVEL OF THE TRIGGER To succeed in this challenge, it was necessary to provide to the trigger processor the cleanest, easiest-to-reconstruct input from the vertex tracking system HENCE THE SILICON PIXEL DETECTOR WAS THE KEY FOR THE SUCCESS OF BTeV Characteristic of b decays

4. Muon EM Cal Straws & Si Strips SM3 Magnet RICH 30 Pixel Detector Stations BTeV detector layout

5. b production angle b production peaks at large angles with large bb correlation b production angle The higher momentum b’s are at larger  ’s   -ln(tan )  2 pp  bb+X Characteristic of hadronic b production

6. Given the importance of the pixel detector, an extensive testbeam program was necessary to demonstrate that the pixel detector developed by FNAL satisfied the BTeV requirements. The testbeam effort started in 2001 but only in May 2004 the beam was available and just one month after we received the detectors with the latest version of read-out chip (FPIX2) to be tested. The main goals that we wanted to achieve were: 1.Measure the spatial resolution of the sensors before and after irradiation 3.Test the read-out chip (ROC) in the real BTeV working conditions (data-driven mode) 4.Build events using only the temporal information (time-stamp) associated to pixel cells without the aid of an external trigger 2.Study the temperature dependence of the spatial resolution BTeV pixel testbeam goals

7. FPIX1 Y measuring planes at 0 deg FPIX1 X measuring planes at 10 deg FPIX2 Detector Under Test Can rotate from 0 to 25 deg in steps of 5 deg 120GeV p 3 FPIX1 pixel detectors upstream and 3 FPIX1 pixel detectors downstream compose the telescope The FPIX2 detector under test is in the center.  it can be rotated from 0 to 25 degrees  a cooling system can lower the temperature to -5° C Y measuring orientation X measuring orientation Experimental setup

9. DetectorPMC (mezzanine card)PTA (PCI Card) Host computer The overall philosophy FPGA It allows users to implement their own code (FIRMWARE) Detector B A mezzanine card for Detector B A generic PCI card Detector A A mezzanine card for Detector A A generic PCI card A host computer Detector C A mezzanine card for Detector C A generic PCI card

10. Readout process Data 1 Data 2 Data 3 Data 5 Data 4 Data 6 Data 7 Data 8 Data 11 Data 10 Data 9 Data 12 EVENT DAQ main features

11. Readout process noise DAQ main features

12. Readout process noise Data 1 Data 4 Data 6 Data 11 Data 10 Data 7 Data 8 DAQ main features

13. Readout process noise Data 1 Data 4 Data 6 Data 11 Data 10 Data 7 Data 8 noise Data 1 Data 4 Data 10 Data 7 Data 1 Data 2 Data 3 Data 5 Data 4 Data 7 Data 10 Data 9 Data 12 ????? DAQ main features

14. CLK particles timestamp 2Mhz 1 2 3456 Data1 Ts 2 Data2 Ts 2 Data3 Ts 2 Data4 Ts 2 Data5 Ts 2 Data6 Ts 2 Data7 Ts 2 Data1 Ts 2 Data2 Ts 2 Data3 Ts 2 Data4 Ts 2 Data5 Ts 2 Data6 Ts 2 Data7 Ts 2 Data1 Ts 6 Data2 Ts 6 Data3 Ts 6 Data4 Ts 6 Data5 Ts 6 Data6 Ts 6 Data7 Ts 6 DAQ main features

15. The read out system works in absence of an external trigger The data collection from the different pixel detectors is therefore asynchronous The DAQ must assemble the events in asynchronous mode Events are built using the time-stamp information DAQ main features In order to balance the different acquisition rates between the detectors, PCI cards and the PC, we took particular care in the design of the FPGA firmware and the read-out software

16. Each PTA card is provided with two memory banks (1 Mb each) and the firmware has been designed to make use of these memories as a data-rate compensation buffer. Signals from the PMC mezzanine cards The Altera programmable FPGA Output to host PCGlobal data path Two memory banks (1 Mb) The PTA design

17. Memoryfull!!! The first task that I was given when I came to Fermilab was the programming of the PTA card in order to handle a continuous data flux from the pixel detectors. Let’s see the strategy that I implemented in the FPGA

18. This process of periodic memory swap and transfer to a shared memory continues indefinetely. A shared memory is a chunk of memory in the host computer which can be accessed by different processes. We have several PCI cards playing this swap game in parallel: in order to be able to build events at a later stage, we needed a syncronization mechanism to keep the event builder as simple as possible. By synchronizing the swapping of all the memories we can build events in a very simple and immediate way. PCI card working mechanism

19. Interrupt handler A Interrupt handler A Interrupt handler B Interrupt handler B Interrupt handler C Interrupt handler C Interrupt handler n Interrupt handler n 01 Banks 1.The PCI card C redirect immediately the data flux to the other empty memory Readout working mechanism 2.The interrupt handler of the PCI card C forces the other cards to swap and then starts flushing its content to the host PC Each PCI card has its own interrupt-handler process listening for the memory-full signal Let’s suppose, for instance, that the PCI card C is the first being filled up.

20. 3.The other cards start flushing their (partially) filled memory banks to the host PC. 01 Banks Readout working mechanism 1.The PCI card C redirect immediately the data flux to the other empty memory 2.The interrupt handler of the PCI card C forces the other cards to swap and then starts flushing its content to the host PC Each PCI card has its own interrupt-handler process listening for the memory-full signal Let’s suppose, for instance, that the PCI card C is the first being filled up. Interrupt handler A Interrupt handler A Interrupt handler B Interrupt handler B Interrupt handler C Interrupt handler C Interrupt handler n Interrupt handler n

21. 01 Banks Readout working mechanism A0C0… n0B0 This architecture guarantees that events with contiguous time-stamps belong to buffers which are also contiguous in the read-out process. Interrupt handler A Interrupt handler A Interrupt handler B Interrupt handler B Interrupt handler C Interrupt handler C Interrupt handler n Interrupt handler n

22. 01 Banks A0C0… n0B0 Readout working mechanism A1C1… n1B1 This architecture guarantees that events with contiguous time-stamps belong to buffers which are also contiguous in the read-out process. Interrupt handler A Interrupt handler A Interrupt handler B Interrupt handler B Interrupt handler C Interrupt handler C Interrupt handler n Interrupt handler n

23. 01 Banks BUF i BUF i+1 A0C0… n0B0A1C1… n1B1 Readout working mechanism This architecture guarantees that events with contiguous time-stamps belong to buffers which are also contiguous in the read-out process. Events with the same time-stamp are contained within the boundaries of this overall buffer (BUF i ), or at least in the next one, BUF i+1, but not in BUF i+2, making the event-builder an implementation of a sorting algorithm. Interrupt handler A Interrupt handler A Interrupt handler B Interrupt handler B Interrupt handler C Interrupt handler C Interrupt handler n Interrupt handler n

24. Event Builder Shared Memory (unordered data) Event buffer (ordered data) Timestamps: Event Every hit with a new timestamp starts a new event (column) in a buffer. Other hits with the same time-stamp are appended to the right column in the buffer When the analysis of the BUFFER i+1 is over, it is reasonable to assume that there are no more data related to an event that begun in BUFFER i. Event builder

25. 7 plane coincidences FPIX1 Telescope FPIX2 Device Under Test XY tracks distribution on the pixel detectors

26. Resolution studies The resolution is calculated taking the RMS value of the distribution obtained subtracting the projected point of the track reconstructed by the telescope and the point measured on the detector under test.

27. Linear charge sharing applied Threshold set at 1800 e - The distribution of the residuals decreases until 15deg and then increases again. The resolution (without removing telescope resolution) varies from 12 to 7.5  m The change in temperature doesn’t change significantly the resolution. Temperature dependence at different angles

28. Linear charge sharing is applied Threshold set at 1800 e - The resolution (without removing telescope resolution) varies from 12 to 7.5  m The same behavior is observed for three different Voltages applied to the sensor. Sensor bias dependence at different angles

29. The angular dependence of the resolution can be easily understood observing the percentage of clusters sizes for different angles. The resolution improves when the number of hits in the cluster is 2 and the charge sharing can be used to determine the position of the hit. Cluster size as function of angle Total Cluster size = 1 Cluster size = 2 Cluster size = 3

30. Results Summary 1.Measure the spatial resolution of the sensors before and after irradiation 3.Test the read-out chip (ROC) in the real BTeV working conditions data-driven mode 4.Build events using only the temporal information (time-stamp) associated to pixel cells without the aid of an external trigger 2.Study the temperature dependence of the spatial resolution Most of the goals that we specified were achieved. The study of the irradiated detectors has been postponed when I’ll find some free time to analyze the data collected last March with one of the detectors that we irradiated last year.

31. From BTeV to CMS After BTeV cancellation, I was particularly attracted by the perspectives offered by the CMS experiment. Here, not only the physics program was very appealing, but we could even profit from our recent experience on BTeV to help on the forward pixel detector. There were several problems which we could work on to help the CMS collaboration and one of them was the low data acquisition rate their system could handle at the testbeam. The CMS telescope is based on silicon strip detectors which have a very slow integration time and a trigger system that has to wait for a confirmation from the software before releasing the veto signal. The result was the collection of ~300 events for each spill ( 0.6 second of beam every ~6 seconds) while, with the same beam structure, we had a rate of ~50000 events. With the new spill structure that is given right now (4 seconds of beam every 2 MINUTES) the collection of high statistic samples is very time consuming.

32. On the other hand the BTeV DAQ has a much higher bandwidth and can collect many more events as the beam intensity increases. The net result is an improvement of the number of collected data, which can be increased by an order of magnitude. Furthermore, the flexibility given by the FPGA micro-programming allow us to adapt the system quickly in case of necessity. From BTeV to CMS

33. CMS testbeam setup

34. CMS detector layout

35. Pixel detector Barrel Forward pixel (USCMS) UC Davis Fermilab Johns Hopkins Mississippi Northwestern Purdue Rutgers PSI ETH U. Zurich U. Basel IHEP Wien RWTH Aachen New Zealand

36. Pixel forward detector 1/2 disk Panels

37. BTeV and CMS pixel detectors comparison Pixel dimensions 50x400 µm 2 BCO is 132ns (~8Mhz) Readout clock is 70Mhz Max readout speed is 840Mbits per seconds over 6 output LINES Binary digital output ADC internal charge conversion Data driven read-out Registers read-back operations Few important adjustable parameters, like thresholds, leakage current compensation and reference voltage. Pixel dimensions 100x150 µm 2 BCO is 25ns (40Mhz) Readout clock is 40Mhz Max readout speed is 240Mbits per seconds over 6 LEVELS 6 levels digital output Analog charge information Triggered read-out No registers read-back Many important adjustable parameters, like threshold, leakage current compensation, analog output gain and offset, levels gain and separation, analog and digital voltages. BTeVCMS

38. The two detectors are designed with completely different philosophies. The first problem that I had to solve to integrate the CMS pixel detector in the BTeV telescope system was the synchronization of the events of the two systems. The BTeV detector does not use any trigger signal. The read-out is data driven and events are reconstructed using the timestamp information provided by the chip. The CMS detector, instead, requires a trigger signal to start the read-out. Events are the collection of the hits associated to a trigger. CMS synchronization with the telescope

39. All the BTeV detectors on the beam are synchronized in order to get the same timestamp from the hits generated when a particle passes through them. Data1 Ts 6 Data2 Ts 6 Data3 Ts 6 Data5 Ts 6 Data6 Ts 6 Data7 Ts 6 CMS synchronization with the telescope

40. 26 bits counter 2Mhz Clock 26 bits counter RST CMS FPIX BTeV FPIX1 6 bits counter Analog output Telescope Timestamp 20 most significant bits mezzanine counter6 FPIX BITS CMS Timestamp 26 bits counter Master PMC card Slave PMC card CMS synchronization with the telescope

41. Data1 Ts 6 Data2 Ts 6 Data3 Ts 6 Data5 Ts 6 Data6 Ts 6 Data7 Ts 6 CoincidenceTBM CMS Data Ts 23 The Timestamp difference between the CMS pixel and the FPIX detector is CONSTANT dependent by the time taken by the scintillators to form the signal, by the cable length, by the TBM and PSI response to a trigger. Scintillators Coincidence DATA CMS synchronization with the telescope

42. Experimental setup 00110010 One of the main problem that we faced was the reconstruction of the data output from the detector. Data are not sent out in binary format but they are encoded on 6 levels. Analog outputADC Binary digital output

43. Original configuration Telescope Host PC PCI bus expander Shared memory Mass storage

44. Modified configuration Telescope PCI bus expander ………………………………………………… CMS COSMO PC in the hut Custom built ADC Host PC Shared memory Mass storage

45. Modified configuration

46. 7 planes coincidences measured on the CMS pixel detector Proof of concept

47. Once we proved that the synchronization scheme was working we started immediately to work on the initialization and control of the chip itself to give us more flexibility in case we needed to study some parameters related to the readout clock. The task was not easy given the big number of parameters that can be tuned for each detector Total integration in the DAQ

48. Summer testbeam Although the effort to fully control the detector was not as simple as we were expecting, after a month, the full integration of the CMS pixel detector in the testbeam DAQ was completed. We requested some beam time in mid July and we started to take data with the new system. We were able to characterize at different angles and different temperatures 3 detectors, before we took them last week to be irradiated. The adapted DAQ was able to collect around 25000 events per spill which reduced by big factor the time needed to collect high statistics for precision studies. This was the first time that the new chip with the analog information was tested on the beam. Here are the very first results of this summer data taking…

49. First results of the summer data taking Charge collection efficiency measured along the pixel dimensions µmµmµmµm

50. Results summary The new DAQ system proved to be faster than the one that CMS was using before. The analysis is still in progress since the last run was taken few weeks ago, nevertheless I showed the preliminary results of the charge collection efficiency in the CMS pixel detector. In the next future we are going to study the charge collection efficiency of the detector that we have just irradiated.

51. After the testbeam performance in May we were asked to evaluate the possibility to develop a new test-stand for the CMS pixel. Based on the experience accumulated in these years I contributed to the design of a new PTA card which will improve considerably the readout bandwidth in order to keep up with higher rates. The new PTA will play an important role in the new test-stand which will be used for the imminent CMS pixel production. This PTA will be also very useful to test the detector in a high intensity beam to simulate as much as possible the real condition of operation in LHC. A new test-stand for the CMS pixel detector The new PTA card

52. Why there is the need for a new test-stand? The problem that the CMS collaboration was facing was again a bandwidth problem since the full characterization of a plaquette ( a module containing more than one chip required 117 hours. After a lot of improvements of their test-stand, now the time needed is reduced to 2 HOURS. This time makes almost impossible a full scale testing of all the components to be installed in the final detector. Based on the recent experience in the testbeam we believed that a new DAQ based on our PCI cards could reduce this time by a lot. A first release of this new test-stand has been delivered to SiDet last week and the total time for a full calibration of a single chip is around 2 MINUTES. This considerable reduction of the time needed for the characterization of the readout chips will make possible to test all the components before being installed in the final detector, making the CMS pixel detector a successful project.

53. Test-stand GUI

54. A highly efficient vertex detector is necessary in modern experiments to perform studies of events containing heavy quarks and to achieve the high precision measurements necessary to challenge the standard model. With the latest development in the pixel detector technology the idea of a trigger capable to select events with heavy-quark secondary vertices is now conceivable also for the LHC experiments where the track density is very high. Fermilab has a key role in the construction of the forward pixel tracker of the CMS experiment and it will have an important role in the upgrade of the detector too. With the expertise accumulated in the BTeV experiment, Fermilab can become the center for the development of a much more efficient trigger system than the one currently designed for CMS. A final remark before the conclusions

55. Conclusions I have been responsible for most of the hardware and software for the BTeV testbeam DAQ for the last 2 years, solving all the problems that we had to face during the data taking. I have successfully integrated the CMS pixel detectors into the BTeV DAQ at the testbeam. We are currently running to characterize them. It took us a few months to produce results. We developed a new test-stand that has been delivered to SiDet and will be crucial to characterize the detectors during the imminent CMS forward pixel production. I have shown the testbeam results for the BTeV and CMS pixel detectors I presented the BTeV results at the last IEEE conference A lot of work still needs to be done… From the preliminary results the new PSI readout chip seems to work very well. This is a great result for all of us.