1. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 1 US CMS Silicon Tracker I&C and M&O planning Joe Incandela University of California Santa Barbara US CMS Silicon Tracker Project Manager July 22, 2005 Fermilab

2. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 2 Overview There have many reviews: US CMS PMG Review of M&O: Fermilab, April 11-12, 2001 US M&O Evaluation Group Review: Fermilab, April 8-10, 2003 US LHC M&O Evaluation Group Meeting: Brookhaven, February 5-7, 2004 US LHC M&O Evaluation Group Meeting: Fermilab, January 27-29, 2005 Will show excerpts from a few Main message is that Sitrk planning has not changed substantially over the years Will show overview of current M&O project file Some I&C is included in the project file Discuss current integration planning and a new initiative for remote effort

3. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 3 Tracker Outer Barrel Silicon Joe Incandela, University of California Santa Barbara US CMS Silicon Tracker Project Manager April 9, 2003 Outline: Basic considerations and assumptions Project end and transition to M&O Maintenance and Operation Plan

4. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 4 CMS Tracker 5.4 m End Caps (TEC 1&2) 2,4 m Inner Barrel & Disks (TIB & TID) Pixels Outer Barrel (TOB) ~100 m 2 volume 24.4 m 3 running temperature – 10 0 C

5. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 5 Basic Considerations Basis of estimates Experience considered: CDF Run 1b and Run 2a D0 Run 2a Scaling factor for resources ~ 1 despite large increase in channel and module counts Considerations TOB = CERN+US CMS but… CERN has large general infrastructure responsibility US CMS group is an obvious choice to play a lead role in M&O team. Operational expertise lies with those having the most testing experience during construction Very useful Tevatron experience Resource driving assumptions: US will assume a major portion of responsibility for TOB M&O Need continuous coverage of 6 main M&O areas Detector should be ready for physics as early as possible (US CMS will now also produce and test >3000 TEC modules…)

6. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 6 CDF/DO Experience CDF and D0 installation & operation A very big job which takes many months and many people Current (~steady state) operations requires on the order of 7 high level experts with backup of ~10 additional operators and monitors We assume that some monitoring can be done remotely for TOB CDF Silicon installation and commissioning group CDF Silicon Cables exiting the bore of the outer tracker “Our current commissioning effort is taking so long partially because of inadequate resources (both physicist and non-physicist). So, I would advise that for commissioning especially, you do not underestimate the need for … resources - ask for enough dedicated engineers to complement as many physicists as you can get. ” -- co-leader of CDF silicon operations group in mid 2002.

7. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 7 Project end & Transition to M&O US must anticipate the need to cover much of the effort for the installation and commissioning (I&C) and probably the lion’s share of the maintenance and operation (M&O) of the TOB. Some I&C resources thru FY05 (now FY06) are included in the construction project. Installation/testing of rods in wheels (Originally) scheduled for Jan 04 – April 05 (Now scheduled for Sep. 05 – May 06) 2-4 FTE Sr. Physicists, 7 Post-docs, students. Testing modules on rods at CERN before installation Limited system tests. 1 FTE Engineer (base). Rod & Module repairs, cooling system, mechanics. Laboratory space and equipment for testing and repairs. In late 2006, large M&O support team must be in place. We estimate ~ 12 FTE physicists at CERN together with continued engineering and technical support & associated M&S and operating costs for maintaining a small laboratory space for testing and repairs.

8. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 8 M&O for Tracker Outer Barrel: 6 Main M&O areas 1.Ops. 2-3 phys. 1/2 eng. 1/2 tech. 24/7 on-call to avoid dead time. Leadership: organize US effort, executive decisions, interpret diagnostic results. Interface with CMS admin. 2.DAQ 1.5 phys. 1/2 eng. 1/2 tech. Front end and higher level modules. FEDs and EB managers. Diagnostics, testing, (repairs, re-testing ?), swapping DAQ modules. 3.Radiation protection and cooling ~1.5 phys. 1/4+1/4 engineer, 1 tech. Monitoring, repair. System issues: cooling tech present or on call at all times to deal with problems as they arise. 4. Crates and power supplies. 1.5 phys, 1/2 eng. 1/2 tech. Testing supplies prior to installation, retest after installation. Safety interlocks. A serious ongoing maintenance issue. 5. Monitoring and calibration databases 4 phys. 1/2 tech. at CERN Additional remote effort (see below). Need to catch problems in data and in calibration techniques. Very significant effort just to maintain a good channel list. 6. Alignment 1.5 phys. Laser alignment runs, hardware maintenance. Offline alignment and monitoring of alignment constants.

9. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 9 Other areas Not included in our resource projections: Module Failures analysis and development of failure prevention methods: CDF/DZero have required considerable resources to Understand why some silicon modules have failed Example: Resonant wirebond breakage Develop failure prevention strategies For resonant wirebond breakage - developed completely new operating procedures to prevent repetitive readout sequences that led to exactly periodic current swings. “If these resources are forced to come from operations, as was the case at CDF, things will be strained and commissioning and/or operational efficiency will be affected.” – CDF silicon operation group co-leader Some of this can be done remotely (see discussion on FNAL remote monitoring/diagnostics station below)

10. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 10 US CMS Silicon Tracker US LHC Detector Maintenance and Operations Evaluation Group Meeting Joe Incandela University of California Santa Barbara US CMS Silicon Tracker Project Manager January 28, 2005 Outline: Project overview and Status Schedule and Project Completion Installation and Commissioning, Maintenance and Operations R&D for upgrades

11. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 11 Project end & Transition to M&O We assume we will be needed to cover at least half of the effort of installation and commissioning (I&C) of the TOB. Installation/testing of rods in wheels (05-06). 2 FTE Sr. Physicists, 7 Post-docs & students. Testing modules on rods before installation. Limited system tests. 1 FTE Engineer (base). Rod & Module repairs, cooling system, mechanics. Laboratory space and equipment for testing and repairs. For maintenance and operation (M&O) of the TOB we need a team to provide support in 6 areas of operation. By late FY06 early FY07, US M&O team must be in place. We estimate 13 physicists at CERN together with continued engineering and technical support & associated M&S and operating costs for maintaining a small laboratory space for testing. Only substantial changes are in the schedule dates…

12. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 12 Project I&C Project I&C now overlaps with M&O in FY06 but this is probably a good thing because more resources are going to be needed to recover schedule.

13. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 13 M&O Cost Estimate

14. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 14 US CMS SITRK M&O Resources Manpower on Base Post-docs, students, and one engineer.. Post-docs all redirected from other programs or the production effort. Manpower on Project All technicians and additional engineering.

15. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 15 M&O Thru 2009 (as of Jan. 2005) 1.1M$ COLA (using Jan. formula with 1.1 FCSH/USD) 1.2M$ Engineering + technical + ALL travel 350k$ R&D

16. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 16 Experience matters First 2 years of operation (FY07-08) will be critical years Difficult problems will have to be solved and will require the most experienced people Establish all steady state operations of the tracker Diagnosis and classification of many failure modes and problems Detailed operating procedures & training guidelines Refine all monitoring, calibration and alignment methods Current uncertainties make it difficult to argue strongly that we absolutely need all of these resources. On the other hand, Tevatron experience tells us to prepare as strong a team as possible

17. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 17 Upgrade R&D Issues CMS silicon has limited lifetime. 10 years of operation (we hope). Further, possibly higher luminosity running will require more rad-hard silicon than currently exists. R&D for current silicon took many years. We need to be involved in future efforts at the level of prototyping and testing silicon or silicon-alternative designs. 65k$ per year M&S FY04 spent 90k$ for NRE and new sensor production Demina (Rochester) Bortoletto (Purdue) FY06+ Some ideas now being developed. Will visit HPK in October.

18. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 18 Summary & Conclusions (Jan. 2005) US Groups now responsible for building and testing roughly 65% of the CMS tracker modules We have lost a year due to component problems. Expanded capacity to recover schedule Components now ok. Based on CDF/D0 experience, significant resources needed for I&C and M&O. We estimate 13 scientists plus engineering and technical support to allow US to have significant involvement in all aspects of operations Nota Bene: The US tracker group now has played a more important role in the construction project than originally anticipated or panned (e.g. problems found and solved with sensors, hybrids, rods… and a major increase in fraction of tracker modules built in US). Nevertheless, we have not made any significant changes to our M&O resource plans… We believe we have a good plan!

19. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 19 TOB Project as of July 2005 US TOB rod integration effort stalled Serious problem found in US at both FNAL and UCsB A very robust solution has been found (Option 7) New interconnect cards are required (see next slides) We pay 80k$ in order to expedite them (saves >1 month) Total schedule hit is going to be around 5 months Module production to continue Need to purchase 400 sensors from HPK (250$) ASAP!!! Need additional module storage costing ~15k$ We are currently storing almost 2000 modules! TOB integration to proceed Use rods modified (Option 6) and fully tested in the US September installation of one full cooling segment of TOB6- at CERN with substantial US involvement (we hope) New Clean room (Tracker Integration Center – TIC) to be ready for support tube in October, fully operational by December (my estimates).

20. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 20 Rods Overview Basic components procured Assembly of frames is DONE (modulo ICC) Integration of modules on rods is understood Rate of ≳ 8/d in the US is not be a problem … Testing was problematic and rod production was halted I2C communication errors seen on 30-50% of rods Recent and very substantial effort to understand and remedy Many thanks to : A. Marchioro, Wim Beaumont, M. Johnson, G. Maggazu, Slawek Tkaczyk… Simple solution (number 6) an improvement but not quite.. A better solution (number 7) adopted and requires new ICC Secondary issues of header errors and failures in LT tests are also under study

21. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 21 Main cause SCL on FE-Hybrid SDA Unequal fall-time of SCL and SDA signals Picture shows sequence of SCL/SDA signals with SDA pulled-down in AOH and measured on FEH

22. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 22 Option 6 AOH FE-Hybrid SCL SDA 82  PSU Control PSU FE LLD APV Jumper creates a common time domain but some recent information from US indicates may still be marginal

23. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 23 Option 7 AOH FE-Hybrid SCL SDA 100 pF Parasitic on AOH C A PSU Control PSU FE ICC CRT245 (Line driver)  22   pF  pF 330  LLD DCU/APV

24. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 24 Effect of “damping” resistor SDA with “damping” SDA without “damping” SCL not show on this slide

25. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 25 Current (limited) production While I2C solution 7 is being prepared we are assembling and testing a limited number of rods with "jumpers" (I2C Solution 6) Test I2C solution 6 To be shipped to CERN to populate 1st cooling segment, and possibly the cosmic rack Gain experience with (somewhat) large scale rod production Somewhat longer testing cycle to double- triple-quadruple-check I2C behavior Gain experience with testing protocol Document/investigate non-I2C failures Refine cut requirements

26. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 26 Part Count as of July 13, 2005 UCSBFNAL Rods jumpered with “solution 6” 1434 Rods populated with modules 1434 Rods tested at SRT 1033 Rods tested with MRT 618

27. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 27 SRT results – out of 43 tested rods 5 with unconnected thermistors (as read out by DCU, i.e., these are not the "hardwired" thermistors) 4 at FNAL, 1 at UCSB 1 fails ring redundancy test FNAL 1 Laser failure FNAL 1 with strange problem, open ring? UCSB, under investigation All 4 SS4 rods (UCSB) have unusually high noise Discussed in TOB meeting at July Tracker Week: agreed upon actions to be taken

28. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 28 But also… Still low rate of I 2 C errors 230 that we thought would have been cured by jumper (Solution 6) Other not-yet-understood I 2 C errors A few other annoyances like header errors HW or SW problems? Experts are actively investigating

29. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 29 Rod Plans After this "limited production" we will stop Take stock of situation, firm up procedures Attack other (non I 2 C) problems Hold a mini-workshop to get the UCSB and FNAL-based people together in August We will likely consider another fairly substantial production exercise Allows one to hunt more rare problems Module storage…

30. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 30 Thermal screen operational Wheel ready for insertion in the tracker support tube and… The tube with thermal screen are already in the assembly clean room. Wheel on assembly chariot Meanwhile …Wheel is ready!

31. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 31 Preparations for TOB integration - where we are as of July 14, 2005 He leak test standOK Rod precabling standOK Cu power cablestwo weeks Cable “bridge”one week Vacuum system for solderingto be commissioned Service supportsin production Cooling manifoldsin production Readout hardwareFEDs/FEC available - firmware 1 week (?) Power suppliesavailable Test softwareto be commissioned on C-Rack Plumbing in 186 (cooling and He)Basically done DB tools defined - only ribbon registration implemented ------ later ------ Alu cablesneed to update layout and start production (lengths OK) Cable test boxin production in Lyon (mid-August) - need updated pinout DOHMsfinalizing tail lengths - start production

32. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 32 US involvement New key personnel Jeff Spalding – will become deputy manager responsible for overseeing initial US integration activities at CERN Very experienced manager Assist the talented but a bit inexperienced CERN team Improve US-CERN communications Slawek Tkaczyk- will initially lead the US testing effort during integration at CERN. Also very experienced Will oversee experienced US physicists at CERN during integration September ~1 month integration Exercise A strong US team at CERN under Spalding and Tkaczyk Two experienced US rod testing physicists Two experienced US Silicon technicians/engineers

33. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 33 Goals of September Exercise Establish detailed procedures Establish rod acceptance tests (upon receipt from US) and rod tests prior to installation on wheel Full installation and commissioning of a rod on wheel Serial installation/commissioning of up to ~20 rods Removal sequence Unsolder all installed rods and remove Batch installation Methods/procedures for batch mode installation of rods Slot in place up to ~10 rods and test connectivity (front ends not powered) Solder cooling pipes all rods and test cooling ring Cable and test all rods and test DAQ ring(s) Use this information to develop resource loaded schedule for full TOB integration We expect to be revising schedule more than once, as other problems develop in commissioning. Information from all integrated experience will be used together with resources/time available to optimize safe assembly…

34. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 34 Integration in 2006 After the September exercises… Plan to have US team start to appear in late ‘05 or early ’06 Depends upon restart of rod production – If smooth, we can transfer some of the experts to CERN. Current management planning is directed toward evolution from sub-system integration teams to full tracker integration team. We are already working with the integration manager (Nicola Bacchetta) to develop TOB integration planning documents that will be reviewed in Autumn. Idea is to develop a uniform set of protocols followed by all subsystems. By the time we have completed the subsystems’ integration we would like the transition to commissioning of the full tracker to be as smooth as possible. This is also facilitated by the new tracker integration center.

35. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 35 Current Status of TOB Module production lines have completed 25% of TOB >99% yield and >99.95% good channels Production of 40-50 modules/day can be achieved, provided that we have adequate parts flow and stable testing Rod frames production in Helsinki is complete Rod integration and testing capacity ≳ 8 rods/day I2C communication problem is understood Rod integration and testing has restarted in the US with solution 6 Solution 7 likely definitive – will learn more in coming weeks. TOB Wheel is ready and other preparations are in train Integration exercises will start soon Very capable team at CERN US may send a team in September (headed by Jeff Spalding)

36. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 36 Tracker DAQ Test Facility at FNAL Being organized by M.Johnson, J. Spalding and S. Tkaczyk. (S. Tkaczyk provided these slides – modified somewhat by J. I.) Motivation: Allows US CMS tracker community to: Participate in online code development for monitoring data quality and detector performance Experience is crucial for the tracker to be usable early in running! Educate new people in using the CMS DAQ HW and SW tools Much more effective to learn this before taking shifts at CERN! Study long term performance of strips and pixels Could determine our upgrade strategy Troubleshoot tracker problems emerging during regular operation of the CMS tracker Pathological modules/hybrids found and fully categorized in US production efforts will be used as a “library of pathology” that could be useful for development of work-rounds for problems arising during data-taking.

37. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 37 TrackerDAQ@FNAL: Configuration HW - a replica of the CMS Tracker DAQ slice to read out a set of RODs (and pixel structures?): Components: FEDs, FEC, TTC, PS, cables, fibers, DCS, interlocks. SW – Run Control, XDAQ shell with tracker- specific extensions for data acquisition, DCS, PS control, calibrations, online data analysis

38. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 38 TrackerDAQ@FNAL: Remote Data Quality Monitoring Allow CMS collaborators to participate in the operation of the tracker in the US. Develop expert level knowledge among US CMS tracker community by means of: Operation of the elements of the tracker (strips, pixels) at FNAL CMS DAQ test facility Development of online software in collaboration with other tracker teams Participation in daily operation and monitoring tasks at CERN through regular shifts at FNAL

39. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 39 TrackerDAQ@FNAL: Implementation Dedicated consoles and workstations (costs included in the Remote Operation Center) Scheduling/oversight of regular tracker shifts at FNAL – 1/4 FTE DAQ test stand at SiDet Will find some eqpt. at FNAL and get some on loan from CMS. (To purchase all is > $100k). HW support – ¼ FTE PC, Software Support – ¼ FTE Comp. Prof. Total Physicists: 6 at FNAL and 3 at CERN Would include post-docs/students from university groups Other items: Dedicated audio/video equipment at CERN and FNAL Travel Support

40. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 40 Preliminary (crude) estimates FY06 $20-50K -DAQ Hardware $10K -Audio and Video Equipment $25K -Analysis Workstations and Data Storage $25K-50K - travel support – (5-10 people) 1/2 FTE support – technician + comp. professional FY07-FY09 $50K/year – travel support $10K-$15K/year - maintenance

41. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 41 Conclusions TOB module production is on schedule Rod production halted to fix major problem Integration planning is underway Schedule must be revisited (this will not be the last time…) US beginning to structure a strong team at CERN Jeff Spalding the US tracker project deputy in residence Slawek Tkaczyk will oversee US testing team Critical integration exercise at CERN in September US I&C and M&O planning is stable and adequate Key will be to see if we actually get the support we need to fulfill our plan which has been stable for many years. TrackerDAQ@FNAL: Early stages of planning for an FNAL-based training/monitoring/online-development facility for the CMS tracker An excellent and necessary US tool!

42. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 42 Additional HPK Silicon for TOB Modules NeededNeededOrdered 3528OB270567058 1680OB133603180 YieldsCentralOptimisticPessimistic Assembly99.4%100.0%98.8% ARCS Tests99.2%99.6%98.8% LT Tests99.9% Rods100.0% Total98.5%99.5%97.5%

43. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 43 Qty for various scenarios If 1% loss on rods If 0.5% loss on rods Needed (No spares) OB2OB1OB2OB1OB2OB1 Central 716734137239344772033430 Optimistic 709533797167341371313396 Pessimistic 723934477312348272763465 Rod Loss0% 1% 0.5% Remaining Order OB2OB1OB2OB1OB2OB1 Central109233181267145250 Optimistic 3719910923373216 Pessimistic 181267254302218285

44. M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 44 Proposal Cost per sensor is 428 Euros 171,200 euro @$1.22/Euro 207k$ 20% for currency fluctuations & contingency etc…  250k$ ProposalOB2OB1 150250