Rods Schedule and Issues J. Incandela University of California Santa Barbara for the TOB Rod Group Tracker Integration Meeting October 27, 2005 Slides courtesy of D. Abbaneo, C. Campagnari, A. Dierlamm, Y. Gotra, J. Lamb, others…

TOB Rods- October 27, 2005 – Tracker Integration Review slide 2 Rods Overview Basic components procured Assembly of frames is DONE (modulo ICC) Integration of modules on rods is understood Rate of ~ 8/d in the US is not a problem … Testing was problematic and rod production was halted I2C communication errors seen on 30-50% of rods Very substantial effort to understand and remedy Solution required making new Interconnect cards (ICC). Now in production. First 25 sets arrive in US today (?) Larger batches arrive at CERN starting early November (?) Rod production scheduled to restart this week.

TOB Rods- October 27, 2005 – Tracker Integration Review slide 3 Recent rod production Meanwhile we implemented a simple fix and built ~75 rods Simple fix allows long term testing in Multi-Rod Test (MRT) stands It is not robust enough to be a final solution We shipped rods to CERN for integration studies For integration studies Develop/document procedures for installation/removal of rods

TOB Rods- October 27, 2005 – Tracker Integration Review slide 4 Mechanical Assembly procedures well developed Trained personnel (including backups) at both sites Takes ~ hours to assemble a rod Add 0.5 hour for ICC swap Not a bottleneck Long term testing ~6 rods/2 days or 4/day if necessary No issues – everything is in place

TOB Rods- October 27, 2005 – Tracker Integration Review slide 5 Electrical testing General idea Assemble rod Quick test of single rod Single Rod Test stand (SRT) Long term test long is really not that long (~ 1-2 days) Multi Rod Test stand (MRT)

TOB Rods- October 27, 2005 – Tracker Integration Review slide 6 Single Rod Test (SRT) Tests of Hardwired thermistors and humidity sensors CCU ring redundancy DCU readouts Leakage current Pedestals & noise Just looking for gross problems No channel level characterization At room temperature FNAL UCSB

TOB Rods- October 27, 2005 – Tracker Integration Review slide 7 SRT (continued) Most tests done using LtStruct currently V30 Hardwired sensors tested with simple custom box Root macro similar to module test macro generates plots Plots etc. uploaded to e-log Output root and text files saved for ALL tests including version number, if tested more than once Only piece missing: protocol for testing digital functionality, experts to provide (?)

TOB Rods- October 27, 2005 – Tracker Integration Review slide 8 Sample Rod output

TOB Rods- October 27, 2005 – Tracker Integration Review slide 9 Multi Rod Test-stand (MRT) (Semi) long term test ~ 1-2 days 6-8 SS rods/load Thermal cycle Full channel level characterization FNAL UCSB

TOB Rods- October 27, 2005 – Tracker Integration Review slide 10 Original MRT test cycle - have since developed shorter one Warm Test Cold Warm Cold TestWarm Test Cold TestWarm Test

TOB Rods- October 27, 2005 – Tracker Integration Review slide 11 MRT status Hardware: all OK, robust Several different coolant leaks at UCSB in last few weeks – hopefully now solved DAQ software (LtStruct) Much progress in the last months Thanks to great cooperation from Wim et al.

TOB Rods- October 27, 2005 – Tracker Integration Review slide 12 MRT status (cont.) Analysis SW DefectAnalyzer Makes plots and XML file First pass cuts defined Will iterate using data from early production Custom perl script Compares bad channels on rod with bad channel lists from module tests Also flags DCU problems, etc Custom root macro To make plots similar to defect analyzer OK, but will be retired

TOB Rods- October 27, 2005 – Tracker Integration Review slide 13 Workshop in August We had an internal workshop in August Try to converge on remaining loose ends Minutes Some of the issues discussed Original default long term standard scenario 2 day, go cold 4 times "Finalize" cuts to flag bad channels Module grading - same as ARCS A: < 1% bad channels B: < 2% bad channels C: > 2% bad channels F: bad HV

TOB Rods- October 27, 2005 – Tracker Integration Review slide 14 Workshop (continued) Rod grading: Pass: only A or B modules ≥ 1 out of 3 non-DCU thermistors OK I(leak) for all groups < 20  A Fail:otherwise XML file for DB Went over it line-by-line Decided on several changes additions e.g., added thermistor information For details see minutes of workshop Now implemented Still a few problems uploading

TOB Rods- October 27, 2005 – Tracker Integration Review slide 15 Towards production: what was missing Demonstration that we can sustain production rates on a daily basis for a long time To this end, we were thinking of going through a production exercise, starting asap ~ 40 "type 6" rods/site ICC Swap – where will this be done? Definition of digital tests May have already been completed ? Some DB details Should be OK

TOB Rods- October 27, 2005 – Tracker Integration Review slide 16 Rod Status Now - UCSB and FNAL While waiting for ICC’s to start production, the two sites have focused on slightly different issues FNAL: testing of new SW (FEC V3, Scientific Linux, Lt V30) UCSB: production exercise to shake down remaining operational issues Fine-tuned scenario to match 4 rods/day rate Freezer takes max 8 SS4, 6 SS6, 4 DS  need one day scenario (so much for long term  ) Only two thermal cycles W  C  W  C  W CalProfRuns only with inverter-on to save time Added HV cycling at each temperature This is not working reliably, may have to ditch

TOB Rods- October 27, 2005 – Tracker Integration Review slide 17 General 4 sets (2/site) of OFECs in transit from CERN FNAL will "commission" this system first; UCSB will follow when it works First batch of ICC cards for 25 rods also in transit Held up at Memphis, expected delivery - today ½ FNAL, ½ UCSB Will mount on existing rods, then put modules on rods, then test, then ship. ASAP

TOB Rods- October 27, 2005 – Tracker Integration Review slide 18 FNAL Status (1) Over 30 production scenarios run on MRT since July SS6 rods and four DS4 rods were assembled (using jumpers on pre-production ICC cards), tested and burned-in Migration of rod test software running to Scientific Linux with FEC software V3 was successful Long term testing with latest release of the FEC Software showed good stability of software and the control ring 1 upload/download error in 24 hours with 4 DS4 rods Latest release of the LtStruct (v30) was extensively tested Number of production scenarios with Single and Double Sided rods

TOB Rods- October 27, 2005 – Tracker Integration Review slide 19 UCSB Production Exercise: Mechanical Built 24 rods last week (22 SS6, 2 DS) Warmup 1st two days, then built 4 rods/day Bottom line: OK Some Issues: Dirty rod boxes, Al dust Tail bending a la FNAL: OK. ICC cards swap: adds minutes/rod DS rods: worry about exposed non-encapsulated bonds Designing cover with cutouts for ICC cards to enable handling of connectors and cables with minimal risk

TOB Rods- October 27, 2005 – Tracker Integration Review slide 20 UCSB Production Exercise: Testing 20/24 rods through SRT Mounted some modules with known pinholes Flagged them without a problem 1st MRT scenario OK, 2nd failed Failure in newly introduced HV cycle. CAEN got stuck. Not a new problem. A couple of laser failures (?) – Worrisome… Currently under investigation – expect info later this week. Some noise oddities to be investigated Some ICC/header errors. Hopefully will be better with new SW, new OFEC (Almost) successful 1st upload of rod XML file to DB Need some small name changes

TOB Rods- October 27, 2005 – Tracker Integration Review slide 21 UCSB Production Exercise: Short Term Plans Stop making rods We learned what we needed to learn Continue exercising testing cycle HV cycle XML uploads

TOB Rods- October 27, 2005 – Tracker Integration Review slide 22 Rod Reception Status Preparing rod reception testing for high rate Automating flagging of bad channels and validation comparison to US Installed/running DefectAnalyzer with the same cuts as US Bad channels agree so far Mostly concerned with making sure there is no shipping damage or other exceptional problems All but two of the forty “production” rods from the US have been tested

TOB Rods- October 27, 2005 – Tracker Integration Review slide 23 Rod Reception Status High current encountered on some modules Annoying but not fatal Few modules on rods from US exhibit high current - humidity effect with a longer “memory” than usual A module that has been exposed to room air for several hours will show a current ramp-up to up to 1mA, a few minutes after bias turned on Then the current falls back to normal levels over a time of about an hour If the module is kept in dry air the high current doesn't reappear if the bias is turned on and off Seems to be correlated with a problem in mixing the Sylgard 186 encapsulant, which left a “stain” on the sensors.

TOB Rods- October 27, 2005 – Tracker Integration Review slide 24 Rod Reception Status

TOB Rods- October 27, 2005 – Tracker Integration Review slide 25 Rod Reception CERN Typical Noise

TOB Rods- October 27, 2005 – Tracker Integration Review slide 26 “Straw” Rod Production Schedule (Sep. 05) Month TOB+TOB-  (3/d)  (4/d)Integral  (4/d) Oct020 0 Nov Dec Jan Feb Mar Apr Totals

TOB Rods- October 27, 2005 – Tracker Integration Review slide 27 Summary Lots of progress Can build and test rods in the US Some problems Production startup will be a little rocky Hope to converge to stability by end of year, prior to peak production rates Robustness of stands has not yet been fully tested, and we will likely encounter some problems and short delays over time. Schedule On paper, we can get all rods here in time In practice this looks extremely challenging. I would expect that a more realistic schedule would have 1-2 months contingency