D0 Collaboration Meeting July 2002 L2: The Road Ahead James T. Linnemann MSU D0 Oklahoma Workshop July 9, 2002 Special Thanks to Terry Toole for status slides

D0 Collaboration Meeting July 2002 Trigger Organization CTT PS CAL CTT CFT/CPS STT MUON FPD CAL MUON PS CTT GLOBAL Detectors L1 Trigger Framework/SCL L2 Trigger FPS CPS/St CAL CFT CPS Ax L0 SILICON SCINT MDT/PDT

D0 Collaboration Meeting July 2002 ● Making decisions since May 1 st (On since Nov. 2001) ● Rejection done on em, jet, and muon objects ● 4 L2 Crates online: Global, Cal, Mu Forward, Mu Central ● 1 Alpha per crate – Limits L2 input to ~.5kHz debug – Probably ~1.5-2KHz maxopt if no optimizer problems) ● Ability to disable individual L2 inputs ● L2 Software Framework improved – More robust operation in real system – Downloading works! ● Crate configurations, Trigger scripts ● Expert & Non-expert monitoring GUIs in control room ● L2 Experts on call Level 2: Status

D0 Collaboration Meeting July 2002 Level 2: Status L2Cal ● Em and Jet algorithms, Admin run simultaneously on 1 alpha ●  coverage to  0.8 (commissioning  2.4 in progress) ● Adjusting L1Cal inputs with L2 Software (performance hit) – pedestals, gains, seed mask, L2 headers/trailers – Recently updated pedestals, gains to reflect L1Cal mods ● Jet parameters  Et cut ● Em parameters  Et cut, EM fraction, (isolation) L2Muon: Rejecting on Muons ● Loose, Medium, Tight ● Pt cut ● No track matching yet

D0 Collaboration Meeting July 2002 L2 Future: Short Term ● Continue improving smoothness of operation ● Online bit-by-bit verification ● Improvements to online monitoring and debugging – Start running L2 trigger examines – Error logging ● Add New Triggers: dimuon trigger soon; others to follow. ● Work to Increase Rate Capabilities – Speed up code – Overlap data input & processing ● Currently in lockstep mode – Want to overlap DMA, L3 readout, and processing ● Multiple alphas per crate

D0 Collaboration Meeting July 2002 Rate Limits: It takes a task force ● Trigger framework and muon FE crate bug fixes recently – Big decrease de-synch problems – Now would like automatic response to SCL_INIT requests (now hangs!) ● L2 code fixes – properly responds to SCL_INIT, config_crate w/o reboot! – Was causing wakeup call to L2 expert every 2 hours—not acceptable ● Limit at 200 Hz: de-synch problems in muon pixels FE – Muon group working on this —many players, not all on site ● Some indications of Cal problems around 400 Hz ● L2 probably can run >500 Hz—no known problems ● Currently: running w/o L2Mu in trigger – Crate in run; diagnose + fix problems, but not in global – Hope ready to turn back on this week – Special running conditions in this configuration ALSO needed debugging ● FE crates couldn’t raise busy if L2Global not making decisions

D0 Collaboration Meeting July 2002 What makes progress possible ● Improving error messages – Details from SLIC DSP’s of which channels de-synched – Errlogger messages tagged with tick and turn by framework – DIAGNOSIS—you can’t fix it by guessing whose fault it is ● L2 expects same inputs every event, and all fragments from same event ● Central L2 error logs (eventually, to SES) ● TCC (oversubscribed); working on phase 1: – collect messages from all crates’ dualport memories – Time stamp them (5 sec blocks) – Merge into single file ● Better timing information now available (offline, online) – Speedup in I/O, cleanup of many layers of jet code (Roger) – Consulting from Computing Division (Fischler) – L2 headers and candidate bit masks avail from L1Cal would help

D0 Collaboration Meeting July with all 10 inputs? maxopt ~3 x faster Slower than prototype code tests—are you surprised? Add < 50usec for DMA, L3, monitoring, etc

D0 Collaboration Meeting July 2002 Level 2: Extended Plans ● Full  coverage for L2Cal (as inputs become available) ● Commission L2CTT, L2PS, L2STT – Mid-August: L1CTT added to L1Muons ● L2muon rejection drop? – L2CTT: Inputs may be available September – L2PS inputs ● CPS stereo may be available in August ● CPS axial, FPS available this Fall (Oct 1 earliest) – L2STT begins commissioning this Fall; ● 30 degree sector by Oct 1; all by Nov? – Expect 6-8 weeks to commission a given processor once inputs are available—we will need ID group help – Will expand capabilities of L2Global (code works on MC) ● m, e - triggers from track matching ● g, e - triggers from PS

D0 Collaboration Meeting July 2002 Level 2: Extended Plans II Alpha PIO rewrite (August??) DMA overlap with processing Multiple alphas per crate (L2Cal, e.g.) Betas (850 MHz PIII cpu; Replacements for Alphas) ● 2 nd prototype expected July 20 th – Test at D0 during August ● Upgrade to 1 GHz PIII  3x faster than Alphas ● Assuming success with pre-production tests: – Start production in September – Hardware checks in November – Begin installation in December

D0 Collaboration Meeting July 2002 Assessment ● Good: – Rejecting where we have inputs (since May) – All code running from releases – Trigsim code matched with online code ● Ntuples can be used for studies ● Bad – Late Start (we had our own problems, but we also had “help”) ● alpha hardware, and arrival of inputs ● 7EE’s left the project before full commissioning of their subsystem ● Commissioning took longer than we’d hoped (remember this phrase) – L2 Monitoring behind commissioning needs – Can’t run at spec of 5kHz – Can’t look at SLIC inputs yet – Trigsim needs an expert to run it (even to make ntuples) – No thumbnail

D0 Collaboration Meeting July 2002 Assessment, II: The Ugly ( Frustrations, mostly) No tracking inputs Helping debug missing or de-synched inputs “we can’t run because of Level 2” (!) Operating without designed-for environment reasonable priority decisions, but costs time, enthusiasm SCL_INIT request connected to aether Code not designed to chat with outside world while for SCL_INIT Some functionality missing from framework/COOR Various flavor of Mark&Pass support Monitoring collection No common object interface across trigger levels to Reco, MC Makes certification harder than necessary Run summary, including pass fractions for L2 (not yet…)

D0 Collaboration Meeting July 2002 L2 Manpower ● 1 postdoc leaving soon, 1 joining ● Loss of experience ● Muon group very thin ● 2 other postdocs ramping down ● I’m on sabbatical starting January – There is a certain level of fatigue ● This is the cost of slow turn-on – Alpha hardware; no tracking inputs ● People who have been doing near 100% service for several years need to be turning more of their attention towards physics – We can build hardware and code – We can’t do all the studies

D0 Collaboration Meeting July 2002 Areas where help needed ● Thumbnail for L2 in particular, Trigger in general ● Release Regression Testing (base on examine?) – Share technology with L3, RECO? – Need data samples from physics groups ● Common Object Interface for certification ● Certification for CPS, FPS, CTT, STT! ● Code maintenance for CPS, FPS? ● IOGen: multiple version reading; long-term support ● Hardware maintenance—we can provide training – FIC (and Alpha to lesser extent) ● L2 expert shift—we can provide training

D0 Collaboration Meeting July 2002 Trigger/DAQ Priorities (Lipton) 1. Keep experiment running (but take necessary time for development) 1. come out of the October shutdown with the D0 Trigger/DAQ operating as designed 2. Improvement of DAQ rates 1. Find problems in multibuffer mode 2. Implement automatic SCL inits, error logging 3. CTT – L1muon, L1, L2 track trigger terms 4. L1 Cal extensions and debugging 5. L2 Improvements 1. Tune filters, improve timing 2. L1 seeds (and headers) to L2 (speedup for L2) 3. Multiple alphas/crate (issue at ~2 kHz) 4. L2  6. Improve monitoring/examines/error logging …

D0 Collaboration Meeting July 2002 Some L2 Priority Choices ● Multiple CPU/crate, DMA overlap with Alpha or Beta? – Rate pressure, and who does which work – Could try maxopt and wait for betas, hoping we don’t: ● Find optimizer bugs ● Have to interpret debug output (maybe use test stand) ● How best to get started with tracks in L2? – L2CTT, then switch to L2STT, or skip L2CTT? – Tracks vs. displaced vertices within L2STT ● We may be short of manpower for L2PS – L2axial on L2CTT tracks for electrons – L2CPS stereo arrives before independent L2PS axial (for photons) – L2FPS arrives last – Anyone interested enough in photons to help? – Defer in favor of tracks? ● New functionality vs. Smoother Running, Verification

D0 Collaboration Meeting July 2002 Readout

D0 Collaboration Meeting July 2002 CTT ● L1  CTT used as basis for L1 tracks – ~6 week schedule delay (~mid August) ● Rough schedule for other components: – L1, L2 Tracks: No sooner than September 1 – CPS stereo clusters: No sooner than August 1 – FPS clusters: No sooner than October 1.

D0 Collaboration Meeting July 2002 Level 2 ● L2 requires inputs for every crate for which L2 is configured in any run – Sensitive to problems in other systems – Needed substantial work for trigger database and COOR to provide proper support Adding better error logging to diagnose FE problems that lead to L2 hangs ● L2 Cal – Em and Jet algorithms, Admin run simultaneously on 1 alpha –  coverage to  0.8 (commissioning  2.4 in progress) ● L2Muon – Rejecting on Loose, Medium, Tight muons wit Pt cut ● Future ● Full  coverage for L2Cal – Commission L2CTT, L2PS, L2STT – Automate SCL init requests – L2 beta – increase L2 input rate to >2.5 Khz

D0 Collaboration Meeting July 2002