L2 Status and Plan Matt Worcester, Heather Ray, Monica Tecchio, Myron Campbell, Jane Nachtman, David Saltzberg, Tom Wright, Steve Kuhlmann, Karen Byrum, Masa Tanaka, Fred Neill, Peter Wittich, Nathan Goldschmidt, Ted Liu, Alexei Varganov, John Dawson, Bill Haberichter, Greg Feild, Stephen Miller December 7, 2001 L2 Review

12/7/012 L2 Status L2 WORKS!

12/7/013 L2 Status Current L2 Crate Configuration –1 Alpha –L1 –Clist –Isolist –Tracklist –Svtlist –Reces(x4)

12/7/014 L2 Status Current L2 Operations –1 Alpha, L1, Clist, SVTlist, Tracklist, Isolist, Reces(w/o readout) –Currently used for data taking with L2 cutting on jets –Tested at 20KHz with cosmics test –All the interface boards work without sync and data errors.

L2 Goals and Plans Stephen Miller September 7, 2001 L2 Workshop Slide From L2 Workshop 9/7/01

12/7/016 Goals Before Shutdown L2 Testing Milestones –Run cutting on Jets at L2 Using Clist and L1 board run at 2-20Khz for 10M L1A Problems to fix: Clist sync error. L1 bits error –Run cutting on SVT track information Using SVTlist and L1 board run at 2-20Khz for 10M L1A Problems to fix: L1 bits error. SVTlist dstrobe error Other work: Correct d0. Write code for simple trigger. –Run cutting on Electrons Using Clist, Tracklist, L1 Problems to fix: L1 bits error. Clist sync error. Clist-Tracklist collisions. Other work: Finish electron code. Slide From L2 Workshop 9/7/01 DONE ALMOST DONE

12/7/017 Goals Before Shutdown L2 Testing Milestones cont.. –Run with L1, Clist, SVTlist, Tracklist with high rate L1A Main goal is to check data integrity. –Test of Reces data Low rate test with beam Useful for understanding SMXR trigger data Ideally: Would have Alpha-VME errors fixed for high rate test –Test with Clist and Isolist Need test with beam to compare Isolist and Clist data –Test with all interface boards Only consider this if all other problems have been fixed Would first do the test with Cosmics. Slide From L2 Workshop 9/7/01 DONE DONE (after shutdown)

12/7/018 Goals Before Shutdown L2 Trigger/Physics Milestones –Cutting on Jets Just Clist and L1 boards in system Point is to improve data quality at higher Luminosity –Cutting on SVT track information Without Clist? And no cutting on Jets? Is there a target sample size useful for Physics groups? –Cutting on Jets, Electrons and SVT tracks Ideal goal before shutdown –Work to do Establish stable L2 operations Have error handling code in L2 to quickly find errors Slide From L2 Workshop 9/7/01 DONE SVX errors Required tagging mode DONE

12/7/019 Plans during shutdown Magic Bus –Install new Magic Bus –Change termination –Study Magic Bus collisions with cosmics Need to remove firmware workarounds to create collisions Alpha Tests –Add additional processors – First test with subset of interface boards –Move to full crate test using cosmics Other –Finish debugging known problems Slide From L2 Workshop 9/7/01 DONE Work in Progress DONE Major Improvements in Clustering Crates

12/7/0110 L2 Status Fixed Problems –Enormous improvement for all boards since the L2 Review in Sept. –List of solved problems too long to cover –Only known problems are listed below Unresolved issues –Errors from Clustering Crate Appearance of single bit energy errors – fixed with jumpers on backplane Isolation boards have 1% eta errors from Clustering Crates –Error reproduced in teststand – caused by Clique board –(Worst Case) Can find error in Alpha code by comparing to Clist »Gives 1% over efficiency for isolation triggers –Major crate issue is Magic Bus arbitration (work in progress) Important for achieving design rate, not for operation at 20KHz –Need to get reliable Reces readout Really a problem with Alpha fpga firmware –Muon board is only missing interface board

12/7/0111 Software Status Current Software Status –Alpha executable used for making L2 decision Has been running and sending L2A since beam started Jet cutting algorithm is used for data taking. L2 autoaccepts used for other triggers SVT based trigger tested. Electron and SVT triggers are written, but need testing Monitoring software will create error condition if interface boards fail to send data or get out of sync –Alpha code organization Alpha software stored in online CVS repository Executable is generated automatically using cuts from the TriggerDB

12/7/0112 Software Status cont.. Current Software Status –People working on Alpha code Peter Wittich, Masa Tanaka, Heather Ray, Stephen Miller, Tom Wright Starting a L2 software meeting to focus on L2 code writing. –Monitoring software TrigMon software used online by shift crew. Offline for analyzing data Has code for all interface boards L3 filter to check for data corruption in TL2D Uses simulation of trigger for exact comparison of TL2D contents with expected (Code will become basis for offline L2 trigger simulation)

12/7/0113 Short Term L2 Plan Maintain stable L2 operation with current configuration Proposed L2 Crate Configuration for mid-January –3 or 4 Alphas, L1, Clist, SVTlist, Tracklist, Isolist, Reces(with readout) running at 20KHz L1A rate Rate well matched to expected luminosity 3 Alphas have been tested in crate in cosmic test at 20 KHz –Need software work to ensure robust running with multiple alphas –Need to process triggers in multiple alphas Reces readout problem can likely be fixed with Alpha firmware change –Will be able to work on problem in teststand at Michigan

12/7/0114 Short Term L2 Plan cont.. Software expectations for mid-January –Jet, SVT and Electron (with Isolation) triggers written and tested with beam –Reces based triggers ready by end of January –Improved tools for code development and testing Masa wrote tool for testing triggers with TL2D data –Most of CDF4718 will be in place by end of January –Will start optimization of triggers with high L1A rates

12/7/0115 L2 Software and CDF4718 L2 options written are highlighted in Blue Many still need to be tested

12/7/0116 L2 Software and CDF4718 cont..

12/7/0117 L2 Tests until mid-January –Multiple Alpha tests –Reces Readout tests –Work on Low level errors in clustering system –Tests of new trigger algorithms –Test of new alpha firmware –Tests of spare boards –Tests with SVT-SVX (not L2 specific) –High Rate tests with beam (change L1 prescales) – CDF test Tests initially done without beam, but may request beam time for final verification

12/7/0118 Path to Higher Rate Defining the Problem: –Current L2 crate runs at only 20KHz L1A rate This rate should be sufficient for Luminosities < 5x10^31 Rate limit due to delays for interface boards to send data on the backplane –SVT board has 34us delay. (Isolation has 50us delay but doesn’t always send data) –Some delay due to overhead in code for controlling crate, interfacing with TS –For higher luminosities, expectation is 50KHz L1A rate Ideal is 10us for loading data, 10us for processing event with both happening at same time. –Solution: Fix Magic Bus Arbitration, Improve Alpha firmware, Optimize Trigger Code, Run with 4 Alphas Magic Bus arbitration –Noise on arbitration lines causes interface boards to take the bus at same time Currently use time delays for each board to send data. Gives limit of 20KHz Delays for boards chosen arbitrarily. Possibility of reducing delays. –New MagicBus is being tested – at Michigan and lower crate Will explore other arbitration options if needed

12/7/0119 Path to Higher Rate Improve Alpha firmware –All control currently done in software- gives 5-10us overhead per event –Some improvements possible with optimizing some code –New firmware written to handle control signals. Will give 1-2 us of overhead Partially tested at Fermilab. Need to verify TS handshake test. New code to match firmware has been written. Needs debugging. –Additional firmware changes could reduce overhead to < 1 us Software would just check for data having been loaded and sending decision to TS. Trigger code optimization –Plan is to first write reliable code and then improve performance – Timing studies can be done in lower crate Assume all improvements will be studied/solved independent of beam –Tests done in teststands, lower crate, or upper crate with cosmics –Ideally test new code/configuration in last hour of a store. Only after extensive testing with cosmics. Would use < 1% of luminosity for tests

12/7/0120 Plan for Muon Board Muon board being designed by Fred Neill at Michigan –Muon data fpga design done and mezzanine board built –9U board layout done –MagicBus interface fpga under design (Tom Wright) Plans for testing –Firsts tests at Michigan with testclock, Muon trigger test card, Alpha for testing board –Cosmic ray tests at B0 Test at high rate standalone Test with full L2 crate –No beam time will be requested for debugging problems –Testing to be done by Fred, Eric, Tom, Stephen

12/7/0121 Plan for Operations Requirements for Reliable L2 operations –L2 monitoring – Check of TrigMon summary. Running validation code on all events from several runs a week (or validation as part of production). Keeping up “ L2 with Beam” web page – describes crate performance for each run. –L2 pager coverage – Response to errors found by shift crew. Instructing aces on any special L2 requirements. Reporting status at 8am meeting. Currently handled full time by Heather Ray. Assembling list of names for additional personnel. Other people will be in place for start of physics data taking. –Spares At least 1 spare exists for each board. Require at least 2 spares per board. Need to finish assembly and testing of some spares. Each institution must have an expert for repair of its boards

12/7/0122 Plan for Operations cont. Requirements for Reliable L2 operations cont… –Documentation Documentation being collected and organized on web pages Schematics and some written documentation exists for all boards General non-expert document has been written Expert documentation/web page under development Proposal –Monitoring and pager coverage can be combined for L1 and L2 trigger Much of monitoring and validation code is already combined Helps in distinguishing L1 and L2 problems Adds additional manpower to both groups and extends expertise across systems

12/7/0123 Summary L2 works! –Able to run at 20KHz with a full crate –Used for data taking with cutting on jets –Major improvement since September workshop Many errors have been fixed. (Most not mentioned in this talk) Short Term Plan –Maintain reliable L2 operation –Work on remaining issues needed to improve performance Multiple Alphas Reading Reces Higher Rates – Magic Bus Arbitration –Writing and testing trigger algorithms for Physics Team of people and tools are in place and being used. Longer Term Plan –L2 debugging and improvements will continue without using beam –Assemble group of people and knowledge base for maintaining L2 trigger