1. Start-up & synchronization sequence for Front-End LHCb Electronics Upgrade Meeting 13 February 2014 F. Alessio, CERN

2. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 2 Scope & Outline Aim at describing the way in which the system would globally: -Synchronize the readout of events at the beginning of a run («Start-of-run fast sequence») o to ensure TELL40 code is able to align to incoming data stream o to ensure FE is aligned to correct BXID -Resynchronization mechanisms After a desynchronization Preventive resynch to avoid loss of data Usage of FE Reset and Synch command

3. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 3 Synch and alignment of TFC links 1. Synch and align the TFC links: send Synch pattern in 8b/10b Firmware will check synchronization of buffer and alignment of content of frame  When done, set synch bit via ECS.

4. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 4 2. Synch and align TFC-TELL40 links: send Synch pattern in 8b/10b Synch and alignment of TFC links Firmware will check synchronization of buffer and alignment of content of frame  When done, set synch bit via ECS.

5. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 5 3. Synch and align TFC-FE links: send TFC commands in GBT according to each FE’s mapping  SOL40 must be configured before FE and clock transmission must be up (minimal SODIN configuration) Synch and alignment of TFC links GBT does the job.  When synch’d, can start configuring FE.

6. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 6 3bis. Special mode : relay TFC commands back to SOL40  measure latency Synch and alignment of TFC links SOL40 will measure transmission delay by comparing BXID  Interested only in BXID latency, not fine phase  However if at the limit might see some desynch and adjust...

7. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 7 4. Synch and align FE-TELL40 links: send Synch Pattern in GBT Synch and alignment of TFC links TELL40 LLI (GBT decoding+ GX buffer) will synchronize the links  Special Synch Pattern is used to align the data stream to TELL40 processing logic

8. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 8 4bis. Special mode : relay TFC commands back to TELL40  measure latency Synch and alignment of TFC links TELL40 will measure transmission delay by comparing BXID received by SOL40 and FE  Interested only in BXID latency, not fine phase  However if at the limit might see some desynch and adjust...

9. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 9 ECS start-of-run sequence Just like now, current LHCb experiment CONFIGURE SOL40s get configured first, then FE gets configured, via SOL40. Everything goes “READY” (if ok…) (very quickly…) Links are synch’d across the whole system READY FE is free running in this state (idle state at FE), but ready to react on TFC commands. On output link send random frames to keep GBT link up ECS issues “RUN”. Some configuration might go via SOL40 to FE (if) RUNNING Everything is ready to take data, SODIN is still in ACTIVE. ECS issues “GO” Fast TFC start-of-run sequence starts. Then, data taking!

10. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 10 TFC start-of-run fast sequence

11. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 11 TFC start-of-run sequence FE Reset is the first thing issued via control system: asynchronous!  Issued when TFC receives command «GO»  Ensure that FE got the right BXID value before synch-ing to TELL40 This sequence is the same every time a FE Reset is issued. If FE Reset is ussed, change Run #. For «in-Run» resynch, use Synch command.

12. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 12 What to do on SYNCH? Synch command is meant to be sure that (whole) system is synchronized… in a synchronous way!  FEs send Synch Patter for the same BXIDs everywhere FE frees its memory : delete its content, read and write pointers back to empty FE sends Synch Pattern for as many as Synch commands are set by TFC Next event, starts at bit 0 (LSB) – might be header only TELL40 will align to corresponding frame and BXID  TELL40 closes all events before and sends them out truncated  TELL40 does not know what BXID to expect. Synch pattern says it. Note: Synch command can be used on lab tests without TFC. For few links, system can align independently of TFC (just program your FE to send it and TELL40 to receive it). TELL40 would be in pass-through (i.e. accept all). As soon as you need synchronicity across all links, then you need TFC. TFC will ensure that synch command is sent out for same BXID everywhere automatically aligning TELL40 readout. Send something like this!

13. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 13 What to do on SYNCH? Double usage (in AND or in OR): 1.Periodically: i.e., SYNCH command sent every n Hz  a la FE Reset, but without resetting  a bit inefficient to clear the FE buffer (could be like this only at the beginning and when very few bunches – lots of empty-empty crossing) 2.Asynchronously: i.e. when a desynch is detected, like TELL40 detects wrong frames, wrong packing  needs fast diagnostics in TELL40 codes  makes sense to clear the FE buffer in this case  could be sent only for a local sub-detector from SOL40 (slow) triggered by ECS (TELL40 or FE set a desynch bit, ECS sends command to SOL40) or by TELL40 via SOL40 transmitting an info field regarding this w/ throttle protocol (we have plenty of BW)  Resynchronization sequence Send something like this!

14. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 14 Resynchronzation sequence Synch command was requested  From ECS or fast via TELL40 Note: Synch command is entirely programmable in frequency, length and location around the orbit. Header Only is entirely programmable in length.

15. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 15 Conclusion TFC documents updated, please review and comment. LHCb-PUB-2012-001LHCb-PUB-2012-001 (TFC specs) LHCb-PUB-2012-017LHCb-PUB-2012-017 (FE and BE specs)

16. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 16 Backup

17. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 17 System and functional requirements 1.Bidirectional communication network 2.Clock jitter, and phase and latency control At the FE, but also at TELL40 and between S-TFC boards 3.Partitioning to allow running with any ensemble and parallel partitions 4.LHC interfaces 5.Events rate control 6.Low-Level-Trigger input 7.Support for old TTC-based distribution system 8.Destination control for the event packets 9.Sub-detectors calibration triggers 10.S-ODIN data bank Infomation about transmitted events 11.Test-bench support

18. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 18 The S-TFC system at a glance DATA S-ODIN responsible for controlling upgraded readout system Distributing timing and synchronous commands Manages the dispatching of events to the EFF Rate regulates the system Support old TTC system: hybrid system! SOL40 responsible for interfacing FE+TELL40 slice to S-ODIN Fan-out TFC information to TELL40 Fan-in THROTTLE information from TELL40 Distributes TFC information to FE Distributes ECS configuration data to FE Receives ECS monitoring data from FE STORAGE

19. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN S-TFC concept reminder 19

20. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN The upgraded physical readout slice - ATCA Common electronics board for upgraded readout system: AMC40 cards fitted in an ATCA motherboard S-ODIN & SOL40  AMC cards LLT & TRIG40  AMC cards TELL40s  AMC card LHC Interfaces  specific AMC card 20

21. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN The upgraded physical readout slice – PCIe Common electronics board for upgraded readout system: PCIe-Gen3 card fitted on a host PC SODIN & SOL40  PCIe card LLT  PCIe ard TELL40  PCIe card LHC Interfaces  specific PCIe card 21

22. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN TFC protocol to TELL40 22 «Extended» TFC word to TELL40 via SOL40:  64 bits sent every 40 MHz = 2.56 Gb/s  packed with 8b/10b protocol (i.e. total of 80 bits)  no dedicated GBT buffer, use ALTERA GX simple 8b/10b encoder/decoder THROTTLE information from each TELL40 to SOL40: 1 bit for each AMC board + BXID for which the throttle was set  merged and aligned in SOL40  same GX buffer as before (same bidirectional transceiver) Constant latency after BXID MEP accept command when MEP ready:  Take MEP address and pack to FARM  No need for special address, dynamic

23. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN TFC protocol to FE 23 TFC word on downlink to FE via SOL40 embedded in GBT word:  24 bits in each GBT frame every 40 MHz = 0.98 Gb/s  all commands associated to BXID in TFC word Put local configurable delays for each TFC command GBT does not support individual delays for each line Need for «local» pipelining: detector delays+cables+operational logic (i.e. laser pulse?)  DATA SHOULD BE TAGGED WITH THE CROSSING TO WHICH IT BELONGS! TFC word will arrive before the actual event takes place To allow use of commands/resets for particular BXID Accounting of delays in SODIN: for now, 16 clock cycles earlier + time to receive Aligned to the furthest FE (simulation, then in situ calibration!) TFC protocol to FE has implications on GBT configuration and ECS to/from FE see specs document!

24. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 24 SODIN firmware v1r0 – block diagram

25. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN Timing distribution 25 From TFC point of view, ensured constant: LATENCY: Alignment with BXID FINE PHASE: Alignment with best sampling point Some resynchronization mechanisms envisaged:  Within TFC boards  With GBT No impact on FE itself Loopback mechanism:  re-transmit TFC word back  allows for latency measurement + monitoring of TFC commands and synchronization

26. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 26 How to decode TFC in FE chips? Usage of TFC+ECS GBTs in FE is 100% common to everybody!!  dashed lines indicate the detector specific interface parts  please pay particular care in the clock transmission: the TFC clock must be used by FE to transmit data, i.e. low jitter! Kapton cable, crate, copper between FE ASICs and GBTX FE electronics block

27. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 27 The TFC+ECS Master GBT These clocks should be the main clocks for the FE 8 programmable phases 4 programmable frequencies (40,80,160,320 MHz) Used to: sample TFC bits drive Data GBTs drive FE processes

28. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 28 The TFC+ECS GBT protocol to FE  TFC protocol has direct implications in the way in which GBT should be used everywhere 24 e-links @ 80 Mb/s dedicated to TFC word as a baseline (effectively 48 bits) use 80 MHz phase shifter clock to sample TFC parallel word TFC bits are packed in GBT frame so that they all come out on the same clock edge Modification are possible in order to satisfy sub-detector’s FE requirements  Leftover e-links dedicated to GBT-SCAs for ECS configuring and monitoring (see later)

29. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 29 Words come out from GBT at 80 Mb/s In simple words: Odd bits of GBT protocol on rising edge of 40 MHz clock (first, msb), Even bits of GBT protocol on falling edge of 40 MHz clock (second, lsb)

30. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 30 TFC decoding at FE after GBT This is crucial!!  we can already specify where each TFC bit will come out on the GBT chip  this is the only way in which FE designers still have minimal freedom with GBT chip if TFC info was packed to come out on only 12 e-links (first odd then even), then decoding in FE ASIC would be mandatory! which would mean that the GBT bus would have to go to each FE ASIC for decoding of TFC command  there is also the idea to repeat the TFC bits on even and odd bits in TFC protocol would that help? FE could tie logical blocks directly on GBT pins… Or to select a minimal set of TFC commands and repeat them to profit from fan-out possibilities

31. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 31 Now, what about the ECS part? Each pair of bit from ECS field inside GBT can go to a GBT-SCA One GBT-SCA is needed to configure the Data GBTs (EC one for example?) The rest can go to either FE ASICs or DCS objects (temperature, pressure) via other GBT-SCAs GBT-SCA chip has already everything for us: interfaces, e-links ports..  No reason to go for something different! However, «silicon for SCA will come later than silicon for GBTX»…  We need something while we wait for it! FPGA emulator (working on it)

32. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 32 Protocol drivers build GBT-SCA packets with addressing scheme and bus type for associated GBT-SCA user busses to selected FE chip  Basically each block will build one of the GBT-SCA supported protocols Memory Map with internal addressing scheme for GBT-SCA chips + FE chips addressing, e-link addressing and bus type: content of memory loaded from ECS SOL40 firmware

33. LHCb Electronics Upgrade Meeting, 26/07/12 F. Alessio, R. Jacobsson Usual considerations … 33 TFC+ECSInterface has the ECS load of an entire FE cluster for configurating and monitoring  34bits @ 40 MHz = 1.36Gb/s on single GBT link ~180 Gb/s for full SOL40 (132 links) Single CCPC might become bottleneck… Clara & us, December 2011  How long to configure FE cluster? how many bits / FE? how many FEs/ GBT link? how many FEs / TFC+ECSInterface?  Numbers to be pinned down soon + GBT-SCA interfaces and protocols.

34. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN Old TTC system support and running two systems in parallel 34 We already suggested the idea of a hybrid system: reminder: L0 electronics relying on TTC protocol  part of the system runs with old TTC system  part of the system runs with the new architecture How? 1.Need connection between S-ODIN and ODIN (bidirectional)  use dedicated RTM board on S-ODIN ATCA card 2.In an early commissioning phase ODIN is the master, S-ODIN is the slave  S-ODIN task would be to distribute new commands to new FE, to new TELL40s, and run processes in parallel to ODIN  ODIN tasks are the ones today + S-ODIN controls the upgraded part In this configuration, upgraded slice will run at 40 MHz, but positive triggers will come only at maximum 1.1MHz… Great testbench for development + tests + apprenticeship… Bi-product: improve LHCb physics programme in 2015-2018… 3. In the final system, S-ODIN is the master, ODIN is the slave  ODIN task is only to interface the L0 electronics path to S-ODIN and to provide clock resets on old TTC protocol

35. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN SODIN on Marseille’s ATCA board 35

36. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN TFC+ECSInterface on Marseille’s ATCA board 36

37. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 37 Reminder: your (generic) FE For details, see LHCb-INT-2011-011 Compress (zero-suppress) data already at the FE reduce # of links from ~80000 to ~12500 (~20 MCHF to ~3.1 MCHF) data driven readout (asynchronous) + variable latencies! Efficiently use data link bandwidth pack data on data link continuously with elastic buffer extensive use of GBT (robust FEC vs WideBus mode) evaluate choices based on complexity vs robustness NO TRIGGER to FE!  Only commands, clock and slow control

38. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 38 Fast & Slow Control to FE Separate links between controls and data A lot of data to collect Controls can be fanned-out (especially fast control) Compact links merging Timing, Fast and Clock (TFC) and Slow Control (ECS). Extensive use of GBT as Master GBT to drive Data GBT (especially for clock) Extensive use of GBT-SCA for FE configuration and monitoring On detector Off detector 4.8 Gb/s TFC ECS Data TFC ECS Data 4.8 Gb/s Off detector

39. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 39 Reminder: generic FE data flow scheme Compression/suppression logic can have dynamic or static latency Applies changes to data FE buffer for data Tag data with TFC commands and pipe them across compresson/suppression logic block Modify data according to TFC commands + BufferFull then pack continuously onto GBT Data available needed only if compression / suppression is dynamic

40. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 40 The code: FE data generator

41. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 41 The code: FE buffer manager

42. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 42 The code: GBT dynamic packing Very important to analyze simulation output bit-by-bit and clock-by-clock!

43. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 43 Reminder: dynamic packing algorithm 0123401234 Average event size = link bandwidth Buffer depth Average event size 0123401234 Link bandwidth 0123401234 BX0 BX1 BX2 BX3 BX4 BX0 BX1 BX2BX 3BX4 Header is the unique identifier for each event in frame Compulsory (tag for each crossing ), partly programmable (must contain length of frame+BXID) Difficult buffer management, but almost no truncation. Flexible against occupancy problem (what if your estimate is wrong?). Maximum exploitation of bandwidth. Readout Board uses Header to decode and separate frames  lots of resources. + = 

44. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 44 Reminder: fixed packing algorithm 0123401234 Average event size /= link bandwidth  Truncation! Buffer depth Average event size 0123401234 Link bandwidth 0123401234 BX0 BX1 BX2 BX3 BX4 BX0 BX1 BX2 BX3 BX4 This is different: one clock cycle  one event  one GBT frame Header more flexible: you can add addresses, hitmaps… Very simple buffer management, but truncation has to happen eventually. Not flexible against occupancy problem (again, what if your estimate is wrong?). Loses a bit of bandwidth as empty spaces must be padded to be sent out. Readout Board uses Header to decode and separate frames  much fewer resources + = 

45. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 45 Reminder: fixed vs variable length header in dynamic packing Dynamic packing with fixed length header. Dynamic packing with dynamic length header (fully flexible!)

46. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 46 Few comments to start with: -BX Veto and Header Only commands are identical from FE point of view  ORed -TFC commands are synchronous wrt to BXID Reset  once we align BXID Reset with beam, TFC commands come ALWAYS at the same latency (wrt to BXID Reset, hence BXID)!  Compression/suppression logic should act accordingly to TFC command (why would you want to compress/suppress if that crossing is rejected a priori? Especially if your pre-processing is dynamic…) -Data is filtered according to TFC commands and the FE buffer status -Data is packed onto the GBT link in a continuous fashion FE flow control scheme

47. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 47 Data Valid GBT can accept DATA or IDLE frame:  Send IDLE frame whenever a GBT frame is not ready to be sent!  IDLE frame can contain whatever your sub- detector wants to send. See TELL40 fw specs, coming soon… Data Valid signal to distinguish between DATA and IDLE frame:

48. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 48 Data Valid Be careful to rise synchronize the Data Valid signal to the right rising edge when using the 80 MHz clock (or 160 or 320…) GBTX would split the frame in this case!! Synchronize your DV signal to the beginning of the GBT frame!

49. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 49 Data Valid Some sub-detectors will connect more FEs to the same GBT transmitter: -Each FE with its own memory -Can happen that one can send DATA, the other cannot! (IDLE vs DATA in the same packet) Keep DV always high! You HAVE to indicate whether the packet was DATA or IDLE, by sacrificing one bit your DATA/IDLE frame

50. LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 50 Data Valid Some sub-detectors will connect more FEs to the same GBT transmitter: -Each FE with its own memory -Can happen that one can send DATA, the other cannot! (IDLE vs DATA in the same packet) Keep DV always high! You HAVE to indicate whether the packet was DATA or IDLE, by sacrificing one bit your DATA/IDLE frame