1. Peter JansweijerMROD Design Review: November 12, 2003Slide 1 MROD-1 Hardware Overview MRODin MRODout MROD-1 = 3 x MRODin + 1 x MRODout

2. Peter JansweijerMROD Design Review: November 12, 2003Slide 2 Memory SHARC Memory FPGA VME64x TTC interface MRODout MRODin FPGA Memory SHARC Memory FPGA Memory SHARC Memory FPGA 1 2 4 3 1 2 4 3 1 2 4 3 SHARC A 1 2 43 0 SHARC B 1 2 4 30 5 5 LVDS In LVDS Out SHARC Links C D E MRODin 6x S-Link or GOLA MROD-1 Hardware Overview

3. Peter JansweijerMROD Design Review: November 12, 2003Slide 3 MRODin Buffer memory, Separate partition per TDC, Dataflow into the buffer memory Tetris register, I2O-FIFO, Fault tolerance, Error signaling and handling, Resynchronization Dataflow out of buffer memory, DMA, Zero suppression, Length FIFO Data format, TDC-ID, Parity errors Interrupt sources FPGA registers

4. Peter JansweijerMROD Design Review: November 12, 2003Slide 4 1 MB ZBT Buffer Memory SHARC Data FIFO Tetris Register Input Output I2O FIFO Control Control/Status Error signaling 4 Sharc links 40 (80) MB/S Each FIFO Length FIFO FPGA Channel A FPGA Channel B MRODin Overview S-Link or GOLA S-Link or GOLA Channel B Memory

5. Peter JansweijerMROD Design Review: November 12, 2003Slide 5 MDT Readout chain wooden model

6. Peter JansweijerMROD Design Review: November 12, 2003Slide 6 Buffer Memory Partitions Dataflow into buffer memory STNTTSTTTT Time Division Multiplexed STTTTNSTN T TT T T T TT T T T T 120345120345120345120345 TDC 0TDC 1TDC 2TDC 3TDC 4TDC 5 S N Separator= Start new TDM Cycle No Data= Time slot filler In real life: 18 Buffer Memory Partitions (8 K words each) T TDC data wordTDC trailer Note: TDC headers are omitted for simplicity!

7. Peter JansweijerMROD Design Review: November 12, 2003Slide 7 Tetris Register Normal operation TDC 0TDC 1TDC 2TDC 3TDC 4TDC 5 Example Tetris Register= 6 x 4 In real life, Tetris Register= 18 x 16 Expected Event-ID n n+3 n+1 n+4 I2O-FIFO All trailers of Event-ID n arrived Event-ID n

8. Peter JansweijerMROD Design Review: November 12, 2003Slide 8 Tetris Register TDC trailer missed TDC 0TDC 1TDC 2TDC 3TDC 4TDC 5 Expected Event-ID n n+3 n+2 n+5 I2O-FIFO All trailers of Event-ID n+1 arrived Event-ID n Event-ID n+1

9. Peter JansweijerMROD Design Review: November 12, 2003Slide 9 Tetris Register Notorious absence of TDC (panic mode) TDC 0TDC 1TDC 2TDC 3TDC 4TDC 5 Expected Event-ID n n+3 n+1 n+4 I2O-FIFO Arrival of a trailer for Event-ID n+3 Event-ID n

10. Peter JansweijerMROD Design Review: November 12, 2003Slide 10 Tetris Register Early / Late TDC 0TDC 1TDC 2TDC 3TDC 4TDC 5 Expected Event-ID n+1 n+4 n+1 n+4 Early TDC1 trailer Late TDC 1 trailer Notes: Expected Event-ID can be set by the SHARC during MRODin initialization Expected Event-ID is hardware updated Resynchronization for an MRODin channel should be possible.

11. Peter JansweijerMROD Design Review: November 12, 2003Slide 11 Buffer Memory Partitions Dataflow out of buffer memory T TT T T T TT T T T T TDC 0TDC 1TDC 2TDC 3TDC 4TDC 5 Read from I2O-FIFO: TDC Link Present (TLP) word 10234 TTTTT 5 T Trailer and Word Count (TWC) Word Count and Event-ID To Output Data FIFO (128) Event-ID n To Length FIFO (128) When a TDC is not present then don’t read the partition! You’ll never encounter a Trailer! This information is signaled in the data stream (TLP word)

12. Peter JansweijerMROD Design Review: November 12, 2003Slide 12 Output Data FIFO is read with continuous DMA by the SHARC Length FIFO provides event summary While reading TDC data from buffer memory, data can be zero suppressed (i.e. TDC header immediately followed by TDC trailer) Dataflow out of buffer memory Continued

13. Peter JansweijerMROD Design Review: November 12, 2003Slide 13 When the MRODin encounters a TDC Header, it reformats the TDC-ID bits: Data Format: TDC-ID in TDC Header word 31-2827-2423-0 TDC-ID (4-bit) Upper Nibble = TDC data type 0x1010TDC Header 0x1011End Of TDC Group (used by AMT-1; Not used by AMT-2/3 chip) TDC word received by CSM 31-29 5-bit TDC-ID based on TDM time slot TDC Header 28-2423-0 With 5-bits, 18 TDCs can be coded See: T. Wijnen, “The MROD data format”, (ATL-COM-MUON-2003-011) http://www.hef.kun.nl/atlas/

14. Peter JansweijerMROD Design Review: November 12, 2003Slide 14 31-28 TDC word received by MROD 272625-2423-0 2 bits, residue from TDC-ID CSM -> MROD Parity bit TDC -> CSM Parity Error Upper Nibble = TDC data type 31-2827-2423-0 TDC-ID (4-bit) Upper Nibble = TDC data type TDC word received by CSM 31-28 Error Replacement Code 27-2423-0 Data Format: Parity errors Upper Nibble When the MRODin encounters a TDC Parity Error or an Input Link Parity Error, it replaces the upper nibble with an Error replacement code (programmable)

15. Peter JansweijerMROD Design Review: November 12, 2003Slide 15 MRODin Interrupts IRQ0 X I2O_FIFO Full (128 entries) X S Buffer Memory Partition Full (8 K words) S Read Out Maximum TDC Parity error (+ Overrun) IRQ1Input Link Parity error (+ Overrun) Input Link Down IRQ2TDC Early, Late (+ Overrun) Notes: X: Fatal S: Shutdown Read-out for particular TDC All interrupt sources are individually mask-able

16. Peter JansweijerMROD Design Review: November 12, 2003Slide 16 MRODin FPGA Registers Separator –Pattern –Control Bit Pattern –Mask –Control Bit Mask TDC Header –Pattern –Control Bit Pattern –Mask –Control Bit Mask TDC Trailer –Pattern –Control Bit Pattern –Mask –Control Bit Mask No Data –Pattern –Control Bit Pattern –Mask –Control Bit Mask Error-Code Replace Patterns msb’s MRODin Header Pattern (TLP) msb’s MRODin Trailer Pattern (TWC) Event Length FIFO Interrupt Control IRQ0 –I2O_FIFO Full –Buffer Memory Partition Full –Read-out Maximum –TDC Parity Error (Overrun) TDC Parity Error Individual Interrupt Mask Input Link Interrupt IRQ1 –Input Link Parity error (Overrun) –Input Link Down Early and Late Event-ID IRQ2 –TDC number, Event-ID (Overrun) Test & Input Link Control Status Register Test Link Data Register Test Link Control Register Maximum Event Size Expected Event-ID TDC Mask Register Partition Read-out Enable Separator Flags Register

17. Peter JansweijerMROD Design Review: November 12, 2003Slide 17 Event building of TDC data Tetris register creates a fault tolerant design FPGAs supply SHARCs with appropriate (error) information Testable by SHARC via access to registers and memory Summary MRODin

18. Peter JansweijerMROD Design Review: November 12, 2003Slide 18 VME64x Interface TTC Interface via TIM FIFO and Flow control via ROL Interrupt sources FPGA registers MRODout VME64x TTC interface MRODout FPGA SHARC A 1 2 43 0 SHARC B 1 2 4 30 5 5 LVDS In LVDS Out SHARC Links

19. Peter JansweijerMROD Design Review: November 12, 2003Slide 19 VME64x Slave Interface VME64x Guideline; Chris Parkman, “ATLAS Read Out Driver VMEbus Implementation”: http://atlas.web.cern.ch/Atlas/GROUPS/FRONTEND/documents/ROD_VME83.pdf CR/CSR (AM 0x2F) –A24/D32Single CycleMandatory –A24/D32/D16/D08(EO)/D08(O)Single Cycle, RMWOptional SHARCs –A32/D32Single Cycle, RMW, BLTPreferred CRfull VME64x range, currently filled with the VME64 subset CSRBAR/BitSet/BitClr registers SHARC may generate a VMEbus Interrupt –Selectable IRQ level [1..7] {I(n) D08(O) where n=1..7} –Programmable 8-bit Status-ID –ROACKPreferred AM 0x10 (User Defined) used as HARD Reset for the SHARCs

20. Peter JansweijerMROD Design Review: November 12, 2003Slide 20 TTC interface TTC Interface TIMDesigned by University College London http://www.hep.ucl.ac.uk/atlas/sct/tim/ Uses a special P3 backplane with a so called TTC-bus (8 serial signals) TIM Distributes: –Event-ID / Bunch-ID Put into a FIFO that can be read by the SHARC –Trigger-Type Put into a FIFO that can be read by the SHARC –ECR Connected to SHARC IRQ1 –BCR not used by MROD –L1A not used by MROD TIM Incorporates Busy Logic to throttle the CTP

21. Peter JansweijerMROD Design Review: November 12, 2003Slide 21 FIFO and Flow control via ROL Output S-Link interface 160 MB/s FPGA contains S-Link Output FIFO (256) to avoid local bus stall due to an XOFF SHARCs can transfer output data through a chained DMA

22. Peter JansweijerMROD Design Review: November 12, 2003Slide 22 MRODout Interrupts IRQ0S-Link Return Lines (LRL) Change S-Link LDOWN IRQ1Event Counter Reset (ECR) IRQ2Event/Bunch-ID FIFO Full Trigger Type FIFO Full All interrupt sources are individually mask-able

23. Peter JansweijerMROD Design Review: November 12, 2003Slide 23 MRODout FPGA Registers VMEbus IRQ –IRQ Level –8-bit Status-ID Pattern VMEbus BAR and IRQ –Trigger VMEbus IRQ –IRQ Pending Status –BAR S-LINK Status and Interrupt Register –Link Return Lines (LRL) –LRL Change Interrupt –Link Down Interrupt TTC Control/Status and Interrupt Register –Event-ID/Bunch-ID/Trigger Type FIFO Empty/Full status –Event-ID/Bunch-ID/Trigger Type FIFO Full Interrupt –FIFO Flush select (Software or ECR) Resets and LEDs

24. Peter JansweijerMROD Design Review: November 12, 2003Slide 24 Event building of data from 3 (/4) MRODin boards Formatting of the output data for the ROL Testable by SHARC via access to register Summary MRODout

25. Peter JansweijerMROD Design Review: November 12, 2003Slide 25 Questions? MRODin MRODout

26. Peter JansweijerMROD Design Review: November 12, 2003Slide 26

27. Peter JansweijerMROD Design Review: November 12, 2003Slide 27 MROD-1 Reset Topology