1. RD53A DAQ/TEST system Silab, Bonn
M. DAAS, T. Hemperek, J. Janssen,
H. Krüger, D.-L. Pohl, M. Vogt

2. RD53A Test System - Hardware
MMC 3: Kintex 7 module on top of a custom design base board (4 layers)
Gigabit Ethernet and USB 3 interface,
2x MGT, voltage and current monitors,
8x RJ45 for FE-I4s.
Design is being modified to support:
4 Multi-Gigabit-Transceivers “MGT”,
High speed connectors for RD53A data,
10 Gbit/s optical Ethernet module port (SFP+).
Displayport connectors
 RD53A
MMC3 system
SFP+ (10G Ethernet)
RJ45
 FE-I4
Debug
MMC3 „RD53A version“
DUT board
RD53A
FIFO DDR3
FPGA
SFP+
GbE, USB 3
MGT PC
Voltage or current source
Data
Supply

3. RD53A Test System - Firmware
Block diagram: IP cores, DAQ HW components, external devices
Internal control signals: Simple 32-bit wide “Basil bus”
Main data flow: AXI4 and AXI4-Stream standard
Firmware is implemented, except the DDR3 memory controller  BRAM for now
Computer
CMD encoder
Aurora receiver
Memory controller
Virtual FIFO
BRAM FIFO
SiTCP Ethernet
AXI
Basil bus
DDR3 memory
RD53A
I²C
Reference clock
Ethernet PHY
Xilinx Kintex 7 FPGA
Basil bus master
DAQ system
AXI stream

4. RD53A Test System - Testbench
For behavioral simulation, replace the interface part with a testbench
Replace the USB/Ethernet interface with the Basil “SiSim”, a virtual interface driver for CocoTB (Example:
Include RD53A Verilog sources “digital sea” from the GitLab project repository
Run tests, written in Python and using the Basil framework
Memory
SiSim
State machine
DUT Interface
RD53A (Model)
Basil bus
Include model PC
Testbench
Core firmware

5. RD53A Test System - Hardware
Planned support for XILINX KC705 evaluation board
Kintex 7 (XC7K325-2), DDR3 module, USB-JTAG
1G and SPF+ module cage for 10G Ethernet, 2 FMCs (1 HPC, 1 LPC)
RD53A FMC-adapter-card is being developed at CERN
Connects to the RD53A Single Chip Card
Can be plugged to “any” FPGA evaluation board with FMC connectors
Will be used for YARR and RCE test system as well
RD53A FMC Card (Luis Miguel Jara Casas (CERN))
XILINX KC705 evaluation board

6. DIGILENT Genesys 2 Board
Features:
Xilinx Kintex-7™ FPGA (XC7K325T-2FFG900C)
50,950 logic slices (up 7x), each with four 6-input LUTs and 8 flip-flops
Close to 16 Mbits of fast block RAM (up 7x)
Ten clock management tiles, each with phase-locked loop (PLL)
840 DSP slices (up 17x)
Internal clock speeds exceeding 450MHz
On-chip analog-to-digital converter (XADC)
Up to Gbps gigabit transceivers
1800Mbps DDR3 data rate with 32-bit data width
Commercial -2 speed grade

7. RD53A testing – Status and Outlook
RD53A submitted end of August, chips available ~ November
Preparation for the first chip tests:
Contribution to verification for the digital design before the submission – several Aurora communication issues discovered 
Software/Firmware development side-by-side with the digital design, to have the test system available until the chips arrive
Aurora receivers for 1,2,4-lane mode implemented 
Command encoder implemented 
DDR3 FIFO module works in a test bench , hardware needs debugging 
Development of test system hardware for the RD53A chips
Single Chip Card layout is ready, being reviewed. First batch of PCBs will be ordered soon 
New DAQ base board variant of MMC3 until November 
Needle probe card for our 12” probe station is currently being developed 
Characterization, first test beam campaigns starting until the end of 2017
Preparations for module hardware development in 2018

8. M. DAAS, T. Hemperek, J. Janssen, H. Krüger, D.-L. Pohl, M. Vogt
Characterization and Verification Environment for the 65 nm Pixel Readout-Chip RD53A
M. DAAS, T. Hemperek, J. Janssen,
H. Krüger, D.-L. Pohl, M. Vogt
TWEPP, Santa Cruz

9. Introduction
RD53A is a large scale demonstrator for a hybrid pixel detector readout chip in a 65 nm process, common ATLAS and CMS effort
Features
400 x 192 pixels (400 x 384 for the production design), 50x50 µm pixel pitch
20 mm x 12 mm size (20 mm x 20 mm for the production design)
Digital design with ~ 1 billion transistors
Serial powering with Shunt-LDOs
Fast data link to the readout system
CML drivers
Aurora 64b66b protocol standard
Configurable data rate options: 4x 1.28 Gb/s or 1x 1.28 Gb/s, or slower
“Analog island” - Full custom design AFE
“Digital sea” - Synthesized Verilog code
Quad
Analog “island”
Digital “sea”
4x4 pixel core
Pixel

10. RD53A Test System
Main purpose: Evaluation of the RD53A prototype chips
Electrical characterization (single chip)
Test beam performance
Multi-chip (module) tests
Wafer-level tests with a probe station
Software, Firmware, Verification
For multiple scenarios, a lot of firmware and software to be written
Testing the test system itself during development is crucial
New concept of including the RD53A Chip into development
Hardware
Commercial Enclustra KX1 FPGA board
New custom design base board, based on MIO3/MMC3
New RD53A Single Chip Card

11. USBpix
Versatile DAQ system for pixel detector readout, developed in SiLab
Over 100 systems shipped worldwide
Applictions:
FE-I4 single & multi-chip tests
CMOS pixel development
FE65_P2 65nm ATLAS prototype chip
Integration of EUDET/AIDA telescope R/O
Basil
Python-based modular DAQ an system testing framework
Library of firmware components and drivers for lab equipment (meters, oscilloscopes etc.)
PyBar
Readout and analysis software for ATLAS FE chips
USBpix wiki:
FE65_P2 setup
EUDET telescope setup

12. USBpix Hardware Upgrade (Multi-Chip R/O)
FE-I4 Adapter Card
Multi-IO board “MIO3”
FE-I4 adapter card
Burn-In card for multiple chips
GPA card with ADCs, voltage sources etc.
Based on commerial FPGA module
Enclustra KX1, Kintex XC7K160T
USB Gbit-Ethernet
1Gbyte Mbyte DDR3 memory
4 MGT transceiver, 6 Gbit/s max
Multi-IO Board 3.0 (MIO3)
Burn-In Card
Enclustra KX-1
General purpose analog card

13. RD53A Test System - Firmware
Block diagram: IP cores, DAQ HW components, external devices
Internal control signals: Simple 32-bit wide “Basil bus”
Main data flow: AXI4 and AXI4-Stream standard
Firmware is implemented, except the DDR3 memory controller  BRAM for now
Computer
CMD encoder
Aurora receiver
Memory controller
Virtual FIFO
BRAM FIFO
SiTCP Ethernet
AXI
Basil bus
DDR3 memory
RD53A
I²C
Reference clock
Ethernet PHY
Xilinx Kintex 7 FPGA
Basil bus master
DAQ system
AXI stream

14. RD53A Test System - Firmware
S/W and F/W based on Basil
Software framework written in Python
FPGA firmware module library written in Verilog
Hosted on Github with Continuous Integration:
We would like to use the same Basil- and (almost) the same FPGA code for the DAQ system and during the chip verification phase.
Approach: Coroutine Co-simulation testbench environment “Cocotb” ( PC
Basil Framework
Drivers
DAQ system firmware
Verilog Modules
Basil Bus
Interfaces
RD53A
Tests
DUT
USB or Ethernet

15. RD53A Test System - Firmware
The synthesizable FPGA firmware cannot be used directly for simulation
Communication uses Ethernet or USB interfaces
Protocols are too complex for a reasonably fast simulation
Involve hardware components: Ethernet PHY or Cypress USB controller
Memory
USB/Eth Interface
State machine
DUT Interface
RD53A (DUT)
Basil bus
I/Os PC
FPGA firmware

16. RD53A Test System - Firmware
First step to enable simulation: Divide the firmware in two parts
Core: Contains functional logic, state machines, Basil F/W modules
Interface: PC interface, I/O buffers
Connection between both: Basil bus, FIFO interface, DUT I/Os
Memory
USB/Eth Interface
State machine
DUT Interface
RD53A (DUT)
Basil bus
I/Os PC
Interface firmware
Core firmware

17. RD53A Test System - Testbench
For behavioral simulation, replace the interface part with a testbench
Replace the USB/Ethernet interface with the Basil “SiSim”, a virtual interface driver for CocoTB (Example:
Include RD53A Verilog sources “digital sea” from the GitLab project repository
Run tests, written in Python and using the Basil framework
Memory
SiSim
State machine
DUT Interface
RD53A (Model)
Basil bus
Include model PC
Testbench
Core firmware

18. RD53A Test System - Verification
Tests are written in Python
DUT: Verilog RD53A digital design model
FPGA firmware, written in Verilog
Testbench contains DUT and FPGA firmware
Access between the Python Test and the Verilog Testbench via easy-to-use, Basil drivers (GPIO, Register, FIFO etc.)
New possibilities: Verification of the DUT itself
#test_HitFile.py (a few examplary lines)
import rd53a
class test_HitFile(unittest.TestCase):
self.chip = rd53a()
def test_file(self):
self.chip['control']['RESET'] = 1
...
self.chip['global_conf']['Latency'] = 110
self.chip.write_ecr()
rawdata = self.chip['fifo'].get_data()
HDL simulator
Testbench.v
Test.py
RD53A Design Model
CSV parser
DUT: RD53A
Event list
(CSV)
FPGA firmware
Basil driver
DUT class commands
Direct DUT stimulus

19. RD53A Test System - Verification
Test hits are generated from text file and injected into DUT
Triggers are generated and send to the FPGA firmware
The CMD encoder in the FPGA generates trigger commands and sends them to the DUT
The DUT sends event frames through the FPGA firmware back to the test routine
The event frames are interpreted and compared with the generated hits
//testdata.csv [BCID, COL, ROW, TOT, TRG]
20,0,0,1,0 21,0,0,3,1
...
//rd53a_tb.v (Verilog testbench)
`include "top/RD53A.sv“
module tb (
input wire [`COLS*`ROWS*`REGIONS*`REG_PIXELS-1:0] HIT,
input wire TRIGGER,
[...]);
RD53A_FW fpga (
.DUT_TRIGGER(TRIGGER),
.DUT_OUT_DATA_P(SER_DATA1G_P),
.DUT_OUT_DATA_N(SER_DATA1G_N),
.CMD_OUT(CMD_OUT),
RD53A dut(
.ANA_HIT(HIT),
.CMD_DATA_P(CMD_OUT),
.CMD_DATA_N(!CMD_OUT),
.SER_DATA1G_P(SER_DATA1G_P),
.SER_DATA1G_N(SER_DATA1G_N), [...]);
#HitDataFile.py (Test)
with open(self.filename) as csvfile:
csv_reader = csv.reader(csvfile, delimiter=‘,')
for row in csv_reader:
while(bcid < int(row[0])):
yield RisingEdge(self.clock)
yield Timer(5000)
[...]
# set pixel
pixn = ((int(row[2]))+(int(row[1])*384))
bv[pixn] = 1
self.hit.assign(str(bv));
self.bus.HIT <= self.hit
# set trigger
trigger = int(row[4])
self.bus.TRIGGER <= trigger
# compare
data = interpret_rawdata(self.chip['fifo'].get_data())
self.assertEqual(np.array_equal(file, data), True)
Instantiates FPGA FW: (GPIOs, CMD encoder, FIFOs, Aurora)
Instantiates DUT: (Analog and digital chip bottom, JTAG, Pixel array) 1 2 4 3 5

20. RD53A Test System - Single Chip Card
One board for multiple use cases and test systems
Designed to be compatible with the other RD53 test systems YARR, RCE
Lab debugging: Access to all signals
Test beam campaigns: Minimal interface
Sensor characterization: Minimal interface with additional outputs (Hit-Or…)
DP1
DP2
Debug signals
PWR IN
OUT
RD53A
DATA1G[3:0]
CMD
HITOR[3:0]
Top Row debug pins
LEMO
Configuration jumpers
AUX
Power: 4-pin Molex connectors
PWR_IN
AUX_PWR (for additional PLL and CMD supply
PWR_OUT (for SP chain)
Analog monitoring
Iref_in/_out
Vref_ADC
Vmux_out
Vinj …
Digital debugging
JTAG
Status signals
Ext. trigger
Reset
Bypass clocks
JTAG

21. RD53A Test System - Single Chip Card
PCB size: 10 cm x 10 cm
Compatible to FE-I4 SCC / telescope setups (mounting holes, chip position)
Thermal management: Thermal vias or cut-out area for direct cooling
RD53A wire bond diagram
RD53A Single Chip Card

22. RD53A Test System - Single Chip Card
Powering options
Direct powering: Shunt-LDO is bypassed, VDDA and VDDD supplied separately
Shunt-LDO mode with constant current supply
Analog and digital power either separately or combined
Serial powering with Shunt-LDO mode
PWR
OUT
RD53A IN
Direct powering
GND VIND
VDDD
VINA VDDA
PWR
OUT
RD53A IN
Shunt-LDO mode
GND VIND
VDDD
VINA VDDA
PWR
OUT
RD53A IN
Serial powering
GND VIND
VDDD
VINA VDDA

23. RD53A Test System - Single Chip Card
Powering configuration
Current path can be selected by jumpers
Separate power lines for analog and digital, can be shorted for serial powering mode
PWR
OUT
RD53A IN
GND
VIND
VDDD
VINA
VDDA
PWR_IN_CON
PWR_IN2
PWR_IN1
VIN_CON

24. RD53A Test System - Single Chip Card
Serial powering chain with SCCs
Daisy-chaining of single chip cards, configuring them with jumpers
Multiple chips are supplied by one single current source
Individual serial links (DATA, CMD are AC-coupled) between FPGA board and SCCs
“Slow Control “signals are not dc-balanced and cannot be ac-coupled
SCC
PWR IN
RD53A
VIN
GND
OUT
Constant current supply
FPGA Board
short PWR_OUT of last module

25. RD53A Test System - Single Chip Card
Needle Probe Station
SCC variant for functional wafer-level tests
To reduce cabling: Port expander on the Needle-SCC for semi-static “slow” signals
Analog MUX for monitoring signals
I²C bus for port expander and mux on Displayport “Debug” connector

26. RD53A Test System - Hardware
"Slow" control- and debug signals: Ribbon cable and IDC connectors
Connector for high speed data:
Displayport 1.4
4 pairs ~ 6 Gbit/s  for RX
1 pair ~ 720 Mbit/s  for TX
Widely available and reasonably priced
Cable lengths up to 7 m available (but not for 6 Gbit/s)
Hit-Or: 4 general purpose LVDS signals, dedicated second Displayport connector
Debug cable can provide two additional clock, in case the RD53 DATA- and CMD PLLs don’t work by changing signals paths on PCB-level
Displayport

27. RD53A Test System - Single Chip Card
Pin
Orig. Signal
SCC DP1
Comment
SCC DP2 1
LANE_0_P
GTX_DATA0_N
HITOR3_N 2
GND 3
LANE_0_N
GTX_DATA0_P
HITOR3_P 4
LANE_1_P
GTX_DATA1_N
HITOR2_N 5 6
LANE_1_N
GTX_DATA1_P
HITOR2_P 7
LANE_2_P
GTX_DATA2_N
Opt. EXT_SER_CLK via jumper
HITOR1_N 8 9
LANE_2_N
GTX_DATA2_P
HITOR1_P 10
LANE_3_P
GTX_DATA3_N
Opt. EXT_CMD_CLK via jumper
HITOR0_N 11 12
LANE_3_N
GTX_DATA3_P
HITOR0_P 13
CONFIG1
NTC1
Temperature sensor
SDA
I²C for probe card 14
CONFIG2
NTC2
SCL 15
AUX_P
CMD_N
EXT_SER_CLK_N
Tbd. 16 17
AUX_N
CMD_P
EXT_SER_CLK_P 18
Hot Plug Detect
RST_B 19
DP_PRW_RETURN
VDDD_SENSE
3.3 V
For probe card 20
DP_PRW
GND_SENSE
3.3 V return

28. THANK YOU