1. Bachir Bouhadef, Mauro Morganti, G. Terreni (KM3-Italy Collaboration)
Parallel Neutrino Triggers using GPUs for an underwater telescope KM3-NEMO
Bachir Bouhadef, Mauro Morganti, G. Terreni
(KM3-Italy Collaboration)
INFN Pisa & Physics Department of Pisa

2. INFN Pisa & Physics Department of Pisa
Outlines
NEMO Phase II Tower
DAQ system for NEMO Tower
Showing a possibility of using a CPU-GPU DAQ for an online muon-track selection.
Proposing a method for Parallelizing the online trigger software NEMO-II Tower.
Test muon triggers in GPUs.
Proposition for KM3Net-It Tower Trigger data handling. 2
INFN Pisa & Physics Department of Pisa 2

3. INFN Pisa & Physics Department of Pisa
NEMO Tower Phase II.
KM3 - NEMO Tower .
8 floors
4 PMT/Floor
8m Floor Arm length
Floor 14
Floor 13
Floor 1
14 floors
6 PMT/Floor
6m Floor Arm
84 PMTs
32 PMTs X 55 kHz 1.7 Mhit/s 3
INFN Pisa & Physics Department of Pisa 3

4. Trigger and Data Acquisition System (NEMO Phase II)
T. Chiarusi & F. Simeone
VLVNT 2013
GPUs in HEP
INFN Pisa & Physics Department of Pisa 4 4

5. Trigger and Data Acquisition System (NEMO Phase II)SC FC
Onshore
TS 1
TS 0
time
AFC
TS 0
TS 0
HM 0
TCPU 0
Gbit swich
TCPU 1
HM 1
TS 1
TS 1
TS 1
TS 0
time
TS is Time Slice of 200ms.
The trigger in TCPU: sorting time’s PMTs hits and applying a charge threshold and
time coincidences (simple coincidence, Floor coincidence).
INFN Pisa & Physics Department of Pisa 5 5

6. Scalable Programming Model
Why GPUs ?
Scalable Programming Model
A GPU uses blocks and threads
for parallel programming SC
AFC
Tower
D.U.
INFN Pisa & Physics Department of Pisa 6 6

7. INFN Pisa & Physics Department of Pisa
A new parallel trigger for muon detection based on time difference of muon’s hits
Most of the muon track hits are
Within a Time Windows (TD).
We need at least 5 hits in different PMTs
to reconstruct the muon track.
We must look for a number of hits N
in a fixed time windows.
INFN Pisa & Physics Department of Pisa 7 7

8. INFN Pisa & Physics Department of Pisa8 8

9. INFN Pisa & Physics Department of Pisa
We propose a DAQ system using CPU-GPU Architecture
Gbit swich
Onshore
Storage Unit
INFN Pisa & Physics Department of Pisa 9 9

10. INFN Pisa & Physics Department of Pisa
TGPU-CPU
TCPU is replaced by TGPU-CPU, and every second the TGPU-CPU will receive
5 Time slices of 200ms each.
INFN Pisa & Physics Department of Pisa
GPUs in HEP 10 10

11. INFN Pisa & Physics Department of Pisa
5 TTS from network thread
TTS4
TTS3
TTS2
TTS1
TTS0
Network THRD
Step 1
5 CPU threads to put PMT hits in correct time interval, and the hits are time-order by thread
Structure is ready to be treated in GPU
TTS4
TTS3
TTS2
TTS1
TTS0
Step 2
CPU work
The size of hits are not fixed, we must prepare a new structure for the GPU
Number of threads must be multiple of 5 and 32
We can not predict how many hits per thread there are, so we fixed a max hits
number by thread using the nominal rate X 3 or 6
We have considered also the edge effect between threads and TTSs.
We should avoid threads with a few hits by choosing the optimal thread
time interval.
INFN Pisa & Physics Department of Pisa 11 11

12. INFN Pisa & Physics Department of Pisa
Sort all PMT hits using classical
Algorithms (shell)
Step 1
Structure is ready to Trigger tagging
Step 2
TTS0
TTS1
TTS2
TTS3
Trigger L0
In L1 Trigger all possible trigger can be implemented, and according
to L0+L1 efficiencies the best is chosen to tag the event to be saved.
GPU work
1 (L0) - N7TW1000
(L0) SC or AFC
2 (L1) - N7TW1000 & SC
3 (L1) - N7TW1000 & AFC
4 (L2) - N7TW1000 & SC & AFC
5 (L2) - N7TW1000 & ( (SC & AFC ) || (SC>1) || (AFC>1))
6 (L2) - N7TW1000 & (SC || AFC))
7 (L0) Charge > Charge_THRHD 1 2 3 4
INFN Pisa & Physics Department of Pisa 12 12

13. INFN Pisa & Physics Department of Pisa
Muon triggers tests in GPUs
GPU cards
INFN Pisa & Physics Department of Pisa 13 13

14. INFN Pisa & Physics Department of Pisa
Muon triggers tests in GPUs
Trigger Time execution for one tower of 32PMT/ ~55kHz.
Using 2 different GPU cards and one 1 second of raw data for 32 PMTs at
background rate ~ 55 kHz and applying all triggers.
The back times are measured while the CPU in Idle.
The read times are measured times while the CPU executes other processes
INFN Pisa & Physics Department of Pisa 14 14

15. INFN Pisa & Physics Department of Pisa
Muon triggers tests in GPUs
Using the same two GPU cards, we use one second data of 84
PMTs (6PMT X 14 Floors) at background rate 55 kHz and applying all triggers.
The back numbers are the measured times while the CPU in Idle.
The red numbers are the measured times while the CPU executes other processes
INFN Pisa & Physics Department of Pisa 15 15

16. INFN Pisa & Physics Department of Pisa
We have also tested a new trigger that combines between 7 different hits within 1000ns
and time-space correlation with respect to first hit. In GPU the trigger was applied to
data tower of 84 PMTs, and measured trigger time is 350ms in case of Tesla20c50.
We are also working muon track reconstruction in GPUs.
Sorting in Tesla 20c50
100 X 182 threads
Alessio Bacciarelli
INFN Pisa & Physics Department of Pisa 16 16

17. INFN Pisa & Physics Department of Pisa
Proposed DAQ CPU-GPU for 8 KM3-Ita Tower
INFN Pisa & Physics Department of Pisa 17 17

18. INFN Pisa & Physics Department of Pisa
Conclusion and future work
- GPUs can also take place in optical neutrino telescopes.
- Still other online more selective algorithms can be applied.
- Muon track reconstruction is underway.
- Both Tesla 20c50 and GTX TITAN can be a good choice.
INFN Pisa & Physics Department of Pisa 18 18

19. Thank you for attention
I would like to thank all Members of
the KM3Net-It Collaboration
INFN Pisa & Physics Department of Pisa 19 19