1. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Fast 2D Detector Control at ESRF beamlines David Fernández Alejandro Homs Emmanuel Papillon Manuel Pérez Vicente Rey Cyril Guilloud V. Armando Sole Assen Kirov

2. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Talk outline Introduction  ESRF FReLoN CCDs  2D control arquitecture Espia card based systems  Espia card  FReLoN 2k, Maxipix  Dalsa, Sarnoff, Aviex Latest developments  Driver improvements  Data saving  Common CCD acq. platform

3. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 FReLoN 1000 (1996): 1024 x 1024 - 14 bit 4 parallel readout channels Readout time: 100 ms FReLoN 2000 (1998): 2048 x 2048 - 14 bit Readout time: 220 ms FReLoN – Fast Readout & Low Noise

4. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 FReLoN 1000/2000 acquisition setup Gigabit Ethernet TACO Device server SPEC

5. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Princeton Sensicam, PCO Photometrics Matrox Meteor II [H. Witsch, J. Meyer] Trixell Pixium [L. Claustre] Andor ImagePro [A. Gotz] Direct-Show Other CCD-like devices supported Mar CCD [L. Claustre] IDS Falcon [D. Miraut, J. Meyer] Firewire (IEEE 1394) [A. Gotz] Dalsa Dalstar

6. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 ADSC Q315r for MX beamlines Gigabit Ethernet Local Gigabit

7. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Designed to interface the FReLoN 2k Collaboration  ESRF + SECAD, S.A. 2 Gbps fiber optic link (> 100 m) PCI 64 bits / 66 MHz ~180 MB/s maximum data rate Bus master + Scatter Gather + IRQ Driver source code available  rewritten!  Can write on 32-bit addresses only (4 GB) Espia card

8. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Control workstation (2005) Transtec (2005): Dual Intel Xeon 2.6 GHz, HT 2 GB ECC DDR-RAM 3 x 73.4 GB Ultra320 SCSI 15Krpm SuSE 8.2 + Linux 2.4.20-64GB-SMP Remote NFS through 1 Gbps link  90 MB/s 2 disks in software RAID 0  100 MB/s (8 MB files) 4 disks in software RAID 0  150 MB/s (1 GB files) for ultra-fast tomography

9. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 New FReLoN Generation FReLoN 2k (ATMEL): No Mux – only one fiber optic! Frame Transfer Mode 2048 x 1024 @ 16 fps  66 MB/s FReLoN 4M (KODAK):  Only 4 channels – Full Frame Mode FReLoN HD (PSB-2): 2048 x 1024 @ 32 fps  ~125 MB/s

10. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Connects to the Espia 2 x Camera Link connectors: Base configuration  Multiplexed Multiple pixel packing formats Test image generator @ ~180 MB/s Camera Link: FOCLA

11. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Frame transfer technology 1024 x 1024 @ 60 fps  120 MB/s 1 x Camera Link [Base] Connector Compact!  Image not reconstructed FOCLA does reconstruction Dalsa Pantera 1M60

12. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Dalsa setup Fast online display Image reconstruction

13. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 512 x 512 @ 300 fps  150 MB/s 512 x 128 @ 500 fps  125 MB/s 2 x 8 ADCs – 12 bit 2 x Camera Link [Base] Connectors  Separate RS-232 serial line Sarnoff CAM512

14. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Sarnoff setup Camera Link Fiber optic RS-232

15. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Sarnoff results (2007) 300 fps 75 frames 250 ms 40x slowdown  No sound!  B&W  Flickering (room light) ID15: Tomography in 0.5 sec @ 500 fps

16. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 The real image Espia #1 Espia #2

17. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Image reconstruction in kernel Espia #1 Espia #2

18. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 “Very High memory” support Physical Memory Low RAM 0 MB 3.2 GB PCI Very high RAM 800 MB 4 GB 5.8 GB High RAM 32 bit boundary Driver always has 32-bit aux. buffers If next target frame has 64-bit pages:  Program each Espia to write on aux buffer  Frame end (all adapters finished) copy from aux. into target buffer  Inform waiting threads Adapters are not synchronous Memory copy with IRQ disabled  Kernel thread Implementation in driver High frame rate  many image buffers

19. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Aviex PCCD-8448 2048 x 3584 @ 1.2 fps  18 MB/s Binning 16 x 16  8 fps

20. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Software image reconstruction ADC Per frame operation Warranted only once in a multi-frame (big) buffer Callback thread safe

21. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Medipix2 / Maxipix Pixel detector  photon counting 256 x 256 – 13 bit 0.3 ms readout time 1000 fps  130 MB/s Max: 1400 fps  180 MB/s Chipboard: University of Geneve PRIAM: ISG/ESRF Interfaced to the Espia Espia

22. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Maxipix results (2007) 1000 fps !!

23. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Multichip Maxipix Medipix2 array 5 x 1  1280 x 256  1000 fps  650 MB/s  PRIAM multiple FO links  Future configs: 2 x 2, 5 x 2

24. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 New control workstation OPALE server from Ecrin Systems (FR): Trenton Single Host Board and backplane Dual Quad Core Intel Xeon @ 2.33 GHz 8 GB of FB DDR2 3 x 140 GB SAS disks @ 15 Krpm Chelsio T320 10 Gbps Ethernet controller

25. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Driver evolution 32-bit application  3GB address space  8GB RAM  Dynamic mapping of buffers Move to 64-bit architecture  No so trivial ESRF/Linux 2.1 [Red Hat Enterprise Linux 5] Kernel version 2.6.18 Copy to very-high-memory: 1.3 GB/s

26. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Sarnoff @ ID15 512 x 512 12 bits @ 300 fps  150 MB/s Write to local disks: 300 MB/s Write to remote disks: 300 MB/s NetApp FAS6070 10 Gbps Ethernet

27. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 “La Bestia” Dell PowerEdge 2900 Similar to Ecrin/Trenton + 16 GB RAM 4 + 6 = 10 disks  530 MB/s (800 MB/s raw) Super Sarnoff: 2x FOCLA test image Acquisition rate: ~360 MB/s Running

28. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Turbo Bestia 4 + 6 + 8 = 18 disks  Single RAID 0 array  840 MB/s raw 3 RAID 0 arrays + parallel write  1.5 GB/s raw 4x 6x 8x

29. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Espia Next Generation  PCI-Express 8x lanes  4 fiber optic links @ 250 MB/s  1 GB/s  Full access to 64-bit memory  Maximum register compatibility

30. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Common CCD acq. Library Reuse of common code  generic procedures + interfaces Software “features” fallback if hardware has limited capabilities High level Python implementation Common TACO CCD interface  Interfaces: implicit  explicit  Image processing and data reduction as plugins (C++)  Extensible interface through user-defined “features” (Maxipix)  Generic TANGO interface  Collaboration framework (ALBA)

31. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Current status x 10 + 1[+1] (C++) x 1 (Python) x 20 x 5 + 1

32. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Conclusions Significant number of supported 2D detectors  This number keeps raising …  New detectors demand high performance control Experience in high speed acquisitions Stable low level driver (Sarnoff failed only once…) Functional high level interface  Never-ending project … (sub-projects) Common image acquisition library will help!

33. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Acknowledgements SECAD, S.A.  F. Hereson Instrument Support Group:  J.C. Labiche  E. Collet  L. Siron  J.J. Thevenin  D. Pothin  C. Ponchut  J.M. Rigal  J. Clement  P. Fajardo Computing Services:  P. Makijarvi  B. Regad  P. Pinel  L. Garçon  F. Andrei  D. Gervaise  S. Ohlsson  B. Lebayle

34. A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008 Thank you!