1. PS TFB
Upgrade possibilities for the Hardware of the LEIR type Beam Control Information from: M.E. Angoletta, P. Baudrenghien, E, Bracke, A. Butterworth, J. Ferreira, J. Molendijk, R. Olsen, F. Pedersen, T. Rohlev, J. Sanchez
LHC – LEIR approach
Present LEIR Hardware
Possible upgrades
Planning
A.Blas AB/RF/FB
Team meeting 21/1/2008

2. Beam Control Typical architecture
Both the LHC and LEIR system start with the same initial structure
Typical constraint for the signal processing delay:
Loop computation time + Hdw delay < 1/(8.Fmod) [phase max Fmod < π/4]
< 21 us for PSB & LEIR phase loop (3.Fs=6kHz)
A.Blas AB/RF/FB
Team meeting 21/1/2008

3. Beam Control Signal processing
In LEIR the FPGA-DSP-FPGA triptych is always used
In LHC it depends.
Note that using the DSP in a loop implies major timing problems when a precise loop delay is required (TFB, 1TFB) (see example in Elettra)
A.Blas AB/RF/FB
Team meeting 21/1/2008

4. Beam Control Board layout
Interconnect architecture shared by the LHC and LEIR hardware
The Central FPGA (fpga array in Leir) routs the data from and to the different locations
The special function board can be in the form of an on-board specific circuit, a daughter cards or a dedicated VME board
A.Blas AB/RF/FB
Team meeting 21/1/2008

5. Beam Control Clocking LHC:
A central VCO, controlled by a central processor, outputs the rf (400MHz +/- e) feeding the cavity. This rf signal is used to create the 380, 80, 40, 20, Frev which are dispatched within the hardware.
LEIR:
A central DDS, controlled by a central processor, outputs the tagged rf clock distributed to all rf sources or demod inputs.
(+) Absolute phase control over distant inputs/outputs without (except practical) limitations on their number.
(+) Synchronous change of parameters on all boards.
(+) The Clock harmonic can be changed on the fly (factor 1, 2, 4 and 8) for use on a wide frequency span (factor 16)
(+) synchronous signal acquisition
A.Blas AB/RF/FB
Team meeting 21/1/2008

6. Beam Control Hardware standards
LHC: VME 64 crate, h: 9U front slots, 160 mm depth with standard P1/P2 bus
15 front slots, 220 mm depth with dedicated P2 bus
Spec. power supplies setup
6 kCHF + 10 kCHF CPU
LEIR: VME 64X crate, h: 9U 23 front slots, 160 mm depth
23 back slots, 160 mm depth
All standard P0, P1, P2 connectors
Standard Power supply
8 kCHF + 10 kCHF CPU
A.Blas AB/RF/FB
Team meeting 21/1/2008

7. Beam Control Communication links
Samtec QSE, 80 (4 small ribons of 20 coax cables on a single connector
160 Gb/s (at 2 GHz)
3.2 Gb/s (at 40 MHz)
Used in LHC hardware
Gigabit serial link with 8b/10b coding
16bit// 50 MHz -> 1Gb/s serial
800Mb/s
Full duplex at the same rate
Very common standard
DSP bus + mother board to daughter card bus at 40 MHz and 32 bit word: 1.28 Gb/s (to be divided by the number of wait states +2 = 4)
320 Mb/s
Used in LEIR
DSP <-> DC
Link port clocked at 40 MHz (can be used at 80 MHz)
DSP <-> DSP
A.Blas AB/RF/FB
Team meeting 21/1/2008

8. FPGA comparison Xilinx Virtex 2 XC2V2000- 6FG676C (Tuner loop)
Altera Stratix
EP1S20F
484C5
(Leir daughter cards)
Virtex 4
XC4VLX40
Beam Ph
+ rf Mod.
Altera
Stratix II
EP2S90F
1020 C4
LHC 1TFB
Logic cells
24,192
18,460
41,472
90,960
RAM [kb]
1,000
1,600
1728
4414
18 x 18 multipliers 56 80 64
192
I/O
456
361
640
758
Price [USD]
278
350
267
2890
A.Blas AB/RF/FB
Team meeting 21/1/2008

9. DSPs comparison Model Clock [MHz] MMACS Link ports Serial Links
(not used)
Unit price
for 1000
(USD)
ADSP-21160M
(Leir) 80
160 6
80 MHz max 2
40 MHz max
159
ADSP-21160N
(upgrade Leir)
100
200
100 MHz max
50 MHz max
170
ADSP-TS101S
(LHC Tuner loop)
300
2100 4
125 MHz max
Incompatible with Sharc DSP!?
Dual edge = 250 MHz eq
186
ADSP-TS201S
(available novelty)
600
4800
500 MHz max
242
A.Blas AB/RF/FB
Team meeting 21/1/2008

10. DSP chip ADSP 21160M (Leir at present)
80 MHz, 4Mbit on chip SRAM, 2.5V core
A.Blas AB/RF/FB
Team meeting 21/1/2008

11. DSP chip Possible LEIR upgrade ADSP 21160N
Same as M version except 20% faster (100 MHz) and 1.8V core
A.Blas AB/RF/FB
Team meeting 21/1/2008

12. DSP chip Tiger Sharc ADSP-TS101 (LHC) A.Blas AB/RF/FB
Team meeting 21/1/2008

13. Leir BC <-> LHC BC
Modified VME 64 crate
160 mm (CO)+ 220 mm (rf)
Private Backplane bus with:
Power, ECL Clocks, Timing, Function Gen, Interlocks, Alarms, LVDS data bus, JTAG, Auto slot addressing, Module serial number bus
Standard VME 64X crate
160mm with rear access to bus
Cadence blocks:
Tiger sharc + environment
Virtex 4 + environment
1 Gb serial link + environment
Tagged clock + distribution
(connectors are fragile; why??)
RTM for timing, data, clock, link ports
High speed parallel connection between VME neighbors
Samtec QSE 80 // coax lines
Mother-daughter card structure interconnected via the DSP bus
A.Blas AB/RF/FB
Team meeting 21/1/2008

14. Leir BC <-> LHC BC
Pros:
Standard VME 64X crate with front and rear access
Very modular approach; with 3 DC (DDC, SDDS, MDDS), 1 MB and 1 RTM, you have the base for all type of applications (except yet for 1TFB and TFB)
Synchronous control over all the hardware
LHC
Pros:
Power supplies adapted to low noise requirements
More up-to-date circuits and connectors
A.Blas AB/RF/FB
Team meeting 21/1/2008

15. Leir Beam Control Clock core
Output rf frequency <12 MHz (for a swept frequency - without up-conversion)
<25 MHz at fixed frequency (without up-conversion)
Limitations due to the basic structure (max ADC DAC sampling + tag creation circuit)
A.Blas AB/RF/FB
Team meeting 21/1/2008

16. Leir Beam Control Acquisition
A.Blas AB/RF/FB
Team meeting 21/1/2008

17. Leir Beam Control Outputs
A.Blas AB/RF/FB
Team meeting 21/1/2008

18. Beam Control General architecture (LEIR)
A.Blas AB/RF/FB
Team meeting 21/1/2008

19. Beam Control Constraints Data flow
Rule of thumb:
The loop will be sufficiently stable if its delay leads to a phase lag < p/4 at the unity loop gain frequency
 Loop computation time + Hdw delay < [1/(8.Fmod)]
< 21 us for PSB & LEIR (3.Fs=6kHz)
< 26 us for PS (3.Fs=4.8 kHz)
In LEIR, the in-loop DSP is sampling the data every TS-DSP = 12.5 us (80 kHz).
The loop delay within the DSP = import data from DDC (<50ns) + compute error (<7us) + send error to MDDS or SDDS (via another DSP or not < 150 ns) + equivalent 1st order S/H delay (6.25 us) => ≈14us delay within the DSP
With a 80 kHz DSP sampling clock, an averaging (CIC) of MHz-samples in the DDC would be adequate. We actually use 256, which means <6.4 us extra delay. This means that we are approaching the reasonable limits required for the LEIR and PSB phase loop.
The DSP process time in LEIR is the most time consuming and multiplying by a factor 2 this process speed would almost double the possible bandwidth.
A.Blas AB/RF/FB
Team meeting 21/1/2008

20. Leir Beam Control Data Flow
Here, the central DSP B has all its links ports used (limiting factor)
In case of more DSP boards, we would need to add links to the leading DSP or implement a pipe lined communication protocol through DSP boards or make all DSPs share a single bus with a communication protocol to be defined.
In terms of bandwidth, A gigabit link could replace the link ports and the DSP-to-daughter card link
A.Blas AB/RF/FB
Team meeting 21/1/2008

21. Leir Beam Control Daughter cards
The present DDC (ADC) and SDDS (DAC) daughter cards having 4 channels are limited in terms of FPGA logic cells.
The signal monitoring circuit could not be implemented.
The 4 channels of the DDC need to share 2 LOs
The DDC FIR could not be implemented
The present chip: Altera Stratix EP1S20F 484C5 should be replaced with a version with 6 times more logic elements to have a 50% loading at most. 18k LE -> 110 k LE and twice as much RAM (1.6 Mb -> 3.2 Mb) and same I/O: 361.
The Stratix 3 EP3SL150 can do the job: 142 kLE, 5.5 Mb, 480 or 736 I/O, 1400 euros (Spoerle)
A.Blas AB/RF/FB
Team meeting 21/1/2008

22. DSP board Background 04/2002 Original design by Joe Delong (BNL)
03/2005 Original version imported in the CERN database (EDA 00990) and called version 1
13 units constructed and tested. Followed-up by J. Ferreira-Bento.
08/2006 Upgrade to version 3 by Tony Rohlev, after a faulty version 2
which had a bad set of connections on a FPGA
Main improvements: latched registers interfacing the VME bus + adequate power-up sequence.
(see AB-Note RF for a detailed description of the modifications)
09/2007 Version 4. Cleaned-up version : removed 4 daughter card connectors (only 2 DC slots now), 1 FPGAs, the Event link circuit, the link-port connections to the DC connectors (reflections), Added DSP address bus lines to the DC to avoid paging of the SRAM address + allow use of DMA (DMA lines added) + use of delayed SELN bits for the register address decoding, all 16 RTM signal connected (6 before).
A.Blas AB/RF/FB
Team meeting 21/1/2008

23. DSP board Version 1 Block Diagram Problem: When the DSP SRAM
is controlled by the VME bus
The DSP is blocked
A.Blas AB/RF/FB
Team meeting 21/1/2008

24. DSP board Version 3 Block Diagram Here the DSP SRAM data can be stored
on the VME bus to allow the “long” ~1us
VME Read process while letting the DSP run
A.Blas AB/RF/FB
Team meeting 21/1/2008

25. DSP board Some limitations
DSP blocked when the VME (which has priority) accesses the DSP SRAM (improved in version 3), Problem solved in Leir with pre-defined time slots for each access. Not convenient for long cycling machine as AD where operational data should be accessed during the cycle.
Bus lines arbitration by 5 (4 in V4) distinct FPGAs => not flexible and uneasy to maintain
DSP limited computing power. Impairs the rf loops pipe-line delay and signal quality (aliases),
requires assembly coding (instead of a more universal C coding)
DSP configuration device not accessible via VME for remote programming,
nor daughter card JTAG programming.
Only 6 timing (or digital) inputs from J2/P2 connector (RTM allows 16) (solved in V4)
4 MB DSP memory + 4 MB for measurement memory foreseen to be too little (-> 32 MB total)
VME base address selected by FPGA coding (-> put switches also)
A.Blas AB/RF/FB
Team meeting 21/1/2008

26. LHC tuner loop DSP board
The Tiger Sharc (tested) circuit can be re-used.
For the FPGA there might be better choices in other circuit using Virtex 4.
A.Blas AB/RF/FB
Team meeting 21/1/2008

27. Proposed upgrade for the LEIR type DSP mother board
Maria Elena’s note
Single FPGA interconnecting all buses, timing, resets and clocks.
Fast VME/DSP arbitration for memory access.
Additional FPGA as link port multiplexer.
Upgrade / doubling of the DSP chip
Increase memory size (4x -> 32 MB)
Additional High speed inter-processor links
Increase the number of lines between RTM and J2/P2
Misc….
Other possible improvements
Add a couple of gigabit serial links from central FPGA to front panel. This would allow using future specific boards as we do presently with daughter cards
A DAC could be connected to the Central FPGA for monitoring/test purposes
Provide a connection port to a stand alone VME daughter card (Samteq QSE) same as for LHC boards
VME offset address selected by jumpers instead of hardcoding
Suppress link ports lines to daughter card sockets (electric reflections) (done V4)
Increase the address bus width connecting the daughter cards to avoid paging (DMA) and avoid the delayed SELN address decoding bit (done V4)
Use better connectors for the tagged clock distribution
Implement a communication protocol to chain DSP boards to ease Beam control upgrades and avoid piling-up of cables (critical in the CPS)
JTAG remote programming as in the CO with a dedicated FPGA-EPROM pair used as a VME-to-JTAG interface. All the FPGAs should be in a single JTAG loop accessible on the front panel for local programming.
A.Blas AB/RF/FB
Team meeting 21/1/2008

28. Leir Type DSP BC upgrade. Summary
Some CadenceTM blocks can be used from LHC designs with improved specs (tiger shark) + Gigabit link + general interconnect philosophy
The Virtex 4 or Stratix 2 circuits could also be imported but we should move to Virtex 5 or Stratix 3 (lower power dissipation 65nm, serial link interface not bugged + higher performances). To be used on DSP board and daughter cards.
The standard VME 64X remains a good choice. Noise issues have still to be checked for AD (linear Power Supplies).
The tagged clock and its distribution are a real plus for synchronous operation but the actual connectors are too fragile and could be replaced the RJ45 model used in LHC rf.
Add all the hardware details mentioned on the previous slide.
A.Blas AB/RF/FB
Team meeting 21/1/2008

29. Leir Type DSP BC upgrade.
Possible scenario
Install a test beam control in the PSB for the 2008 run, using our present stock of Hardware. A few units are still to be tested. Can be ready in mid 2008.
Analysis of the software requirements if using a new DSP (MEA)
Definition of an adequate inter-DSP link (h/w + protocol) (1man.month)
Update the mother board with all the new features keeping the compatibility with the present daughter cards. 8 man-month for the hardware, 6 man-month for the software + 40 kCHF. Ideally the software should be finished before we start the hardware.
Update of the daughter boards (DDC + SDDS): 3 man-month x 2 for the hardware + 1 man-month x 2 for the software + 40 CHF -> 1 working prototype for each function.
Total: 2 man-year + 80 kCHF for a “final” DSP beam control prototype hopefully fulfilling the requirements of all the Injectors on the Meyrin site.
A.Blas AB/RF/FB
Team meeting 21/1/2008

30. DSP BC upgrade Generic view
A.Blas AB/RF/FB
Team meeting 21/1/2008

31. A.Blas AB/RF/FB
Team meeting 21/1/2008

32. Leir Beam Control
A.Blas AB/RF/FB
Team meeting 21/1/2008

33. Misc Arctan computation:
FPGA Cordic John: 17 clock periods for a 16 bit resolution
DSP Pawel: 33 clock period for a 18 bit resolution
A.Blas AB/RF/FB
Team meeting 21/1/2008