1. RT2010 & xTCA workshop a (personal) summary
Markus Joos
CERN-PH/ESE

2. xTCA ATCA uTCA / AMC General
A success in the telecom industry (market volume ~109 $)
80% of the boards use GBE in the fabric network (15% PCIe)
Workshop had a focus on control (IPMI, S/W) aspects
AXIe (
uTCA / AMC
Mid-size (4HP) most popular AMC height
Nothing revolutionary new
General
Little information on real-life experience with HA solutions (technology still too young?)
Performance
No much about actual transfer speeds and latencies of e.g. GBE or PCIe at system level
Recommendation for PCIe: Avoid “reads” (use remote write)

3. uTCA / AMC for physics RTMs defined
ADF/ZD connector, 20 (compact AMC) or 30 LVDS pairs
RTM ~same size as AMC
RTM gets 12 V (+3.3 V) from AMC
Power: AMC + RTM = 80W (recommended: W)
AMC + RTM = 457 cm2 (6U VMEbus = 345 cm2)
Mechanical key
AMC / RTM panels according to rugged uTCA spec (insertion force)
Special backplane (zone 1&2) in preparation
Backplane in “zone 3” possible
Two types of clocks:
High precision radial (TCLK[A/B] as defined in AMC.0)
Low speed bussed (ports 17-20)
Still some issues to be resolved. (e.g. Control of RTM fans)
First products available:
Elma, Schroff: crates
SIS8300: ADC (10 channel, 16 bit, 125 MSPS) with RTM

4. uTCA backplane for physics Taken from draft document (
NOT YET APPROVED
A possible configuration:
Port 0 should be used for base Ethernet interface
Port 1 is connected to the optional (redundant) MCH
Port 4 to 7 should be used for PCIe
Port 12 to 15 may be used for application specific wire-ring
Ports 17 – 20 should be wired as a bus
FCLKA (Clk3) should be used for PCIe clock distribution as defined in MTCA.0 R1.0 (main MCH only)
TCLKA and TCLKB should be wired from the main MCH to all AMC slots
TCLKC and TCLKD should be wired from the redundant MCH to all AMC slots

5. xTCA for physics ATCA xTCA S/W Mechanical standards: 90% done
Zone 3 connector (for RTMs) defined
3 ADF/ZD connectors (40 LVDS pairs, 5 GHz) for data transfer
1 combined connector for:
Power: 12V, 50W (+ 3.3V management power)
Management (I2C, JTAG, card detection)
Mechanical keying
Definition of timing signals via Zone 2 connector
For dual-dual star topologies
xTCA S/W
No physics standard planned but recommendations in preparation
Mechanical standards: 90% done
Specification for clock distribution: 90% done
Submission to PICMG for approval planned for September
Questionnaire circulated to identify current xTCA projects

6. xTCA users & products ATCA uTCA / AMC
Several plasma (fusion) experiments
ITER not yet fully convinced, will also evaluate PXIe
XFEL LLRF control system
Panda, BELLE II experiments
PCIe switch blade developed by IST (Lisbon)
uTCA / AMC
XFEL LLRF control system (non redundant)
DESY did a physics compliant, general purpose AMC with RTM
Offer to share design or sub-systems (e.g. MMC)

7. xTCA projects mentioned @ RT2010
Development of
Experiment
Link
ATCA CPU blade (Based on ATX mother board)
COMPASS tokamak (
ATCA ADC blade
IPMC (Intelligent Platform Management Controller)
Nuclear fast fusion plant
Timing & Trigger AMC
XFEL
ATCA blades
XFEL LLRF controls
10 Gbit uplink AMC
PANDA (FAIR)
ATCA computing blade
Several ATCA blades
Fusion Control
AMC carrier
AMC with FMC site
Timing AMC
IPMI
XFEL LLRF
ATCA switch
Proposed to ATLAS
uTCA system
Accelerator controls

8. IPMI Nobody likes it; everybody uses it
PICMG have added features to IPMI for xTCA systems
State machine for hot swapping
Power budgeting
E-Keying
Cooling policy (fan control commands)
F/W upgrade
IPMI spec (600 pages) very ambiguous. This often leads to interoperability problems as manufacturers interpret the spec in different ways
DESY did a “dummy” ATCA AMC carrier as a platform for their IPMC development. They have also built a real-time IPMI analyzer

9. IPMI design Tips
Taken from slides of Pete Hawkins (Performance Technologies)
Always Provide a Debug Interface
Serial Port and/or JTAG
Temperature Sensors
Remember Power and Ground Planes Conduct Heat
PCB mounted sensors don’t measure ambient temperature, but that is how everyone implements them
Headroom is More Important Than Absolute Temp
Max Temp – (Ambient + Conduction + Heat Dissipation)
Thresholds
Don’t be Overly Aggressive
“Non-Critical” thresholds are intended to just be informative, but people tend to think they mean something is wrong
Be Especially Careful with “Non-Recoverable” Thresholds
Management software may take action to prevent damage –i.e. may turn off power to a FRU
Especially true for temperature sensors

10. IPMI pitfalls
Taken from slides of Pete Hawkins (Performance Technologies)
Management Power Budget
15W for ATCA Boards, which includes AMCs for carriers and conversion losses
150mA for AMCs
In-Rush Current
Applies to Management Power and Payload Power
AMC.0 limits must be followed
IPMB Capacitive Loading / Hot Swap Glitches
Follow the PICMG specs carefully
IPMI Controller Resets
Payload Function cannot be affected
Some I/O needs to be latched
Payload Controls, E-Keying Controls, LEDs, etc.
Power Boundaries
IPMI Controller can be powered while payload is not
Connections between payload and IPMI controller may need isolation
PCIeReset / FCLKA
PCIeRequires PERST# to remain active for 100 μs after the reference clock is stable
FCLKA is Part of AMC edge connector, but PERST# is not
Follow Requirements in AMC.1 R2.0 carefully
0000h vs. FFFFh
IPMI often uses 0000h to mean first item and FFFFh to mean end of list
Programmers are used to using null pointers to end lists

11. Other Information Exhibitors (only ~5) Other conference highlights
CAEN
still not convinced by xTCA
also hesitating to go for VXS
VMEbus 2eSST in the pipeline
Agilent
VXS based 6U high speed ADC (RocketIO on VXS)
Schroff / Elma
“uTCA for physics shelves are useful not only for physics”
Other conference highlights
As many new VMEbus designs as xTCA modules
Use of GPUs in trigger systems
High precision timing systems
High speed ADCs

12. Links & events xTCA Interoperability Workshop xTCA workshop slides
Week of October 18, 2010, Schroff Training Center at Straubenhardt, Germany
xTCA workshop slides
RT2010 presentations

13. TWEP2010 & xTCA work group meeting a short (personal) summary

14. xTCA for physics Other observations
Standards (PICMG 3.8 & MTCA.4) still in draft status but submission for ballot may happen soon
Still a lot of work to be done on the S/W side (such as replacing licensed S/W by open source code)
Apparently there are also efforts to standardize ATCA RTMs in Telecoms
Wiener also planning to develop a uTCA (MCA.4) shelf
Other observations
uTCA proposed for XFEL, FAIR and Linac SLAC
Lots of good results from signal integrity tests at xTCA frequencies
Still lots of VMEbus based designs (also for long term projects)
One VXS card (GANDALF for COMPASS) but nobody really compared VXS and uTCA / ATCA

15. xTCA projects mentioned @ TWEP 2010
Development of
Experiment
Description
AMC with MMC on mezzanine
CMS
HCAL upgrade (sLHC)
MCH
Secondary MCH for switching of custom protocol
AMC
LHCb
Read-out board for sLHC
MHC mezzanine
ATCA blade
ATLAS
ROD for Pixel/CSC upgrade. New RCE for meshed backplanes (no switch blade) in the pipeline
RTM for SIS8300 / DESY AMCs
XFEL experiments
MTCA.4 RTM
Lar ROD (for phase II)
“GOLD” card for L1 trigger

16. xTCA Work Group meeting
J. P. Cachemiche
MMC soon ready (stable) for use in other projects
Modifications for MMC F/W of DESY and CPPM not synchronized
CPPM willing to manufacture boards
Request for CERN to manage user support
R. Larsen
New requests for PICMG 3.8 (RTM) functionality slow down completion of the standard and may lead to a proliferation of connector types
Still discussions on timing distribution in ATCA
Discussions on a standard way of reconfiguring FPGAs in radiation environments (may require fast LAN add-on to IPMI)
G. Iles
The PCI clock makes the uTCA backplane non symmetrical. Apparently the MTCA.4 team did not spot this
Meeting results
Participants agreed to found a “xTCA Interest Group”
Plan to meet twice per year (during appropriate conference or via EVO)
The list will be used for communication
A Web site ( has been set up at CERN. All developers of xTCA H/W & S/W will be invited to add a few words about their projects