1. Synchronous Device Interface at NSLS-II Yuke Tian Control Group, NSLS-II, BNL (May 1, 2009 EPICS Collaboration Meeting, Vancouver)

2. Outline 1.Synchronous, deterministic and reliable data distribution 2. Synchronous Device Interface 3. Synchronous Device Interface Extension 4.MicroTCA 5.Summary EPICS Collaboration Meeting, Vancouver 2009

3. Synchronous, deterministic and reliable data distribution Requests to control systems from accelerator physics Distribute data in accelerator complex with: ● Synchronism: Data acquisition and distribution systems run off a single master oscillator. ● Deterministic : Data distribution with a fixed delay. ● High reliability: Data distribution with redundancy, no single point of failure. NSLS-II Accelerator Complex EPICS Collaboration Meeting, Vancouver 2009

4. Goal: run all circuitry in the accelerator off a single master oscillator Advantages  Accelerator are driven by many subsystems (RF, magnet/PS, diagnostics etc). Global synchronism offers repeatable, stable behavior. For example, fast orbit feedback.  Synchronous stimulus-response measurement (such as response matrix, tune measurement etc) are easy in a fully synchronous system.  Global synchronism offers global timestamp.  There is no added jitter from crude local clock. Synchronous, deterministic and reliable data distribution: synchronism EPICS Collaboration Meeting, Vancouver 2009

5. Synchronous, deterministic and reliable data distribution: synchronism EPICS Collaboration Meeting, Vancouver 2009 Clock Domains In Accelerators

6. Synchronous Device Interface: challenges for FOFB EPICS Collaboration Meeting, Vancouver 2009

7. Synchronous Device Interface: answers for NSLS-II FOFB EPICS Collaboration Meeting, Vancouver 2009

8. Synchronous Device Interface: cell controller EPICS Collaboration Meeting, Vancouver 2009

9. Synchronism in NSLS-II  125MHz clock (in sync with 500 RF master clock) and fiducia will be distributed.  We already have a very fine, low jitter clock. We can synchronize all circuit with this RF clock. We can timestamp every event with 2ns resolution. All the system are talking with the same clock.  We will push the synchronism into power supply control system. Then, all subsystem (BPMs, cell controller, PS control etc) will be fully synchronous with RF clock. This will provide a lot of benefits for accelerator physics. Synchronous Device Interface: synchronism EPICS Collaboration Meeting, Vancouver 2009

10. Synchronous Device Interface: deterministic Data per BPMData per cellTotal data to be distributed GTP 1.25Gbps 8-bit datawidth GTP 2.5Gbps 16-bit datawidth 10 bytes (X:4; Y: 4; Status:2) 120 Byte (8 BPM 4 XBPM) 120 * 30 = 3.6Kbyte 3.6Kbye/1Gbps = 28.8 us FPGA: 125MHZ 3.6Kbye/2Gbps =14.4 us FPGA: 125MHZ Latency calculations: EPICS Collaboration Meeting, Vancouver 2009

11. Synchronous Device Interface: deterministic GTP interface test: 1.25Gbps 8-bit datawidth EPICS Collaboration Meeting, Vancouver 2009

12. Synchronous Device Interface: deterministic GTP interface test: 2.5Gbps 16-bit datawidth EPICS Collaboration Meeting, Vancouver 2009

13. Synchronous Device Interface: reliability Synchronous Device Interface reliability:  There are two 2.5Gbps links running in opposite directions. One is the main link and the other is backup link.  When a link (or a pair of links) between two cell controllers is broken, the backup link is used to keep connectivity between any pair of cell controllers in the ring. The system has localized the broken link and re- configured automatically to maintain connectivity using backup link.  The BPM data distribution is abstracted from upper application levels, and therefore corrector setting calculations are not altered when the system falls into the fault recovery operation mode. The fault is identified by the system, and error flags are delivered to the operator level. EPICS Collaboration Meeting, Vancouver 2009

14. Synchronous Device Interface: extension EPICS Collaboration Meeting, Vancouver 2009

15. Synchronous Device Interface: Extension to PS Control EPICS Collaboration Meeting, Vancouver 2009

16. MicroTCA EPICS Collaboration Meeting, Vancouver 2009 In 1979, Motorola was developing their new Motorola 68000 CPU and one of their engineers, Jack Kister, decided to set about creating a standardized bus system for 68000-based systems, which he called VERSAbus. http://en.wikipedia.org/wiki/VMEbusMotorolaMotorola 68000Jack KisterVERSAbus Request from distributed control/computing system (looks familiar ?)

17. MicroTCA EPICS Collaboration Meeting, Vancouver 2009

18. ANSI/VITA 1.1 80 MB/s ANSI/VITA 1.5 320+ MB/s VITA 41 3 to 30 GB/s VME with Switch Fabric on P0 VITA 46 VME and/or Switch Fabrics 3U & 6U IEEE1014 40 MB/s VME…VME64…VME64x…VME2eSST…VXS… 1981 1994 1997 2003 2005 VME…VME64…VME64x…VME2eSST…VXS… 1981 1994 1997 2003 2005 Ethernet…10BASE5…10BASE2…10BASE-T…100BASE-T…1000BASE-X…10GBASE-X… 1982 1983 1984 1990 1994 1998 2002 Ethernet…10BASE5…10BASE2…10BASE-T…100BASE-T…1000BASE-X…10GBASE-X… 1982 1983 1984 1990 1994 1998 2002 MicroTCA

19. EPICS Collaboration Meeting, Vancouver 2009 VME backplane evolution MicroTCA How about a cleaner solution ? MicroTCA

20. EPICS Collaboration Meeting, Vancouver 2009 Kontron OM9140 Emerson OM5080 Schroff MicroTCA crate Elma Blu!Smart MicroTCA

21. EPICS Collaboration Meeting, Vancouver 2009 TechnologyBusBandwidthCPUOSCostVendors VMEParallel + Serial VME64: 80MB/sPowerPCLinux VxWorks RTEMS Crate: $3-7K CPU: $2-5K CPU: Emerson Crates: Wiener, Dawn, Elma, Schroff, Rittal MicroTCASerials (GigE. PCIe, RapidIO) With PCIex4 1GB/s Intel multi core PowerPC Linux Windows WindRiver? $5K (crate, MCH, and CPU board) CPU: Emerson, Kontron, GE Fanuc, more Crates: Dawn, Elma, Schroff, Rittal Compare of VME and MicroTCA Challenges of MicroTCA for accelerator control community: 1)IO modules 2)Software development. MicroTCA

22. Summary SDI status:  LBNL group (Larry Doolittle group) is leading the SDI core design. The communication through fiber links are done.  BNL group is designing the SDI extension at PS control system.  Get BPM data into cell controller by using Libera grouping features is under development.  We are evaluating MicroTCA IOC. It will be test on EPICS/Linux. Then we will try to migrate to EPICS/RTEMS if necessary. EPICS Collaboration Meeting, Vancouver 2009

23. Summary SDI features:  Use the same clock to synchronously distribute data around nodes;  Two directions transmitter and receiver to make it redundant and single node/connection fail safe;  Serial chain link requires minimum wiring;  Protocol independent of carrier/link speed: Rocket IO 2.5Gbps fiber link; 100Mbps Ethernet link; SDI will be an open source design. EPICS Collaboration Meeting, Vancouver 2009

24. Acknowledgement Larry Doolittle (LBNL) Bob Dalesio (BNL) Carlos Serrano (LBNL) Joseph Mead (BNL) Thank you ! EPICS Collaboration Meeting, Vancouver 2009