1. www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Reliable Synchronization for Multi-Hop Networks and its Realization in FPGA Wilfried Steiner wilfried.steiner@tttech.com TTTech Computertechnik AG MAPLD 2009 – Session A CoMMiCS

2. Page 2 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Problem Statement 1.Reliable Synchronization 2.Multi-Hop Network Topology 3.TTEthernet - Realization in FPGA 4.Model-Based Development

3. www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Reliable Synchronization for Multi-Hop Networks and its Realization in FPGA

4. Page 4 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Synchronization Strategy Clock Synchronization Service Startup/Restart Service Clock Synchronization Service is executed during normal operation mode to keep the local clocks synchronized to each other. Startup/Restart Service is executed to reach an initial synchronization of the local clocks in the system. Integration/Reintegration Service is used for components to join an already synchronized system. Clique Detection Services are used to detect loss of synchronization and establishment of disjoint sets of synchronized components.

5. Page 5 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Two-Step Clock Synchronization Service For now, let us assume that we operate in a single-hop network:  End Systems operate as Synchronization Masters/Clients.  Switches are configured as Compression Masters. Synchronization Strategy operates in two steps for clock synchronization during normal operation mode. Step1: Synchronization Masters send synchronization messages to Compression Masters. Step2: Compression Masters send synchronization messages back to the Synchronization Masters and Clients.

6. Page 6 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Four-Step Startup/Restart Service Step1: Synchronization Masters send out Coldstart Frames (CS) Step2: Compression Masters forward all CS frames to all ports Step3: Each Synchronization Master that receives a CS frame will answer with a Coldstart Acknowledgement Frame (CA) Step4a: Compression Masters will forward all CA frames to all ports (for multiple failure tolerant configurations). Step4b: Compression Masters will forward only one compressed CA frame (for single failure tolerant configurations).

7. Page 7 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Formal Verification Activities (Synchronization Strategy) TTEthernet Executable Formal Specification  Using symbolic and bounded model checkers sal-smc and sal-bmc  Focus on Interoperation of Synchronization Services (Startup, Restart, Clique Detection, Clique Resolution, abstract Clock Synchronization) Verification of Lower-Level Synchronization Functions  Permanence Function  verified with the infinite-bounded model checker sal-inf-bmc  using disjunctive invariant and k-induction  Compression Function  verified with the infinite-bounded model checker sal-inf-bmc  using abstraction and 1-induction Formal Methods have been applied as early as in the requirements capturing phase Finalization and Completion of the formal assessment within the CoMMiCS Project  Complexity Management for Mixed-Criticality Systems  European Communities FP7 project [FP7/2007-2013] no. 236701] CoMMiCS

8. www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Reliable Synchronization for Multi-Hop Networks and its Realization in FPGA

9. Page 9 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 TTEthernet in a Nutshell  … is all about smart ways to integrate data flows with different characteristics in a single communication infrastructure.  While various groups started with standard Ethernet and aim at enhancing it with real-time and fault tolerance properties, TTTech comes from the opposite direction:  Our know-how is in real-time and fault-tolerant data flows.  We are applying our know-how on top of Ethernet and optimize our concepts with respect to Ethernet features (switches, full-duplex, etc).  In principle we can enable any carrier protocol with time-triggered technology, … … but there are some good reasons for using Ethernet.

10. Page 10 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 The Motivation for Ethernet  Ethernet hardware is low cost.  Ethernet is a well-established open-world standard and very scaleable.  The OSI reference model gives a well-structured classification of concepts that can be built on top of Ethernet.  Existing tools can be leveraged as cost-efficient diagnosis tools.  As all messages in TTEthernet are standard Ethernet compliant, existing tools can be leveraged for time-triggered messages as well.  Standard web servers can be leveraged for maintenance and configuration.  TTEthernet allows hosting of IEEE 1588 clock synchronization clients.  Engineers learn about Ethernet at school. Ethernet compatibility enables the usage of technology that is established, tested, and verified.

11. Page 11 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Different applications have different communication needs: fault, tolerance, real time, high speed, high bandwidth, low latency, low jitter, active redundancy, hot standby, … TTEthernet provides following traffic classes for applications:  Time-Triggered (TT): Bandwidth in TTEthernet networks can be highly utilized due to the possibility of strictly deterministic (vs. probabilistic) traffic scheduling of high-priority periodic traffic.  Rate-constrained (RC): Event-triggered traffic with priority levels (ARINC 664) and transmission guarantees.  Non-critical standard Ethernet (BE): Low priority traffic (e.g. data download) served during network idle times; can also be scheduled explicitly. Dataflow in TTEthernet TTEthernet allows applications with different communication requirements to share a single physical network.

12. Page 12 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 TTEthernet Integrated Dataflow Dataflow – Integration - Time-Triggered (TT) - Rate-Constrained (RC) - Standard Ethernet (BE) Protocol Control Frames (PCFs) are transmitted on the same physical network as dataflow.

13. Page 13 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Integration Problem When two (or more) messages compete for relay to the same outgoing port, the switch (or ES) has to serialize these messages. If there are messages of same priority the messages will be serviced according First-Come First-Served. What happens if there is a Data message (Data) in relay, when a Protocol Control Frame (PCF) becomes ready for relay?  If a Data message is relayed by a switch when a PCF arrives, the PCF message is delayed until the relay process of the Data message is finished.  Hence, in the worst case the PCF is delayed for a maximum sized Data message (plus other queued PCFs).  Protocol Control Frames record their delay through the network.  All components in the network that impose a delay on a PCF will add this delay into a field in the PCFs.

14. Page 14 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Single-Hop Synchronization Flow The (Dispatch, Permanence) – Pair hides the network jitter for PCFs almost entirely. ~constant dynamic

15. Page 15 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Multi-Hop Synchronization Flow Multi-hop requires only a small add-on to the existing mechanism: an intermediate switch will not only consume the PCF, but also forward it.

16. www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Reliable Synchronization for Multi-Hop Networks and its Realization in FPGA

17. Page 17 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Synchronization Master / Client i Protocol Control Frames (PCFs) from the Compression Master flow in from the left. PCFs produced by the Synchronization Master flow out on the right. Filter: removes PCFs that are semantically incorrect (e.g. wrong type, outdated, etc.). Frame Permanence: holds PCFs in buffers until their age equals the configured max. transparent clock value (respecting the transparent clock field as part of their age). Sync State Machine: this module will process all the PCFs that made it past the Frame Permanence module as specified by the synchronization services. In particular this module is in charge of maintaining the local copy of the synchronized cluster time.

18. Page 18 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Synchronization Master / Client ii The size of the Synchronization Unit depends upon others on the number of Synchronization Masters that have to be supported. For sixteen Synchronization Masters:  the complete block (incl. interfaces to MACs) takes ~5,000 flip-flops (including diagnostics) and ~7,000 ALUTs  this is 12% and 15%, respectively, of the total size of a 3 channels GBits/s End System controller in an Altera Stratix III FPGA. For four Synchronization Masters:  the block needs ~2,200 flip-flops and ~3,500 ALUTs.

19. Page 19 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 TTE Evaluation System (2 channels, 1 Gbit/s)

20. Page 20 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 TTE Evaluation System Takes you from first evaluation steps to full-blown prototypes  4 Linux-based PCs with triple channel TTEthernet capability  Optical Gigabit Ethernet (1000Base-X) compliant interfaces  One or more high-performance TTEthernet switches with 4 ports and configuration, monitoring and diagnostic interfaces  Optical cables supplied for easy connection to existing LAN infrastructure or directly to the user’s development PC (needs 1000Base-X interface)  TTEthernet end system software libraries offering an API (application programming interface) to the TTEthernet protocol hardware  TTEthernet tools for network planning, schedule inspection, and configuration  PC-based monitoring tool supports logging/recording for analysis of network traffic and network-based applications  Power supply 110-240 VAC  Example application, documentation

21. Page 21 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 TTEthernet Hardware Products - Summary  TTE Development Switches  TTE Development Switch 1Gbit/s  TTE Development Switch 100Mbit/s  TTE End Systems  TTE End System PCIe  TTE End System PMC  TTE End System XMC  TTE Evaluation Systems  TTE Evaluation System 1Gbit/s  TTE Evaluation System 100 Mbit/s  TTE Test Equipment  TTE Monitoring Switch 1Gbit/s  TTE Monitoring System

22. www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Reliable Synchronization for Multi-Hop Networks and its Realization in FPGA Model-Based Development

23. Page 23 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 TTEthernet Executable Formal Specification i

24. www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Reliable Synchronization for Multi-Hop Networks and its Realization in FPGA Summary & Conclusion

25. Page 25 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 Summary Reliable Synchronization  We showed the two-step clock synchronization service and the four-step startup/restart service and discussed their failure-tolerance capabilities. for Multi-Hop Networks  We discussed the problem of integrating Control-Flow and Dataflow on the same physical network.  We showed that the permanence function is a very simple mean to establish a strong interface with clean semantics for the synchronization services. and its Realization in FPGA  We introduced the conceptual design and discussed the implementation details in an Altera Stratix III FPGA. Model-Based Development  We introduced the Executable Formal Specification and sketched how the worst-case stabilization time of TTEthernet has been calculated.

26. Page 26 www.tttech.comCopyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009 QUESTIONS?