1. September 2.001

2. ENGINEERING ETHERNET BASED SOLUTIONS This presentation is based in GTD experience integrating control and supervision architectures for Science, Space, Defense and Industrial applications. By the Software Support Team (GTD St.Genis) and the Cryogenics Team (GTD BCN) Projects with relevant GTD involvement: XMM-Newton Space Telescope, Eurofighter, Arianne V launcher, Spaceport Control-room and Accelerators (Grenoble and CERN)

3. ENGINEERING ETHERNET BASED SOLUTIONS By the Software Support Team (GTD St.Genis) and the Cryogenics Team (GTD BCN) 1)SOLUTIONS: … to a problem. This presentation discuss suitable problems for ETHERNET application. 2) ENGINEERING: No general solution (at least for the most relevant problems). Criteria to create suitable solutions, identify major constraints and defined the proper integration strategy are also introduced.

4. PROBLEM OVERVIEW 1.1 Displaying Large amount of data in human real-time scale (i.e.: seconds) Registering Non real-time large amount of data with high time-stamp accuracy … … … Processing/Control Variable amount of data in process real-time ** scale (1ms – 500ms) * Data Supply for: - Displaying - Registering … … … *) Ranges for industrial (PLC based) control systems in GTD **)REAL-TIME: known, accurate - i.e. 1ms - & controllable Acquisition Variable amount of data in process real-time scale ** (1ms – 500ms) * Data Supply for: - Processing/Control - Displaying - Registering … … … Requesting Reduced amount of data in human time scale (seconds) Configuring/Backing-up Large amount of data non real-time event-driven (i.e. start-up) … … … Commanding Variable amount of data in process real-time ** scale (1ms – 500ms) * … … … Actuation Variable amount of data in process real-time ** scale (1ms – 500ms) * … … … the Process the Control the Supervision

5. Synchronous / Time[ms] Synchronous / Time[s] Asynchronous requests commands records: events, trends, … sub-sampling status data supply the Process the Control the Supervision PROBLEM OVERVIEW 1.2

6. synchronous asynchronous Max Data Quantity Time Schedule Max Data Quantity Time Limits data Real-Time Non Real-Time Bw1Bw iBw j Bw n Application Domain of Ethernet as Fieldbus PROBLEM OVERVIEW 1.3 dataTIME acquisitionprocessing known & fixed T ± T

7. THE ENGINEERING APPROACH THE CONCEPT 2.1 PROBLEM URD … TOPOLOGY ARCHITECTURE ADD For Processes affecting multiple devices: Nature (synchronous/asynchronous) Real Time Constraints Bandwidth (Bw) FUNCTIONALITIES  validated 1 on 1 ETHERNET approach  + complexity + performance + new functionalities + data availability*/richness + flexibility may block (partially or totally) Ethernet applicability

8. Synchronous (1 (typically Closed Control Loops signals – “feedback” and “output”) Real Time Requirements 0< Tx <10ms Tiny Bandwidth Limited (Field)bus participants  Local Processing (eventually by distributing processing effort) 10ms< Tx <50ms Limited Bandwidth Limited (Filed)bus participants  Deterministic Fieldbus Device’s efficiency for TCP/IP packaging is limited under 50ms 50ms < Tx  Ethernet Applicable (Bw occupation 1:10) for “determinisms” New topologies (system redesign!) Greater Bw Packets oriented transmission for Bw optimization (Protocol “Overhead”)  Fieldbus still Applicable Less Bw available Less flexible topologies (several distance constraints) 1) Synchronous: Deterministic time control on processes concurrently running at different devices. THE ENGINEERING APPROACH THE CONCEPT 2.2

9. Asynchronous Real Time Requirements 0< Tx <10ms [i.e. Open Loop Control – Alarm Handling, Time-stamping, …] Tiny Bandwidth Limited (Field)bus participants * Deterministic Fieldbus > Polling / Packages > Slow!  Local Processing (eventually by distributing processing effort) 10ms< Tx <50ms [i.e. Open Loop Information refresh] * “Messages” better than “token” * Token > Polling / Packages limited efficiency (overhead)  TCP/IP with PROTOCOL + Tx Policy 50ms < Tx[i.e. (very) large systems for info refresh, buffered data, configurations’ up/download, …] * fully package oriented  Ethernet Best Choice (Bw occupation 1:10) for “confidence” New topologies (system redesign!) Greater Bw  TCP/IP with PROTOCOL + Tx Policy THE ENGINEERING APPROACH THE CONCEPT 2.3

10. Real Time Requirements 10ms100ms50ms0 SYNCHRONOUS ASYNCHRONOUS for Increasing Bandwidth ETHERNET TCP/IP Deterministic FieldBus suitability Recommendable … at the limit Not recommendable THE ENGINEERING APPROACH THE CONCEPT 2.4

11. THE ENGINEERING APPROACH ENGINEERING A SOLUTION 3.1 ETHERNET TCP/IP 10MBits 100MBits Industrial Control & I/O Devices But … … … What is the effective Bandwidth occupation? What are the constraints? How shall the information exchange be organized?

12. Ethernet (IEEE 802.3) TCP/IP Middleware Services 1 2 3 4 5 6 7 THE ENGINEERING APPROACH ENGINEERING A SOLUTION 3.2 Physical Data Link Network Transport Session Presentation Application Layer High Level Protocols Information Organization APPLICATION throughput More data managed (maybe not more knowledge!) ENGINEERING: Superprotocol + BW management Some constraints: somehow inherited structures Some immaturity adding TCP/IP services to certain PLC’s Operating Systems. Some immaturity degrees in the firmware connected to proprietary architectures. (is it becoming a PLC weakness?!?) hardware

13. THE ENGINEERING APPROACH ENGINEERING A SOLUTION A case study... the Control System for the LHC Cryogenics 3.3 supervision layer control layer field layer FIRST IDEA... Derived literally from Specification Synchronous Time Stamping 10ms <9.000 Binaries ( 275.000 Binar.) Visualization Synchronous ~ 2s <50.000 Binaries <22.000 Floats ( 700.000 Binar.) ( 300.000 Floats) Synchronous Process Ts=500ms (150ms) < 9.000 Binaries <15.000 Floats ( 275.000 Binar.) ( 105.000 Floats.) Events Asynchronous <acceptable delay from 700.000 Binaries Max 500 events/s ( 4.000 events/s) Synchronous Process Ts=500ms (100ms) < 3.500 Binaries < 3.500 Floats ( 155.000 Binar.) ( 160.000 Floats.) Asynchronous Manual Requests ~ 1s Human Feeling < 3.500 Binaries ( 155.000 Binar.) 1 3 Bw~8MBits Bw~10,5MBits 2

14. THE ENGINEERING APPROACH ENGINEERING A SOLUTION A case study... the Control System for the LHC Cryogenics 3.4 IMPLEMENTATION (as it is today!) Visualization Asynchronous (worst case 2s) <50.000 Binaries <22.000 Floats ( 700.000 Binar.) ( 300.000 Floats) Process Asynchronous (worst case 150ms) < 9.000 Binaries < 5.000 Floats ( 275.000 Binar.) ( 105.000 Floats.) Events Asynchronous <acceptable delay from 700.000 Binaries Max 500 events/s ( 4.000 events/s) Process Asynchronous (worst case 100ms) < 3.000 Binaries < 2.000 Floats ( 155.000 Binar.) ( 160.000 Floats.) Asynchronous Manual Requests ~ 1s Human Feeling < 3.500 Binaries ( 155.000 Binar.) TIME STAMPING X 165 X 64 X 8 (redundant) 2 x TCP/IP cards * * * * 1 2 3 4 SAMPLING

15. 3.5 THE ENGINEERING APPROACH ENGINEERING A SOLUTION A case study... the Control System for the LHC Cryogenics summarizing the chosen approach … 1)Distribution of the Time-stamping process. Time-stamping at the source eliminates synchronous communication. >>> It is feasible to move to a full ETHERNET TCP/IP solution >>> Requires specific Supervision system 2)Duplication of networks at the Control Layer to make synchronous exchanges “deterministic” and/or reduced collisions (moreover, dimensioning of the number of devices) >>> Measure performances of the Hardware 3)Eliminate unnecessary throughputs by directly addressing the information item to the target (favored by the flexible Ethernet architecture) 4) Bandwidth Management: Synchronous >>> Asynchronous! (further explained)

16. 3.6 TCP/IP 100MBits Open Modbus CPS211 00 CPU534 14 NOE771 00 Percentage of the PLC cycle time occupied by communications related tasks [%] for n parallel ports/channels and different PLC cycles. Performance (Bw)[Kbit/s] for n parallel managed communication’s ports/channels. Curves represent different PLC cycles (40ms.. 100ms) THE ENGINEERING APPROACH ENGINEERING A SOLUTION A case study... the Control System for the LHC Cryogenics HARDWARE evaluation (example … at the Control Layer)

17. 3.7 Duty Cycle / Band Width ratio for different PLC Cycles. Duty Cycle / Band Width ratio for different number of parallel opened Ports/channels (MASTERS) Good conditions found for 16 parallel managed ports/channels in a 50ms PLC cycle, thus resulting in approx. 256 Kbits/s of bandwidth and less than 20% PLC cycle occupation due to communication’s management (approx. 10ms). THE ENGINEERING APPROACH ENGINEERING A SOLUTION A case study... the Control System for the LHC Cryogenics HARDWARE evaluation (example … at the Control Layer)

18. Event Driven Protocols / Tx Policies: >>> Oriented to Asynchronous strategies a) People use to neglect “worst case” conditions b) Steady conditions shall be carefully understood >>> Event Driven use to be optimized for reduced pieces of INFO Worst conditions use to mean: large quantities! >>> Optimization is achieved by the probabilistic fact of “little changes” “worst case” and “close to worst case” are not negligible >>> Oriented to Synchronous strategies It is feasible by: a) Dimensioning for the worst case b) Sampling to regularize data flow >>> What is the advantage? Bandwidth Management a) To support concurrent, less demanding, Asynchronous communication processes. b) To reduce collision probability in steady conditions 3.8 THE ENGINEERING APPROACH ENGINEERING A SOLUTION A case study... the Control System for the LHC Cryogenics PROTOCOL & BANDWIDTH MANAGEMENT Ti Tk Ts process Ts

19. THE ENGINEERING APPROACH ENGINEERING A SOLUTION A case study... the Control System for the LHC Cryogenics 3.9 IMPLEMENTATION (as it is today!) Maximum 8 Control Units (PCUs) for each DS Maximum 4 Field Interfaces (FIs) for each PCU 8 Redundant Data Servers (Bi-Pentium) Events Visual Trends Requests Events Visual Requests Process Events Visual Trends Process Requests SAMPLING TIME STAMPING Example PCU: Quantum 534 14 + 2x NOE 771 00 Event Driven (by GTD) on Open Modbus Steady “worst conditions” (= Polling) Bw = 256 Kbit/s available - needed for Superv. 200KBit/s - needed for Process.160KBit/s Better than “worst conditions” - Bandwidth availability for true Asynchronous processes.

20. ENGINEERING ETHERNET BASED SOLUTIONS conclusions 1. Synchronous-Asynchronous 2. Data Quantity 3. Time Frame Our recommendation: Try to use Ethernet for 1. Synchronous, over 75ms 2. Asynchronous, over 25ms Still difficulties: 1. Certain immaturities 2. Inherited standards 3. Large/Complex applications (more than necessary !?!) > Engineering Effort a) To adapt resources b) To create Tx policies c) To benefit from Ethernet added values. Case Study: Characterized by … 1. Application “size” 2. Information flow 3. Synchronous/Asynchronous Ethernet TCP/IP not well suited for synchronous, fast processes

21. September 2.001