High Speed Interconnect Project May08-06

Problem Statement At this time, the maximum real-world throughput of 10 Gbps network configurations is unknown.

Need Statement Lockheed Martin (LM) needs a test plan designed and executed to measure the maximum real-world throughput of a 10 Gbps network composed of COTS components.

Project Goals Create and test a network capable of reaching 10 Gbps with Commercial Off The Shelf (COTS) components Topology has to use fiber optics Remain within approx. $3500 budget

Possible Strategies Use Advanced TCA or µTCA Architectures Use PCI Express Network Cards with a XFP Switch Use PCI-X Network Cards with a XFP Switch

µTCA Node Strategy

µTCA Configuration Single Node composed of the following Three 10 Gbps Network Interface Cards MicroTCA Carrier Hub Power module Nodes can be connected in any way possible Testing would be done with a single node due to the cost of components

PCI Express Slot Speeds Implementation Encoded Data Rate Unencoded Data Rate x1 5 Gbps 4 Gbps (500 MB/sec) x4 20 Gbps 16 Gbps (2 GB/sec) x8 40 Gbps 32 Gbps (4 GB/sec) x16 80 Gbps 64 Gbps (8 GB/sec) Source: http://www.dell.com/content/topics/global.aspx/vectors/en/2004_pciexpress?c=us&l=en&s=corp System PCI-E slots must be at least x4 to allow for 10 Gbps testing

PCI Express Node Strategy Node to Node Testing Configuration Final Testing Configuration

PCI Express Configuration Composed of three systems and a Ethernet switch Initial testing will be completed with two systems directly connected Final testing will be done with whole system

PCI-X Configuration Same node strategy as PCI Express Bus speed max of approx. 4.25 Gbps Rarity of cards and slots achieving this bandwidth This configuration would be used only as a backup plan

Feature Comparison of uTCA & PCI-E Advanced TCA and µTCA PCI Express Advantages Modular design allows for expansion 262.5 Gbps maximum throughput for Advanced Mezzanine Cards Disadvantages Network Interface Card Technology for Advanced Mezzanine Cards hasn’t reached 10 Gbps Costly components Advantages Readily available optical 10 Gbps NICs Variety of 10 Gbps XFP Switches Relatively low cost components Disadvantages PCI-E bus bandwidth does not reach 10 Gbps Lack of systems that support PCI-E NICs Source: http://www.compactpci-systems.com/columns/Tutorial/pdfs/4.2005.pdf

System Interface Speed Comparison Maximum Transfer Rate PCI 33-MHz 2.13 Gbps (266 MB/sec) PCI 66-MHz 4.26 Gbps (532 MB/sec) PCI-X 100-MHz 6.4 Gbps (800 MB/sec) PCI-X 133-MHz 8 Gbps (1 GB/sec) PCI-E x1 4 Gbps (500 MB/sec) PCI-E x4 16 Gbps (2 GB/sec) PCI-E x8 32 Gbps (4 GB/sec) PCI-E x16 64 Gbps (8 GB/sec) uTCA (AMC) 262.5 Gbps (32.8 GB/sec) Source: http://www.dell.com/content/topics/global.aspx/vectors/en/2004_pciexpress?c=us&l=en&s=corp

Presented Hardware Setup NXB-10GXxR Intelligent NIC® 10 Gigabit Ethernet PCIe Adapter with pluggable XFP optical interface (http://www.netxen.com/products/boardsolutions/NXB-10GXxR.html) Node 1 Node 2 Node 3 TigerSwitch 10G 8-Port Standalone XFP 10Gigabit Ethernet Managed Layer 2 Switch SMC Networks, Inc.

Host System: Dell PowerEdge SC440 Capable of PCI-E x1/x4/x8 Operating System Linux Approximate Cost: $500/system Separate system needed for each node

NetXen NXB-10GXxR NIC Pluggable XFP optical interface 10GBASE-SR and –LR support PCIe Ver 1.1 Interface x1/x4/x8 compatible 4 GB/s throughput Linux and Window OS supported Source: NetXen website http://www.netxen.com/products/boardsolutions/NXB-10GXxR.html

XFP – 10 Gigabit Small Form Factor Pluggable Used for 10 Gbps connections Operate at 850 nm (short range), 1310 nm (long range), or 1550nm (extended range).

TigerSwitch™ 10G Standalone 8-port XFP 10G Managed Layer 2 switch Model Number: SMC8708L2 Supports up to 8 XFP ports Delivers 10-Gigabit Ethernet Switching fabric – 160Gbps AC Input – 100 to 240 V, 50 – 60 Hz, 2 A

Estimated Resource Costs Quantity Unit Cost Total Cost Optical NICs 2 $1000 $2000 1 Donated1 XFP Switch Donated Fiber optic cables 3 $802 $240 Host System $500 $15003 Total $3740 1 One optical NIC will need to be donated if the team must purchase the host systems 2 Source: http://www.myri.com/Myri-10G/product_list.html#cables-SR 3 ISU ECpE Department’s update of the Senior Design lab may cover this cost

Testing Specifications Bandwidth Measurement Channel Capacity Bandwidth Efficiency (Throughput) Switching Time Measurement Latency Measurement Quality of Service Measurement Load Balancing Measurement

Bandwidth Measurement Node 1 Node 2 What is the maximum Channel Capacity between two nodes? What is the effective bandwidth efficiency or amount of information transmitted versus overhead (frame formatting, coding, etc.)?

Switching Time Measurement Node 1 Node 2 Node 3 How quickly can the source for information to Node 3 be switched?

Latency Measurement Node 3 Node 1 Node 2 What is the time delay between output on Node 1 or Node 2 to input on Node 3?

Quality of Service Measurement Node 1 Node 2 Node 3 If two Nodes are transmitting to the same Node how is bandwidth allocation or QOS handled?

Load Balancing Measurement Node 1 Node 2 Node 3 If two Nodes are transmitting to the same Node how is bandwidth balanced between the two sources?

Questions? ?