High Speed Optical Interconnect Project May08-06

Client: Lockheed Martin Team Information Client: Lockheed Martin Team Members Team Leader: Adam Jackson Communication Coordinator: Nick Ryan Bader Al-Sabah David Feely Richard Jones Faculty Advisor Dr. Ahmed Kamal Client Contacts Aaron Cordes Rick Stevens

Problem Statement At this time, the maximum real-world throughput of 10 Gbps network configurations is unknown.

Need Statement Lockheed Martin (LM) needs a test plan designed and executed to measure the maximum real-world throughput of a 10 Gbps network composed of Commercial Off the Shelf (COTS) components.

Project Goals Create and test a network capable of reaching 10 Gbps with COTS components Topology has to use fiber optics Remain within approx. $3500 budget

Possible Strategies PCI Express (PCI-E) Network Cards with an XFP Switch PCI Extended (PCI-X) Network Cards with an XFP Switch Advanced TCA or MicroTCA (µTCA) Architectures

PCI-E Node-to-Node Configuration Testing will be completed with two systems directly connected Used for testing bandwidth and bandwidth efficiency Graphic inspired by previous HSOI team

PCI-E Switched Configuration Composed of three nodes and a Ethernet switch Used for testing switching time, latency, and quality of service Graphic inspired by previous HSOI team

PCI-X Configuration Same node strategy as PCI-E Bus speed max of approximately 8 Gbps Client requirement of 10 Gbps makes this an unfeasible solution

µTCA Configuration Testing should be done with a single node due to the high cost of components Single Node composed of the following Three 10 Gbps Network Interface Cards µTCA Carrier Hub Power module Control Processor Switching Fabric Nodes can be connected in various ways

µTCA Node Strategy Diagram courtesy of LMCO

Feature Comparison of µTCA & PCI-E Advantages Modular design allows for expansion 262.5 Gbps maximum throughput for Advanced Mezzanine Cards (AMC) Disadvantages AMC Network Interface Cards at 10 Gbps are not readily available Costly components Advantages Readily available optical 10 Gbps NICs Variety of 10 Gbps XFP Switches Relatively low cost components Disadvantages Lack of PCI-E systems at ISU Source: http://www.compactpci-systems.com/columns/Tutorial/pdfs/4.2005.pdf

System Interface Speed Comparison Maximum Transfer Rate PCI-X 100-MHz 6.4 Gbps (800 MB/sec) PCI-X 133-MHz 8 Gbps (1 GB/sec) PCI-E x1 4 Gbps (500 MB/sec) PCI-E x4 16 Gbps (2 GB/sec) PCI-E x8 32 Gbps (4 GB/sec) PCI-E x16 64 Gbps (8 GB/sec) uTCA (AMC) 262.5 Gbps (32.8 GB/sec) Source: http://www.dell.com/content/topics/global.aspx/vectors/en/2004_pciexpress?c=us&l=en&s=corp

Decision: PCI-E (x4 or better) µTCA will not fit into budget µTCA components may not be available in time PCI-X is not fast enough

Host System: Dell PowerEdge SC440 Capable of PCI-E x1/x4/x8 Operating System Dual-Boot (Windows, Linux) Approximate Cost: $500/system Separate system needed for each node

NetXen NXB-10GXxR NIC Pluggable XFP optical interface 10GBASE-SR and –LR support PCI-E Ver. 1.1 Interface x1/x4/x8 compatible 32 Gbps throughput Linux and Windows OS supported Source: NetXen website http://www.netxen.com/products/boardsolutions/NXB-10GXxR.html

TigerSwitch™ 10G Standalone 8-port XFP 10G Managed Layer 2 switch Model Number: SMC8708L2 Supports up to 8 XFP ports Delivers 10-Gigabit Ethernet Switching fabric – 160Gbps AC Input – 100 to 240 V, 50 – 60 Hz, 2 A http://www.pcworld.com/product/pricing/prtprdid,9311286-sortby,retailer/pricing.html

XFP – 10 Gigabit Small Form Factor Pluggable SMC10GXFP-SR TigerAccess™ XFP 10G Transceiver 1-Port 10GBASE-SR (LC) XFP Transceiver Used for 10 Gbps connections http://ecx.images-amazon.com/images/I/11jHA98YEEL._AA160_.jpg

Presented Hardware Setup (As recommended by Lockheed Martin.) NXB-10GXxR Intelligent NIC® 10 Gigabit Ethernet PCIe Adapter with pluggable XFP optical interface (http://www.netxen.com/products/boardsolutions/NXB-10GXxR.html) Node 1 Node 2 Node 3 TigerSwitch 10G 8-Port Standalone XFP 10Gigabit Ethernet Managed Layer 2 Switch SMC Networks, Inc.

Availability Purchased by Team Purchased by Client Provided by Department NICs 2-3 0-1 - Switch 1 XFP Transceiver (for use on switch) Optical Cabling As needed Available, details unknown Computer Systems If necessary and available in budget Supplied to senior design lab

Estimated Resource Costs Quantity Unit Cost Total Cost Optical NICs 2 $1000 $2000 1 On loan from client1 XFP Switch $6500 On loan from client XFP Transceiver 3 $1880 Fiber optic cables $80 $240 Host System $500 $15002 Total $3740 1 One optical NIC will need to be borrowed if the team must purchase the host systems 2 ISU ECpE Department’s update of the Senior Design lab may cover this cost

Network Testing Software Qcheck Packet generation program Can be used to test bandwidth, bandwidth efficiency, and latency Ethereal Packet capture program Can be used for bandwidth efficiency testing IP Traffic Test & Measure Network testing suite Can be used for quality of service and latency testing

Test Equipment Testing will be predominantly software based, the test bench will be executed on the computer system described previously. If issues arise and the signal needs to be observed, an Agilent 86100A oscilloscope is available from the department. Availability of splitters is unknown.

Performance Metrics Bandwidth Measurement Switching Time Measurement Channel Capacity Bandwidth Efficiency (Throughput) Switching Time Measurement Latency Measurement Quality of Service Measurement

General Test Plans Execute each test multiple times to ensure precise results Provide appropriate statistics from results Use both UDP and TCP protocols when possible Vary data size to avoid skewed results due to packet header overhead

Test Results Bandwidth Compare link usage for each node under varying workload types Bandwidth Efficiency Show a comparison of the amount of OSI Layer 1 data sent for different OSI Layer 7 data block sizes Switching Time Compare switching time and link load for cases when 2 and 3 nodes are connected to the network

Test Results Latency Compare the latency between nodes under different network loads Quality of Service Show the amount data received from each sending node for each endpoint node over time

Questions? ?