Visible Human Networking Update Thomas Hacker June 3, 2000

Outline Latest News U-M Network Connectivity Existing CPC Network Topology Proposed CPC Network Topology Performance Improvement Program NPACI Storage Resource Broker Performance Goals

Network Status In the works … –CPC Upgrade from OC-3 155Mbps to OC Mbps –CPC Router Purchase OC-12 line installed from Media Union to Arbor Lakes, needs some tuning to improve performance

VBNS UM BIN FDDI ITD FDDI CAEN DOE NSI DREN Abilene X MERIT BIN OCR MCIT Mich Giga PoP OC Mb/sec gigabit ethernet CPC X 5 OC-12 FW m 17m Mich NET GW Quin Cleveland POP VBNS/Abilene Research Commercial/Net FW University of Michigan Network Topology Firewalls

2 Processor 2 GB Mem New ADSM/Tivoli Mass Storage Sever IBM F-50 NPACI AFS Storage Server IBM F-50 Existing CPC High Performance Network Topology 5/2000 CPC IBM SP2 Complex (48) 66 MHz Nodes Switchless (Training Nodes) (48) 166 MHz Nodes (Production 1 GB Mem Nodes) 4 Processor 2 GB Mem Network Performance Measurement Station IBM F-50 4 Processor 2 GB Multi-Resident/ Resident AFS Mass Storage Server IBM F-50 2 Processor 2 GB Mem New Nodes SP2 Control Workstation IBM F-50 2 Processor 1 GB Mem New ADSM/Tivoli Robotic Tape Controller IBM F-50 SGI 12 GB, 16 Processors SGI Origin 2000 Switched 100 BaseT Lines 48 Switched 10 BaseT Lines 4 EtherChannel Bonded Switched 100 BaseT Lines Switched 10 BaseT Existing ADSM Mass Storage Server 1 Processor 1 GB Mem New ADSM/Tivoli Robotic Tape Controller IBM F-50 SGI Powerchallenge 4 GB, 16 Processors IBM NPACI Globus Research Server Existing ADSM Robotic Tape Controller 1 Processor 512 Mb Mem CPC-CISCO 5500 mr CPC CISCO 5500 Ethernet Switch UMMU-cis LINUX Cluster Firewall CPC BEOWULF LINUX Cluster 48 port 10BaseT Hub BaseT Lines 32 Switched 100BaseT Lines SP2 Control Workstation SP2 Build/ Submit Node UMMU-CIS 1010 Media Union ATM Switch CISCO Lightstream 1010 Chry-CIS 1010 Chrysler ATM Switch CISCO Lightstream 1010 ATM OC Mb/sec OC-12 To Abilene 622 Mb/sec Switched 10 BaseT 2 Ether-Channel Bonded Switched 100 BaseT Lines Switched 100 BaseT Line CISCO 2900XL 10/100 Switch 48 Switched 100 BaseT Lines 100 Mb/sec FDDI OC-3 ATM 155 Mb/Sec Switched 100 BaseT CISCO CDDI/FDDI Concentrator 100 Mb/sec FDDI 144 GB Local Disk OC-3 ATM 155 Mb/Sec Dual Ppro 200 MHz 128 Mb RAM per node 2 Processor 2 GB Mem New Tivoli/ADSM Backend Server + Engineering Research Host Backup IBM H-50 (64) 166 MHz Nodes (SDSC) June 2000 (16) 166 MHz Nodes (SDSC) June 2000 Switched 100 BaseT IBM GB CD-R Juke Box FWD SCSI DIFF SCSI STK Timberwolf TB 216 GB Dual Channel RAID 1 FWD SCSI 100 GB ADSM Data Vole LVD SCSI FWD SCSI CPC Beowulf Complex 100 Mb/sec FDDI UMMU-CIS GB RAID 1 LVD SCSI LVD SCS I 200 GB Backup/Archive

4 Processor 2 GB Mem Network Performance Measurement Station IBM F-50 Proposed CPC High Performance Network Topology 2 Processor 2 GB Mem New Tivoli/ADSM Backend Server + Engineering Research Host Backup IBM H-50 OC-48 OC-12 Gigabit Ethernet CISCO 6509 Router ADSM/Tivoli Mass Storage System (42 TB) 4 Processor 2 GB NPACI AFS Server IBM F-50 ADSM/Tivoli Robotic Tape Controller 2 Processor 1 GB IBM F-50 Media Union ATM Router OC Mb/sec Engineering ATM Cloud - or - To ITD- Cooley ATM Switch Arbor Lakes ATM Switch Abilene 1000 Mb/sec Gigabit Ethernet SGI Origin EtherChannel Bonded 100 Mb/sec Switched Ethernet 1000 Mb/sec Gigabit Ethernet UMVH SGI OCTANE Webserver OC Mb/sec OC Mb/sec 1000 Mb/sec Gigabit Ethernet Media Union SGI Onyx2 1,000 Mb/sec Gigabit Ethernet 1,000 Mb/sec Gigabit Ethernet 1,000 Mb/sec Gigabit Ethernet 1000 Mb/sec Gigabit Ethernet CPC IBM SP2 Complex (48) 66 MHz Nodes Switchless (Training Nodes) (48) 166 MHz Nodes (Production 1GB Mem Nodes) (64) 166 MHz Nodes (SDSC) June 2000 Switched 10 BaseT IBM F-50 SP2 Control Workstation SP2 Build/ Submit Node Switched 10 BaseT FWD SCSI Storage Technology Timberwolf 9710 DLT Tape Silo 41 TB 1,000 Mb/sec Gigabit Ethernet 1000 Mb/sec Gigabit Ethernet Nodes (16) 166 MHz Nodes (SDSC) June 2000 Switched 100 BaseT 588 DLT7000 Tape Cartridges 6 DLT7000 Tape Drives 36GB AFS Disk FW SCSI SGI 4 GB, 16 Processors SGI Origin 2000 SGI Power Challenge Server SGI 12 GB, 16 Processors SGI Origin 2000 FWD SCSI 216 GB Dual Channel RAID GB ADSM Data Vol LVD SCSI Dual Ppro 200 Mhz 128 Mb RAM per node LINUX Cluster Firewall CPC BEOWULF LINUX Cluster 48 port 10 baseT Hub CISCO 2900XL 10/100 Switch CPC Beowulf Complex 32 Switched 100BaseT Lines UMVH Production Prototype Compaq Cluster 48 Switched 10BaseT 48 Switched 100BaseT CISCO 5500 Ethernet Switch Switched 100BaseT Lines 1000 Mb/sec Gigabit Ethernet 2 Processor 2 GB Mem SDSC Nodes SP2 Control Workstation IBM F-50 4 Processor 2 GB Multi-Resident/ Resident AFS Mass Storage Server IBM F-50 2 EtherChannel Bonded 100 Mb/sec Switched Ethernet 1000 Mb/sec Gigabit Ethernet 1440 GB Sun A1000 Disk FWD SCSI 200 GB Backup/Archive LVD SCSI 144 GB RAID 1 LVD SCSI

High Performance Networking Upgrade Program Application:Training, ImBench performance testing System: Network: NPACI Storage Resource Broker/ Reagan Moore – Data Intensive Computing Environment Thrust Area University of Texas/University of Michigan SP2 system integration collaboration AFS file system, joint job scheduling, Engineering Materials Database sharing Multi-resident AFS wide area mass storage file systems (AFS server that utilizes mass storage system) Job scheduling/moving large multi-gigabyte files between NPACI and National HPC sites/Grid computing infrastructure (Globus, Legion, Linux Clusters) NPACI thrust areas: Engineering/Education, Outreach and Training/ ISR-Social Science Visible Human/NPACI Neurology Collaboration/ Brian U-M; Human Cochlear Modeling, Ed U-M; Mark UCSD, Platform Applications Network performance testing & tuning Move from 10/100 G.E./ATM Move from OC-3 OC-12 Current Applications That Will Immediately Benefit From Upgrade Program Approach Upgrade physical infrastructure Optimize facilities and operations use by performance tuning & host network adapter upgrade User training & consulting to optimize application utilization of network (Matt Mathis–PSC/NLAR TCP testrig)

1st Read Application Data Transfer DataTransfer.ppt File System Disk B/W 1st File System Read 2nd File System Read bcopy Memory Read B/W Application Network Interface Card TCP/IP B/W Time for Application to Process Data Bi-Directional B/W Route Stability “Bi-Directional” Latency (RTT) Path Length (Hops) Internet Characteristics Application Network Interface Card Network Socket TCP/IP Write Virtual Memory B/W Write File System BW File System Dependent on File System “dirty page” Policy Cached File System Read B/W Router Time for Application to Process Data Link Characteristics – Link B/W – Link RTT – Link Packet Loss Virtual Memory Measure TCP/IP Loopback Network B/W Measure TCP/IP Loopback Network B/W

Networking Goals Get new router and tuned OC-12 connection in place Move critical servers and HPC systems to gigabit ethernet Develop system, network and internet performance tuning toolset and expertise to optimize use of resources