Web100 Basil Irwin National Center for Atmospheric Research Matt Mathis Pittsburgh Supercomputing Center Halloween, 2000

NCAR National Center for Atmospheric Research 2 The Problem

NCAR National Center for Atmospheric Research 3 Problem Explanation A major TCP design feature is to compensate for packet/ack latency by sending and buffering multiple consecutive unacknowledged packets Packets are buffered for possible retransmission until acknowledgement is received, so the maximum size of the TCP transmit buffer determines the maximum amount of data that can be retransmitted prior to acknowledgement

NCAR National Center for Atmospheric Research 4 Problem Explanation (cont.) Transmission will stop if the TCP transmit buffer fills up because none of the transmitted packets have been acknowledged To utilized the full bandwidth of a given connection, packets need to be transmitted continuously, i.e., packet transmission must never stop Therefore, the TCP transmit buffer must be large enough to avoid completely filling up with unacknowledged packets

NCAR National Center for Atmospheric Research 5 Problem Explanation (cont.) To avoid “premature” filling, the TCP transmit buffer must be at least the number of bytes equal to the product of the round trip latency and the bandwidth of the connection The problem is that the TCP transmit buffer size is usually defaulted by the application to the kernel default value, and the kernel default is usually sized for low bandwidth LAN connections with very small latency

NCAR National Center for Atmospheric Research 6 Problem Explanation (cont.) Also, most apps don’t provide a user-parameter to override the default value And even if they did, how does one “know” what the bandwidth of the connection is? So the problem is that most TCP apps repeatedly halt data transmission due to mini-sized TCP transmit buffers, and they don’t provide a way for the user to increase the buffer size, and even if they did, the user doesn’t know what size to use

NCAR National Center for Atmospheric Research 7 A Solution

NCAR National Center for Atmospheric Research 8 Solution Explanation Have the OS/kernel transparently, automatically, and optimally set the TCP transmit buffer size (“autotuning”) for the bandwidth and latency conditions detected at all times during every TCP connection based on congestion feedback information available from the TCP stack This is easier said than done, and it is what Web100 is all about

NCAR National Center for Atmospheric Research 9 What Web100 Is Do “autotuning” via the following method: Implement per-connection TCP MIB in kernel Similar to UNIX netstat information, but per- TCP-connection and more useful information Write-variables will allow user-level TCP- connection performance tuning based on real- time congestion feedback from TCP connection Also allows multitude of user-level display and diagnostic tools regarding TCP behavior during real-time

NCAR National Center for Atmospheric Research 10 The Prototype

NCAR National Center for Atmospheric Research 11 Implementation Information ~1,200 diff lines against Linux API is through /proc pseudo-file system About 80 defined variables right now All counters are cumulative (soon to include statistical measures as well) Counters updated continuously in kernel; /proc updates each time accessed One instance of data structure for each TCP session in /proc

NCAR National Center for Atmospheric Research 12 The Coding Team Matt Mathis at PSC is principal designer John Heffner at PSC is currently the main coder Andy Adams at PSC is working on prototype autotuning daemon John Estabrook at NCSA is working on prototype GUI applications (Jeff Semke formerly at PSC was lead software engineer and programmer)

NCAR National Center for Atmospheric Research ESTABLISHED PktsIn 1974 | PktsOut 1951 | Enabled: DataPktsIn 972 | DataPktsOut 1002 | SACK N AckPktsIn 1975 | AckPktsOut 949 | ECN N DataBytesIn | DataBytesOut | Timestamps N DupAcksIn 0 | PktsRetran 0 | | BytesRetran 0 | loss episodes 0 | cwnd | winscale rcvd 0 timeouts 0 | max cwnd | rwin rcvd TO after FR 0 | ssthresh 0 | max rwin rcvd | min ssthresh 0 | winscale sent 0 | max ssthresh 0 | rwin sent | | max rwin sent rto (ms) 20 | rtt (ms) 1 | mss 1448 | Rate min rto (ms) 20 | min rtt (ms) 0 | min mss 1448 | Out (kbps) 0.1 max rto (ms) 20 | max rtt (ms) 1 | max mss 1448 | In (kbps) Overall rate-controlling effects (only valid if we are the sender): aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa Receiver:(S)topped,(A)pp,(B)ufsize / Path:(C)ongestion / Sender:(b)ufsize,(a)pp Sample Demo Output

NCAR National Center for Atmospheric Research 14 Demo Code Release

NCAR National Center for Atmospheric Research 15 Timetable Alpha release to made today (Halloween)! * Very limited set of handpicked testers * To test install procedures * To test API and API documentation * To provide bug and feature feedback Additional alpha testers are possible, but must provide a test plan Beta release date unknown * Code is in an extremely early state * Project just received funding from NSF on 9/15

NCAR National Center for Atmospheric Research 16 Web100 Project Elements

NCAR National Center for Atmospheric Research 17 Support

NCAR National Center for Atmospheric Research 18 Develop Support Procedures Distribution of source, binary, etc. Feedback from users to developers Website development Answer questions, etc, Much of this is being done by NCSA

NCAR National Center for Atmospheric Research 19 Continued Development

NCAR National Center for Atmospheric Research 20 Code Improvements Continuous kernel, MIB, and feature improvements based on user feedback What we have now is a concept demonstration just to start the feedback process Expect several iterations Projected work includes OS-independent library-based API, autotuning the receive buffer, 64-bit counters, statistical variables

NCAR National Center for Atmospheric Research 21 Autotuning Use the writeable variables to develop user- level dynamic optimal maximum TCP buffer- size algorithms based on TCP congestion- feedback variables

NCAR National Center for Atmospheric Research 22 Tool Development Assist tool developers, collect, package, and distribute tools

NCAR National Center for Atmospheric Research 23 TCP-MIB Development

NCAR National Center for Atmospheric Research 24 TCP-MIB Define, publish, and shepherd new TCP-MIB through the MIB standards process

NCAR National Center for Atmospheric Research 25 Vendor Acceptance

NCAR National Center for Atmospheric Research 26 Work with Vendors Extremely important commercial OS vendors pick this code up as quickly as possible Integral part of Web100 proposal Will work closely with vendors to achieve this Includes working with Linux community to have Web100 code included in standard Linux kernel releases

NCAR National Center for Atmospheric Research 27 Linux Distribution Work with Linux distributors to pick up Web100 tool suite Perhaps publish Web100/NSF Linux release based on one or more common commercial releases, perhaps including other work from other NLANR/I2 development projects

NCAR National Center for Atmospheric Research 28 URLs

NCAR National Center for Atmospheric Research 29 URLs

NCAR National Center for Atmospheric Research 30 Project Status

NCAR National Center for Atmospheric Research 31 Pending NSF Funding Major funding for Web100 has been provided from NSF/CISE/ANIR Web100 also has received $100,000 grant from Cisco Systems.

NCAR National Center for Atmospheric Research 32 Partners National Center for Atmospheric Research Pittsburgh Supercomputer Center National Center for Supercomputing Applications

NCAR National Center for Atmospheric Research 33 Discussion and Feedback!