Interposed Request Routing for Scalable Network Storage Darrell Anderson, Jeff Chase, and Amin Vahdat Department of Computer Science Duke University

Duke University  Department of Computer Science Goals Devise a highly scalable network storage architecture Interpose on a standard file system protocol. –Prototype supports NFS version 3. Distribute responsibilities and data. –Divide functions (e.g., data vs. metadata). –Scale functions by aggregating servers. This talk: Request routing to scale functions.

Duke University  Department of Computer Science In the Beginning... NFS ClientNFS Server Network Client sends and receives standard NFS packets. Server sends and receives standard NFS packets.

Duke University  Department of Computer Science Interposed Routing NFS Client*Server Client sends and receives standard NFS packets. Slice µProxy intercepts and redirects NFS packets to specialized servers. µ*Server

Duke University  Department of Computer Science Outline Interposed routing Slice architecture Functional decomposition Data decomposition Functions Block-I/O Small-file Metadata Request routing Performance

Duke University  Department of Computer Science Slice Architecture file placement policy network storage array small-file servers directory servers name space requests bulk I/O small file read/write name routing striping policy client µproxy

Duke University  Department of Computer Science Functional Decomposition file placement policy network storage array small-file servers directory servers name space requests bulk I/O small file read/write name routing striping policy client µproxy

Duke University  Department of Computer Science Data Decomposition file placement policy network storage array small-file servers directory servers name space requests bulk I/O small file read/write name routing striping policy client µproxy

Duke University  Department of Computer Science Outline Interposed routing Slice architecture Functional decomposition Data decomposition Functions Block-I/O Storage Nodes Small-file Servers Directory Servers Request routing Performance

Duke University  Department of Computer Science Block-I/O Storage Nodes Network storage nodes provide all storage in Slice. Prototype uses a simple object-based model. –Read, write, remove, truncate. Clients access storage nodes directly. –Static striping, or flexible block-maps. –Optional RAID “10” mirrored striping. network storage array bulk I/O striping policy client µproxy

Duke University  Department of Computer Science Handle read and write operations on small files. All I/O requests below threshold (e.g., 64 KB). –Also the initial “small” segments of large files. Absorb and aggregate I/O on small files. –Data backed by storage array. Storage nodes need not handle small files well. Small-File Servers small-file servers file placement policy small file read/write client µproxy network storage array

Duke University  Department of Computer Science Directory Servers Handle name space operations. Associate name with attributes (lookup, getattr). Manage directory contents (create, readdir). –Preserve dependencies between objects. Create affects new object and its parent directory. directory servers name routing policy name space requests client µproxy network storage array

Duke University  Department of Computer Science Outline Interposed routing Slice architecture Functional decomposition Data decomposition Functions Block-I/O Storage Nodes Small-file Servers Directory Servers Request routing Performance

Duke University  Department of Computer Science Request Routing Goals Focus on name space. Spread name space across multiple servers. –Balance capacity and load. (Maybe) keep entries on same server as parent. –Some name space ops involve multiple sites. Create entry, update parent modify time.

Duke University  Department of Computer Science Request Routing Three policies for name space request routing: Volume Partitioning: –Divide the name space into volumes. –Volumes have well defined mount points. Mkdir Switching: –Items on same server as parent directory. –Some mkdirs redirect to another server. Name Hashing: –Name space is a distributed hash table. –Requests hash by name, parent dir.

Duke University  Department of Computer Science Outline Interposed routing Slice architecture Functional decomposition Data decomposition Functions Block-I/O Storage Nodes Small-file Servers Directory Servers Request routing Performance

Duke University  Department of Computer Science Experiment Configuration Hardware Client: 450 MHz P3 with 32 bit 33 MHz PCI. Server: 733 MHz P3 with 64 bit 66 MHz PCI. Server: 8x 18 GB Seagate Ultra-2 Cheetah disks. Gigabit Ethernet with 9 KB “jumbo” frames. Software FreeBSD 4.0-release. Modified NFS stack and firmware for zero-copy. NFS uses UDP/IP with 32 KB MTU. Slice kernel modules; µProxy is IP filter on client.

Duke University  Department of Computer Science Block-I/O Scaling

Duke University  Department of Computer Science Name Space Scaling

Duke University  Department of Computer Science Mkdir Switching Affinity

Duke University  Department of Computer Science SPECsfs97 Throughput

Duke University  Department of Computer Science SPECsfs97 Latency

Duke University  Department of Computer Science Summary Slice interposes between NFS client and server. Simple redirection of NFS version 3 packets. –Slice µProxy inspects and rewrites packets. Separates functions normally for central server. –Functional decomposition for request stream. –Data decomposition to scale each function. Prototype shows performance and scalability.

Duke University  Department of Computer Science EOF

Duke University  Department of Computer Science Handling Failures Approach: write-ahead logging. µProxy logs intentions for “dangerous” operations to coordinator. –Also logs when finished. Coordinator completes or aborts aging operations. –Roll forward, or back. Independent of client, server, and storage nodes. µ CoordinatorNFS Client 4. Safe again 2. Danger! 3. (do it) 1. Request 5. Response