Parallel Virtual File System (PVFS) a.k.a. OrangeFS By Collin Gordon and Lisa Kosiachenko

What is PVFS?

Background information PVFS - parallel, distributed filesystem for Linux clusters Used in High-Performance Computing at large scale (while Hadoop is primarily used in cloud computing) Grew out of research project from 1993 Since 2008 the main development branch is called OrangeFS As of May 2016 release OrangeFS is a part of the Linux kernel!

Advantages of the PVFS Concurrent read/write operations from different processes/threads to a common file PVFS can be used with multiple application programming interfaces (APIs): a native PVFS API, the UNIX/POSIX API, MPI-IO Common UNIX shell commands, such as ls, cp, and rm, work with PVFS files Supports the regular UNIX I/O commands such as read() and write() Existing binaries that use UNIX API can access PVFS files without recompiling

How PVFS works

PVFS System Architecture

File Manager and Metadata Application Process communicates with the manager daemon through TCP only during the following file operations: Open Close Creation Removal Allows application to communicate directly with I/O nodes for read and write operations Implementation problem: No directory access at first, then used NFS, currently have a system that determines whether the requested directory is PVFS and if so, the manager daemon is connected.

File Manager and Metadata Single manager daemon stores all file metadata in the PVFS System File data and metadata stored on local file systems rather than raw devices Metadata components can be user set or default values Metadata components: Base: I/O Node where the file is stored Pcount: Number of I/O Nodes for the file Ssize: Size of each stripe on the I/O Nodes

File Metadata and Striping

I/O Daemons and Data Storage I/O nodes specified at install I/O nodes are not distinct from compute nodes Each I/O node has an ordered set of I/O daemons I/O daemons responsible for using local disk on I/O node to store files Application sends a request for data to I/O daemons which work together to send back the information

I/O Daemons and Data Storage

Rainbow diagram

Trapping UNIX I/O calls System calls are typically made by calling wrapper functions in the standard C library, which in turn pass the parameters to the kernel Straightforward way to trap system calls - provide a separate library to instead of standard C library PVFS implements a library of system-call wrappers that is loaded before the standard C library

Disadvantages

PVFS disadvantages No fault-tolerance at the software level (expects in to be implemented at the hardware level - RAID arrays) Doesn’t have any client-side buffering -> high I/O overhead for small write requests Big limitation - uses TCP for all communications (bottleneck for fast networks)

PVFS vs HDFS

Hadoop Distributed File System (HDFS) Parallel Virtual File System (PVFS) Deployment model Computation and storage are performed on the same node Separate compute and storage nodes Concurrent writes Not supported – allows only one writer per file Can perform concurrent writes into the same file in parallel as long as they are non-conflicting (in different regions of a file) Small file operations Not optimized for small files. Client-side buffering aggregates many small requests to one file into one large request Uses few optimizations for packing small files. But has no client-side buffering or caching and sends all application level write requests directly to I/O server. This may result in high I/O overhead for small write requests.

Hadoop Distributed File System (HDFS) Parallel Virtual File System (PVFS) Buffering Client-side readahead and write-behind staging improves bandwidth, but reduces durability and consistency guarantees No client-side prefetching or caching provides improved durability and consistency for concurrent writers Data layout Exposes mapping of chunks to data-nodes to Hadoop applications Does not expose a file’s object and stripe unit layout between nodes to the application by default Fault tolerance Uses rack-aware replication with at least three copies of every file chunk No replication at the file system level; relies on underlying hardware solutions such as RAID subsystems Compatibility Custom API and semantics for specific users Can be used with multiple APIs: a native PVFS API, the UNIX/POSIX API, MPI-IO

PVFS + HDFS

Hadoop-PVFS extension

Bibliography Ross, Robert B., and Rajeev Thakur. "PVFS: A parallel file system for Linux clusters." Proceedings of the 4th annual Linux Showcase and Conference. 2000. Bonnie, Michael Moore David, et al. "OrangeFS: Advancing PVFS." Tantisiriroj, Wittawat, et al. "On the duality of data-intensive file system design: reconciling HDFS and PVFS." Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, 2011. Tantisiriroj, Wittawat, Swapnil Patil, and Garth Gibson. "Data-intensive file systems for internet services: A rose by any other name." Parallel Data Laboratory, Tech. Rep. UCB/EECS-2008-99 (2008). Wikipedia: https://en.wikipedia.org/wiki/Parallel_Virtual_File_System, https://en.wikipedia.org/wiki/OrangeFS OrangeFS official web-site: http://www.orangefs.org/ “The OrangeFS distributed filesystem” article on LWN.net: https://lwn.net/Articles/643165/

Rainbow diagram Questions?