1. Parallel Virtual File System (PVFS) a.k.a. OrangeFS
By Collin Gordon and Lisa Kosiachenko

2. What is PVFS?

3. Background information
PVFS - parallel, distributed filesystem for Linux clusters
Used in High-Performance Computing at large scale (while Hadoop is primarily used in cloud computing)
Grew out of research project from 1993
Since 2008 the main development branch is called OrangeFS
As of May 2016 release OrangeFS is a part of the Linux kernel!

4. Advantages of the PVFS
Concurrent read/write operations from different processes/threads to a common file
PVFS can be used with multiple application programming interfaces (APIs):
a native PVFS API,
the UNIX/POSIX API,
MPI-IO
Common UNIX shell commands, such as ls, cp, and rm, work with PVFS files
Supports the regular UNIX I/O commands such as read() and write()
Existing binaries that use UNIX API can access PVFS files without recompiling

5. How PVFS works

6. PVFS System Architecture

7. File Manager and Metadata
Application Process communicates with the manager daemon through TCP only during the following file operations:
Open
Close
Creation
Removal
Allows application to communicate directly with I/O nodes for read and write operations
Implementation problem: No directory access at first, then used NFS, currently have a system that determines whether the requested directory is PVFS and if so, the manager daemon is connected.

8. File Manager and Metadata
Single manager daemon stores all file metadata in the PVFS System
File data and metadata stored on local file systems rather than raw devices
Metadata components can be user set or default values
Metadata components:
Base: I/O Node where the file is stored
Pcount: Number of I/O Nodes for the file
Ssize: Size of each stripe on the I/O Nodes

9. File Metadata and Striping

10. I/O Daemons and Data Storage
I/O nodes specified at install
I/O nodes are not distinct from compute nodes
Each I/O node has an ordered set of I/O daemons
I/O daemons responsible for using local disk on I/O node to store files
Application sends a request for data to I/O daemons which work together to send back the information

11. I/O Daemons and Data Storage

12. Rainbow diagram

13. Trapping UNIX I/O calls
System calls are typically made by calling wrapper functions in the standard C library, which in turn pass the parameters to the kernel
Straightforward way to trap system calls - provide a separate library to instead of standard C library
PVFS implements a library of system-call wrappers that is loaded before the standard C library

14. Disadvantages

15. PVFS disadvantages
No fault-tolerance at the software level (expects in to be implemented at the hardware level - RAID arrays)
Doesn’t have any client-side buffering -> high I/O overhead for small write requests
Big limitation - uses TCP for all communications (bottleneck for fast networks)

16. PVFS vs HDFS

17. Hadoop Distributed File System (HDFS)
Parallel Virtual File System
(PVFS)
Deployment model
Computation and storage are performed on the same node
Separate compute and storage nodes
Concurrent writes
Not supported – allows only one writer per file
Can perform concurrent writes into the same file in parallel as long as they are non-conflicting (in different regions of a file)
Small file operations
Not optimized for small files.
Client-side buffering aggregates many small requests to one file into one large request
Uses few optimizations for packing small files.
But has no client-side buffering or caching and sends all application level write requests directly to I/O server. This may result in high I/O overhead for small write requests.

18. Hadoop Distributed File System (HDFS)
Parallel Virtual File System
(PVFS)
Buffering
Client-side readahead and write-behind staging improves bandwidth, but reduces durability and consistency guarantees
No client-side prefetching or caching provides improved durability and consistency for concurrent writers
Data layout
Exposes mapping of chunks to data-nodes to Hadoop applications
Does not expose a file’s object and stripe unit layout between nodes to the application by default
Fault tolerance
Uses rack-aware replication with at least three copies of every file chunk
No replication at the file system level; relies on underlying hardware solutions such as RAID subsystems
Compatibility
Custom API and semantics for specific users
Can be used with multiple APIs:
a native PVFS API,
the UNIX/POSIX API,
MPI-IO

19. PVFS + HDFS

20. Hadoop-PVFS extension

21. Bibliography
Ross, Robert B., and Rajeev Thakur. "PVFS: A parallel file system for Linux clusters." Proceedings of the 4th annual Linux Showcase and Conference
Bonnie, Michael Moore David, et al. "OrangeFS: Advancing PVFS."
Tantisiriroj, Wittawat, et al. "On the duality of data-intensive file system design: reconciling HDFS and PVFS." Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, 2011.
Tantisiriroj, Wittawat, Swapnil Patil, and Garth Gibson. "Data-intensive file systems for internet services: A rose by any other name." Parallel Data Laboratory, Tech. Rep. UCB/EECS (2008).
Wikipedia:
OrangeFS official web-site:
“The OrangeFS distributed filesystem” article on LWN.net:

22. Rainbow diagram
Questions?