1. General Parallel File System
Presentation by:
Lokesh Pradhan

2. Introduction File System
Way to organize data which is expected to be retained after the program terminates by providing procedures to store, retrieve and update data as well as manage the available space on the device which contains it.
As we are juniors and senior standing in comp science, we all know what is File System . However, I want repeat what it means
Way to organize data which is expected to be retained after the program terminates by providing procedures to store, retrieve and update data as well as manage the available space on the device which contains it

3. Types of File System Types Examples Disk file system FAT, exFAT, NTFS…
Optical discs
CD, DVD, Blu-ray
Tape file system
IBM’s Linear tape
Database file system
DB2
Transactional file system
TxF, Valor, Amino, TFFS
Flat file system
Amazon’s S3
Cluster file system
Distributed file system
Shared file system
San file system
Parallel file system
NFS, CIFS, AFS, SMB, GFS, GPFS, LUSTRE, PAS
Here the list of types. They can be classified into many types, but here are only few you might know or will from now from this slide.

4. In HPC world Equally large applications
Large input data set (e.g. astronomy data)
Parallel execution on large clusters
Use parallel file systems for scalable I/O
e.g. IBM’s GPFS, Sun’s Lustre FS, PanFS, and
Parallel Virtual File System (PVFS)
High performance computing is another style of computing at a comparable scale
which process large input data set in parallel
In HPC world, parallel file systems are used for highly scalable storage I/O
Examples of parallel file systems are GPFS, Lustre, PanFS and PVFS
??? What do you think we can propose?

5. General Parallel File System
Cluster: 512 nodes today, fast reliable communication
Shared disk: all data and metadata on disk accessible from any node through disk I/O interface (i.e., "any to any" connectivity)
Parallel: data and metadata flows from all of the nodes to all of the disks in parallel
RAS: reliability, accessibility, serviceability
The General Parallel File System (GPFS) is a high-performance shared disk clustered file system developed by IBM. As it is based on the characteristics of shared disk file system, it has provides concurrent high speed file access to the applications executing on multiple nodes of cluster. It can be used with AIX 5L clusters, Linux clusters, on Microsoft Windows Server, or a heterogeneous cluster of AIX, Linux and Windows nodes. Moreover, GPFS offers tools to manage and administrate the GPFS cluster and hence, allows the shared access to file systems from remote GPFS clusters.
there are the characteristics of GPFS
Cluster: 512 nodes today, fast reliable communication, common admin domain Shared disk: all data and metadata on disk accessible from any node through disk I/O interface (i.e., "any to any" connectivity) Parallel: data and metadata flows from all of the nodes to all of the disks in parallel RAS: reliability, accessibility, serviceability
GPFS is the filesystem of the ASC Purple Supercomputer[2] which is composed of more than 12,000 processors and has 2 petabytes of total disk storage spanning more than 11,000 disks.

6. History of GPFS Shark video server Video streaming from single RS/6000
Complete system, included file system, network driver, control server
Large data blocks, admission control, deadline scheduling
Bell Atlantic video-on-demand trial ( )
Tiger Shark multimedia file system
Multimedia file system for RS/6000 SP
Data striped across multiple disks, accessible from all nodes
Hong Kong and Tokyo video trials, Austin video server products
GPFS parallel file system
General purpose file system for commercial and technical computing on RS/6000 SP, AIX and Linux clusters.
Recovery, online system management, byte-range locking, fast pre- fetch, parallel allocation, scalable directory, small-block random access.
Released as a product /98.
History
Shark video server
Video streaming from single RS/6000
Complete system, included file system, network driver, control server
Large data blocks, admission control, deadline scheduling
Bell Atlantic video-on-demand trial ( )
Tiger Shark multimedia file system
Multimedia file system for RS/6000 SP
Data striped across multiple disks, accessible from all nodes
Hong Kong and Tokyo video trials, Austin video server products
GPFS parallel file system
General purpose file system for commercial and technical computing on RS/6000 SP, AIX and Linux clusters.
Recovery, online system management, byte-range locking, fast prefetch, parallel allocation, scalable directory, small-block random access, ...
Released as a product /98, /98, /00, …

7. What is Parallel I/O?
Multiple processes (possibly on multiple nodes) participate in the I/O
Application level parallelism
“File” is stored on multiple disks on a parallel file system

8. What does Parallel System support?
A parallel file system must support
Parallel I/O
Consistent global name space across all nodes of the cluster
Including maintaining a consistent view across all nodes for the same file
Programming model allowing programs to access file data
Distributed over multiple nodes
From multiple tasks running on multiple nodes
Physical distribution of data across disks and network entities eliminates bottlenecks both at the disk interface and the network, providing more effective bandwidth to the I/O resources

9. Why use general parallel file systems?
Native AIX File System
No file sharing - application can only access files on its own node
Applications must do their own data partitioning
Distributed File System
Application nodes (DCE clients) share files on server node
Switch is used as a fast LAN
Coarse-grained (file or segment level) parallelism
Server node : performance and capacity bottleneck
GPFS Parallel File System
GPFS file systems are striped across multiple disks on multiple storage nodes
Independent GPFS instances run on each application node
GPFS instances use storage nodes as "block servers" - all instances can access all disks
Native AIX File System (JFS)
No file sharing - application can only access files on its own node
Applications must do their own data partitioning
DCE Distributed File System
Application nodes (DCE clients) share files on server node
Switch is used as a fast LAN
Coarse-grained (file or segment level) parallelism
Server node is performance and capacity bottleneck
GPFS Parallel File System
GPFS file systems are striped across multiple disks on multiple storage nodes
Independent GPFS instances run on each application node
GPFS instances use storage nodes as "block servers" - all instances can access all disks
Parallel file systems offer numerous advantages and address some key issues by providing:
Concurrent access to files by multiple nodes of a cluster. This prevents users from having to utilize the local disk of each node and then reassemble the output into either a coherent single file or a collection of multiple files (sometimes referred to as post-mortem reassembly).
Scalable performance. Parallel file systems are designed with scalability in mind. As clusters grow, more disk and more network connections need to be incorporated into the fabric of a file system.
A single disk space where serial files and files created by parallel applications can coexist and be manipulated.

10. Performance advantages with GPFS file system
Allowing multiple processes or applications on all nodes in the cluster simultaneously
Access to the same file using standard file system calls.
Increasing aggregate bandwidth of your file system by spreading reads and writes across multiple disks.
Balancing the load evenly across all disks to maximize their combined throughput. One disk is no more active than another.
Performance advantages with GPFS file system
Using GPFS to store and retrieve your files can improve system performance by:
Allowing multiple processes or applications on all nodes in the cluster simultaneous
access to the same file using standard file system calls.
Increasing aggregate bandwidth of your file system by spreading reads and writes across multiple disks.
Balancing the load evenly across all disks to maximize their combined throughput. One disk is no more active than another.
Supporting very large file and file system sizes.
Allowing concurrent reads and writes from multiple nodes.
Allowing for distributed token (lock) management. Distributing token management reduces system delays associated with a lockable object waiting to obtaining a token.
Allowing for the specification of other networks for GPFS daemon communication and for GPFS administration command usage within your cluster.

11. Performance advantages with GPFS file system (cont.)
Supporting very large file and file system sizes.
Allowing concurrent reads and writes from multiple nodes.
Allowing for distributed token (lock) management. Distributing token management reduces system delays associated with a lockable object waiting to obtaining a token.
Allowing for the specification of other networks for GPFS daemon communication and for GPFS administration command usage within your cluster.

12. GPFS Architecture Overview
Implications of Shared Disk Model
All data and metadata on globally accessible disks (VSD)
All access to permanent data through disk I/O interface
Distributed protocols, e.g., distributed locking, coordinate disk access from multiple nodes
Fine-grained locking allows parallel access by multiple clients
Logging and Shadowing restore consistency after node failures
Implications of Shared Disk Model
All data and metadata on globally accessible disks (VSD)
All access to permanent data through disk I/O interface
Distributed protocols, e.g., distributed locking, coordinate disk access from multiple nodes
Fine-grained locking allows parallel access by multiple clients
Logging and Shadowing restore consistency after node failures

13. GPFS Architecture Overview (cont.)
Implications of Large Scale
Support up to 4096 disks of up to 1 TB each (4 Petabytes)
The largest system in production is 75 TB
Failure detection and recovery protocols to handle node failures
Replication and/or RAID protect against disk / storage node failure
On-line dynamic reconfiguration (add, delete, replace disks and nodes; rebalance file system)
Implications of Large Scale
Support up to 4096 disks of up to 1 TB each (4 Petabytes)
The largest system in production is 75 TB
Failure detection and recovery protocols to handle node failures
Replication and/or RAID protect against disk / storage node failure
On-line dynamic reconfiguration (add, delete, replace disks and nodes; rebalance file system)

14. GPFS Architecture - Special Node Roles
Three types of nodes:
File system nodes
Manager nodes
Storage nodes
Three types of nodes: file system, storage, and manager
File system nodes
Run user programs, read/write data to/from storage nodes
Implement virtual file system interface
Cooperate with manager nodes to perform metadata operations
Manager nodes
Global lock manager
File system configuration: recovery, adding disks, …
Disk space allocation manager
Quota manager
File metadata manager - maintains file metadata integrity
Storage nodes
Implement block I/O interface
Shared access to file system and manager nodes
Interact with manager nodes for recovery (e.g. fencing)
Data and metadata striped across multiple disks - multiple storage nodes

15. Disk Data Structures:
Large block size allows efficient use of disk bandwidth
Fragments reduce space overhead for small files
No designated "mirror", no fixed placement function:
Flexible replication (e.g., replicate only metadata, or only important files)
Dynamic reconfiguration: data can migrate block-by-block
Multi level indirect blocks
Each disk address:
List of pointers to replicas
Each pointer:
Disk id + sector no.

16. Availability and Reliability
Eliminate single point of failures
Designed to transparently fail over token (lock) operations.
Supports data replications to increase availability in the vent of a storage media failure.
Offers time-tested reliability and has been installed on thousands of nodes across industries
Basis of many cloud storage offerings
For optimal reliability, GPFS can be configured to eliminate single points of failure. The file system can be configured to remain available automatically in the event of a disk or server failure.
A GPFS file is designed to transparently fail over token (lock) operations and other GPFS cluster services, which can be distributed throughout the entire cluster to eliminate the need for dedicated metadata servers. GPFS can be configured to automatically recover from node, storage and other infrastructure failures.
GPFS provides this functionality by supporting data replication to increase availability in the event of a storage media failure; multiple paths to the data in the event of a communications or server failure; and file system activity logging, enabling consistent fast recovery after system failures. In addition, GPFS supports snapshots to provide a space-efficient image of a file system at a specified time, which allows online backup and can help protect against user error.
GPFS offers time-tested reliability and has been installed on thousands of nodes across industries, from weather research to broadcasting, retail, financial industry analytics and web service providers. GPFS also is the basis of many cloud storage offerings.

17. GPFS’s Achievement
Used on six of the ten most powerful supercomputers in the world, including the largest (ASCI white)
Installed at several hundred customer sites, on clusters ranging from a few nodes with less than a TB of disk, up to 512 nodes with 140 TB of disk in 2 file systems
20 filed patents
ASC Purple Supercomputer which is composed of more than 12,000 processors and has 2 PB of total disk storage spanning more than 11,000 disks.

18. Conclusion Efficient for managing data volumes
Provides world-class performance, scalability and availability for your file data
Designed to optimize the use of storage
Provide highly available platform for data-intensive applications
Delivering real business needs by streamline data workflows, improvised services reducing cost and managing the risks.
GPFS already is managing data volumes that most companies will not need to support for five years or more. You may not need a multipetabyte file system today, but with GPFS you know you will have room to expand as your data volume increases.

19. References
"File System." Wikipedia, the Free Encyclopedia. Web. 20 Jan <
"IBM General Parallel File System for AIX: Administration and Programming Reference - Contents." IBM General Parallel File System for AIX. IBM. Web. 20 Jan <
"IBM General Parallel File System." Wikipedia, the Free Encyclopedia. Web. 20 Jan <
Intelligent Storage Management with IBM General Parallel File System. Issue brief. IBM, July Web. 21 Jan <
Mandler, Benny. Architectural and Design Issues in the General Parallel File System. IBM Haita Research Lab, May Web. 21 Jan <Architectural and Design Issues in the General Parallel File System>.
"NCSA Parallel File Systems." National Center for Supercomputing Applications at the University of Illinois. University of Illinois, 20 Mar Web. 21 Jan <
Parallel File System. Rep. Dell Inc., May Web. 21 Jan <
Welch, Brent. "What Is a Cluster Filesystem?" Brent B Welch. Web. 21 Jan <
Here are the references.

20. Questions?