1. Large Scale Test of a storage solution based on an Industry Standard
Michael Ernst
Brookhaven National Laboratory
ADC Retreat
Naples, Italy
February 2, 2011

2. Motivation
Though dCache at BNL supports distributed analysis well (up to 5000 concurrent analysis jobs) we are looking into ways to improve the usability of our environment, e.g. include large-scale interactive analysis.
We want to explore to what extent we can build on industrial products as far as the facility part is concerned w/o having to rely on community extensions and interfaces.
NFS 4.1 (pNFS) appealing because of performance, simplicity, level of integration w/ OS
BlueArc successfully used by PHENIX and STAR

3. Areas of Interest

4. Whys is NFS4.1(pNFS) attractive?

5. BlueArc System Performance

6. Traditional Network File System
No Throughput Aggregation
Metadata and data together
BUT
Proven architecture
Enterprise features
Open, standard protocols
Open storage philosophy
BlueArc is about high performance network storage through standards. We deliver our file services via NFS and CIFS for example. So just so we’re on the same page, this is a view of a typical, traditional NFS system.
The benefits are…
At BlueArc we deliver the fastest systems for mixed workloads in this architecture
BUT there what you don’t get here is throughput aggregation to go beyond the fastest nodes we have, and metadata and data are typically residing together on the disks – essentially getting in the way of each other as you scale up in bandwidth performance 8

7. pNFS Architecture
With that said, let’s take a look at the pNFS architecture and I’ll also point out some of the areas where BlueArc’s implementation go beyond the minimum that is defined in the standard.
In this picture you have … walk through the general elements
Essentially pNFS is a protocol standard, it defines the communication between the clients and the metadata server and belween the clients and the data servers (movers). The communication between the metadata server and the data movers is an implementation detail. Some of the items we add to the solution is building on our current technology portfolio. With our hardware accelerated IOPS performance we have a screaming metadata server that will not melt under heavy workloads. We also have the ability through out to cluster nodes wherever needed. This is a cluster of clusters. The performance profile for the data servers is different so we will optimize those for maximum bandwidth performance.
The architecture is essentially limitless, you scale performance (bandwidth) by adding more data server nodes.
Any questions? 11

8. BlueArc pNFS Based NAS Platform
Highly scalable single metadata server
Clustered for HA
Architecture supports multiple clustered metadata servers
Support for heterogeneous data servers
Built on Enterprise Class Platform
Reliability
Full Featured NAS – Quota, Snapshots etc. 12

9. pNFS for HPC Performance Scaling
Data files automatically spread across multiple data FS for load balancing and performance
Individual data files optionally striped across multiple data FS for performance
Extreme Flexibility: A single server can run the entire cluster (acting as both metadata server and data mover)
Can scale performance by adding data movers (and relocate the FS)
Can scale capacity by adding data file systems
Clients
Direct IO
Get Data Location
File A
File B
Meta-Data File System
Separate Data File Systems 13 13

10. Product Availability and Test Bed at BNL
BA’s pNFS – Mid 2011
Up to 40 data movers with up to 8PB capacity each – growing to 128 data movers
Mercury - High performance metadata server
BlueArc Mercury or BlueArc Linux-based data mover appliances
Performance Goals for Test Bed at BNL
Will become DDM endpoint w/ SRM, GridFTP, Xrootd
Up to 2k concurrent Analysis Jobs
Up to 4 GB/s throughput between storage system and clients
Aiming at having ~500 TB of disk space 15 15

11. Test Bed at BNL SRM Xrootd WA Network GridFTP
With that said, let’s take a look at the pNFS architecture and I’ll also point out some of the areas where BlueArc’s implementation go beyond the minimum that is defined in the standard.
In this picture you have … walk through the general elements
Essentially pNFS is a protocol standard, it defines the communication between the clients and the metadata server and belween the clients and the data servers (movers). The communication between the metadata server and the data movers is an implementation detail. Some of the items we add to the solution is building on our current technology portfolio. With our hardware accelerated IOPS performance we have a screaming metadata server that will not melt under heavy workloads. We also have the ability through out to cluster nodes wherever needed. This is a cluster of clusters. The performance profile for the data servers is different so we will optimize those for maximum bandwidth performance.
The architecture is essentially limitless, you scale performance (bandwidth) by adding more data server nodes.
Any questions?
WA Network
GridFTP 16