Building a Parallel File System Simulator E Molina-Estolano, C Maltzahn, etc. UCSC Lab, UC Santa Cruz. Published in Journal of Physics, 2009

Abstract This paper briefly introduced the parallel file system simulator named IMPIOUS (Imprecisely Modeling Parallel I/O is Ususlly Successful). They showed IMPIOUS is capable of simulating the performance effects of different workloads in PFS (They validated it in PanFS). They showed the simulation runtime is acceptable for large scale (e.g., thousands of clients, 512 OSD) simulations on a regular single core machine.

Introduce IMPIOUS IMPIOUS is a trace-driven parallel file system simulator. It uses an abstract simulation model of object-based parallel file systems. It only accounts for just enough detail to reproduce important effects observed in real systems. They hope it to be: A simple, scalable, and general simulation model. A framework for rapid hypothesis testing. A simulator available to the open source community.

Abstract model of PFS's Typical PFS's consist of the same subsystems: OSD – run their own local file system and export a flat name space of objects. Metadata Management – implements the file name space and all metadata operations. Clients – provide the file system interface to applications.

Abstract model of PFS's The key characteristics distinguishing particular PFS's are: Data placement strategy – maps the requests to OSDs using a particular striping strategy. Resource locking protocol – determines which disks or OSDs are locked for a particular client access. Redundancy strategy – determines the kind of redundancy and which subsystem is responsible for maintaining it. Client buffer cache – specifies management of communication between clients and OSDs.

The Major Components of IMPIOUS

The clients are driven by parallel I/O traces. The file system-specific plugins determine fundamental behaviors: The data placement strategy, replication strategy, locking discipline and client cache strategy. Clients communicate with OSDs to perform I/O. Each simulated OSD consists of an instance of NaiveFS – an idealized per-OSD filesystsem. Disk simulation – DiskSim / a simple disk model.

Current status Implemented the models for PVFS, PanFS and Ceph, in the process of implementing Lustre. The validation results are based on PanFS.

Validation 1 Write Alignment vs. Bandwidth Traces used: PLFS Maps by LANL, N-1 checkpoint pattern. For the real experiments: 512 client processes running on LANL's Roadrunner test cluster. 56 OSD's implemented PanFS storage system running on LANL's Roadrunner test cluster. Use PatternIO benchmark running on clients to collect the results.

Validation 1 Real Results Each single write size is multiple of page size(4K) Each single write size is smaller than 4K Each single write size is multiple of stripe size (64K). This achieves the highest bandwidth.

Validation 1 - compare with IMPIOUS

Overall, the relative results from IMPIOUS is useful for analysis. IMPIOUS models the trends precisely. For Disksim, the IMPIOUS results do not match the absolute values because the authors did not use the same performance disk model in Disksim. Simple disk model is developed because Disksim consumes too much of simulation's runtime. For simple disk model, the absolute throughputs are considerably lower. Though the results are not very precise, the authors are refining it.

Validation 2 - performance effects of N-1 checkpoint Due to locking present in an N-1 checkpoint pattern, for multiple clients writing to the same file, the performance degrees significantly. Real experiment: run on LANL's Roadrunner supercomputer, 512 OSDs running PanFS. Simulation: Simple disk model. 2GHz Opteron one core machine; ran for 12 min – 3 hr. In this workload, a number of clients generate an N- 1 strided workload in 47KB writes.

Validation 2 - performance effects of N-1 checkpoint The stair step behavior is because the Roadrunner cluster is partitioned into sub-clusters, each supporting 720 processors, and each with an equal fraction of the total I/O bandwidth. We can see, IMPIOUS is able to correctly predict the relative ordering of these two lines, and the linear growth of the N-1 throughput.

Conclusions Given two different set of experimental results from checkpointing workloads on PanFS, IMPIOUS is able to simulate the important performance effects of those workloads. The wall-clock runtime of IMPIOUS is also encouraging: for experimantal runs consuming 2 minutes across a 512-node storage cluster, the slowest runs took 3 hours on a single core.

Our PFS Simulator

Client trace Metadata Server Stripping Strategy Local FS Disk queue Local FS Disk queue Local FS Disk queue Client trace Client trace Simulated Network OMNeT++ Disksims Data Servers Scheduling Algorithm Socket Connections

Existing Problems Simulation time – Disksim and the communication between Disksim and OMNeT++ is the runtime bottleneck of our simulator. –Optimize synchronization mechanism. –Choose more efficient communication channel other than TCP. –Implement “simple disk model”. We do not have disk models for modern disks. –DIXtrac program is very limited in modeling modern disks.

Thank you!