1 JTE HPC/FS Pastis: a peer-to-peer file system for persistant large-scale storage Jean-Michel Busca Fabio Picconi Pierre Sens LIP6, Université Paris 6 – CNRS, Paris, France INRIA, Rocquencourt, France

2 JTE HPC/FS 1.DHT-based File Systems 2.Pastis 3.Performance evaluation Outline

3 JTE HPC/FS Distributed file systems Client-serverP2P LAN (100) NFS- Organization (10.000) AFSFARSITE Pangaea Internet ( ) -Ivy * Oceanstore * Pastis * scalability (number of nodes) architecture * uses a Distributed Hash Table (DHT) to store data

4 JTE HPC/FS Distributed Hash Tables

5 JTE HPC/FS DHTs logical address space South America North America Australia Asia Europe Asia high latency, low bandwidth between logical neighbors Overlay network

6 JTE HPC/FS Insertion of blocks in DHT 04F B C52A BB2 3A AC78 895D E25A 04F2 3A B BB2 AC78 C52A E25A k = 8958 k = 8959 put(8959,block) root of key 8959 block Address space replica 895D

7 JTE HPC/FS PAST: Storage System PAST: Cooperative, archival file storage and distribution  Layered on top of Pastry Goals:  Strong persistence of the data  High availability  Scalability of the System  Reduced cost (no backup)  Efficient use of pooled resources

8 JTE HPC/FS Insertion of blocks in DHT 04F B C52A BB2 3A AC78 895D E25A 04F2 3A B BB2 AC78 C52A E25A k = 8958 k = 8959 put(8959,block) root of key 8959 block Address space replica 895D replica

9 JTE HPC/FS Insertion of blocks in DHT 04F B C52A BB2 3A AC78 895D E25A 04F2 3A B BB2 AC78 C52A E25A block Address space replica 895D replica k = 8958 k = 8959 get(8959,block)

10 JTE HPC/FS P2P File systems architecture put(key, block) block = get(key)  files and directories  read-write access semantics  security and access control DHash / Past Ivy / Pastis DHT FS - scalability - fault-tolerance - self-organization  block store (DHT)  message routing open(), read(), write(), close(), etc.

11 JTE HPC/FS DHT-based file systems Ivy [OSDI’02]  log-based, one log per user  fast writes, slow reads  limited to small number of users Oceanstore [FAST’03]  updates serialized by primary replicas  partially centralized system  BFT agreement protocol requires well-connected primary replicas primary replicas secondary replicas User A’s log User B’s log User C’s log DHT object DHT object DHT object

12 JTE HPC/FS Pastis

13 JTE HPC/FS Pastis design Design goals  simple  completely decentralized  scalable (network size and number of users) put(key, block) block = get(key) Pastry Past Pastis DHT FS storage routing

14 JTE HPC/FS Pastis data structures Data structures similar to the Unix file system  inodes are stored in modifiable DHT blocks (UCBs)  file contents are stored in immutable DHT blocks (CHBs) metadata block addresses UCB file inode CHB1 CHB2 file contents UCB CHB1 CHB2 replica sets DHT address space Inode key

15 JTE HPC/FS Pastis data structures (cont.)  directories contain entries  use indirect blocks for large files metadata block addresses UCB directory inode CHB file1, key1 file2, key2 … metadata block addresses UCB file1 inode CHB old contents CHB indirect block CHB file contents CHB old contents CHB file contents

16 JTE HPC/FS Content Hash Block (CHB) Content Hash Block  block has to be immutable Solution to check and prevent modification  block contents determine block key  can detect if block is modified data block block key = Hash( block contents ) block contents

17 JTE HPC/FS User Certificate Blocks (UCBs) UCBs are modifiable by the block owner. Question: How to check that the file is modified only by the owner? Protocol  (KB pub, KB priv ) associated to each block  The owner builds a signature of the block using KB priv. Authentication  Verify signature of UCB using the KB pub sign(KB priv ) timestamp UCB block key = Hash( KB pub ) inode contents