1. Implementing CVMFS server using an union file system
Monday, 12th of September 2011
René Meusel
Implementing CVMFS server using an union file system
CERN Summer Student Report
René Meusel
Hasso Plattner Institute Potsdam (Germany)
Supervisors: Predrag Buncic, Jakob Blomer
Good Morning

2. 1 2 3 4 Agenda porting CVMFS to Mac OS X
Monday, 12th of September 2011
René Meusel
Agenda 1
porting CVMFS to Mac OS X
changing it‘s build system to CMake
replacing the shadow directory with a union file system approach
using the low level interface of FUSE for CVMFS 2 3
port to Mac OS, build system, MAIN TASK: union file system, low level API 4

3. Porting CVMFS to Mac OS X
Monday, 12th of September 2011
René Meusel
Porting CVMFS to Mac OS X
Currenty CVMFS runs exclusively on Linux
Porting to Mac OS X was fairly simple
a few system calls had to be abstracted
e.g. stat64, readdir, spinlock-implementation and Thread Local Storage
Mostly implemented in a compatibility-header
Just a proof of concept at the moment
No repositories with Mac OS X binaries
Rewrote build system from automake to CMake
Easier and more convenient user interface
packaging for different platforms
Fully platform independent
packaging requested by some users

4. Kernel module with no real support anymore
Monday, 12th of September 2011
René Meusel
Synchronisation
Release Manager
Shadow Directory
Kernel module with no real support anymore
Redir-FS
CernVM Filesystem
Repository
read only volume
Union Filesystem
Overlay
redir-FS = unsupported on old kernels
User
Server
Client

5. Union Filesystems Mount more than one file system at once
Monday, 12th of September 2011
René Meusel
Union Filesystems
Mount more than one file system at once
Appear layered on top of each other (branches)
Usual use case: Linux Live-CDs
Allows changes to read only file systems
Main features (see next slide):
copy on write
white-out files
opaque directories
UnionFS, AUFS, OverlayFS, UnionFS FUSE, ...
several implementations
no union file system in mainline kernel yet
But: SLC 5 ships with AUFS 1 included
read only
writable overlay
usual use case, main features, support guaranteed

6. Updating CVMFS repo using AUFS
Monday, 12th of September 2011
René Meusel
Updating CVMFS repo using AUFS
repository
writable overlay
AUFS
CernVM Filesystem
appears as ordinary
writable volume
Advanced multi layered Unification FileSystem
AUFS makes CVMFS mutable by the release manager
Changes show up in the overlay directory
We can sync these files into the repository afterwards

7. Usual synchronisation internals
Monday, 12th of September 2011
René Meusel
Usual synchronisation internals
repository (read only)
Directory
foo
bar
file
symlink
CernVM Filesystem
repository
writable overlay
AUFS
appears as ordinary
writable volume
$ rm -rf Directory symlink
$ mkdir Directory
$ touch Directory/newFile
$ ln file hardlink
writable volume
Directory
foo
bar
file
symlink
overlay directory
Directory
newFile
.wh..wh..opq
.wh.symlink
hardlink
extract changes
taking all three volumes into account
new data in overlay directory
simplifies compression
writable volume
Directory
newFile
file
hardlink

8. Union Filesystems Quick Performance test: Simulating a usual use case
Monday, 12th of September 2011
René Meusel
Union Filesystems
Quick Performance test:
Simulating a usual use case
copy on write semantics by union filesystem means
Decompressing Linux 3.0 kernel sources into union volume
77 MB | 400 MB decompressed | files
Delete the resulting directory afterwards
Decompression
Recursive delete
Native (no union)
29.8s
0.9s
AUFS
31.0s
1.5s
UnionFS FUSE *
54.2s
10.3s
AUFS directly decided for write branch  no significant overhead
Deleting: AUFS has to check for file existance in read only directory  lots of stat calls
Underlying branch must be checked for file existence
mean of three trials
No real overhead: underlying branch is marked read only
Performance of the FUSE implementation is not suitable
* FUSE implementation was investigated because it wouldn‘t need a specific kernel but obviously is too slow

9. Problems foo bar file hardlink
Monday, 12th of September 2011
René Meusel
Problems
Shadow directory had normel file system semantics
we just „copied“ them into the CVMFS repository
storage
inode 3
inode 1
inode 2
foo
bar
file
hardlink

10. Problems Shadow directory had normel file system semantics
Monday, 12th of September 2011
René Meusel
Problems
Shadow directory had normel file system semantics
we just „copied“ them into the CVMFS repository
AUFS approach consists of three different file systems
inodes are not consistent over file system borders
mapping of different inodes is not accessible (only present in kernel)
storage 42 12 25
.wh.foo
bar
moep
writable overlay
cache 3 1 2
foo
bar
file
hardlink
read only repository
foo
bar 5
hardlink
file 6
moep 7
union volume

11. Problems Impossible to distingush „edit“ and „replace“
Monday, 12th of September 2011
René Meusel
Problems
Shadow directory had normel file system semantics
we just „copied“ them into the CVMFS repository
AUFS approach consists of three different file systems
inodes are not consistent over file system borders
mapping of different inodes is not accessible (only present in kernel)
Impossible to distingush „edit“ and „replace“
files are copied to the overlay in any case
as file systems differ, inode changes cannot be recognized
hard links become challenging
AUFS supports pseudo hard links over different branches
however hard links are not properly supported by FUSE‘s high level API, resulting in a kernel panic in connection with AUFS
solution: port CVMFS to low level API of FUSE

12. FUSE high vs. low level API
Monday, 12th of September 2011
René Meusel
FUSE high vs. low level API
High level API is designed for easy usage
simple callback architecture
based on file names and paths rather than inodes
provided on top of low level API by libfuse in user space
Low level API provides more control
usual VFS layer functions passed to user space as callbacks
based on inodes  therefore full control over hard links
maybe better performance (to be investigated)
Possibility to export CVMFS via NFS using low level API
requested by Gird sites with limited local disk cache on worker notes

13. Chance for better performance?
Monday, 12th of September 2011
René Meusel
Chance for better performance?
libfuse
CVMFS
High Level API
CVMFS optimised caching
general purpose caching
inode translation
Low Level API
Kernel Space
User Space
Character Device
FUSE Kernel Module
Virtual File System Layer

14. Conclusion Ported CVMFS to Mac OS X
Monday, 12th of September 2011
René Meusel
Conclusion
Ported CVMFS to Mac OS X
Converted the build system to CMake
Implemented repository updates by union file system means
CVMFS on top of the low level FUSE API
as a prototype

15. Future work Integrate CVMFS with FUSE low level API
Monday, 12th of September 2011
René Meusel
Future work
Integrate CVMFS with FUSE low level API
optimise internal caching infrastructure
implement a proper inode allocation method
think about backward compatibility
Overall system hardening
beta test build system
integration tests for newly implemented synchronisation
bring low level FUSE API interface to a stable implementation
Mac OS X could become interesting over time
set up a test repository with Mac binaries