Integration of Singularity With Makeflow Kyle Sweeney University of Notre Dame ksweene3@nd.edu

Main design of an HPC Environment Compute Nodes with small discs In an abstract sense, an HPC cluster is a series of computation nodes, each with a small amount of local storage, and all connected to a massive FileSystem which maintains all of the storage for the system. Massive Parallel File System

Makeflow and Work Queue with HPC Work nodes with Workers Computer running Makeflow and Work Queue Makeflow then uses a work managing system, such as Workqueue, to create workers on the nodes, and distribute jobs. Massive Parallel File System

Anatomy of a Node with Workers Scratch Space Workqueue sets up a central cache on every host machine where workers exist. When a task needs an input file, the file is copied over into this central cache, and then symlinked into the worker’s temporary work space. This enables us to reuse input files, and save of network traffic if multiple tasks require the same input. This is also the default operation, which can be changed when using Workqueue, but is the main assumption of Makeflow, for a general Makeflow script. While in Makeflow we can specify the hardware requirements for tasks (cores, HDD space, memory capacity), it doesn’t allow us to specify software requirements, which is where containers come in, namely , singularity Symlink

Singularity No root Daemon! Does not manage a database of images! Does not attempt any kind of scheduling! No root daemon: simply need to be root to install it Doesn’t have a database of images: users manage the images themselves Doesn’t do scheduling: runs as a user process, behaving just like every other process

Main Benefits of Singularity Democratization of Resource Management! Everything needs to be done by the user And Singularity gives tools to do this Singularity gives tools to create images exactly how users want them, packages, go inside of the images to install and set variables, even import from other container tools, such as docker.

Using Singularity and Makeflow Two methods of using Singularity with Makeflow Global Wrapper: “makeflow –T wq –N mytasks --singularity=<img> script.mf” Will wrap every tasks in “script.mf” in Singularity “singularity exec <img> sh –c ‘task1’ ” Per-Task wrapper: Inside of your makeflow script, you can simply add singularity to the start of you task, and include the image file “foo.out: singularity mytask myfile.img foo.in singularity exec myfile.img mytask foo.in –o foo.out” When you run creating your workers, you need to specify to run on machines that you know has singularity installed.

Singularity and a Node with Workers Scratch Space By default, when running a command in a container with singularity, the image is read-only. This is due to how singularity handles privilege levels inside of the container. To be root inside the container, you need to be root outside of the container. Singularity thus only allows changes to files inside the container if the calling user is root. This is to our advantage though, because now every worker can share the same image for their task. This massively saves on network traffic, as images tend to be large. Symlink

Problems with Singularity Creating and Modifying an image file requires the user to be root Image Files are large Folders linked into Singularity Modifying and creation: Thus, users need to be able to do “sudo” or become root to create their image file for their jobs Large Image Files: Image files by default are 1024MB, but not really, Image files in our version of Singularity are sparse-thus they are really only around 32MB large at the most bare bones (just the OS), but the filesystem and network reads them as 1GB

Biggest issue with integration Singularity attempts to bind 3 important locations into a container /tmp Current working directory $HOME/ When a user requests a worker, that worker’s owner is that user Singularity attempted to bind the user calling Makeflow and Workqueue’s home directory into the container Worker’s owner is the user: Thus all programs called by that worker are owned by the user Attempted to bind /home to container: Due to AFS permissions, this isn’t allowed

Solutions First: change config file on every node; there’s an option in the config files whether to bind the user’s home directory. By default, this is set to True.

Solution Second: set home directory in singularity. We modified it to add a switch, and set it in makeflow global wrapper to use “--change-home `pwd`” This is better because it puts less work on the sys-admins when installing singularity, it allows for more future flexibility, and the developers of singularity wanted this to be a feature anyway

Testing Singularity

CPU Measurement Modified the shrimp workflow Performed this test 3 different ways Natively, Using Docker, and Using Singularity Each worker was requested to have 1 Core, 2000MB memory Performed this test around 150 times Modified shrimp: a bio-informatics workflow that’s CPU heavy, Reduced the workflow to just the first setup task done locally, the first 50 tasks, and the final combination step also done locally, Only looked at the results of the 50 tasks, as those were on the “disc” cluster

X-axis is the test number, and the y axis is how much time each item took to run.

CPU Measurement This is an upclose examination of the data, around the middle. It shows off how singularity and native are basically the same, sometimes going higher than the other, but generally the same. Docker, on the other hand, is always higher.

Disc I/O Measurement Modified a test which tested the overhead of different file systems The test nests singularity calls (e.g singularity exec img1.img singularity exec img2.img my_task.sh) The images were made ahead of time, all existing flat, not nested inside of each other Required sudo to work, thus ran on an Ubuntu VM on my laptop Modified test: nest calls, it reads and writes a certain amount of data, tested it 10 deep but got it to go 20 deep, and the amount of data read/written were each their own category of tests, at 2GB, 4GB, 8GB

Disc I/O Overhead Talk about the X and Y values, this is the 2GB files example

Conclusion Singularity fits into our Makeflow+Work Queue system very well, thanks to Singularity’s democratization of resource management Requires users to create their own images to match their work Negligible CPU overhead, and minor I/O overhead

Singularity: http://singularity.lbl.gov Makeflow + Singularity – out soon! CCL Website: http://ccl.cse.nd.edu Email: ksweene3@nd.edu