1. Integration of Singularity With Makeflow
Kyle Sweeney
University of Notre Dame

2. Main design of an HPC Environment
Compute
Nodes with small discs
In an abstract sense, an HPC cluster is a series of computation nodes, each with a small amount of local storage, and all connected to a massive FileSystem which maintains all of the storage for the system.
Massive Parallel File System

3. Makeflow and Work Queue with HPC
Work nodes with Workers
Computer running Makeflow and Work Queue
Makeflow then uses a work managing system, such as Workqueue, to create workers on the nodes, and distribute jobs.
Massive Parallel File System

4. Anatomy of a Node with Workers
Scratch Space
Workqueue sets up a central cache on every host machine where workers exist. When a task needs an input file, the file is copied over into this central cache, and then symlinked into the worker’s temporary work space. This enables us to reuse input files, and save of network traffic if multiple tasks require the same input. This is also the default operation, which can be changed when using Workqueue, but is the main assumption of Makeflow, for a general Makeflow script.
While in Makeflow we can specify the hardware requirements for tasks (cores, HDD space, memory capacity), it doesn’t allow us to specify software requirements, which is where containers come in, namely , singularity
Symlink

5. Singularity No root Daemon! Does not manage a database of images!
Does not attempt any kind of scheduling!
No root daemon: simply need to be root to install it Doesn’t have a database of images: users manage the images themselves Doesn’t do scheduling: runs as a user process, behaving just like every other process

6. Main Benefits of Singularity
Democratization of Resource Management!
Everything needs to be done by the user
And Singularity gives tools to do this
Singularity gives tools to create images exactly how users want them, packages, go inside of the images to install and set variables, even import from other container tools, such as docker.

7. Using Singularity and Makeflow
Two methods of using Singularity with Makeflow
Global Wrapper:
“makeflow –T wq –N mytasks --singularity=<img> script.mf”
Will wrap every tasks in “script.mf” in Singularity
“singularity exec <img> sh –c ‘task1’ ”
Per-Task wrapper:
Inside of your makeflow script, you can simply add singularity to the start of you task, and include the image file
“foo.out: singularity mytask myfile.img foo.in
singularity exec myfile.img mytask foo.in –o foo.out”
When you run creating your workers, you need to specify to run on machines that you know has singularity installed.

8. Singularity and a Node with Workers
Scratch Space
By default, when running a command in a container with singularity, the image is read-only. This is due to how singularity handles privilege levels inside of the container. To be root inside the container, you need to be root outside of the container. Singularity thus only allows changes to files inside the container if the calling user is root. This is to our advantage though, because now every worker can share the same image for their task. This massively saves on network traffic, as images tend to be large.
Symlink

9. Problems with Singularity
Creating and Modifying an image file requires the user to be root
Image Files are large
Folders linked into Singularity
Modifying and creation: Thus, users need to be able to do “sudo” or become root to create their image file for their jobs Large Image Files: Image files by default are 1024MB, but not really, Image files in our version of Singularity are sparse-thus they are really only around 32MB large at the most bare bones (just the OS), but the filesystem and network reads them as 1GB

10. Biggest issue with integration
Singularity attempts to bind 3 important locations into a container
/tmp
Current working directory
$HOME/
When a user requests a worker, that worker’s owner is that user
Singularity attempted to bind the user calling Makeflow and Workqueue’s home directory into the container
Worker’s owner is the user: Thus all programs called by that worker are owned by the user
Attempted to bind /home to container: Due to AFS permissions, this isn’t allowed

11. Solutions
First: change config file on every node; there’s an option in the config files whether to bind the user’s home directory. By default, this is set to True.

12. Solution
Second: set home directory in singularity. We modified it to add a switch, and set it in makeflow global wrapper to use “--change-home `pwd`”
This is better because it puts less work on the sys-admins when installing singularity, it allows for more future flexibility, and the developers of singularity wanted this to be a feature anyway

13. Testing Singularity

14. CPU Measurement Modified the shrimp workflow
Performed this test 3 different ways
Natively, Using Docker, and Using Singularity
Each worker was requested to have 1 Core, 2000MB memory
Performed this test around 150 times
Modified shrimp: a bio-informatics workflow that’s CPU heavy, Reduced the workflow to just the first setup task done locally, the first 50 tasks, and the final combination step also done locally, Only looked at the results of the 50 tasks, as those were on the “disc” cluster

15. X-axis is the test number, and the y axis is how much time each item took to run.

16. CPU Measurement
This is an upclose examination of the data, around the middle. It shows off how singularity and native are basically the same, sometimes going higher than the other, but generally the same. Docker, on the other hand, is always higher.

17. Disc I/O Measurement
Modified a test which tested the overhead of different file systems
The test nests singularity calls (e.g singularity exec img1.img singularity exec img2.img my_task.sh)
The images were made ahead of time, all existing flat, not nested inside of each other
Required sudo to work, thus ran on an Ubuntu VM on my laptop
Modified test: nest calls, it reads and writes a certain amount of data, tested it 10 deep but got it to go 20 deep, and the amount of data read/written were each their own category of tests, at 2GB, 4GB, 8GB

18. Disc I/O Overhead
Talk about the X and Y values, this is the 2GB files example

19. Conclusion
Singularity fits into our Makeflow+Work Queue system very well, thanks to Singularity’s democratization of resource management
Requires users to create their own images to match their work
Negligible CPU overhead, and minor I/O overhead

20. Singularity: http://singularity.lbl.gov
Makeflow + Singularity – out soon!
CCL Website: