1. A whale of a tale. (embattling your apps with Docker)
Zach Giezen
@cf_man
Topic #U894

2. Who the heck am I? You can call me Zach or cf_man if you like. Sli.do
Hi there, you can call me Zach … or cf_man if you like.
<Click>
I love bikes …
And guns …
And sometimes bikes and guns …
I enjoy printing stuff …
Sorry wrong slide ..
Yes more like this.
I love working on hardware and firmware … (Yes thats a Raspberry Pi!)
Yes Firmware acquisition via SPI … (However now I’ve moved over to a GoodFett)
And I love teaching others ...
(Homeschool, Umbrella School, Computer Science etc.)
#U894

3. Predicates Why am I doing this? How I got here Where I’m going Sli.do
Tired of being that conference attendee that just sat back and said I could do this.
I've been an OWASP Member for some time now and always felt like I should contribute more
Then just my Dues for membership.
Every time I talked to someone about containers, I was surprised on how little they knew.
Really just want to help bring awareness of the Security related settings in containers, specifically Docker
How I got here
I’m a natural born tinkerer
I love BOTH breaking and defending.
Where I’m going
Looking to leverage containers to automate all those
Tasks that should be repeated on code from an AppSec / AppAssurance perspective
i.e. Scanning code with Manticore, ZAP, Burp, FindBugs, OpenSWAMP etc.
#U894

4. #U894 sli.do Mitigation != Remediation Topic’s: What we will cover.
Container configurations and settings specific to security.
Host/Node settings
Runtime engine settings
Invocation settings
If the internal App. is vulnerable, we are not going to change it.
Mitigation != Remediation
4 Clicks for the list !
Some context and what we will cover.
Container configurations and settings specific to security.
Host/Node settings
Runtime engine settings
Invocation settings
If the internal App. is vulnerable, we are not going to change it.
When I thought about creating this talk, I wanted to set it up
Form the point of, ”Hey, I’ve got this aging webApp that we cant change because …”
Vendor is out of business we don’t have access to the source code, or closed source.
App is too business critical and can be modified
Remediation time line is just too great. (will take a while to get the issues fixed)
You pick
<Click>
Everybody say it with me “Mitigation != Remediation” !
We want to mitigate using these controls, and ultimately your container ecosystem should be setup
This way from the start.
#U894
sli.do

5. Sli.do #U894 Securing the “Node” matters! Docker is NOT a hypervisor !
This narrows us down to only two options.
Bare Metal (Just OS and Docker, Orchestration Software)
Docker running within a Hypervisor Guest.
Due to the “access”, the Docker daemon needs to run, i.e. root!
It’s “unsafe” to run anything other then Docker
on that “Host” or “Node”.
Sli.do
Docker is NOT a hypervisor !
With Virtual machines you don’t share kernels (class 2 hypervisors)
<Click>
Due to the “access”, the Docker daemon needs to run, i.e. root!
It’s typically “unsafe” to run anything other then Docker
on that “Host” or “Node”.
This narrows us down to only two options.
Bare Metal (Just OS and Docker, Orchestration Software)
Docker running within a Hypervisor Guest.
#U894

6. Sli.do #U894 Start with logging!
Logging from the perspective of the “run time” engine.
Defaults
Plug-In’s, What you should do.
Logging form the perspective of the container.
Best Practice
PCI and many other regulatory bodies stipulate centralized logging.
Therefor the suggested “log-driver” value should be set to
“syslog” for linux or “etwlogs” for Windows, or a plug-in like splunk”
Default:
Look for your “daemon.json”, it holds the info for your “default” engine AND
container log location, and settings. If you have no “log-driver” stanza in your
file you are defaulting to the json-file format.
(This format is locked to the individual node you are using Docker on.)
This is bad
Settings within the “daemon.json” file can be over written by a “—log-driver”
directive, on a image by image “spin up.
For the love of Pete, set your “mode” option to non-blocking!
By default it’s not!
Sli.do
#U894
Default:
Look for your “daemon.json”, it holds the info for your “default” engine AND container
<Click>
Log location, and settings. If you have no “log-driver” stanza in your file you are defaulting to the
Json-file format. (This format is locked to the individual node you are using Docker on.
PCI and many other regulatory bodies stipulate centralized logging.
Therefor the suggested “log-driver” value should be set to
“syslog” or a plug-in like “Splunk”
Settings within the “daemon.json” file can be over written by a “—log-driver” directive
On a image by image “spin up.
For the love of Pete, set your “mode” option to non-blocking!
By default it’s not!

7. namespaces Cgroups seccomp Resource Controls Hyper-V namespaces
Docker security trifecta
namespaces
Cgroups
seccomp
Resource
Controls
Hyper-V
namespaces
Sli.do
Lets talk about the Docker security trifecta.
<Click>
On Linux it’s
Cgroups
Seccomp
Namespaces
On Windows it’s
resource controls implemented by (Windows host compute service)
Hyper-v
#U894

8. seccomp-bpf:
Kernel dependent. (>= ) armv6,7,8 too 
Must be compiled in.
On a 64bit system there are something like 548 syscalls
There are 306 syscall filters in the docker “default” BPF profile.
CONFIG_SECCOMP=y
CONFIG_SECCOMP_FILTER=y
Docker seccomp Demo
So how do we create a policy for system calls?
Manually create the policy (Guess)
Leverage existing tools
Sli.do
There are other tools too
like “seccomp-tools”
Docker-Slim Demo
seccomp (short for secure computing mode) is a computer security facility in the Linux kernel. It was merged into the Linux kernel mainline in kernel version
Set and limit access to kernel calls.
systemd's "SystemCallFilter" feature is based on seccomp
Chrome
QEMU
As stated this is currently a linux only technology.
If you wanted to achieve a similar technique on windows you would have to make use on EMET and HIPS.
Demo:
Cd into ….. SnowFroc2018/labs/security/seccomp/seccomp-profiles
docker run --rm -it --cap-add ALL --security-opt apparmor=unconfined --security-opt seccomp=deny.json alpine sh
docker run --rm -it --cap-add ALL --security-opt apparmor=unconfined --security-opt seccomp=default.json alpine sh
docker run --rm -it --cap-add ALL --security-opt apparmor=unconfined --security-opt seccomp=default-no-chmod.json
alpine sh
#U894

9. Sli.do #U894 Namespaces: Kernel enforced user space views. Peer:
UTS – No really “this is your hostname”
NET - Docker using this to allow C-2-C and C-2-H IP communication. (own stack, routing etc. i.e. veth)
MNT - used to manage filesystem mount points.
IPC - Widely unused by anyone.
Hierarchies:
PID - Docker makes use of this to setup the inner process tree, i.e. starting at PID # 1 etc.
User - Docker allows you to map or re-map inner container users to “NODE” or “HOST” users.
Cgroups - Designed to “hide” system limits / limitations.
Sli.do
Docker has support for:
Namespaces are basically
Kernel enforced user space views.
sudo unshare --fork --pid --mount-proc bash
Peer
uts – This is your HOSTNAME
net - you see Docker using this to allow C 2 C and container to host IP communication.
because each veth has it’s own stack, they have there own default gateway etc.
mnt - Not to be confused with chroot,
ipc Widely unused my anyone.
IPC namespaces isolate processes from SysV style inter-process communication.
Hierarchies
PID - Docker makes use of this to setup the inner process tree, i.e. starting at PID # 1 etc.
USER - Docker allows you to map or re-map inner container users to “NODE” or “HOST” users.
Cgroups -- Designed to “hide” system limits / limitations.
The pid namespace: Process isolation (PID: Process ID).
The net namespace: Managing network interfaces (NET: Networking).
The ipc namespace: Managing access to IPC resources (IPC: InterProcess Communication).
The mnt namespace: Managing filesystem mount points (MNT: Mount).
The uts namespace: Isolating kernel and version identifiers. (UTS: Unix Timesharing System).
The user namespace;
Cgroups control what a process can use. Whereas namespaces control what a process can see.
Syscalls[edit]
Three syscalls can directly manipulate namespaces:
clone, flags to specify which new namespace the new process should be migrated to.
unshare, allows a process (or thread) to disassociate parts of its execution context that are currently being shared with other processes (or threads)
setns, enters the namespace specified by a file descriptor. (Docker attach)
#U894

10. User Namespace Remapping
Initially added to the Linux kernel around 2013
Docker added support using it in v1.10
As one of the “newish” namespaces it’s made the most noise.
Docker daemon still runs as root, but now “inner” user is not.
This is a daemon level setting i.e. “daemon.json” or via startup
script like this “dockerd –userns-remap=default”.
Not to be confused with the “USER” Dockerfile parameter
or the “--u” / “--user” Docker run options
Let’s try for a quick Demo 
Sli.do
/etc/subuid
/etc/subgid
docker info
dockerd –userns-remap=default
default
docker run –cap-add ALL --rm -it --name dvwa -p 8080:80 -v /etc:/var/www/html/etc vulnerables/web-dvwa
; echo ‘ google.com >> /var/www/html/etc/hosts’; cat /var/www/html/etc/hosts
; cat /var/www/html/etc/hosts
cat /proc/101/uid_map
cat /proc/101/gid_map
USER is the default user within a container. The image developer can create additional users. 
Or pass them in via docker run options
--user 10000:10000
#U894

11. #U894 Sli.do control groups: a.k.a. Cgroups “namespace”
Docker exposes easy access to these via “docker run” options.
Some of these include:
--memory=32m
--memory-swap=64m
--kernel-memory=16m
--cpus=.5
--cpu-shares=256
#U894
Sli.do
Cgroups control what a process can use. Whereas namespaces control what a process can see.

12. Using Linux capabilities
Always always always run “--privileged=false” !!!!
If you are, your doing something wrong 
Drop all capabilities first “—cap-drop=ALL”
Then add them back as you need them
“—cap-add=NET_RAW”
Capability Key
Capability Description
SETPCAP
Modify process capabilities.
MKNOD
Create special files using mknod(2).
AUDIT_WRITE
Write records to kernel auditing log.
CHOWN
Make arbitrary changes to file UIDs and GIDs (see chown(2)).
NET_RAW
Use RAW and PACKET sockets.
DAC_OVERRIDE
Bypass file read, write, and execute permission checks.
FOWNER
Bypass permission checks on operations that normally require the file system UID of the process to match the UID of the file.
FSETID
Don’t clear set-user-ID and set-group-ID permission bits when a file is modified.
KILL
Bypass permission checks for sending signals.
SETGID
Make arbitrary manipulations of process GIDs and supplementary GID list.
SETUID
Make arbitrary manipulations of process UIDs.
NET_BIND_SERVICE
Bind a socket to internet domain privileged ports (port numbers less than 1024).
SYS_CHROOT
Use chroot(2), change root directory.
SETFCAP
Set file capabilities.
Docker can limit access to resources from inside the container.
CPU
Memery
Disk
Network
Kernel
docker run --rm ubuntu capsh --print |grep net_raw
docker run --rm -it --cap-drop ALL -p 8080:8080 python:2.7 python -m SimpleHTTPServer 8080
docker run --rm -it --cap-add ALL --security-opt apparmor=unconfined --security-opt seccomp=deny.json alpine sh
docker run --rm --cap-drop ALL ubuntu capsh --print |wc

13. #U894 Mounting Containers “Read-Only”
Just how powerful “—read-only” flag is
Just how frustrating this same flag is
If you can’t mount read-only, throw away
The running container (they’re immutable right?)
Note:
You can still mount fs resources “on top” of this “read-only”
overlay. Think persistent storage like “REX-Ray” or “ssh-fs”
#U894
By using the “docker diff” command or the “docker history” command you could script analysis of how to
Place the “tmpfs” folders.
docker run –read-only --rm -it --name dvwa -p 8080:80 -v /etc:/var/www/html/etc vulnerables/web-dvwa
Just how powerful “—read-only” flag is
<Click>
Just how frustrating this same flag is
DEMO
docker run -it --rm –p 8081: name node-hack node-hack
docker network disconnect bridge node-hack
docker export --output=”hacked_site.tar" node-hack
docker diff node-hack
$ while :; do test $(docker diff node-hack | wc -l) -gt 0 && docker kill node-hack; sleep 3; done
# in a different terminal:
$ while :; do docker run -it --rm -p 8080: name node-hack node-hack; sleep 2; done
Rinse, repete
Sli.do

14. #U894 sli.do #U894 sli.do #U894 sli.do CIS CE Benchmark
docker run –rm -it --net host --pid host --userns host \
--cap-add audit_control \
-e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
-v /var/lib:/var/lib \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /usr/lib/systemd:/usr/lib/systemd \
-v /etc:/etc --label docker_bench_security \
docker/docker-bench-security
docker run –rm -it --net host --pid host --userns host --cap-add audit_control \ -e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
-v /var/lib:/var/lib \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /usr/lib/systemd:/usr/lib/systemd \
-v /etc:/etc --label docker_bench_security \
docker/docker-bench-security
#U894
sli.do
#U894
sli.do
#U894
sli.do

15. Other Resources And Tools
Anchore
Aqua Security
Twistlock
Docker
Sonatype (Nexus OSS)
And on and on and on (LMGTFY)
If your new to this container thing
And you just don’t know where to start, or where to go
For information.
Here are some idea’s

16. Let’s review … Container file systems should be mounted “Read-Only.”
Sli.do
Let’s review …
Container file systems should be mounted “Read-Only.”
Unless unable to do so, Containers should not be linked.
Inner container process should run as non privileged users.
Avoid direct “host” storage exposer i.e. “bind mount.”
wherever possible, all files and folders should be visible as
“read-only” to inner container processes.
Partition or process level isolation should be enabled.
Storing certificates or credentials of any kind is prohibited.
Validate the image you want to run is the one you are about to run.
docker run -v /path/on/host:/path/on/container:ro my/image
docker run -d --read-only --tmpfs /run --tmpfs /tmp IMAGE
docker run --isolation=hyperv
docker run –isolation=default
docker –d –user bob –name bar foo/bar
docker -d --icc=false ....
docker run –env -d –name foo bar/foo
docker pull
docker service create --name redis \
--secret source=ssh-key,target=ssh \
--secret source=app-key,target=app,uid=1000,gid=1001,mode=0400 \
redis:3.0.6
Max 4 min.
Container file systems should be mounted “Read-Only”
There should be no need for permanent storage within a container.
Unless unable to do so, Containers must not be linked.
Forcing containers to use network resources to communicate is vital as additional ACL’s and monitoring can be put into place.
Inner container process should run as non privileged users.
Because process within a container “share” the kernel with the “host” you should think of the inner container process as running under
The corresponding “host” user.
Avoid direct “host” storage exposer “bind mount”
Sometimes you need to expose files or information to the process inside the container, these “bind mounts” should be avoided
If possible, if not; they should be mounted as “read-only”.
wherever possible, all files and folders must be visible as “read-only” to inner container processes.
Reducing “attack surface” is critical. If everything except log files and essential “write to files” are “read-only”
The attacker cannot gain a foothold.
Partition or process level isolation should be enabled.
Most container runtime engines allow for “isolation technologies”.
Storing certificates or credentials of any kind is prohibited.
“bolt-on” tools and orchestration platforms allow for retrieval of key’s and credentials during the “spin-up” process.
Passing keys or credentials in via environmental variables is not ok.
Validate the image you want to run is the one you are about to run.
typically container images are made up of a compressed file system, some also digitally hash each layer.
these images can be validated by each layer, image as a whole AND signer’s signature.
#U894

17. Slides / Demos available Here: https://goo.gl/mqK8ai
Zach Giezen
@cf_man
Topic #U894