1. Security Practices in OpenShift
as Amadeus
OpenShift Commons Briefing
© Amadeus IT Group and its affiliates and subsidiaries
Nenad Bogojević
Amadeus S.A.S.
Diogenes Rettori
Red Hat
2017

2. Amadeus Provides IT services for travel industry
In one slide
Provides IT services for travel industry
Operates e-commerce web sites, payment processing, b2b services in travel
Using OpenShift 3 since 2 years
In own datacenters, in public clouds
© Amadeus IT Group and its affiliates and subsidiaries

3. Why security Protecting assets Personal information
And not the one like in picture
Protecting assets
computing capacity, data
Personal information
General Data Protection Regulation (GDPR)
e-Commerce & payment processing
PCI/DSS
© Amadeus IT Group and its affiliates and subsidiaries

4. How? OpenShift & Containers Lot of things are changing
To be better than the one like in picture
OpenShift & Containers
Lot of things are changing
Old rules may not be applicable
Risks are still out there
© Amadeus IT Group and its affiliates and subsidiaries

5. EVERYONE ON BOARD How? OpenShift & Containers
To be better than the one like in picture
OpenShift & Containers
Lot of things are changing
Old rules may not be applicable
Risks are still out there
EVERYONE ON BOARD
© Amadeus IT Group and its affiliates and subsidiaries

6. Infrastructure
© Amadeus IT Group and its affiliates and subsidiaries

7. OpenShift Architecture
In one slide
© Amadeus IT Group and its affiliates and subsidiaries

8. Preparing infrastructure
And security
Use OpenStack on our hardware
Or public cloud providers
Pre-constructed VM images
mirrored repositories & registries
scanned using OpenSCAP
Network design
Where are DMZ and layered protection?
OpenStack – security groups
Access control (bastion server)
Upgrade policy
Rebuild vs rolling
Bi-weekly/monthly
Insert new photo: Click on Insert photo-icon on slide and Insert new photo. Use Crop Tool under Picture Tools to pan and zoom
To update the text, just click on the text
More slides with this layout: Copy slide and paste
© Amadeus IT Group and its affiliates and subsidiaries

9. OpenShift Security Architecture
Different kind of network zones
Bastion
App Nodes
DevOps
SSH
Masters
HTTPS
CI/CD
SDN
© Amadeus IT Group and its affiliates and subsidiaries
Users
Infra Nodes
Developers
Repository
pull
Repository @

10. OpenShift
© Amadeus IT Group and its affiliates and subsidiaries

11. Let’s login!
© Amadeus IT Group and its affiliates and subsidiaries

12. Let’s login! oc login -u system:admin
© Amadeus IT Group and its affiliates and subsidiaries

13. Let’s login!
© Amadeus IT Group and its affiliates and subsidiaries

14. OpenShift Security Introduction
Diogenes says that he knows everything about security and that he runs openshift on his own laptop as an admin.
Then Nenad will stop him, call him crazy and talk about how things are done in the real world.
LDAP, RBAC, Ordinary Developers can view, DevOps are gods
© Amadeus IT Group and its affiliates and subsidiaries

15. OpenShift Secrets Way of managing & distributing sensitive information
Decoupling sensitive information from applications
Way of managing & distributing sensitive information
keys, certificates, passwords, usernames
Separate sensitive information management from application pods
Secured delivery to nodes (TLS)
Only present in memory on openshift nodes
Centralized management
Easy access from application
Environment variables
Volumes
© Amadeus IT Group and its affiliates and subsidiaries

16. Using Secrets Security as code Masters apiVersion: v1 kind: Pod
metadata:
  name: use-secret-pod
spec:
  containers:
    - name: secret-test-container
      image: myapp
      env:
        - name: SECRET_USERNAME
          valueFrom:
            secretKeyRef:
              name: top-secret
              key: username
  restartPolicy: Always
apiVersion: v1
kind: Secret
metadata:
  name: top-secret
data: 
  username: bmVuYWQ= 
  password: aWtuZXd5b3V3b3VsZHRyeXRoaXM=
© Amadeus IT Group and its affiliates and subsidiaries

17. OpenShift Secrets – „Less Great Things“
Handbrake for certification
Stored in (almost) clear
in etcd on masters
on tmp storage on nodes
accessible through API
How about vaults?
© Amadeus IT Group and its affiliates and subsidiaries

18. Some solutions
It’s not show-stopper
You already have big issue if someone compromised your infrastructure
Encrypt disks
Store in vault, with decryption service
Side-car or init containers
Security as a service
Compensating controls
© Amadeus IT Group and its affiliates and subsidiaries

19. OpenShift Audit Log
OpenShift provides log of activities that have affected system by individual users, administrators, or other components of the system.
Activate on master /etc/origin/master/master-config.yaml
AUDIT: id="5c3b8227-4af a c3887b" ip=" " method="GET" user="nenad" as="<self>" namespace="someproject" uri="/api/v1/namespaces/someproject/secrets" AUDIT: id="5c3b8227-4af a c3887b" response="401"
© Amadeus IT Group and its affiliates and subsidiaries
auditConfig:
   enabled: true

20. auditd introduction audit.log Alerting system rsyslog auditd rules
© Amadeus IT Group and its affiliates and subsidiaries
auditd rules
-a always,exit -S <syscall>
-w <filename>

21. auditd rules for masters
Monitoring etcd
OpenShift master - know if someone plays with etcd
-a always,exit -F arch=b64 -S creat -S open -S openat S open_by_handle_at -S truncate -S ftruncate –F dir=/var/lib/etcd k openshift_etcd
© Amadeus IT Group and its affiliates and subsidiaries

22. ..and on nodes Secrets mounted as tmpfs inside /var/lib/opesnift.
Monitoring secret
Secrets mounted as tmpfs inside /var/lib/opesnift.
When new secret is mounted add it to auditd rules
When new secret is unmounted remove it to from auditd rules
All monitorable secrets must have certain string in name
(e.g. secret~example)
If you open or close secrets often, it may generate a lot of messages
findmnt --list --noheadings --types tmpfs --poll --output ACTION,TARGET | grep secret~example | awk ‘$1 == “mount” { print $2 }‘ | xargs -L 1 -i auditctl --a always,exit -F arch=b64 -S creat -S open -S openat
-S open_by_handle_at -S truncate -S ftruncate -F dir={} -k openshift_secret
© Amadeus IT Group and its affiliates and subsidiaries
findmnt --list --noheadings --types tmpfs --poll --output ACTION,TARGET | grep secret~example | awk ‘$1 == “unmount” { print $2 }‘ | xargs -L 1 -i auditctl --d always,exit -F arch=64 -S creat -S open -S openat
-S open_by_handle_at -S truncate -S ftruncate -F dir={} -k openshift_secret

23. More use of auditd With the help of openscap
© Amadeus IT Group and its affiliates and subsidiaries

24. Service Signing Certificate
Secure communication inside or outside your cluster
Service annotated with service.alpha.openshift.io/serving-cert-secret- name=name
Certificate automatically generated and provided as a secret to pod
Clients can rely on automatically mounted CA /var/run/secrets/kubernetes.io/serviceaccount/ service-ca.crt
clientpod
mongodb.myproject.svc
© Amadeus IT Group and its affiliates and subsidiaries

25. Containers
© Amadeus IT Group and its affiliates and subsidiaries

26. Containers & Developers
We want to empower developer
Let’s be agile!
Run as root
Privileged containers – hostpath
port < 1000
Running old containers
FROM httpd:2.4.12
There’s this cool blackhat/jboss container on docker hub, let’s pull it
© Amadeus IT Group and its affiliates and subsidiaries

27. Containers & Developers
When will they learn!
We want to empower developer
Let’s be agile!
Run as root
Privileged containers – hostpath
port < 1000
Running old containers
FROM httpd:2.4.12
There’s this cool blackhat/jboss container on docker hub, let’s pull it
© Amadeus IT Group and its affiliates and subsidiaries

28. Root Access Support arbitrary user ids
Not allowed
Support arbitrary user ids
Use root group chown -R someuser:root /app && chmod –R g+rwX /app
Your application needs to listen on port 80?
Can’t you change it?
Use SCC (Security Context Constraint)
privileged containers, host paths, user id, FS Groups, selinux, capabilities
seccomp if you want to restrict even more
apiVersion: v1
kind: Service
metadata:
  name: httpd
spec: 
  ports:
- port: 80
targetPort: 10080
protocol: TCP
© Amadeus IT Group and its affiliates and subsidiaries

29. Image control Secured source All images come from internal registry
Using RHEL as base images
RedHat repository mirrored into internal
Other images must be built internally from source code
No automatic access to docker hub from build machines
Production access it’s own repository with only validated images
access.redhat.com/containers
© Amadeus IT Group and its affiliates and subsidiaries

30. Old images and security vulnerabilities
image-inspector
Can we run security scan on image before it runs?
image-inspector
oscap-docker
Run OpenSCAP on a docker image and serve result
docker run -ti --rm --privileged -p 8080: v /var/run/docker.sock:/var/run/docker.sock openshift/image-inspector --image=some-application: path=/tmp/image-content --serve : scan-type=openscap
Used during build process
Diogenes: What would be a better way to do it?
Objective: verify if images are safe /up2date and do something that would alert/prevent them from running.
Image Streams
Pulling latest fixes
Diogenes: show container catalog with 'freshness' on the image.
© Amadeus IT Group and its affiliates and subsidiaries

31. Guiding thoughts
Platform can be secured from container vulnerabilities
containers do bring risk, but it can be managed
Platform will not solve application vulnerabilities
but it can help
true multitenancy is complex
Start with the principle of least access
grant new capabilities to applications only when needed
© Amadeus IT Group and its affiliates and subsidiaries

32. What we miss Encryption of Secrets!
This might be roadmap
Encryption of Secrets!
Network policies – internal and egress
Generic/pluggable image-inspector?
More fine-grained RBAC.
© Amadeus IT Group and its affiliates and subsidiaries

33. Thank you! AmadeusITgroup amadeus.com amadeus.com/blog
© Amadeus IT Group and its affiliates and subsidiaries
amadeus.com amadeus.com/blog
You can follow us on:
AmadeusITgroup