1. Federated Cluster Selector
Kubernetes 1.7
Federated Cluster Selector
Dan Wilson, Principal Architect, Concur

2. Agenda
Why k8s & CoreOS?
Concur
Federated Cluster Selector
Q & A

3. 20+ 6,800 120 70% $87B +45M 8K Years as industry leader
Headquartered in Bellevue, WA.
Offices in North America, Europe and Asia Pacific
6,800
employees worldwide
120
Countries with client representation
70%
Of Fortune 500 & 100 companies use Concur
$87B
of expenses processed in 2016
+45M
End users 8K
TripLink clients
Concur has been the industry leader for more than 20 years, since our founding in Today, we have 6,800 employees worldwide and offices in North America, Europe, and Asia Pacific. 70 percent of Fortune 500 and 100 companies use Concur. In 2016, we processed $87 billion in expenses. 3

4. Registered mobile users
Concur growth
Expense transactions
Bookings growth
103
million
110+
million
2016
2017
Provisioned users
45+
million
2016
Registered mobile users
7.1
million 8+
million
2011
2012
2013
2014
2015
2016
2017
2018
2016
2017

5. A little about me Principal Architect
at Concur since 1998 in many roles - DBA, Ops Manager, Architect, etc.
working w/ k8s since mid 2015
contributor to k8s and participant in federation SIG
danwilson on github, k8s slack & gmail
Tripit
Hipmunk

6. Why Kubernetes? Largest community support Gifted to Linux Foundation
Protection from shifts in container technology
Pluggable API
Declarative syntax for the complete make up of services
Capable of isolation between namespaces
Built in logging, service discovery, etc.
Networking design eliminates port conflicts of containers

7. Why CoreOS? Simple updates by channel Designed to run as a cluster
Low overhead
Best documentation for k8s

8. Why CoreOS? Simple updates by channel Designed to run as a cluster
Low overhead
Best documentation for k8s
Explain hypercar components and how they relate to building a kubernetes cluster.
CoreOS makes it easy by already having the pieces tested together (kernel, flannel, etcd, systemd, etc)

9. CoreOS

10. Concur
staging
prod CI
N clusters

11. Concur CI
N clusters

12. k8s @ concur Load Balancers API svc K8s Cluster Master worker1 coreOS
workerN
coreOS
Master
master
coreOS
Monitoring
svc
svc
svc
Master
Master
newRelic & prometheus
etcd
coreOS
ELK
Logstash forwarder
flannel

13. High Availability All things fail, yes everything!
K8s HA guide:
cluster etcd – 3, 5 or 7 nodes
replicated + load balanced api servers
many minions
Plan for unknown unknowns
Locking thresholds or bad query plans are known unknowns

14. High Availability
If cluster == alwaysOnline { glog.V(1). Infof(“nothing to do here!”) }
kubelet to container which messed up dns settings

15. k8s @ concur – Multi-clusterLB
VIP
pool
LBaaS
API
K8s 1 P
K8s 2 P
K8s 3 P
svc
svc
svc

16. Sample project https://github.com/concur/kubegowatcher
A sample golang program that leverages kubernetes watch endpoints using the client-go library.
Designed to run as a container on k8s
example of setting annotation on the service
example of checking event delay
Apache 2.0 license
Add your own business logic
Contribute back improvements to make it better
Docker golang setup for kubernetes f5 plugin

17. F5 Kubernetes Integration
Docker golang setup for kubernetes f5 plugin

18. K8s @ Concur – k8s upgrades
Location US
EUROPE
Zone A B C
Cluster v.1.2
us12a
us12b
us12c
eur12a
eur12b
eur12c
Best way to get to v.1.3?
Blue/green

19. K8s @ Concur – k8s upgrades
Location US
EUROPE
Zone A B C
Cluster v.1.2
us12a
us12b
us12c
eur12a
eur12b
eur12c
Cluster v.1.3
us13a
us13b
us13c
eur13a
eur13b
eur13c

20. Deployment Approaches
Individual projects run kubectl commands
Spec Repo
Job to pull and process the spec repo
“kubectl apply”
Works well if all kubernetes objects are deployed to all clusters
Custom Pipeline
Built into the CI/CD system
Runs kubectl or makes API calls
Custom API

21. K8s @ Concur Deployment Challenges Stale kubectl scripts
Stale kubectl config
Handling credentials for each ci job
Many commands for each cluster
Handling proxies
Pod -> rc -> deployment
So, we created skipper

22. Concur

23. CI Deployment API K8s API calls for service & deployment
test14a, b & c
Deploy petshop:v1 to test
So what does this skipper thing do anyways?

24. CI Deployment API US14a, b & c K8s API calls for service & deployment
EMEA14a, b & c
Deploy petshop:v1 to prod
So what does this skipper thing do anyways?
APAC14a, b & c

25. Config API Add/remove cluster EMEA14a, b & c
So what does this skipper thing do anyways?

26. Setup API
Output kubectl config commands for all cluster zones requested
curl /setup
| bash
So what does this skipper thing do anyways?

27. Concur

28. K8s @ Concur Adding a feature to kubernetes federation
v1.7 – deploying to clusters based on label #29887
Other potentials…
Location specific overrides (PodPreset for federation?)
Feature restrictions (for example, don’t allow rc’s)
Docs:

29. Cluster Selector

30. Setup Federation
federation-kubefed/

31. Label Clusters kubectl label cluster cluster123 key1=value1 …
kubectl label cluster cluster124 environment=prod location=europe

32. Add the cluster selector to your yaml
apiVersion: v1
kind: Service
metadata:
annotations:
federation.alpha.kubernetes.io/cluster-selector: '[{"key": "location", "operator":
"in", "values": ["europe"]}, {"key": "environment", "operator": "==", "values":
["test"]}]'

33. v1.7 supported objects Configmap Secret Daemonset Services Ingress
Replicaset
Deplopyment

34. federation.alpha.kubernetes.io/cluster-selector

35. federation.alpha.kubernetes.io/cluster-selector

36. Federation v1.7
Federation deploys to clusters based on ClusterSelector
US14a, b & c CI
EMEA14a, b & c
Deploy petshop:v1 to prod
So what does this skipper thing do anyways?
APAC14a, b & c

37. Running a Service only on PCI Compliant Clusters
PCICompliant == true
US14a, b & c CI
EMEA14a, b & c
Deploy petshop:v1 to pci clusters
So what does this skipper thing do anyways?
APAC14a, b & c

38. Deploy to all locations except for production
Location != prod Or
Location in (test, integration, e2e)
US14a, b & c CI
EMEA14a, b & c
Deploy secret1 to us
So what does this skipper thing do anyways?
APAC14a, b & c

39. Different config/secret by zone
Location = US
US14a, b & c CI
EMEA14a, b & c
Deploy secret1 to us
So what does this skipper thing do anyways?
APAC14a, b & c

40. Different config/secret by zone
Location = EMEA
US14a, b & c CI
EMEA14a, b & c
Deploy secret2 to emea
So what does this skipper thing do anyways?
APAC14a, b & c

41. Different config/secret by zone
Location = APAC
US14a, b & c CI
EMEA14a, b & c
Deploy secret3 to APAC
So what does this skipper thing do anyways?
APAC14a, b & c

42. Different config/secret by zone
No cluster selector or Location in US, EMEA, APAC
US14a, b & c CI
EMEA14a, b & c
Deploy service to all clusters
So what does this skipper thing do anyways?
APAC14a, b & c

43. Cluster Selector Clusters are labeled environment = test
location = europe
pciCompliant = true
Objects in kubernetes can specify label selection criteria based on an annotation “federation.alpha.kubernetes.io/cluster-selector”
location in europe
pciCompliant exists
Environment != test
Leverages apimachinery for matching logic

44. Q & A Thank you @marun @nikhiljindal
danwilson on github, k8s slack & gmail