When your security tools cost more than the thing they protect
Lets say you have a micro-services app. Its got a bunch of containers that you’ve orchestrated out with Kubernetes. Deployments, Pods, Daemonsets all over the place. Autoscaling. You are happy. Now it comes time to implement that pesky ‘security’ step. You are a bit nervous, there’s no internal firewall, all the services listen on port 80, no encryption. All the passwords are hard-coded and in the global environment. No one would guess your l33t mysql password right? So you google ‘how is secur networx’. And you click I’m feeling lucky.
Good thing for you google was watching your previous searches and had the microphone on, so it not only corrected your txt-speak spelling but also selected Istio for you.
But suddenly you need to triple the capacity of your cluster. Lets take a look. Here’s kubectl top from my cluster. The lines in red are associated with the securing + auditing. See that last column? Seems we are using 8144MiB for monitoring the thing that is using 2259MiB. And don’t get me started on the CPU.
I said it before, the cloud doesn’t scale down.
Let’s look. istio-system + logging + monitoring == nearly all the resources!
NAMESPACE NAME CPU MEMORY
default ingress-nginx-ingre 4m 146Mi
default ingress-nginx-ingre 0m 3Mi
istio-system istio-citadel-84fb7 0m 12Mi
istio-system istio-egressgateway 2m 35Mi
istio-system istio-galley-655c4f 13m 39Mi
istio-system istio-ingressgatewa 3m 37Mi
istio-system istio-pilot-6cd69dc 8m 84Mi
istio-system istio-policy-77f684 89m 419Mi
istio-system istio-policy-77f684 97m 521Mi
istio-system istio-policy-77f684 99m 492Mi
istio-system istio-policy-77f684 62m 345Mi
istio-system istio-policy-77f684 63m 351Mi
istio-system istio-sidecar-injec 13m 27Mi
istio-system istio-statsd-prom-b 34m 23Mi
istio-system istio-telemetry-77f 76m 440Mi
istio-system istio-telemetry-77f 105m 559Mi
istio-system istio-telemetry-77f 109m 525Mi
istio-system istio-telemetry-77f 106m 574Mi
istio-system istio-telemetry-77f 79m 437Mi
istio-system prometheus-84bd4b97 51m 689Mi
kube-system cert-cert-manager-6 2m 22Mi
kube-system heapster-6c4947855f 0m 41Mi
kube-system kube-dns-v20-5fd69f 18m 27Mi
kube-system kube-dns-v20-5fd69f 18m 28Mi
kube-system kube-proxy-5rhch 3m 36Mi
kube-system kube-proxy-dxk9f 3m 42Mi
kube-system kube-svc-redirect-d 11m 156Mi
kube-system kube-svc-redirect-z 5m 110Mi
kube-system kubernetes-dashboar 0m 15Mi
kube-system metrics-server-64f6 0m 26Mi
kube-system tiller-deploy-895d5 0m 45Mi
kube-system tunnelfront-7794f9f 21m 16Mi
logging elasticsearch-867b4 567m 1420Mi
logging fluent-bit-56d6z 21m 11Mi
logging fluent-bit-8cbnl 17m 12Mi
logging logging-fluentd-69f 1m 59Mi
logging logging-kibana-7684 1m 152Mi
logging sysctl-conf-92l84 0m 0Mi
logging sysctl-conf-hb2vn 0m 0Mi
monitoring alertmanager-monito 1m 15Mi
monitoring monitoring-exporter 3m 37Mi
monitoring monitoring-exporter 1m 14Mi
monitoring monitoring-exporter 1m 10Mi
monitoring monitoring-grafana- 0m 35Mi
monitoring monitoring-promethe 2m 30Mi
monitoring prometheus-monitori 7m 176Mi
socks carts-6994d7d589-6j 5m 340Mi
socks carts-db-7dd64bfd7b 5m 96Mi
socks catalogue-849865789 4m 47Mi
socks catalogue-db-6d6667 3m 236Mi
socks front-end-855684fd8 4m 118Mi
socks orders-7d9cf5cb46-d 5m 350Mi
socks orders-db-6db4678bf 5m 93Mi
socks payment-6cdc5b656-8 4m 48Mi
socks queue-master-7b99db 5m 301Mi
socks rabbitmq-7c5fbf778d 7m 127Mi
socks session-db-fdd649d6 3m 52Mi
socks shipping-5b9ffdbdfb 5m 321Mi
socks user-84ccd5fd57-2vp 4m 47Mi
socks user-db-7dcc9649dc- 4m 83Mi