24 Oct, 2017

Securing Kubernetes: Going from k8s to k8sec

by Jack Mannino

Continuing our blog series on technologies we love and use internally at nVisium, let’s take a stroll through the park with Kubernetes. We’ve implemented Kubernetes to deploy and scale our containerized microservices, which allows us to magically easily spin up and manage new services and focus on new features rather than managing containers and infrastructure. However, the speed and simplicity of how you can deploy complex applications with a few strokes of the keyboard can also be our enemy if we don’t bake security into our design. This post is the first in an eight-part series that will explore the broad surface and internals of securely designing your systems as powered by Kubernetes.

What is Kubernetes?

Kubernetes is a lot of things. The Kubernetes website has a great overview that answers this question. It allows us to deploy our containers and Kubernetes handles the hard stuff like making sure our containers auto-scale and rebuild themselves when they fail. Without a container orchestration system, managing the lifecycle for containers as well as managing service discovery and load balancing are tedious tasks.

If you are brand new to Kubernetes, here a few useful resources to help expedite your knowledge:

Kubernetes Architecture

When you spin up a new Kubernetes cluster, there are few things you’ll need to understand right out of the gate. By default, Kubernetes creates several components with that are deployed to your environment. We will review each of these in great depth in the next few posts, but let’s explore them at a high level. Here are some terms and important components and terms to familiarize yourself with:

  • Master - does all of the things you would expect a master to do, like making cluster decisions and scheduling.
  • Worker - VM or physical machine that can run Nodes.
  • Namespace - virtual boundary and scope where users and services can be isolated.
  • Pod - group of one or more containers that share networking and storage and are commonly deployed based on business function.
  • Container - a single container running within a pod; pods can run multiple containers.
  • Workloads - sets of containers deployed as Pods; includes Deployment, DaemonSet, ReplicaSet, StatefulSet, etc.

Kubernetes Architecture

Kubernetes Reference Architecture

Now, the fun starts. What are we building today and how are we getting there? We are hosting Kubernetes ourselves rather than using Azure, AWS and/or Heptio, or GCP implementations. As we mentioned before, we have four microservices being developed by three different teams. Not all of these services require direct API access to each other. Rather, they’re loosely coupled through Apache Kafka, where we subscribe to message topics and publish messages as events occur. Instead of directly communicating with each other, they hook up to Kafka and publish/subscribe as needed. We also run Kafka on top of Kubernetes, which means network segmentation happens at the cluster level rather than at the cloud infrastructure level. Here’s a high-level look at our architecture microservice architecture before we add Kubernetes:

Reference Kubernetes Architecture

Securing Kubernetes

As we’ve seen, the Kubernetes attack surface spans across our cloud services providers, our Kubernetes implementation, the services running on top of it, and our developer ecosystem (CI/CD systems, container registries, infrastructure-as-code, etc.). In order to build secure and robust systems that will scale securely as you continue evolving your architecture, thinking about security early will slow you down less later on.

Network Segmentation And Filtering

When you’re designing a Kubernetes architecture, it is important to think of the current and future intended use cases and to build an architecture that will naturally support them. Some implementations are fairly flat where a single namespace is used, and minimal filtering happens between services. In more complex implementations, you may be leveraging a multi-cloud federated implementation, where you’ll have many more things to consider. At a more granular level, isolation occurs between containers in a Pod as well as isolation from other Pods and Workloads inside of a Namespace. These rules can be implemented within a PodSecurityPolicy and propagated to containers running within your Pods.


Kubernetes provides user and service accounts as well as authenticating nodes to the cluster. User accounts are used for API access, while service accounts are used for intra-Pod operations.

We should be heavily restricting access to the api server, but we also have people who need access to perform various tasks related to deployments and maintenance. Hence, we need a way to authenticate them.

The frameworks, tools, and templates we use to create our k8s clusters can have a significant impact on security as well. For example, if you’re using the Heptio Quick Start CloudFormation template on AWS, did you know that your kubeadm tokens are generated using Python’s random module? random is not cryptographically secure, and as a result, the implementation is weakened:

Python random + kubeadm token

You can find this example within the CloudFormation template on GitHub.

Authorization & Role-Based Access Control (RBAC)

Authorizing user actions happens at a few different layers: Service-level access control within containers in a Pod In 1.7 and above, Role-Based Access Control (RBAC) for user account access via the API (apiserver) Webhook endpoint authorization for alerting and events

Using the SecurityContext object within a Pod’s configuration, we can apply access control to objects using standard Linux UID and GID permissions as well as limit privileges with attributes such as MustRunAsNonRoot.

Roles and rights can be applied for API users. Roles can be configured to be valid across the cluster through ClusterRoleBindings and within a specific namespace with a RoleBinding.

Secure Network Communications

Network communications for Kubernetes occur from an external perspective as well as internally between nodes and Kubernetes nodes, such as the apiserver. In some scenarios, TLS certificates are not validated by default. In other scenarios, communications are configured to utilize plain text protocols by default. In the post on Secure Network Communications, we will teach you how to build an end-to-end encrypted system where your user data is protected using every tool at your disposal.

Secrets Management & Data Storage

Kubernetes provides its own distributed data store with etcd, and can also be configured to leverage your cloud storage infrastructure (e.g. - EBS volumes). Often, your services and applications require sensitive passwords and cryptographic keys at runtime to perform privileged tasks. You want to ensure that these values are passed between services by properly authenticating the requesting container, passing the credentials over the network securely, and by ensuring the secrets arrived with their integrity preserved. You want to ensure that keys and passwords aren’t accessible to just anyone; rather, you want to limit access on a need-to-know basis. Kubernetes offers a secrets management API that gives you the ability to leverage underlying platform features to seamlessly pass your credentials to containers securely at runtime only to the users, containers, and processes you want to access them. As of Kubernetes 1.7, encryption of secrets at-rest is available as a feature. You can even take this a step further and integrate it with services such as AWS Key Management Service (KMS) to ensure full end-to-end encryption of secrets.

Container Security

Kubernetes deploys and runs software packages as containers. For our implementation and examples, we’re using Docker. Contrary to popular belief though, there are also other container types that exist in the world, and they’re also supported by Kubernetes. If you haven’t heard of Container Runtime Interface (CRI-O), go take a look right away.

CRI-O Architecture

We should be asking ourselves questions like:

  • Where is our container registry?
  • Who has access to build and release new containers?
  • What are we doing in between to make sure we’re keeping things up to date?

In addition to just worrying about the Kubernetes piece, a good container management strategy is key in adopting a secure Kubernetes architecture. Why? Your containers may be running very out of date software packages with known vulnerabilities that will result in a breach even though your custom code and Kubernetes setups were solid. Additionally, you want to make sure containers are being built with as little attack surface as possible. This includes using an appropriately small footprint image, minimizing the amount of packages and executables for post-exploitation, and limiting the state and useful data to be exfiltrated from a running container.

Logging and Monitoring

With the high rate of change for a microservices architecture running on Kubernetes, we want to make sure we have a realistic way to keep up with the security-related events happening all day and night. This requires building visibility in at all layers of our architecture. This typically requires considerations at the following layers:

  • Infrastructure/Platform-as-a-Service - For example, if you are an AWS customer and you are using a mix of AWS services and services running on Kubernetes, you should make sure you are maximizing the value you get from CloudTrail.
  • Kubernetes - Security events at the namespace and pod level can be extracted as well as pushing events to a Webhook endpoint as they happen.
  • Application Logging - You should still be performing logging inside of your own services; use OWASP Top 10 2017 RC2 for reference: Insufficient Logging & Monitoring

Coming Up Next

Over the next seven posts, we will take a closer look at each major area involved in securing a Kubernetes architecture. We will examine the architectural, programmatic, and declarative techniques you’ll need to use to be successful. Please come back soon for the next post in this series where we’ll learn how the core Kubernetes components work together and how we can secure them.