How To Do DevSecOps for Kubernetes

In this article, we’ll provide an overview of security concerns related to Kubernetes, looking at the built-in security capabilities that Kubernetes brings to the table.

Kubernetes at the center of cloud-native software

Since Docker popularized containers, most non-legacy large-scale systems use containers as their unit of deployment, in both the cloud and private data centers. When dealing with more than a few containers, you need an orchestration platform for them. For now, Kubernetes is winning the container orchestration wars. Kubernetes runs anywhere and on any device—cloud, bare metal, edge, locally on your laptop or Raspberry Pi. Kubernetes boasts a huge and thriving community and ecosystem. If you’re responsible for managing systems with lots of containers, you’re probably using Kubernetes.

The Kubernetes security model

When running an application on Kubernetes, you need to ensure your environment is secure. The Kubernetes security model embraces a defense in depth approach and is structured in four layers, known as the 4Cs of Cloud-Native Security:

1. Cloud (or co-located servers or the corporate datacenter)
2. Container
3. Cluster
4. Code

Security at outer layers establishes a base for protecting inner layers. The Kubernetes documentation reminds us that “You cannot safeguard against poor security standards in the base layers by addressing security at the Code level.”

At the Cloud layer, security best practices are expected of cloud providers and their infrastructure. Working inward to the Cluster layer, cluster components need to be properly secured, as do applications running in the cluster.

At the Container level, security involves vulnerability scanning and image signing, as well as establishing proper container user permissions.

Finally, at the innermost layer, application code needs to be designed and built with security in mind. This is true whether the application runs in Kubernetes or not.

In addition to the 4 C’s, there are the 3 A’s: authentication, authorization, and admission. These measures apply at the Cluster layer. Secure systems provide resource access to authenticated entities that are authorized to perform certain actions.

Authentication

Kubernetes supports two types of entities: users (human users) and service accounts (machine users, software agents). Entities can authenticate against the API server in various ways that fit different use cases:

• X509 client certificates
• Static tokens
• Bearer tokens
• Bootstrap tokens
• Service account tokens
• OpenID Connect tokens

You can even extend the authentication process with custom workflows via webhook authentication.

Authorization

Once a request is authenticated, it goes through an authorization workflow which decides if the request should be granted.

The main authorization mechanism is role-based access control (RBAC). Each authenticated request has an HTTP verb like GET, POST, or DELETE, and authenticated entities have a role that allows or denies the request. Other authorization mechanisms include attribute-based access control (ABAC), node authorization, and webhook mode.

Admission

Admission control is a security measure that sets Kubernetes apart from other systems. When a request is authorized, it still needs to go through another set of filters. For example, an authorized request may be rejected by an admission controller due to quotas or due to other requests at a higher priority. In addition to validation, admission webhooks can also mutate incoming requests as a way of processing request objects for use before reaching the Kubernetes API server.

In the context of security, pod security admission might add an audit notation or prevent the scheduling of a pod.

Secrets management

Secrets are an important part of secure systems. Kubernetes provides a full-fledged abstraction and robust implementation for secrets management. Secrets are stored in etcd—Kubernetes’ state store—which can store credentials, tokens, SSH keys, and any other sensitive data. It is recommended to store small, sensitive data only as Kubernetes Secrets.

Data encryption

When you want to store a large amount of data, consider using dedicated data stores like relational databases, graph databases, persistent queues, and key-value stores. From the vantage point of security, It’s important to keep your data encrypted both at rest (when it is simply sitting in storage) as well as in transit (when it is sent across the wire). While data encryption is not unique to Kubernetes, the concept must be applied when configuring storage volumes for Kubernetes.

Encryption at rest

There are two approaches to encryption at rest. The first approach uses a data store that encrypts the data for you transparently. The other approach makes the application responsible for encryption, then storing the already-encrypted data in any data store.

Encryption in transit

Eventually, you’ll need to send your data for processing. Because the data is often (necessarily) decrypted at this point, it should be sent over a secure channel. Using  HTTPS, STCP, or SFTP for secure transit of data is best practice.

Kubernetes services can be configured with specific ports like 443 for HTTPS.

Managing container images securely

Kubernetes orchestrates your containers. These containers are deployed as images. Many Kubernetes-based systems take advantage of third-party images from the rich Kubernetes ecosystem. If an image contains vulnerabilities, your system is at risk.

There are two primary measures to safeguard your system. First, use trusted image registries, such as Google Container Registry, AWS Elastic Container Registry, or Azure Container Registry. Alternatively, you may run your own image registry using an open-source project like Harbor and curate exactly which trusted images you allow.

The other measure is to frequently scan images for vulnerabilities as part of the CI/CD process.

Defining security policies

Kubernetes and its ecosystem provide several ways to define security policies to protect your systems. Note that the built-in Kubernetes PodSecurityPolicy resource is deprecated and will be removed in Kubernetes 1.25. At the time of this writing, the Kubernetes community is working on a lightweight replacement. However, the current recommendation is to use a robust third-party project—for example, Gatekeeper, Kyverno, or K-Rail—as a policy controller.

Policies can be used for auditing purposes, to reject pod creation, or to mutate the pod and limit what it can do. By default, pods can receive traffic from any source and send traffic to any destination. Network policies allow you to define the ingress and egress of your pods. The network policy typically translates to firewall rules.

Resource quotas are another type of policy, and they’re particularly useful when multiple teams share the same cluster using different namespaces. You can define a resource quota per namespace and ensure that teams don’t try to provision too many resources. This is also important for security purposes, such as if an attacker gains access to a namespace and tries to provision resources (to perform crypto mining, for example).

Monitoring, alerting, and auditing

We have mostly discussed preventative measures thus far. However, a crucial part of security operations is detecting and responding to security issues. Unusual activity could be a sign that an attack is in progress or that a service is experiencing degraded performance. Note that security issues often overlap with operational issues. For example, an attacker downloading large amounts of sensitive data can cause other legitimate queries to time out or be throttled.

You should monitor your system using standard observability mechanisms like logging, metrics, and tracing. Kubernetes provides built-in logging and metrics for its own components. Once a serious problem is discovered, alerts should be raised to the relevant stakeholders. Prometheus can provide metrics monitoring and alerting, while Grafana provides dashboards and visualizations for those metrics. These tools, along with AppDynamics or countless others, can serve as effective Kubernetes monitoring solutions.

When investigating an incident, you can use the Kubernetes audit logs to check who performed what action at a particular time.

Conclusion

As a robust platform, Kubernetes comes out of a developer community that takes security seriously. Kubernetes provides many security-oriented features and capabilities. However, the onus is still on the developers, administrators, and security personnel to utilize these features, configuring them and using them correctly to protect the system, the users, and their data.

Related resources

We’d love to hear what you think. Ask a question or leave a comment below.
And stay connected with Cisco DevNet on social!

LinkedIn | Twitter @CiscoDevNet | Facebook |  Developer Video Channel

Share: