Kubernetes Secrets – The good the bad and the ugly

Secrets, by definition, should be kept secret, whichever tool you’re using. While there are plenty of best practices for keeping your Kubernetes Secrets actually secret, there are some loopholes that can compromise their security, and might be taken advantage of by malicious entities. This post will cover prevalent best practices for securing your secrets on Kubernetes along with some new approaches for secrets management.

Kubernetes Secrets vs. default handling of sensitive information 

Kubernetes Secrets takes care of storing sensitive and confidential information, such as passwords, OAuth tokens, and ssh keys so that they are not accessible publicly and are encrypted. It is always safer to store the confidential information in a Secret than in the Pod definition or in the Docker Image itself which will be used to create a Pod. 

If K8s Secrets are not used, the credentials are hardcoded in the application code or will be  saved in a file. This application code when pushed to Source Code Management systems will have credentials hardcoded in it. This is not at all recommended, as the credentials can be compromised.

*Red dots represent your secrets

What is K8s Secrets and how does it work?

Kubernetes Secrets is a mechanism that allows for storing secrets within a centralized repository named etcd, essentially a key-value store that holds the complete information of the cluster. These secrets can then be mounted as data volumes or exposed as environment variables to be used by a container in a Pod. 

So the big benefit of K8s Secrets is mainly to decouple secrets from the application code and help reuse them whenever needed inside the cluster.

*Red dots represent your secrets

Beware of the default K8s Secrets

Problem: Etcd stores Secrets in a non-encrypted form – base64, which is an encoding method, not encryption.

Here are five ways to mitigate the above-mentioned problem:

1. Enable Encryption-at-rest. To store the data in etcd in encrypted form, Kubernetes supports encryption at rest for the data in etcd, which by default it is not enabled. This setting can be enabled to encrypt the data stored in etcd. 
2. Limit access to etcd for admin users only. Etcd is a distributed, consistent and highly-available key value store used in Kubernetes to store all cluster data and may contain sensitive information related to access of the deployed services. To restrict the access to this information, only admin users should have access to the etcd database.
3. Enable TLS/SSL between Etcd and your pods. Internal communication in the Kubernetes Cluster can be secured by enabling TLS/SSL between Etcd and pods. 
4. Wipe disks. It’s possible for someone to recover secrets if the disks are not disposed of properly. To ensure that the data on the drive is no longer readable, the disks should be wiped off. 
5. Don’t share or check YAML / JSON files containing base64 secrets into a repo. Anyone who has access to the repository can run away with your credentials. 

For more details regarding the above precautionary measures, refer to Encrypting Secret Data at Rest and Operating etcd clusters for Kubernetes. 

Yet that’s not all…

Even after all of this, unfortunately, the Secrets are still exposed for all containers within a certain pod. Meaning, anyone with Admin permissions to your pod would be able to read your secrets. This is problematic (to say the least) since no proper separation of duties exists as multiple applications with different security content are able to run within the pod. What happens as a result is that the least privileges principle isn’t maintained and malware can simply copy these secrets and exfiltrate them.

To mitigate this, you can choose one of the following:

1. Follow K8s best practice by avoiding running several containers within a pod. This is called “one-container-per-Pod” model. To learn more about this, refer to Kubernetes Documentation here.
2. Use a Secrets Management solution that supports Container Isolation mode.

Furthermore, you should  definitely consider hardening your Admin access by:

1. Making sure admins are who they are by enforcing MFA to Kubectl commands.
2. Use Just-in-Time access solution, where no one has standing permissions to access your K8s cluster and access is given only upon demand for a certain amount of time, using short-lived TLS Certificates.
3. Audit any Kubectl command that any Admin is using.

Additional issues K8s Secrets won’t solve by design

Even after all of the above was configured, there are two by-design problems that will still leave your secrets exposed:

1. Applications tend to expose secrets in audit logs and monitoring systems
2. The same secrets might also be exposed in your other DevOps platforms (CI/CD/CM).

Just-in-Time access: Eliminate the risk of compromised secrets, once and for all

Just-in-Time Access is a concept that enables you to stop worrying about standing permission, meaning secrets. It’s an approach for secrets management that we at Akeyless work by and believe to be the only reasonable approach for today’s complex working environments that are multi-cloud and multi-region.

How does it work? By providing access to whoever asks for it (human or machine) only for the time they need it while using temporary access permission / credentials that are created ad-hoc or on-demand. Next, when a short-lived-secret expires after usage, it eliminates the problem of potential leaks because secrets are created only when needed and then expire.

Imagine this: An entire environment with 0 standing permissions, where no one can access any resource. No need for cleaning processes, and no secrets that can be severely compromised.

Now let’s see how the Just-in-Time approach would be applicable with Kubernetes. For each and every container that is requesting a secret, it will actually get an ephemeral secret that was created only when the container was brought to life. Whenever that container is down, that particular secret would either be deleted or simply expire. 

This way, even if someone grabs a secret, either through an administrator or via log files that the application is writing to – the potential damage is limited to the period of time of the ephemeral secret.