Nicolas is the main developer advocate at Ondat. He is an experienced hands-on technologist who has worked in cloud native technologies, open source software, virtualization and data center networking for 17 years. You’ll often see him speak at global tech conferences and online events.
The biggest problem with Kubernetes Secrets? They are neither secret nor secure.
When users want to store sensitive data such as authentication tokens, SSH (Secure Shell Protocol) keys, passwords, etc. in Kubernetes they use the open source platform secret object. Unfortunately, Kubernetes does little to secure this information once it is stored in a secret object. In fact, the data stored in a secret object is simply coded in Base64 – which is simple to decode and encode – and are not crypt at all.
An alternative approach can be to add a static encryption key to the encryption configuration file. But since this file is usually hosted on control plane nodes, anyone with access to those nodes can crack all the secrets.
So how do we actually secure sensitive information in Kubernetes?
We use a Key Management Service (KMS) provider like Hashi Corp. To jump.
HashiCorp Vault Overview
HashiCorp Vault has a variety of security applications, but it’s most commonly used for:
- Identity-Based Access: Using role-based access control (RBAC), Vault grants or restricts access to creating secrets or other users’ access. Vault also allows the generation of temporary secrets to provide machine or user access to secrets or servers on a limited basis, as needed, ensuring that there are no unnecessarily permissive access privileges available for exploitation. .
- encryption: Vault offers both in-transit and at-rest encryption, which is one of the main services we care about when it comes to protecting Kubernetes secrets. In transit, Vault uses TLS encryption, while at rest it uses AES 256-bit CBC encryption.
- Secrets management: Vault’s main use case in relation to Kubernetes is its secrets management capabilities. Vault can store API keys, database credentials, environment variables, and more.
While the encryption and secrets management features that Vault offers its users are attractive for a Kubernetes use case, there is still a problem with how HashiCorp Vault works with Kubernetes secrets. In fact, this challenge is common to most, if not all, KMS services: they are not native to Kubernetes.
The challenge of managing Kubernetes secrets
By default, Kubernetes stores Secrets objects in the etcd database. KMS service providers like Vault, however, tend to store secrets externally in their own systems by default.
From the perspective of a third-party KMS provider, this makes sense. By storing secrets outside of the Kubernetes etcd database, they can do more with those secrets and build a product roadmap around the more comprehensive secrets management capabilities that external storage allows.
It’s great for the vendor – it forces developers and infrastructure managers to spend more time with their tool, they become more sticky with their customers, and more users are forced to develop skills in all of the features of their solution. There’s nothing wrong with mastering a great solution, but most developers and infrastructure managers prefer to limit the number of tools they use.
Ideally, users could perform infrastructure and development-related tasks, such as testing, data protection, failover and availability, etc., in the same location where they perform security-related tasks, like managing secrets, i.e. in Kubernetes.
Keeping the workflow as much as possible in Kubernetes allows for more efficiency, increases the number of steps that can be automated, and limits the burden of additional training and maintenance. Crucially, Kubernetes also works the same no matter which cloud provider(s) you use. In contrast, secrets management within HashiCorp Vault works differently for each cloud provider, requiring cloud-specific skills and increasing complexity. And when it comes to security, a simple workflow creates far fewer risks than a complicated workflow.
Staying within Kubernetes allows you to become agnostic towards public cloud providers. Kubernetes is the de facto cloud operating system and can be used as the standard to build modern applications and infrastructure regardless of cloud, using the same technologies and patterns for all. Therefore, you can focus on Kubernetes skills, and the cloud-specific skills required are limited. This allows customers to build a solid multicloud strategy.
How to Run Vault on Kubernetes with Kube-Native Secrets Management
Fortunately, Vault gives us a way to at least get started with a Kube-native approach to secrets management.
Vault has a variety of different Secret Engines. These are essentially different approaches to storing, generating and encrypting data. These engines can focus on connecting to specific services like Amazon Web Services or Google Cloud, simply storing and reading data, providing specific forms of encryption and more.
Our interest lies in the Transit Secrets engine.
Transit operates as an encryption engine as a Secrets service – it does not store data sent to its engine but simply encrypts or decrypts data in transit.
This allows us to manage secrets natively in Kube. Because Kubernetes wants and is designed to store data in etcd, all we have to do is request encryption services from Vault through the Transit Secrets engine whenever we need them. Managing Kubernetes Secrets.
Create secrets with Vault in a Kube-native way
Vault was not designed for native Kube applications, and Kubernetes only provides a generic high-level API that offloads encryption/decryption tasks to an external KMS provider. Thus, DevOps professionals must do work in order to integrate the two solutions. To speed up this integration as much as possible, Ondat is sponsoring an open source project called Trousseau.
Keychain acts as a proxy between Kubernetes and Vault (or any other KMS provider). Using Keychain, users and workloads can manage Kubernetes secrets using the native Kubernetes API and the kubectl CLI to access encryption as a service from a KMS provider. In the specific use case we’re discussing today, Keychain would allow a user or workload to use the kubectl CLI to request encryption from Vault’s Transit Secrets engine.
Keychain works because Kubernetes already offers a plugin to support KMS providers. Implementing Keychain with this plugin allows users to quickly connect to any KMS provider and encrypt secrets on the fly using native Kube tools and practices that your team is already familiar with.
Once you create a secret in Kubernetes, the Kube API calls Keychain, which then sends the encryption request to a KMS provider (Vault in this case). Then the KMS provider performs the encryption without storing the associated data. Instead, it sends it back to Keychain, which then sends the data to the Kube API, which in turn stores the encrypted resource in etcd.
The result is faster time to market, greater security, and reduced costs associated with training your team on new tools and practices. You can learn more about the Trousseau project hereas well as a hands-on lab that walks you through setting up Trousseau with Vault and Kubernetes, and receives information on how to join the open source project.
The selected image via Pixabay.