AWS Secrets in EKS— part II: How to securely and easily pass secrets to the cluster?
The million-dollar question is: How to securely and easily pass secrets to the cluster?
On one hand, the right approach is a GitOps mindset: all the cluster’s configurations and resources are stored in Git, making everything readable, dynamic, and recoverable. But on the other hand, we don’t want to expose our secrets in Git.
There are many solutions to this, but in this article, we’ll focus on one of them — the Secrets Store CSI Driver. It is a service that can be installed on the cluster as a Helm chart and allows us to consume secrets directly from our secret provider. This service can work with multiple providers, and in this article, we’ll concentrate on AWS Secrets Manager.
The first step to working with AWS Secrets Manager is to configure the required permissions for the cluster. You can find the complete explanation of how to do it in the first part of my blog.
How does Secrets Store CSI Driver work?
By default, the driver doesn’t generate Kubernetes secrets but simply injects the secret as a volume into the pod, to the path you specify. This provides a more secure approach but may be less convenient since you need to extract environment variables from the secret into the pod, making it less user-friendly. Nonetheless, it can also be configured to generate Kubernetes secrets in addition to the volume.
In this blog, I will explain the basic mechanics of the Secrets Store CSI Driver, and then mention two basic things that are missing in the chart and how I address them.
so, how does Secrets Store CSI Driver work?
Secrets Store CSI Driver contains a custom resource, which is called a Secret Provider Class and is designed to transfer the secrets from the secrets manager to the cluster.
Here is an example of how to create a SecretProviderClass:
- objectName: "staging-mysql-url" #the name of the secret in aws secrets manager
Now, I create a deployment:
- name: aws-secrets
- name: my-app
- name: aws-secrets
In the volumes section we define the volume that refers to the secretProviderClass that we’ve created.
In the volume mounts we define the path that the secrets will be stored inside the pod.
Now, if we exec into the pod to the path /mnt/secrets, a file named
“staging-mysql-url” will be there, and it will contain the value of this secret in aws secrets-manager.
Create Kubernetes Secret
To create a Kubernetes secret, we will add it to our SecretProviderClass:
- objectName: "staging-mysql-url"
objectAlias: db_url #alias that should be different from the object name
- key: DATABASE_URL
objectName: db_url #the alias name
secretName: db-url #the name of kubernetes secret that will be created
Now a secret named db-url will be created, and its data will include this:
DATABASE_URL: <value of the secret, base64>
Of course, it’s possible to create more than one data, or more than one Kubernetes secret. it’s a list of lists.
Your next steps to finalize the process
So far, so good. Now, what is missing in the chart? Two simple things that, once added, will make it perfect.
First, in order to work with AWS Secrets Manager, you need to add an additional installation of a few files that are located here.
The second thing, as I mentioned before, is that you can add a configuration that generates a Kubernetes secret. However, when I initially tried it, it didn’t work. There is a pod where I can see the file and its secret value, but the Kubernetes secret is not created. It turns out (and for some reason, it’s not mentioned in the documentation) that the cluster role of the chart (named
secretproviderclasses-role) does not have permission to create secrets. So, what do we do? We add it to the cluster role of the chart, and it works perfectly.
kubectl edit clusterrole secretproviderclasses-role
and add this block to the cluster role:
But as we said, we want all the cluster configuration to be in Git without the need to run kubectl commands to make it work.
The simple solution is to create a custom Helm chart that uses the official chart as a dependency, and there the AWS installation files are present, along with a cluster role that has permissions for secrets. This cluster role overrides the original cluster role of the chart. Here you can see an example that I created.
Now, when we install this chart, we have a simple and functional system for managing our secrets in Kubernetes.
Did you miss my first article demonstrating how to use Terraform to grant the cluster the required access to the relevant services, so that it is ready with every access it needs when it goes live? You can read it here