Declarative Configuration of Repository Credentials for ArgoCD, Using External Secrets Operator

ArgoCD is a popular tool in Kubernetes for continuous delivery. As ArgoCD is used to sync git repositories into Kubernetes, it requires secure handling of the repository credentials. In Kubernetes, handling sensitive info like passwords and keys is not straightforward. The default secrets management is pretty basic, so there’s a big need for a more secure solution. This is where tools like the External Secrets Operator (ESO), Sealed Secrets, and Secret Configuration Interface (SCI) come into play. 

Each tool has its own advantages, but here, we will be focusing on ESO. The advantage of ESO is its ability to seamlessly integrate with external secret management systems like AWS Secrets Manager, automatically create Secret resources in the Kubernetes cluster and continuously synchronize the values. As ArgoCD uses Kubernetes Secrets to store repo creds, ESO is a great way to secure the process.

In this walkthrough, I’ll show you how you can safely store your repo creds in AWS Secrets-Manager, and use ESO to sync these secrets right into your EKS cluster, maintaining a declarative approach with Terraform. Although this example focuses on AWS, the same principles can be applied with minimal effort to synchronize secrets from various external secret management systems such as HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, CyberArk Conjur, etc, and integrate ESO with any Kubernetes cluster, whether hosted on cloud platforms like Azure AKS and Google GKE or managed independently.

Prerequisites 

A Terraform module with EKS cluster

IAM Permissions

We will first start by creating an IRSA (IAM Roles for Service Accounts) for the External Secrets Operator. An IRSA is a feature for EKS that allows you to associate an IAM role with a Kubernetes service account. This integration enables pods running on EKS to use AWS IAM roles, granting them specific permissions to access AWS resources. For the ESO, we need to allow access to secrets in AWS Secrets Manager. We will use the iam-role-for-service-accounts-eks terraform module for creating the IRSA. You can find more info and usage examples for this module in this repo.

```
module "irsa_for_external_secrets" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "5.33.0"

  role_name                                          = "external-secrets"
  attach_external_secrets_policy                     = true
  external_secrets_secrets_manager_arns              = ["arn:aws:secretsmanager:*:*:secret:*"]
  external_secrets_secrets_manager_create_permission = false

  oidc_providers = {
    ex = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["external-secrets:external-secrets"]
    }
  }

  tags = local.tags
}
```

Notice that if you want to use ESO for exposing SSM parameters or KMS keys inside your Kubernetes cluster, you can add the external_secrets_ssm_parameter_arns and external_secrets_kms_key_arns attributes. Also, you would probably want to limit access to only specific ARNs, but for this demo, we will keep it simple.

Then, we will create the external-secrets namespace:

```
resource "kubernetes_namespace" "external_secrets" {
  metadata {
    name = "external-secrets"
    labels = {
      "name" = "external-secrets"
    }
  }
}
```

Now, we will create a service account in the external-secrets namespace, and associate it with the IAM role we created with the IRSA module:

```
resource "kubernetes_service_account" "external_secrets" {
  metadata {
    name      = "external_secrets"
    namespace = "external_secrets"
    labels = {
      "app.kubernetes.io/name"      = "external_secrets"
      "app.kubernetes.io/component" = "controller"
    }
    annotations = {
      "eks.amazonaws.com/role-arn"               = module.irsa_for_external_secrets[0].iam_role_arn
      "eks.amazonaws.com/sts-regional-endpoints" = "true"
    }
  }

  depends_on = [
    kubernetes_namespace.external_secrets
  ]
}
```

Deploy ESO

Now we can deploy the ESO. I will be using helm to install it:

```
resource "helm_release" "external_secrets" {
  name             = "external_secrets"
  repository       = "<https://charts.external-secrets.io>"
  chart            = "external-secrets"
  version          = "0.9.11"
  namespace        = "external_secrets"

  values = [jsonencode(local.external_secrets_config)]
}

locals {
  external_secrets_config = {
    serviceAccount = {
      create = false
      name   = kubernetes_service_account.external_secrets.metadata.name
    }
  }
}
```

The values attribute is used to override the default values of the helm chart. We want the ESO to use our service account that we just created, so we will define a local external_secrets_config which sets serviceAccount.create to false, and uses the kubernetes_service_account.external_secrets that we just created instead.

Now, our ESO is all set up and configured to access secrets in AWS Secrets manager and inject them into Kubernetes secrets inside the cluster. Lets now install ArgoCD and configure the repository credentials using ESO resources. 

Install ArgoCD

Install ArgoCD using helm:

```
resource "helm_release" "argo-cd" {
  name             = "argocd"
  repository       = "<https://argoproj.github.io/argo-helm>"
  chart            = "argo-cd"
  version          = "5.52.2"
  namespace        = "argocd"
  create_namespace = true
}
```

We now want to connect ArgoCD to our private repository, using an access token for authentication. Argo reads repository credentials from Kubernetes secrets labelled with argocd.argoproj.io/secret-type: repository

Create Secret in Secrets Manager

Instead of directly creating the secrets resource, we will create a Secrets in Secrets Manager, and use ESO to fetch its content and create the secret resource for us. First, you will have to manually create the secret in Secrets Manager. You can use Terraform for that too, but as this secret has sensitive data it is best to do it manually. Name the secrets however you want, in this example we will name it RepoCreds. Just make sure that the secret ARN can be accessed by the IAM role that we defined for the ESO’s service account (look at external_secrets_secrets_manager_arns in module.irsa_for_external_secrets ) The secret should have the following fields:

Sync Secret with Cluster

Now we will create our first ESO resources. The External Secrets Operator in Kubernetes defines two primary resources: ExternalSecret and SecretStore. The ExternalSecret resource is used to specify exactly which secrets to fetch from Secrets Manager (or any other external secrets store). It includes details like the secret name in the external store, the specific keys to fetch, and the target name for the secret in Kubernetes. The SecretStore resource defines how to access Secrets Manager, detailing the authentication and configuration required to connect to it, such as credentials and endpoint information. This resource is namespaced. For a cluster-wide SecretStore for all namespaces you can use The ClusterSecretStore resource. It functions similarly to SecretStore but is cluster-scoped, allowing a single ClusterSecretStore to serve multiple namespaces. In our example we will use ClusterSecretStore for configuring the connection to Secrets Manager, and ExternalSecret for creating our ArgoCD repo creds secrets.

Let’s start with ClusterSecretStore :

```
resource "kubernetes_manifest" "cluster_secret_store" {
  manifest = {
    "apiVersion" = "external-secrets.io/v1beta1"
    "kind"       = "ClusterSecretStore"
    "metadata" = {
      "name" = "cluster-secret-store"
    }
    "spec" = {
      "provider" = {
        "aws" = {
          "auth" = {
            "jwt" = {
              "serviceAccountRef" = {
                "name"      = "external_secrets"
                "namespace" = "external_secrets"
              }
            }
          }
          "region"  = data.aws_region.current.name
          "service" = "SecretsManager"
        }
      }
    }
  }
}
```
And the ExternalSecret :
```
resource "kubernetes_manifest" "external_secret" {
  manifest = {
    "apiVersion" = "external-secrets.io/v1beta1"
    "kind"       = "ExternalSecret"
    "metadata" = {
      "labels" = {
        "argocd.argoproj.io/secret-type" = "repository"
      }
      "name"      = "repo-creds"
      "namespace" = "argocd"
    }
    "spec" = {
      "data" = [
        {
          "remoteRef" = {
            "key"      = "RepoCreds"
            "property" = "url"
          }
          "secretKey" = "url"
        },
        {
          "remoteRef" = {
            "key"      = "RepoCreds"
            "property" = "type"
          }
          "secretKey" = "type"
        },
        {
          "remoteRef" = {
            "key"      = "RepoCreds"
            "property" = "password"
          }
          "secretKey" = "password"
        },
        {
          "remoteRef" = {
            "key"      = "RepoCreds"
            "property" = "username"
          }
          "secretKey" = "username"
        },
      ]
      "refreshInterval" = "1m"
      "secretStoreRef" = {
        "kind" = "ClusterSecretStore"
        "name" = "cluster-secret-store"
      }
      "target" = {
        "creationPolicy" = "Owner"
      }
    }
  }
}
```

A few points to notice:

• For serviceAccountRef provides the name of the External Secrets Operator and the namespace where it’s running.
• The ExternalSecret must have the label argocd.argoproj.io/secret-type: repository so ArgoCD will recognize it as a repo secret. Also, create the secrets in the ArgoCD namespace.
• Reference the ClusterSecretStore from the ExternalSecret resource, so that ESO will know how to access the secrets in AWS. This is done within the secretStoreRef block.
• The refreshInterval sets the sync intervals for refreshing the values of the secret. If the values are changed from the AWS side, the changes will be automatically recognized and synced into the cluster by ESO when the refresh interval is over.

And that’s it! Once you apply this to your cluster, you will see that an ExternalSecret resource was created, but also a Secret resource will be available, with all the keys and values from the AWS secrets manager injected inside. ArgoCD will recognize this secret as a repository, and if you go to the ArgoCD UI, you will see the repository listed in the repositories section.