The Best Way to Install Apps on Multiple Clusters in ArgoCD
The Challenge: Managing Apps Across Dozens of Clusters
Imagine managing 40–50 Kubernetes clusters, each running a wide variety of applications and tools. Sounds overwhelming, right? Now, think about needing to update one of those tools—maybe it’s a new version or a configuration change. Without a centralized system, you’d have to go into each cluster, make the update manually, and hope nothing breaks along the way. It’s a recipe for errors, wasted time, and frustration. But it doesn’t stop there. Each cluster serves a unique purpose—some run on different cloud platforms, others are for development, staging, or production. That means the tools and versions required for each cluster aren’t the same. On top of that, you’ll need a way to customize settings for each environment using separate `values.yaml` files while still keeping everything organized and manageable. The complexity is undeniable: how do you maintain control over all clusters while allowing flexibility for unique needs? When we faced this challenge, it was clear we needed a solution that could handle this scale and complexity without becoming a bottleneck.
The ultimate solution for managing apps across clusters
To address the complexity of managing applications across multiple clusters, we implemented a dynamic and scalable solution using ArgoCD ApplicationSets. First, we created an ApplicationSet to dynamically generate a list of applications for each cluster. This process uses a matrix generator, which iterates through the list of clusters, pulling their details based on specific labels defined in the cluster’s secret within ArgoCD. These labels determine which clusters are targeted for the application deployment. For each cluster, the ApplicationSet dynamically generates a list of applications and tools to be deployed. To simplify and centralize the management, we encapsulated this logic within Helm charts. This approach allows us to define the list of applications in a modular and reusable way, making it easy to adjust configurations, add new applications, or upgrade existing ones. Using this setup, every cluster receives the correct set of applications and configurations dynamically. The Helm chart ensures that all deployments are managed from a single centralized source, enabling seamless updates and efficient version control. This solution not only reduces manual effort but also ensures consistency across all environments while allowing for environment-specific customizations through separate values files.
Code Optimization
Optimizing cluster management with labels in Argo CD
This section explains how using labels in Argo CD improved cluster management, making it more dynamic and flexible.
Identifying and mapping clusters with labels
Labels, defined within the secret of the cluster in Argo CD, are a simple and effective way to identify and map clusters in the ApplicationSet, making cluster selection easy and automated. By using labels like pim/instance-name and argocd.argoproj.io/secret-type in the secret of the specific cluster, you can filter and target specific clusters for deployment without any manual effort. This approach removes the need for hardcoding, making it effortless to scale and saving time and effort by avoiding the creation of a separate ApplicationSet for each cluster.
Example Secret Template:
Example Secret Template:
apiVersion: v1
kind: Secret
metadata:
name: my-cluster-secret
labels:
pim/instance-name: my-clusters
argocd.argoproj.io/secret-type: cluster
type: Opaque
data:
# Your secret data here
In this example, the labels are defined within the secret, allowing Argo CD to automatically target and map the cluster for deployment based on the metadata.
Here’s how it looks in the ApplicationSet:
selector:
matchLabels:
pim/instance-name: my-clusters
argocd.argoproj.io/secret-type: cluster
In this example, matchLabels selects clusters based on their instance name and secret type, simplifying the process of choosing the right cluster for deployment.
Using labels for customizing applications
Labels provide a simple way to make your configurations more flexible by adding metadata. This allows you to control application names and behavior better. For example, labels like pim/region let you create unique and meaningful application names, making it easier to track them across different environments and regions. Additionally, you can use metadata.labels in any part of the ApplicationSet configuration to tailor it to your needs.
Example Code:
metadata:
name: 'cluster-{{`{{ index .metadata.labels "pim/region" }}`}}'
Managing applications with ApplicationSet
ApplicationSet allows you to define and deploy a list of applications to a single cluster using reusable templates. It simplifies managing multiple applications by automating the deployment process and reducing manual work.
Example Configuration:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: example-applicationset
namespace: argo-cd
spec:
generators:
- list:
elements:
- releaseName: reloader
chart: reloader
repoURL: https://stakater.github.io/stakater-charts
version: v1.0.*
namespace: common
- releaseName: keda
chart: keda
repoURL: https://kedacore.github.io/charts
version: 2.15.*
namespace: keda
template:
metadata:
name: '{{ .releaseName }}-{{ .namespace }}'
spec:
destination:
namespace: '{{ .namespace }}'
source:
repoURL: '{{ .repoURL }}'
chart: '{{ .chart }}'
targetRevision: '{{ .version }}'
syncPolicy:
automated:
prune: true
selfHeal: true
Key benefits of using ApplicationSet:
- Simplified management: Deploy a list of applications to a single cluster without creating separate ArgoCD applications.
- Reusability: Use templates to easily deploy multiple applications with configurable parameters like versions and namespaces.
- Scalability: Easily add more applications to the list as needed.
- Consistency & automation: Ensures consistent deployments across applications with automated syncing and self-healing.
ApplicationSet streamlines the process of deploying and managing applications within a cluster, saving time and reducing errors.
Deploying multiple applications across multiple clusters
The Matrix generator in Argo CD’s ApplicationSet simplifies deploying multiple applications to multiple clusters. It dynamically generates configurations by combining cluster-specific details with an application list, ensuring consistent, scalable deployments while eliminating manual effort and reducing errors.
This approach addresses exactly the challenge we’re facing! Managing deployments across various environments with multiple clusters and applications can be tedious and error-prone. The Matrix Generator eliminates this complexity by automating the entire process, allowing us to scale deployments effortlessly and with confidence.
How the Matrix generator works
The configuration below demonstrates the Matrix Generator:
The configuration below demonstrates the Matrix Generator:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: infra-apps
namespace: argo-cd
spec:
generators:
- matrix:
generators:
- clusters:
selector:
matchLabels:
pim/instance-name: my-clusters
argocd.argoproj.io/secret-type: cluster
- list:
elements:
- releaseName: cert-manager
chart: cert-manager
repoURL: https://charts.jetstack.io
version: v1.16.*
namespace: cert-manager
- releaseName: external-dns
chart: external-dns
repoURL: https://charts.bitnami.com/bitnami
version: 8.3.*
namespace: cert-manager
template:
metadata:
name: 'cluster-{{ index .metadata.labels "pim/region" }}'
namespace: argo-cd
spec:
project: infrastructure
sources:
- repoURL: '{{ .repoURL }}'
path: '{{ or .path "" }}'
chart: '{{ or .chart "" }}'
targetRevision: '{{ .version }}'
helm:
valueFiles:
- $values/infra-apps/{{ .releaseName }}.yaml
releaseName: '{{ .releaseName }}'
destination:
name: '{{ index .metadata.labels "pim/cluster-name" }}'
namespace: '{{ .namespace }}'
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
selfHeal: true
prune: true
Cluster selection
The clusters generator selects clusters automatically using matchLabels by targeting clusters with the labels pim/instance-name: my-cluster-dev and argocd.argoproj.io/secret-type: cluster ensuring that only the desired clusters are included in the deployment
Application list
The list generator specifies multiple applications along with their Helm chart details, such as name repository and version or Git repository paths. For example, the cert-manager application retrieves its chart from https://charts.jetstack.io and targets version v1.16.*
Dynamic application configuration
The Matrix Generator creates unique deployment configurations for each combination of cluster and application by leveraging the template field to dynamically set the application’s name namespace Helm values and destination cluster using parameters like releaseName repoURL and version
Automation and sync policies
Deployments are automated with sync policies that include selfHeal: true and prune: true to maintain consistency and correct any resource drift automatically.
What does the Matrix generator provide
This setup eliminates manual configuration for multi-cluster deployments. By defining applications and clusters in a single ApplicationSet, you can:
- Scale effortlessly to support hundreds of clusters and applications.
- Ensure consistency across deployments with minimal effort.
- Adapt dynamically to new clusters or applications by simply updating the selectors or application list.
The Matrix Generator makes deploying and managing applications across multiple clusters simpler, faster, and more reliable.
Transforming the configuration to dynamic with Helm Chart
Imagine you need to deploy 10 applications across 40 clusters in 3 environments. You would typically need to define multiple ApplicationSet configurations—one for each environment or cluster group. Furthermore, whenever an upgrade is required or a new application needs to be added, you’d have to go through the files and make the necessary changes manually.
We’ve developed a solution to address this challenge that makes the entire deployment process dynamic using a Helm chart. By defining variables for clusters, environments, and applications, the Helm chart dynamically generates the required configurations, saving hours of manual effort and ensuring consistency. This approach transforms what would otherwise be static and repetitive configuration files into a fully adaptable setup, making your deployments highly efficient, scalable, and adaptable to any changes in your infrastructure or application setup.
The Value of Dynamic Helm Chart Configuration in ArgoCD ApplicationSet
- Dynamic Adaptation: The configuration can be adjusted dynamically for different clusters, environments, or applications without modifying the core logic.
- Scalability: Instead of manually configuring deployments, you can scale easily by updating variables.
- Consistency: It ensures all deployments follow the same structure, reducing misconfigurations and errors.
- Reusability: The same Helm chart can be reused across different projects, saving development time.
- Flexibility: It accommodates various needs, from adding new applications to deploying in new environments, by simply updating input variables.
Apologies for the omission! Here’s the updated version with all the requested sections included, along with additional explanations:
Achieving Dynamic Deployment with Helm and ArgoCD
{{- range $defaultName, $_ := .Values.infra_apps }}
{{- if or .create (eq .create nil) }}
{{- $name := .name | default $defaultName }}
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: {{ $name }}-{{ $.Values.instance_id }}
namespace: argo-cd
spec:
goTemplate: true
generators:
- matrix:
generators:
- clusters:
selector:
matchLabels:
pim/instance-name: {{ $.Values.instance_name }}
argocd.argoproj.io/secret-type: cluster
{{- with .clusters_selector }}
{{- toYaml . | nindent 18 }}
{{- end }}
- list:
elements:
{{- range $releaseName, $_ := .helm_charts }}
{{- if or .enabled (eq .enabled nil) }}
- releaseName: {{ .releaseName }}
{{- if hasKey . "path" }}
chart: ""
path: {{ .path }}
{{- else }}
chart: {{ .chart | default $releaseName}}
path: ""
{{- end }}
repoURL: {{ .repoURL}}
version: {{ .version }}
namespace: {{ .namespace | default $releaseName }}
extraSyncOptions: {{ .extraSyncOptions }}
ignoreDifferences: {{ (.ignoreDifferences) | toYaml | nindent 18 }}
{{- with .parameters }}
parameters:
{{- range $name, $value := . }}
- name: {{ $name | quote }}
value: {{ $value | quote }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
syncPolicy:
preserveResourcesOnDeletion: true
template:
metadata:
name: '{{`{{ .releaseName }}`}}-{{`{{ index .metadata.labels "pim/region" }}`}}-{{ $.Values.instance_id }}'
namespace: argo-cd
spec:
destination:
name: '{{`{{ .name }}`}}'
namespace: '{{`{{ .namespace }}`}}'
syncPolicy:
syncOptions:
- CreateNamespace=true
- '{{`{{ .extraSyncOptions }}`}}'
{{- with .automated_sync }}
{{- if .enabled }}
automated:
{{ toYaml .settings | nindent 10 }}
{{- end }}
{{- end }}
project: {{ $.Values.project }}-infra
templatePatch: |
spec:
{{`{{- with .ignoreDifferences }}`}}
ignoreDifferences:
{{`{{ . | toYaml | nindent 4 }}`}}
{{`{{- end }}`}}
sources:
- repoURL: '{{`{{ .repoURL }}`}}'
chart: '{{`{{ .chart }}`}}'
path: '{{`{{ .path }}`}}'
targetRevision: '{{`{{ .version }}`}}'
helm:
releaseName: '{{`{{ .releaseName }}`}}'
{{`{{- with .parameters }}`}}
parameters:
{{`{{ . | toYaml | nindent 10 }}`}}
{{`{{- end }}`}}
valueFiles:
- $values/config/default/pi-instance/infra-apps/{{`{{ .releaseName }}`}}.yaml
- $values/config/environments/{{ $.Values.project }}/pi-instance/infra-apps/{{`{{ .releaseName }}`}}.yaml
- repoURL: {{ .config_repo_url | default $.Values.defaults.config_repo_url}}
targetRevision: {{ .config_branch }}
ref: values
{{- end }}
---
{{- end }}
Cluster selection (selectors and matchLabels)
The clusters section in the generators part of the template uses label selectors to dynamically match clusters. This is done based on labels like pim/instance-name and argocd.argoproj.io/secret-type: cluster, which are dynamically populated from the Values object. The use of .clusters_selector further enhances this by allowing additional customizations to specify which clusters to match at runtime. This flexibility ensures that the template can adapt to different clusters and environments without needing static definitions.
Helm Chart List
This section dynamically iterates over the .helm_charts array defined in the Values object. Each Helm chart’s configuration—including parameters like releaseName, chart, repoURL, version, and an optional enabled flag—is extracted directly from the Values file.
The configuration in the Values object allows for a highly flexible and environment-specific setup. By defining the list of applications in the Values file, we can control which charts are deployed, their specific configurations, and any additional customizations. This approach ensures that the deployment process adapts dynamically to different environments, such as development, staging, or production, without requiring changes to the template itself.
This design simplifies Helm chart management across environments, centralizing control and enabling rapid adjustments directly through the Values file.
Dynamic Template Metadata (name and namespace)
The template.metadata.name and template.metadata.namespace are dynamically constructed using values like releaseName, region, and instance_id from the Values object. This ensures that each deployment has a unique identifier, which is essential for managing multiple deployments in the same namespace. The use of these Helm variables guarantees that resources are created with the appropriate names and namespaces, maintaining organization and avoiding conflicts.
Sync policy
The syncPolicy is configured to ensure that resources are preserved during deletion by setting preserveResourcesOnDeletion: true. This is a critical feature for managing environments where accidental deletion of resources could have significant consequences.
In dev and stg environments, it is recommended to enable automated synchronization by setting automated_sync.enabled to true. This ensures that changes are applied automatically when updates are detected, making deployments more efficient and reducing the need for manual intervention.
In contrast, for prd (production), we disable automatic synchronization to ensure greater control over what gets deployed. This allows for thorough monitoring and manual approval of changes before they are synced with the production environment.
The ability to enable or disable synchronization dynamically is controlled through the Values file, providing flexibility to adjust the behavior based on the requirements of each environment.
goTemplate and templatePatch
The goTemplate flag enables Go templating within the Argo CD ApplicationSet, allowing us to insert dynamic values and apply conditional logic, such as selectively including or excluding Helm charts based on runtime parameters. This makes the configuration highly flexible and adaptable to different environments.
The templatePatch allows us to customize the default template further. In this case, it adds the ignoreDifferences section, instructing Argo CD to disregard specific differences, such as annotations or labels, between local and live configurations. This prevents unnecessary syncs, improving deployment efficiency and providing more control over the generated resources.
Here is a minimal example of a values.yaml file:
instance_id: ""
instance_name: ""
# -- The project to create the ApplicationSet in
project: ""
defaults:
# -- The URL of the Git repository to use for configuration
config_repo_url: git@github.com:YourOrg/your-config-repo.git
# -- The list of applications to create
infra_apps:
# Example application group
example-app-group:
config_branch: main
automated_sync:
enabled: true
settings:
selfHeal: true
prune: true
helm_charts:
redis:
repoURL: https://charts.bitnami.com/bitnami
version: "20.2.*"
nginx:
chart: ingress-nginx
repoURL: https://kubernetes.github.io/ingress-nginx
version: 4.11.*
namespace: ingress
custom-app:
path: custom-app
repoURL: git@github.com:YourOrg/your-helm-charts.git
version: HEAD
parameters:
app.customParam: "customValue"
