When your cloud provider doesn’t offer a VPC and your nodes are exposed to the internet, securing your Kubernetes cluster becomes more than a checklist task — it becomes a creative challenge. This article walks you through how we built a scalable, GitOps-friendly “firewall without a firewall” using Calico on LKE — and how you can too.
During a migration to LKE (Linode Kubernetes Engine) for a client, we faced an interesting challenge: how can you enforce node-level network security in a cloud that doesn’t provide a VPC and whose native firewall isn’t built for dynamic Kubernetes workloads? Linode does offer a Cloud Firewall, but it lacks Kubernetes integration and requires manually managing static instance IDs—none of which scale for auto-scaling node pools. Securing nodes is usually routine, but in this case it became a creative challenge. Ultimately, we devised what you might call a “firewall without a firewall” using Kubernetes-native tools.
As of early 2024, Akamai Connected Cloud (formerly Linode) introduced VPCs in public beta—adding support for private networking and distributed regions (per their blog). Unfortunately, when our team migrated to LKE, this feature wasn’t available to us yet, so we had to improvise.
At that time, LKE’s default configuration left our cluster open to the world – a serious security concern. The official docs did offer manual hardening steps, but we needed something more robust that wouldn’t crumble in a dynamic, auto-scaling environment. In the end, we secured our nodes using Kubernetes-native tools, without relying on a VPC or the native cloud firewall. In the following sections, we’ll walk through how we did it—and how you can apply a similar approach to protect your own clusters if needed.
To get the most out of this article, you should be comfortable with the following tools/concepts:
- Kubernetes CNI (Container Network Interface)
- Calico (Kubernetes networking plugin)
- CRDs (Custom Resource Definitions)
- ArgoCD (GitOps continuous delivery tool)
- Helm charts (for Kubernetes packaging)
(This article was tested with Calico v3.25.0 and Kubernetes v1.32, originally created with Kubernetes v1.27.)
The journey to secure our cluster when the traditional approach is unavailable – step by step
Setting up a firewall (and why it wasn’t viable)
Well, we just need to secure the nodes, right?
We saw that Linode has a native firewall service, so we tried deploying it on all our nodes. We quickly realized it requires the Linode instance ID of each node — something static that won’t adapt to an auto-scaling LKE cluster. How would it handle new nodes or the removal of existing ones?
Time to get creative. With no VPCs and a non-viable native firewall, what were our options? We manage our infrastructure with Terraform (using Linode’s provider), so perhaps we could use a Terraform data source to fetch the node IDs and then dynamically create a firewall (or set of firewalls) for each node. However, that approach still wouldn’t work. With auto-scaling, whenever a node is added or removed, we’d have to rerun ‘terraform apply’ to update the firewall rules. We could set up a webhook to trigger Terraform on each change, but that would introduce significant complexity and overhead — essentially a maintenance nightmare. This path clearly wasn’t going to work for us.
There’s a saying: “8 hours of debugging can save you 10 minutes of reading documentation.” Someone else must have hit this issue before us, so there had to be an existing solution out there. Sure enough, the official documentation had a section on securing LKE nodes that pointed us to a community forum post about this very problem.
Enter Calico
Then we stumbled upon this forum answer which suggested using Calico’s globalNetworkPolicies to secure our nodes.
Calico (aside from being a three-colored cat 🐈) is a Container Network Interface plugin, and in fact LKE’s default CNI. Essentially, the forum advice was to use a Calico policy to protect the hosts, which is really neat – it is basically a “firewall without a firewall”!
In order to achieve this, we’ll proceed in three stages:
1 – Create a basic policy and make sure it works.
2 – Automate it.
3 – Optimize by packaging into a chart.
Entering the first stage – creating the basic policy.
Stage 1 – Creating a Basic Policy (Calico GlobalNetworkPolicy)
According to the official Calico documentation, “A global network policy resource (GlobalNetworkPolicy) represents an ordered set of rules which are applied to a collection of endpoints that match a label selector.”
We need rules to protect our nodes, which means our policy’s selector must match all the nodes in the cluster. We are running on LKE, so our selector is:
selector: has(lke.linode.com/pool-id)Using that selector, we created a basic “default-deny” policy. Essentially, we allow only necessary traffic and then deny everything else. Here’s what the policy (named waf-rules) looked like:
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: waf-rules
spec:
egress:
- action: Allow # (Allow all egress by default in this example)
ingress:
- action: Allow # Allow specific essential ports and networks (see below)
destination:
ports: [10250,10259,10257,179,"30000:32767",6443,6666,6667,"2379:2380"]
nets: ["192.168.128.0/17", "10.0.0.0/8"]
protocol: TCP
- action: Allow # Allow specific UDP ports (nodePort range, WireGuard)
destination:
ports: ["30000:32767", 51820]
nets: ["192.168.128.0/17", "10.0.0.0/8"]
protocol: UDP
- action: Deny # Deny everything else
order: 20
selector: has(lke.linode.com/pool-id)
We denied all traffic except for the Allow rules above. A quick explanation of those allowed ranges:
Nets:
- `192.168.128.0/17` – as recommended by the community thread, allowing any host in our private network to communicate with the cluster.
- `10.0.0.0/8` – the general pod network CIDR. This permits full pod-to-pod and pod-to-node communication within the cluster.
Ports:
- We included ports like 2379, 2380, 6443, etc., based on Kubernetes documentation, the Linode community thread, and Tigera’s official guidelines. These cover Kubernetes control plane components (API server, etcd), kubelet, BGP, nodeports (
30000-32767), etc, ensuring the cluster’s internal operations and nodeport services continue to function.
However, as the official Calico documentation warns, these rules alone won’t take effect on host interfaces without Calico HostEndpoints (HostEndpoints are Calico’s representation of node network interfaces). In other words, we needed to configure HostEndpoints for each node to enforce the above policy at the node (host) level.
Stage 1 (Continued) – Enforcing Policy with Calico HostEndpoints
To enforce the policy on host interfaces, we enabled Calico HostEndpoints. Calico can automatically create a HostEndpoint for each node if you toggle a configuration. We ran:
calicoctl patch kubecontrollersconfiguration default --patch='{"spec": {"controllers": {"node": {"hostEndpoint": {"autoCreate": "Enabled"}}}}}'
to let Calico generate HostEndpoints for all nodes (making sure we had calicoctl installed locally).
With HostEndpoint auto-creation turned on and the policy defined, we moved on to test the setup — starting with the obvious choice:
kubectl apply -f network-policy.yaml
But it failed. The error was:
“`error: resource mapping not found for name: “waf-rules” namespace: “” from “network-policy.yaml”: no matches for kind “GlobalNetworkPolicy” in version “projectcalico.org/v3”
ensure CRDs are installed first“`
Which was… confusing. Calico is installed by default on LKE, including its CRDs. So why was kubectl acting like the resource type didn’t exist? (Don’t worry, the answer awaits in the automation step below)
To unblock ourselves, we tried Calico’s official suggestion:
calicoctl create -f network-policy.yaml
And that worked. The policy was applied successfully, and we saw traffic being filtered at the node level. That confirmed we were on the right path — but it raised a bigger question: how do we automate this in a GitOps-compliant way?
Stage 2 – Automation
How do we automate the creation of Calico’s HostEndpoints?
We’re strong believers in Everything-as-Code (EaC), and our deployments are managed by ArgoCD — so calicoctl shell commands weren’t going to cut it long term. We needed something declarative that could live in Git, be tracked by ArgoCD, and apply automatically on deployment or rollback.
Here’s where it gets tricky: the GlobalNetworkPolicy is just a manifest — we can package it into a Helm chart like any other Kubernetes object. But the HostEndpoint configuration isn’t really a resource. It’s a controller toggle — a patch. So we can’t just declare it like we would a Deployment or Secret.
Instead, we wrote a Kubernetes Job that runs calicoctl patch inside a container. And to ensure it runs at the right times, we used Helm lifecycle hooks: post-install, post-upgrade, and post-rollback. That way, HostEndpoints are always enabled when the chart is active — and never left dangling if it’s removed.
# This job is used to create host endpoints automatically (managed by calico) in the cluster.
# It takes about 10s for the automatic endpoints to create.
# If this does NOT run, then the network rules set in this chart will NOT apply even if present in the cluster.
# This will run on any deployment or upgrade of the chart.
apiVersion: batch/v1
kind: Job
metadata:
name: calico-patch-job
annotations:
"helm.sh/hook": post-install,post-upgrade,post-rollback
spec:
template:
metadata:
name: calico-patch-pod
spec:
containers:
- name: calicoctl
image: calico/ctl:v3.25.0 # Calicoctl version that matches the client on the cluster
command: ["calicoctl", "patch", "kubecontrollersconfiguration", "default", "--patch", "{\"spec\": {\"controllers\": {\"node\": {\"hostEndpoint\": {\"autoCreate\": \"Enabled\"}}}}}"]
restartPolicy: Never
And to clean up after ourselves, we added a pre-delete hook to disable auto HostEndpoints when the chart is uninstalled:
apiVersion: batch/v1
kind: Job
metadata:
name: calico-patch-uninstall-hook
annotations:
"helm.sh/hook": pre-delete
spec:
template:
metadata:
name: calico-patch-uninstall-hook-pod
spec:
containers:
- name: calicoctl
image: calico/ctl:v3.25.0 # Calicoctl version that matches the client on the cluster
command: ["calicoctl", "patch", "kubecontrollersconfiguration", "default", "--patch", "{\"spec\": {\"controllers\": {\"node\": {\"hostEndpoint\": {\"autoCreate\": \"Disabled\"}}}}}"]
restartPolicy: Never
How do we automate applying GlobalNetworkPolicies?
We still wanted to apply the GlobalNetworkPolicy with kubectl, not calicoctl, so it could live inside our Helm chart and be handled declaratively by ArgoCD. But every time we tried kubectl apply, we saw the same CRD error as before.
Eventually, we realized what was wrong — the apiVersion was off. kubectl expected a different API group than what calicoctl used.
We found it out by checking the existing CRDs, looking for GlobalNetworkPolicy specifically:
There it was — but under a different group: crd.projectcalico.org.
We dug deeper by inspecting one of the policies we had already created using calicoctl:
So that was the issue. All we had to do was update the apiVersion in our YAML to match the CRD:
apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
name: waf-rules
spec:
egress:
- action: Allow
ingress:
- action: Allow
destination:
ports: [10250,10259,10257,179,"30000:32767",6443,6666,6667,"2379:2380"]
nets: ["192.168.128.0/17", "10.0.0.0/8"]
protocol: TCP
- action: Allow
destination:
ports: ["30000:32767",51820]
nets: ["192.168.128.0/17", "10.0.0.0/8"]
protocol: UDP
- action: Deny
- action: Log
order: 20
selector: has(lke.linode.com/pool-id)
And just like that, applying it with kubectl worked — no extra tooling required.
Trial run (the need for a cluster role)
With everything in place, we attempted to deploy our manifests to the cluster. We immediately hit an authorization error: the Job’s pod was not allowed to access certain Calico custom resources — first complaining about KubeControllersConfiguration, then about ClusterInformation. This made sense in hindsight, as our Calico patch job lacked the necessary Kubernetes permissions.
To resolve this, we added the proper RBAC (Role-Based Access Control) rules so the job could do its work. The calicoctl container needed to patch Calico’s KubeControllersConfiguration (which also involves reading the ClusterInformation CRD), so we created a ClusterRole to allow get, update, and patch on those resources. We also defined a ServiceAccount for the job to run under, and a ClusterRoleBinding to tie the two together. Here are the RBAC manifests we applied:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: calico-patch-cluster-role-binding
subjects:
- kind: ServiceAccount
name: calico-config-manager
namespace: {{ .Release.Namespace }}
roleRef:
kind: ClusterRole
name: calico-patch-cluster-role
apiGroup: rbac.authorization.k8s.io
apiVersion: v1
kind: ServiceAccount
metadata:
name: calico-config-manager
Finally, we updated the Job spec to run under this new ServiceAccount. Below is the revised Job manifest with the serviceAccountName added:
apiVersion: batch/v1
kind: Job
metadata:
name: calico-patch-job
annotations:
"helm.sh/hook": post-install,post-upgrade,post-rollback
spec:
template:
metadata:
name: calico-patch-pod
spec:
serviceAccountName: calico-config-manager # Added service account for the pod
containers:
- name: calicoctl
image: calico/ctl:v3.25.0
command: ["calicoctl", "patch", "kubecontrollersconfiguration", "default", "--patch", "{\"spec\": {\"controllers\": {\"node\": {\"hostEndpoint\": {\"autoCreate\": \"Enabled\"}}}}}"]
restartPolicy: Never
Stage 3 – Packaging as a Helm Chart
At this point, we had all the pieces needed to build a Helm chart for our solution. We packaged the GlobalNetworkPolicy and the accompanying patch jobs into a basic Helm chart so that everything would be applied automatically to the cluster. This chart was designed for our LKE environment, but if you plan to use it, make sure to adjust the values to suit your own cluster’s needs.
In fact, you can make the chart more flexible by parameterizing its values. By moving configuration specifics (like port ranges, CIDRs, or the Calico version) into the values.yaml, you allow users to tailor the chart via simple configuration changes instead of modifying the templates. Our example chart already uses this approach for some settings (like calicoctlVersion and the network policy rules), and it can be extended further for reusability. Below is an example of the Helm chart we created for LKE, showing the values and template files:
values.yaml:
calicoctlVersion: 3.25.0
globalNetworkPolicy:
egress:
- action: Allow
ingress:
- action: Allow
destination:
ports:
- 2379
- 2380
- 10250
- 10259
- 10257
- 179
- "30000:32767"
- 6443
- 6666
- 6667
- "2379:2380"
nets:
- "192.168.128.0/17"
protocol: TCP
- action: Allow
destination:
ports:
- "30000:32767"
- 51820
nets:
- "192.168.128.0/17"
protocol: UDP
- action: Allow # enable pod-to-pod and pod-to-node communication
destination:
nets:
- "10.0.0.0/8"
protocol: TCP
- action: Allow # enable pod-to-pod and pod-to-node communication (UDP)
destination:
nets:
- "10.0.0.0/8"
protocol: UDP
- action: Allow # enable `kubectl exec` (ephemeral ports to localhost)
protocol: TCP
source:
nets:
- 127.0.0.1/32
destination:
ports:
- "32767:49999"
- action: Deny # deny everything else
- action: Log
additionalEgressRules: []
additionalIngressRules: []
templates/service_account.yaml:
apiVersion: v1
kind: ServiceAccount
metadata:
name: calico-config-manager
templates/clusterrole.yaml:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: calico-patch-cluster-role
rules:
- apiGroups: ["crd.projectcalico.org"]
resources: ["kubecontrollersconfigurations","clusterinformations"]
verbs: ["get", "update", "patch"]
templates/clusterrolebinding.yaml:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: calico-patch-cluster-role-binding
subjects:
- kind: ServiceAccount
name: calico-config-manager
namespace: {{ .Release.Namespace }}
roleRef:
kind: ClusterRole
name: calico-patch-cluster-role
apiGroup: rbac.authorization.k8s.io
templates/globalnetworkpolicy.yaml:
apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
name: waf-rules
spec:
egress: # include ICMP to allow pings (optional)
- action: Allow
protocol: ICMP
- action: Allow
protocol: ICMPv6
{{- .Values.globalNetworkPolicy.egress | toYaml | nindent 4 }}
{{- with .Values.globalNetworkPolicy.additionalEgressRules }}
{{- . | toYaml | nindent 4 }}
{{- end }}
ingress:
- action: Allow
protocol: ICMP
- action: Allow
protocol: ICMPv6
{{- .Values.globalNetworkPolicy.ingress | toYaml | nindent 4 }}
{{- with .Values.globalNetworkPolicy.additionalIngressRules }}
{{- . | toYaml | nindent 4 }}
{{- end }}
order: 200
selector: has(lke.linode.com/pool-id)
A quick note on ICMP:
Allowing ICMP (and ICMPv6) can be controversial, but it depends on your environment. In our case, we allowed it to support basic diagnostics (e.g., ping) and monitoring within our internal network. If you’re exposing services to the public internet or working in a high-security environment, you may want to deny ICMP entirely or limit it to specific CIDRs. Always tailor your network policies to your threat model.
templates/patch_job_disable.yaml:
(Helm pre-delete hook job to disable HostEndpoints on uninstall)
apiVersion: batch/v1
kind: Job
metadata:
name: calico-patch-uninstall-hook
annotations:
"helm.sh/hook": pre-delete
spec:
template:
metadata:
name: calico-patch-uninstall-hook-pod
spec:
serviceAccountName: calico-config-manager
containers:
- name: calicoctl
image: calico/ctl:v{{ .Values.calicoctlVersion }} # Calicoctl version that matches the client on the cluster
command: ["calicoctl", "patch", "kubecontrollersconfiguration", "default", "--patch", "{\"spec\": {\"controllers\": {\"node\": {\"hostEndpoint\": {\"autoCreate\": \"Disabled\"}}}}}"]
restartPolicy: Never
templates/patch_job_enable:
(Helm hooks to enable HostEndpoints on install/upgrade)
# This job is used to create host endpoints automatically (managed by calico) in the cluster.
# It takes about 10s for the automatic endpoints to create.
# If this does NOT run, then the network rules set in this chart will NOT apply even if present in the cluster.
# This will run on any deployment or upgrade of the chart.
apiVersion: batch/v1
kind: Job
metadata:
name: calico-patch-job
annotations:
"helm.sh/hook": post-install,post-upgrade,post-rollback
spec:
template:
metadata:
name: calico-patch-pod
spec:
serviceAccountName: calico-config-manager
containers:
- name: calicoctl
image: calico/ctl:v{{ .Values.calicoctlVersion }} # Calicoctl version that matches the client on the cluster
command: ["calicoctl", "patch", "kubecontrollersconfiguration", "default", "--patch", "{\"spec\": {\"controllers\": {\"node\": {\"hostEndpoint\": {\"autoCreate\": \"Enabled\"}}}}}"]
restartPolicy: Never
Chart.yaml:
(basic chart metadata)
apiVersion: v2
name: calico-net-conf
description: A basic chart for configuring Calico network policies and other Calico resources.
type: application
version: 0.1.0
And that’s it! At this stage, our “firewall without a firewall” solution was fully packaged and ready to deploy using our favorite CD tool (ArgoCD in our case) or simply via a Helm install.
Lessons Learned and Best Practices
Know your CRDs
Whenever you deploy a third-party application in your cluster, chances are it comes with its own Kubernetes Custom Resource Definitions (CRDs). These CRDs are frequently where installations fail if they’re not handled properly. We even encountered a couple of F.A.B. (“Frequently Asked Bugs” 🙂) related to CRDs:
The CRDs are not installed– Make sure to install any required CRDs (usually found in the app’s GitHub repository) before applying the application’s manifests. Using the correct version of the CRDs for your app is critical.The CRDs are not in the same API version– This one often masquerades as the first issue: you have installed the CRDs, but your manifest is using the wrong API group/version for them. In our case, we initially referenced the Calico policy CRD under the wrong API group, causing Kubernetes to complain that the resource didn’t exist. (See the earlier section “Automating globalNetworkPolicies creation” for how we discovered and fixed this.) Ensuring your manifests use the exact apiVersion that the CRD expects will save you a lot of headache.
Always package into helm
I once heard someone quip, “What is your job exactly? You’re just running helm install.” Well, I am – and it’s awesome. I always strive to bundle applications and infrastructure configs into Helm charts. That way, the next time we need to deploy them, it’s literally just a git push and voilà – ArgoCD takes care of the rest. If something goes wrong, we can roll back with minimal effort. And if we need to see what changed (or who made a change), we simply check our GitOps repo for a clear history. In short, using Helm charts and GitOps means deployments are reproducible, auditable, and easily revertible.
Consider helm hooks
If you need to run certain tasks on your cluster during chart installation, upgrade, or deletion, consider using Helm hooks. They’re straightforward to implement – just add the appropriate hook annotations to your Kubernetes manifests. (For a concrete example, see the “Automating hostEndpoints creation” section above, where we used Helm hooks to run a Job at install/upgrade time.) Helm hooks allow you to automate cluster-side actions tied to your chart’s lifecycle events.
However, a word of caution: don’t overuse Kubernetes jobs as a band-aid for deeper problems. It might be tempting to automate a job to run a script that “fixes” an issue (such as restarting a component or deleting a resource) when things don’t work as expected. Resist that urge. Using jobs in place of a proper solution can prevent you from understanding and addressing the root cause – and that quick fix may come back to haunt you later.
For example, if we had relied solely on a Helm hook Job to apply our Calico network policy, we might never have realized that our manifest’s apiVersion was wrong for the GlobalNetworkPolicy CRD. The policy would have been created outside of ArgoCD’s purview, so ArgoCD wouldn’t show it or monitor its health. Remember: resources applied purely via jobs aren’t managed by ArgoCD. A hook Job might complete successfully even if the thing it created fails silently afterward – and you’d get no native alert about that failure. Furthermore, when you uninstall the chart, any resources that were created by jobs (and thus not tracked in Git) will not be removed; they’ll linger in your cluster unless you manually clean them up.
In short, use jobs with Helm hooks to automate necessary setup/teardown tasks, but make sure the end state is something your GitOps or CD tool can still manage and observe. Jobs with hooks should enhance your deployment, not obscure it.
Conclusion
In the end, we achieved our goal: the cluster is secure, and deploying our solution is as simple as a git push — in our case, that turned out even smoother than managing a traditional cloud firewall. More importantly, this journey taught us to resist the urge to rush forward with ad-hoc fixes. Sometimes it pays to pause and spend a short block of time (even a 25-minute Pomodoro sprint) investigating better solutions that already exist and figuring out how to integrate them. By doing so in this project, we arrived at a cleaner, more robust approach – essentially a “firewall without a firewall.” This story is a testament to how a bit of creativity and Kubernetes-native tooling can fill the gaps when your cloud provider’s features fall short.
References and further read:
- Protect hosts and VMs | Calico Documentation – Official guide on protecting host network interfaces (and avoiding cutting off all host connectivity).
- Get started with Calico network policy – Explanation of using standard
kubectlversus Calico’scalicoctlfor applying network policies. - Securing k8s cluster (19155) | Linode Questions – Linode Q&A thread discussing how to secure LKE clusters (source of the Calico GlobalNetworkPolicy idea).
- Global network policy | Calico Documentation – Documentation for GlobalNetworkPolicy resource and its fields.
- Apply Calico policy to Kubernetes node ports – Official doc on how Calico policies interact with NodePort services.
- Helm | Chart Hooks – Documentation on using Helm hooks for running tasks during chart lifecycle events.
- Configure calicoctl | Calico Documentation – Guide to setting up and using
calicoctlto manage Calico components. - Get started with Calico network policy – Tutorial for getting started with Calico network policy on Kubernetes.
- Calico HostEndpoints (Calico docs) – Reference for Calico HostEndpoint resources (how Calico represents node interfaces for policy enforcement).
