Scaling to Zero Is Easy. Waking Up Is the Hard Part.

Idle Kafka workers on Planet Nine’s dev cluster now scale to zero and wake on demand in about half a minute, across every developer environment, with zero change to staging or production.

Planet Nine defends governments, enterprises and critical infrastructure from AI-driven and cyber-enabled threats. I’m a DevOps engineer at Develeap, embedded in their platform team.

Over two phases in August, we got the developer environments on Planet Nine’s dev cluster to let go of what they weren’t using. While idle, their workers now reserve 78% less CPU and 74% less memory than before, which I estimate at about $1,000 a month. Getting there was mostly about one question: how do you make sure a worker that’s asleep always wakes up?

The problem: dev runs everything, all the time

Planet Nine runs a pipeline of about thirty Kafka-driven workers on EKS: crawlers, feature extractors, inference workers, writers. On the dev cluster every developer gets their own environment in its own namespace, running the part of the pipeline they need. At the time there were seven of them, averaging around twenty workers each.

That’s great for isolation, and it means a lot of pods. A developer runs a scan, the pipeline does its work, and then the workers sit there holding CPU and memory until the next one.

The goal: every idle Kafka consumer on dev should scale to zero, wake up by itself when a message arrives, and do that reliably on both queues each worker reads, the normal one and the high-priority one.

What we considered

KEDA was the obvious engine. The design question was how to switch it on for dev without touching anything else.

All workers share one Helm chart, and most of them declare replicas: 1 in a base values file that production also reads. The chart rendered a KEDA ScaledObject only for workers with no replicas key, so I needed a way to hand the replica count to KEDA on dev alone. There were four options:

1. Make scaling always win over replicas. The simplest change, but rendering it against every cluster showed it would also switch seven staging workers to KEDA. Not acceptable.
2. Null out replicas in the dev override. Helm’s merge can’t delete a key, so this can’t work.
3. Delete replicas from the shared base file. That would change production.
4. An opt-in flag. scaling.overridesReplicas: true. When it’s set, the deployment stops emitting a static replicas: and the ScaledObject renders. When it’s not set, nothing changes.

The flag won. The whole chart change came to two one-line edits. Before merging, I rendered all 250 workers on staging, the other non-dev clusters and both production clusters, and diffed them against main. Zero differences.

How we rolled it out

Phase 1 was six workers in one developer’s environment, with no shared config touched, to prove the wake path end to end.

Phase 2, the next day, moved the configuration into the values file that only the dev cluster loads, so all 30 consumers got it in every dev environment. Staging and production read different files and can’t pick it up.

The trigger is a Prometheus query over kafka_consumergroup_lag, which plugs straight into the chart’s existing step-function scaling (lagSteps). One trigger covers both queues with a topic regex, <topic>(_high_priority)?. I verified it by waking the same worker from each queue independently. Workers return to zero after a 600-second cooldown. Only the floors changed: every ceiling and every lag step stayed as it was, so a browser crawler that needs to fan out can still scale to 32.

Message on <topic> or <topic>_high_priority
  → kafka_exporter → kafka_consumergroup_lag → Prometheus
  → KEDA ScaledObject (lag ≥ 1 → wake, per lagSteps)
  → Deployment 0 → N → Karpenter adds a node if needed
  → Idle for 600 seconds → back to 0

What made waking up reliable

Scaling down is trivial. The engineering went into making sure a sleeping worker always notices its work.

Every partition has to be visible

kafka_exporter publishes consumer lag only for partitions where the consumer group has a committed offset. A partition the group has never read from reports no lag at all, so a scaler watching it would never fire. That matters most on low-traffic queues like high-priority topics, where a partition can go a long time without a commit.

So I extended the platform’s existing “warmup” action. Before it scales a worker up, warmup now commits the log-end offset for every partition that doesn’t have one yet. It has to happen before the pod joins, because Kafka rejects offset changes once a group has active members. Partitions that already have an offset are left untouched, so it never rewinds anyone’s progress.

Running warmup across all seven environments brought the browser crawler from 1 of its 64 partitions visible to all 64, and took the number of workers with no usable consumer group from 26 down to 4.

Every scaler has to watch a real topic

When a scaling block doesn’t name a topic, the chart defaults to the worker’s own name. I checked every Kafka scaler on dev against the declared topic list, and set the topic explicitly on the three workers whose topic is named differently from the worker. For two GPU-backed LLM workers with the same kind of mismatch, I repointed the scalers on every cluster. That changed which topic the scaler watches, not how many replicas run.

Cold starts, measured instead of guessed

Two inference workers ship ~3.2 GB images, and an earlier run had seen them take over six minutes to come up. The first instinct was bigger nodes, so I measured controlled cold pulls of the real image on fresh nodes before changing anything:

A 4× larger instance saved about twenty seconds, so bigger nodes weren’t worth paying for. By my reconstruction, most of the rest is node provisioning and scaling overhead, which a dev cluster can absorb. That meant the heavy workers could go to zero like everything else, with no standing capacity kept around for them.

Where it landed

The 135 counts deployments across seven environments: the same thirty workers, in every environment that runs them. That repetition is exactly where dev’s idle cost came from, and it’s what now drops to zero between runs.

Together those deployments requested roughly 145 vCPU that no longer sits reserved while idle. That’s the 78% in the table: it covers the developer environments’ own workers, not shared cluster services such as Kafka and monitoring. The $1,000 a month prices that CPU at spot rates, allowing for the time workers are actually awake. It’s an estimate, not a line on the bill.

The pattern carries over to any Kafka-driven platform: make the change opt-in so it can’t leak into production, and make sure every partition is visible before you let anything sleep.