Software · 02 / 06 · Kubernetes · Terraform · Helm
Cloud-Native Platform
My job queue run the way production runs it: Kubernetes built by Terraform, autoscaled on queue depth, with a node killed mid-run and nothing lost.
- Kubernetes
- Terraform
- Helm
- KEDA
- CloudNativePG
- PgBouncer
- Prometheus
- Grafana
Why I built this
Most services don't fail in the code, they fail in how they're run: a node dies, traffic spikes, the database runs out of connections at 2 a.m. Platform and SRE teams exist to make that boring, with autoscaling, failover, and alerts that fire before customers notice.
My job queue survived killed workers on a laptop. Would it survive a whole machine dying inside a real cluster, while it scaled itself up and down? I wanted to run it the way companies run services, then pull a node out from under it.
What I built
A five-node Kubernetes cluster created end to end by Terraform, with the job queue packaged as a Helm chart. KEDA scales the workers on Redis stream lag, CloudNativePG runs PostgreSQL with a hot replica, and Prometheus and Grafana track SLOs with burn-rate alerts.
The first load test OOM-killed the PostgreSQL primary: ten autoscaled workers each opened up to 20 connections. The replica took over by itself and no job was lost. PgBouncer now keeps connections flat however many workers run.
How it works
The whole platform builds from nothing with one `make up`, in about 10 minutes. An AWS version of the Terraform (VPC, EKS, ECR) is validated in CI but not applied yet.
How it stays correct
Worker leases expire and other workers take the jobs over; fencing tokens stop the dead node's writes.
KEDA reads Redis stream lag, not CPU, so workers follow the backlog.
PgBouncer in transaction mode caps connections at 67.
SLOs with burn-rate alerts in Prometheus, dashboards in Grafana.
Measured
- Load
- 20,000 jobs at 400/s, each 500 ms of work, on a laptop cluster
- Node kill
- 117 jobs taken over from the dead node, 20,000 / 20,000 done, 0 duplicates
- Autoscaling
- Workers 1 → 10 → 1; backlog drained 84 s after the first job
- Failover
- Primary OOM-killed in the first run: 18,832 accepted jobs, none lost or run twice