Software · 03 / 06 · Python · Distributed systems
Distributed Job Scheduler
A Celery-style job queue that never runs a job's side effects twice, even when workers are killed in the middle of a job.
- Python 3.12
- FastAPI
- React
- PostgreSQL
- Redis Streams
- asyncio
- TypeScript
- Docker
Why I built this
Every product runs work in the background: sending emails, charging cards, generating reports. Workers crash, deploys restart them, networks time out. If the queue retries a half-finished job carelessly, a customer gets two emails or, worse, two charges. Payment companies, e-commerce and banks all have to solve this.
How do you make sure a job runs exactly once, even if the worker dies halfway through? A repeated email is annoying; a repeated payment is a refund ticket. I wanted to build that guarantee myself, then try hard to break it.
What I built
A job platform in the spirit of Celery: jobs run now or on a schedule, retry with backoff, and land in a dead-letter queue when they run out of retries. PostgreSQL is the source of truth, Redis only carries hints, and every worker holds a heartbeated lease with a fencing token, so a crashed or frozen worker can never commit a job twice.
The test was never “does it run jobs”. It was: kill a worker every minute for 25 minutes, fail 5% of attempts on purpose, then audit the database for duplicates.
How it works
A scheduler promotes due jobs, reaps expired leases and re-sends anything Redis lost. There is no leader: every component can run as many copies as you like.
How it stays correct
Committed before the 2xx. Leases and a reaper recover dead workers; a sweeper recovers lost messages.
Idempotency-Key plus a unique constraint. Same key, different body: 409.
A fencing token on every write. Side effects commit in the same transaction as the fenced completion.
FOR UPDATE SKIP LOCKED and conditional updates everywhere.
Measured
- Load
- 50,000 jobs at 2,000/min, a worker SIGKILLed every minute, 5% of attempts failing
- Recovered
- 2,633 retries and 41 crash-interrupted jobs, all finished
- Enqueue latency
- p95 7.1 ms, p99 8.2 ms
- Tests
- 88 tests against real Postgres and Redis, 94% coverage, chaos tests with real processes
- Dashboard
- React + TypeScript operations dashboard; a Playwright worker-kill test committed 500 / 500 jobs exactly once