Distributed Task Queue

A production-grade distributed task queue built in Go. Designed for reliable async job processing with priority scheduling, at-least-once delivery guarantees, dead worker detection, and real-time observability.

---

Architecture

Producers submit jobs via gRPC. The API gateway writes to PostgreSQL (for durability) and Redis (for fast dispatch). Workers pull jobs, write heartbeats, and update job status on completion. A background detector watches for dead workers and requeues their jobs. Prometheus scrapes metrics from the task queue process.

---

Requirements

Functional

• Submit jobs with priority (HIGH / NORMAL / LOW)
• Cancel pending jobs (PROCESSING jobs return an error)
• Query job status
• At-least-once delivery — no job is silently dropped
• Dead worker detection and automatic job recovery
• Exponential backoff retries with DLQ for exhausted jobs
• Idempotency keys to prevent duplicate submissions

Non-Functional

• Horizontally scalable — multiple server and worker instances
• Durable — jobs survive Redis restarts via PostgreSQL
• Observable — queue depth, worker health, job latency via Prometheus/Grafana
• Graceful shutdown — in-flight jobs complete before worker exits

---

Key Design Decisions

PostgreSQL as source of truth, Redis as dispatch layer Redis is fast but volatile. Writing to PostgreSQL first means jobs are never lost even if Redis goes down. Redis holds only what's needed for dispatch: the queue lists, in-flight registry, and heartbeats. A reconciliation cron (planned) can heal any gaps between the two.

Write order: idempotency check → PostgreSQL → Redis The order is intentional and non-negotiable. Writing Redis first creates ghost jobs — job IDs in the queue with no backing record, unrecoverable on failure. Idempotency keys (stored in Redis with 24hr TTL) are checked atomically via Lua before any write, preventing duplicate submissions under concurrent requests.

Three priority queues with weighted Lua polling (70/20/10) A simple strict-priority queue starves low-priority jobs under load. The weighted random approach gives high-priority jobs 70% of worker attention while ensuring low-priority jobs still make progress. The Lua script runs atomically on Redis — no distributed locks, no race conditions. When the selected queue is empty, it cascades down.

Heartbeat + in-flight registry for dead worker detection Workers write worker:{id}:heartbeat with a short TTL every 2 seconds. A separate detector goroutine scans inflight:{worker_id} entries and requeues any job whose worker's heartbeat has expired. Lease tokens prevent ghost completions — if a "dead" worker somehow finishes, its status update is rejected because the lease token no longer matches.

Exponential backoff via Redis Sorted Set Failed jobs aren't immediately requeued — they're pushed to a delayed sorted set with score = now + 2^attempt seconds. A background poller moves them to the priority queue when their time comes. This keeps the main queues clean and prevents hot retry loops from hammering downstream services.

Worker semaphore for bounded concurrency Each worker goroutine uses a buffered channel as a semaphore to cap the number of concurrently executing jobs. This creates natural backpressure — workers stop pulling new jobs when all slots are full — preventing memory exhaustion under burst load.

Graceful shutdown with WaitGroup On SIGINT/SIGTERM, the worker pool cancels its context (stops pulling new jobs) and calls wg.Wait() to drain all in-flight goroutines. No job is abandoned mid-execution.

Go over TypeScript/Python Goroutines, built-in race detection, and tight memory control make Go the right fit for a concurrent, latency-sensitive queue consumer. It also signals systems programming literacy to anyone reading the code.

---

Stack

• Go — core implementation
• Redis — priority queues, in-flight registry, heartbeats, delayed jobs
• PostgreSQL — durable job store (source of truth)
• gRPC + protobuf — API layer
• Prometheus & Grafana — observability
• Docker Compose — local stack

---

Observability

Prometheus metrics exposed at :2112/metrics:

Metric	Type	Labels	Purpose
taskqueue_queue_depth	Gauge	priority	Queue buildup per priority bucket
taskqueue_workers_alive	Gauge	—	Current active worker count
taskqueue_worker_deaths_total	Counter	—	Cumulative dead workers detected
taskqueue_dlq_jobs_total	Counter	—	Jobs that exhausted all retries
taskqueue_job_duration_seconds	Histogram	priority, status	p50/p95/p99 execution time

• Prometheus runs on port 9090
• Grafana runs on port 3000 (Default Credentials: admin / admin)

---

API (gRPC)

Defined in proto/task.proto:

service TaskQueue {
    rpc SubmitJob    (SubmitJobRequest)    returns (SubmitJobResponse);
    rpc CancelJob    (CancelJobRequest)    returns (CancelJobResponse);
    rpc GetJobStatus (GetJobStatusRequest) returns (GetJobStatusResponse);
}

SubmitJob — takes payload, priority, idempotency_key. Checks idempotency key atomically via Lua before writing. Returns existing job_id on duplicate, new job_id with status: queued otherwise.

CancelJob — checks current status first. PENDING jobs are marked CANCELLED. Jobs in any other state return a FailedPrecondition gRPC error.

GetJobStatus — reads current status from PostgreSQL.

---

Run Locally

Prerequisites: Docker, Docker Compose, Go 1.26+

Option 1: Run Services Locally (Development)

Use Docker Compose to run only the dependencies, and run the Go services directly on your host machine:

# Start Redis, PostgreSQL, Prometheus, Grafana
docker compose up -d redis postgres prometheus grafana

# Run migrations
psql $POSTGRES_CONN_STR -f migrations/001_create_jobs.sql
psql $POSTGRES_CONN_STR -f migrations/002_create_dlq.sql

# Start the gRPC server
go run ./cmd/server

# Start workers (separate terminal)
go run ./cmd/worker

# Run the test client to submit jobs (separate terminal)
go run ./cmd/client

Option 2: Run Everything via Docker Compose

Build and run the entire stack (including the gRPC server) inside Docker containers:

# Start the full stack
docker compose up -d

# Run the test client from your host machine to interact with the containerized server
go run ./cmd/client

Environment variables:

REDIS_ADDR=localhost:6379
POSTGRES_CONN_STR=postgres://user:password@localhost:5432/task_queue?sslmode=disable

---

Project Structure

cmd/
├── server/main.go       — gRPC server entrypoint
├── worker/main.go       — worker pool entrypoint
└── client/main.go       — interactive client tool to test queue functionality
internal/
├── api/api.go           — gRPC handler implementations
├── db/db.go             — PostgreSQL queries
├── queue/queue.go       — Redis queue operations (push, pop, heartbeat, delayed)
├── worker/worker.go     — worker pool, semaphore, graceful shutdown
├── detector/detector.go — dead worker detection and job requeue
└── metrics/metrics.go   — Prometheus metric definitions
proto/
├── task.proto           — protobuf service definition
├── task.pb.go           — generated
└── task_grpc.pb.go      — generated
migrations/
├── 001_create_jobs.sql
└── 002_create_dlq.sql

---

Roadmap

• Reconciliation cron — heal stale PENDING jobs with no Redis entry
• ECS Fargate deployment with CloudWatch + Prometheus remote write
• gRPC interceptor for request logging and tracing (OpenTelemetry)