Background Job Processing System

A production-ready background job processing system built with Golang using Clean Architecture.

Features

• Concurrent Job Processing: Configurable worker pool for asynchronous task execution.
• Real-time Tracking: Real-time progress and status updates via Server-Sent Events (SSE).
• Intelligent Resumption: Paused jobs correctly resume from their last completed step.
• Job Control: Pause, Resume, and Cancel jobs mid-execution via REST API or Dashboard.
• Reliable Retries: Automatic retry mechanism (up to 3 retries) on simulated failures.
• Distributed Ready: Uses PostgreSQL SELECT FOR UPDATE SKIP LOCKED for safe concurrent processing across multiple instances.
• Clean Architecture: Strict separation of concerns (Domain, UseCase, Repository, Delivery) for high maintainability.
• Configuration Management: Environment-based configuration using .env files and godotenv.
• Minimal Dashboard: Responsive vanilla HTML/JS dashboard for real-time monitoring.

Tech Stack

• Backend: Go 1.23+, go-chi/chi, GORM
• Database: PostgreSQL 15
• Frontend: Vanilla HTML/JS/CSS, Nginx
• Infrastructure: Docker, Docker Compose

Getting Started

Prerequisites

• Docker and Docker Compose installed.

Configuration

1. Copy the example environment file:

   cp .env.example .env

2. Adjust variables in .env as needed (e.g., NUM_WORKERS=5).

Running the System

1. Start the services:

   docker-compose up --build

2. Access the dashboard at http://localhost.
3. The API is available at http://localhost:8080/api.

Running Tests

To run unit tests and check coverage:

make coverage

Current statement coverage: 67.2% (Target: > 60%). A detailed coverage report is available in the coverage.report file.

API Documentation (Standard v1)

Method	Endpoint	Description
POST	/api/v1/jobs	Create a new background job
GET	/api/v1/jobs	List all jobs
GET	/api/v1/jobs/{id}	Get details of a specific job
POST	/api/v1/jobs/{id}/pause	Pause a processing job
POST	/api/v1/jobs/{id}/resume	Resume a paused job
POST	/api/v1/jobs/{id}/cancel	Cancel a pending/processing/paused job
GET	/api/v1/jobs/events	SSE endpoint for real-time updates

Project Structure (Clean Architecture)

The project strictly follows Uncle Bob's Clean Architecture principles to ensure separation of concerns, high testability, and independence from frameworks.

• cmd/api: Main entry point and Dependency Injection (DI) container.
• internal/config: Configuration loading and environment variable management.
• internal/domain: Entities layer: Core business models and repository interfaces (independent of any library).
• internal/usecase: Use Cases layer: Application-specific business rules (Job orchestration, retries, and resumption).
• internal/repository/postgres: Interface Adapters layer: PostgreSQL data access implementation using GORM.
• internal/delivery/http: Interface Adapters layer: REST API handlers and SSE streaming logic.
• internal/worker: Background worker pool management.
• ui/: Static dashboard frontend served via Nginx.