Skip to content

feat(learning): add the workers and redis job queue roadmap#82

Open
OthmaneZ05 wants to merge 1 commit into
mainfrom
feat/r5-roadmap
Open

feat(learning): add the workers and redis job queue roadmap#82
OthmaneZ05 wants to merge 1 commit into
mainfrom
feat/r5-roadmap

Conversation

@OthmaneZ05

Copy link
Copy Markdown
Collaborator

Summary of Changes

Adds the third catalog roadmap, “Workers & the Redis job queue” (redis-queue-workers, EN + FR, 8 steps, ~40 min, intermediate) in the Nimbus Books universe. The learner feels a 2-second synchronous checkout, decouples it with a producer pushing orders onto a real Redis list, writes a blocking BRPOP worker, archives receipts into PostgreSQL (stateless workers / shared state), floods the queue with a 300-order flash sale to watch backpressure live, scales consumption with an auto-scaling group cloned from the worker (a security-group rule with the ASG node as Source covers every replica), and finishes with a poison message handled by retry counters shared in Redis plus a dead-letter queue. Step 7 is an explicit "validate before fixing" beat, and every validator was chosen to remain stable when earlier steps are re-validated after completion.

Uses existing validators only: container_running, http_get_contains, edge_exists, redis_key_exists, table_exists, port_denied, asg_replicas.

Types of Changes

  • New feature / node type addition
  • Bug fix (non-breaking change resolving an issue)
  • Refactoring / structural cleanup
  • Documentation update

Verification & Testing

Automated Checks

  • Run npm run lint successfully with no errors
  • Run npm run build successfully with no compilation errors
  • Run npm test successfully (all tests pass) — backend 400/400

Manual Verification

npm run roadmap:validate OK for both language files; a script asserts EN/FR ids and validators are strictly identical.

Played the full roadmap against real Docker with an API driver that mirrors the canvas exactly (network config with default subnet routes, per-node security groups, ASG deploy via the same endpoints as the modal) and executes in the containers the exact code blocks extracted from the roadmap JSON. Result: 34/34 checks green

  • every step validated fail first, then pass (e.g. the producer step fails while the old synchronous code runs; the flash-sale step fails during the flood, before the ASG);
  • flash sale: 300 orders queued in seconds, backlog drained to 0 by worker-1 + 3 ASG replicas; SELECT worker, COUNT(*) FROM receipts GROUP BY worker shows 4 distinct container hostnames (67/99/67/68);
  • poison message: 3 shared-counter retries then parked in queue:orders:dead, checkout still accepting;
  • revisit stability: all 8 steps re-validated green after completion;
  • kill & restart mid-journey: web then worker-1 stopped/restarted one at a time (canvas-style) — servers revive via the baked boot line and all 8 steps re-validate green;
  • catalog lists both language variants.

Adjacent findings (not addressed here, per scope rules)

  1. The derssa/backend-lab-mongo:v1 image is currently unusable on a Fedora 44 / kernel 7.0.9 host: mongod silently segfaults (exit 139) 30–90 s after boot, including a bare docker run outside Torollo (reproduced 3×). This is why the receipt archive uses PostgreSQL (table_exists) instead of the originally considered Mongo (mongo_collection_exists stays unexercised by the catalog). The image likely needs a rebuild on a newer mongod.
  2. Stopping two containers simultaneously and restarting them can reshuffle their subnet IPs; peer containers' firewall rules then keep the stale source IPs even after a config re-save (the persisted nodeIpMap wins over live state), silently breaking cross-node traffic. A single-container stop/start (the canvas gesture) keeps its IP and re-wires correctly — verified. Worth a backlog entry.

Checklist

  • My code follows the repository's code style and lint standards
  • I have updated the documentation or instructions if necessary
  • All unit and integration tests are passing

@OthmaneZ05
OthmaneZ05 requested a review from Derssa as a code owner July 22, 2026 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant