From 257a295e0bd5516c16aaac48fbeaee58b4c5e710 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Mon, 13 Jul 2026 17:23:50 +0300 Subject: [PATCH 01/20] docs: add architecture diagram Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 140 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 docs/architecture.md diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 00000000..7245d491 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,140 @@ +# Forge Architecture + +## System Overview + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ External Systems │ +│ │ +│ ┌──────────┐ ┌──────────┐ ┌───────────┐ ┌────────────────┐ │ +│ │ Jira │ │ GitHub │ │ Deep │ │ Langfuse │ │ +│ │ │ │ │ │ Agents │ │ (Observability)│ │ +│ └────┬─────┘ └────┬─────┘ └───────────┘ └────────────────┘ │ +│ │ │ │ +└────────┼───────────────┼────────────────────────────────────────────────┘ + │ webhooks │ webhooks + ▼ ▼ +┌─────────────────────────────────────────────────────────────────────────┐ +│ FastAPI Server (:8000) │ +│ │ +│ POST /api/v1/webhooks/jira POST /api/v1/webhooks/github │ +│ GET /api/v1/health GET /api/v1/metrics │ +│ │ +└────────────────────────────┬────────────────────────────────────────────┘ + │ enqueue events + ▼ +┌─────────────────────────────────────────────────────────────────────────┐ +│ Redis Streams ("forge:events") │ +│ │ +│ Consumer group: forge-workers State: AsyncRedisSaver │ +│ Per-ticket thread IDs (AISOS-123) LangGraph checkpointing │ +│ │ +└────────────────────────────┬────────────────────────────────────────────┘ + │ dequeue + dispatch + ▼ +┌─────────────────────────────────────────────────────────────────────────┐ +│ LangGraph Orchestrator │ +│ │ +│ Orchestrator.resume(event) → StateGraph(WorkflowState) │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ Workflow Nodes │ │ +│ │ │ │ +│ │ CLASSIFY ─┬─► [Bug] TRIAGE → RCA_OPTIONS → RCA_GATE → RCA │ │ +│ │ │ │ │ │ +│ │ └─► [Feature] ──────────────────────────┤ │ │ +│ │ ▼ │ │ +│ │ PRD → PRD_GATE ─► SPEC → SPEC_GATE │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ PLAN → PLAN_GATE ─► TASKS → TASK_GATE │ │ +│ │ │ │ │ +│ │ ▼ │ │ +│ │ IMPLEMENT ─► PR_CREATE ─► CI_GATE ◄─┐ │ │ +│ │ │ │ │ │ +│ │ CI_EVALUATOR┤ CI_FIX (retry) │ │ +│ │ │ │ │ │ +│ │ ▼ │ │ │ +│ │ HUMAN_REVIEW_GATE───┘ │ │ +│ │ │ │ │ +│ │ MERGE → DONE │ │ +│ │ │ │ +│ │ ⏸ Gates = human checkpoints (skipped with forge:yolo label) │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +│ │ +└───────────────┬─────────────────────────────┬───────────────────────────┘ + │ │ + ▼ ▼ +┌──────────────────────────┐ ┌──────────────────────────────────────────┐ +│ Claude LLM │ │ Podman Container (ephemeral) │ +│ │ │ │ +│ Anthropic API │ │ ┌─────────────────────────────────┐ │ +│ or Vertex AI │ │ │ Deep Agents + MCP tools │ │ +│ │ │ │ .forge/task.json │ │ +│ Used by: classify, │ │ │ workspace mounted │ │ +│ triage, RCA, PRD, │ │ │ commits changes locally │ │ +│ spec, plan, CI eval │ │ └─────────────────────────────────┘ │ +│ │ │ │ +└──────────────────────────┘ └──────────────────────────────────────────┘ +``` + +## Ticket Lifecycle + +``` +Jira ticket created + labeled "forge:managed" + │ + ▼ +CLASSIFY ── Claude determines: bug or feature? + │ + ├── Bug ────► TRIAGE → RCA_OPTIONS → ⏸ RCA_OPTION_GATE → RCA_DEEP_DIVE + │ (user picks ">option N") + │ + └── Feature ─────────────────────────────────────┐ + │ + ◄─────────────────────────────────────────────────┘ + │ + ▼ +PRD ── generate requirements ── ⏸ PRD_GATE (Jira or proposals PR) + │ + ▼ +SPEC ── generate tech design ── ⏸ SPEC_GATE + │ + ▼ +PLAN ── generate impl plan ──── ⏸ PLAN_GATE + │ + ▼ +TASKS ── break into work items ─ ⏸ TASK_GATE + │ + ▼ +IMPLEMENT ── Podman container + Deep Agents executes code + │ + ▼ +PR_CREATE ── open GitHub PR + │ + ▼ +CI_GATE ── wait for checks ──► CI_EVALUATOR + │ │ + │ ┌── pass ────────┤ + │ │ └── fail ──► ATTEMPT_CI_FIX ─┐ + │ │ (retry ≤N) │ + │ │ ◄────────────────────────────┘ + ▼ ▼ +HUMAN_REVIEW_GATE ── ⏸ wait for PR approval + │ + ▼ +MERGE ── merge PR ──► DONE ✓ + +⏸ = human checkpoint (auto-approved when forge:yolo label is set) +``` + +## Data Flow Summary + +``` +Inbound events: Jira/GitHub webhooks → FastAPI → Redis Streams +State persistence: Redis (LangGraph AsyncRedisSaver, keyed by ticket) +LLM calls: Orchestrator nodes → Claude (Anthropic API / Vertex AI) +Code execution: IMPLEMENT node → Podman container → Deep Agents +Outbound actions: Jira (comments, labels, transitions) + GitHub (PRs, branches, reviews) +Observability: Langfuse (LLM traces, workflow spans, costs) +``` From 4530c3a9719b809da063c4d1c4e13251c6d3b871 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Mon, 13 Jul 2026 19:10:10 +0300 Subject: [PATCH 02/20] docs: fix architecture diagram inaccuracies Verified against actual source code and corrected: - Remove fictional CLASSIFY node (routing is done by WorkflowRouter) - Show Feature and Bug as separate graphs with correct node names - Fix Redis stream names (forge:events:jira, forge:events:github) - Fix metrics endpoint path (/metrics, not /api/v1/metrics) - Fix container task path (/task.json, not .forge/task.json) - Add Gemini as a supported LLM backend - Add missing nodes: local_review, update_documentation, setup/teardown_workspace, task_router, completion/aggregation Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 187 ++++++++++++++++++++++++++++--------------- 1 file changed, 124 insertions(+), 63 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 7245d491..149f94b4 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -18,122 +18,183 @@ │ FastAPI Server (:8000) │ │ │ │ POST /api/v1/webhooks/jira POST /api/v1/webhooks/github │ -│ GET /api/v1/health GET /api/v1/metrics │ +│ GET /api/v1/health GET /metrics │ │ │ └────────────────────────────┬────────────────────────────────────────────┘ │ enqueue events ▼ ┌─────────────────────────────────────────────────────────────────────────┐ -│ Redis Streams ("forge:events") │ +│ Redis │ │ │ -│ Consumer group: forge-workers State: AsyncRedisSaver │ -│ Per-ticket thread IDs (AISOS-123) LangGraph checkpointing │ +│ Streams: forge:events:jira State: AsyncRedisSaver │ +│ forge:events:github LangGraph checkpointing │ +│ Consumer group: forge-workers Per-ticket thread IDs (AISOS-123) │ │ │ └────────────────────────────┬────────────────────────────────────────────┘ │ dequeue + dispatch ▼ ┌─────────────────────────────────────────────────────────────────────────┐ -│ LangGraph Orchestrator │ +│ WorkflowRouter │ │ │ -│ Orchestrator.resume(event) → StateGraph(WorkflowState) │ +│ Selects graph based on Jira issue type: │ +│ Feature/Story → FeatureWorkflow (StateGraph) │ +│ Bug → BugWorkflow (StateGraph) │ │ │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ Workflow Nodes │ │ -│ │ │ │ -│ │ CLASSIFY ─┬─► [Bug] TRIAGE → RCA_OPTIONS → RCA_GATE → RCA │ │ -│ │ │ │ │ │ -│ │ └─► [Feature] ──────────────────────────┤ │ │ -│ │ ▼ │ │ -│ │ PRD → PRD_GATE ─► SPEC → SPEC_GATE │ │ -│ │ │ │ │ -│ │ ▼ │ │ -│ │ PLAN → PLAN_GATE ─► TASKS → TASK_GATE │ │ -│ │ │ │ │ -│ │ ▼ │ │ -│ │ IMPLEMENT ─► PR_CREATE ─► CI_GATE ◄─┐ │ │ -│ │ │ │ │ │ -│ │ CI_EVALUATOR┤ CI_FIX (retry) │ │ -│ │ │ │ │ │ -│ │ ▼ │ │ │ -│ │ HUMAN_REVIEW_GATE───┘ │ │ -│ │ │ │ │ -│ │ MERGE → DONE │ │ -│ │ │ │ -│ │ ⏸ Gates = human checkpoints (skipped with forge:yolo label) │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ +│ ┌──────────────────────────────┐ ┌────────────────────────────────┐ │ +│ │ Feature Workflow │ │ Bug Workflow │ │ +│ │ │ │ │ │ +│ │ generate_prd │ │ triage_check │ │ +│ │ → ⏸ prd_approval_gate │ │ → ⏸ triage_gate │ │ +│ │ generate_spec │ │ analyze_bug ◄─┐ │ │ +│ │ → ⏸ spec_approval_gate │ │ → reflect_rca ─┘ (≤3x) │ │ +│ │ decompose_epics │ │ → ⏸ rca_option_gate │ │ +│ │ → ⏸ plan_approval_gate │ │ plan_bug_fix │ │ +│ │ generate_tasks │ │ → ⏸ plan_approval_gate │ │ +│ │ → ⏸ task_approval_gate │ │ decompose_plan │ │ +│ │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ +│ │ task_router (per-repo) │ │ Shared implementation path: │ │ +│ │ setup_workspace │ │ (same nodes as Feature) │ │ +│ │ implement_task │ │ │ │ +│ │ local_review │ │ Post-merge: │ │ +│ │ update_documentation │ │ post_merge_summary → END │ │ +│ │ create_pr │ │ │ │ +│ │ teardown_workspace │ └────────────────────────────────┘ │ +│ │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ +│ │ wait_for_ci_gate │ │ +│ │ ci_evaluator ◄── ci_fix │ │ +│ │ ⏸ human_review_gate │ │ +│ │ complete_tasks │ │ +│ │ aggregate_epic_status │ │ +│ │ aggregate_feature_status │ │ +│ │ → END │ │ +│ └──────────────────────────────┘ │ +│ │ +│ ⏸ = human checkpoint (auto-approved with forge:yolo label) │ │ │ └───────────────┬─────────────────────────────┬───────────────────────────┘ │ │ ▼ ▼ ┌──────────────────────────┐ ┌──────────────────────────────────────────┐ -│ Claude LLM │ │ Podman Container (ephemeral) │ -│ │ │ │ -│ Anthropic API │ │ ┌─────────────────────────────────┐ │ -│ or Vertex AI │ │ │ Deep Agents + MCP tools │ │ -│ │ │ │ .forge/task.json │ │ -│ Used by: classify, │ │ │ workspace mounted │ │ -│ triage, RCA, PRD, │ │ │ commits changes locally │ │ -│ spec, plan, CI eval │ │ └─────────────────────────────────┘ │ +│ LLM Backends │ │ Podman Container (ephemeral) │ │ │ │ │ +│ Anthropic API (Claude) │ │ ┌─────────────────────────────────┐ │ +│ Vertex AI (Claude) │ │ │ Deep Agents + MCP tools │ │ +│ Vertex AI (Gemini) │ │ │ /task.json (task description) │ │ +│ │ │ │ /workspace (repo mounted) │ │ +│ Used by: PRD, spec, │ │ │ commits changes locally │ │ +│ plan, RCA, triage, │ │ └─────────────────────────────────┘ │ +│ code review, CI eval │ │ │ └──────────────────────────┘ └──────────────────────────────────────────┘ ``` -## Ticket Lifecycle +## Feature Ticket Lifecycle ``` -Jira ticket created + labeled "forge:managed" +Jira ticket created + labeled "forge:managed" (type: Feature or Story) + │ + ▼ +generate_prd ── AI drafts Product Requirements Document + │ + ▼ +⏸ prd_approval_gate ── wait for approval (Jira comment or proposals PR) + │ + ▼ +generate_spec ── AI drafts Technical Specification + │ + ▼ +⏸ spec_approval_gate │ ▼ -CLASSIFY ── Claude determines: bug or feature? +decompose_epics ── AI breaks spec into Jira Epics │ - ├── Bug ────► TRIAGE → RCA_OPTIONS → ⏸ RCA_OPTION_GATE → RCA_DEEP_DIVE - │ (user picks ">option N") + ▼ +⏸ plan_approval_gate + │ + ▼ +generate_tasks ── AI creates implementation Tasks under Epics + │ + ▼ +⏸ task_approval_gate │ - └── Feature ─────────────────────────────────────┐ - │ - ◄─────────────────────────────────────────────────┘ + ▼ +task_router ── group tasks by target repo + │ + ▼ (per repo) +setup_workspace ── clone repo, create branch │ ▼ -PRD ── generate requirements ── ⏸ PRD_GATE (Jira or proposals PR) +implement_task ── Podman container + Deep Agents writes code │ ▼ -SPEC ── generate tech design ── ⏸ SPEC_GATE +local_review ── AI reviews diff, fixes issues (≤2 passes) │ ▼ -PLAN ── generate impl plan ──── ⏸ PLAN_GATE +update_documentation ── update stale docs │ ▼ -TASKS ── break into work items ─ ⏸ TASK_GATE +create_pr ── open GitHub PR │ ▼ -IMPLEMENT ── Podman container + Deep Agents executes code +teardown_workspace ── cleanup (loop back if more repos) │ ▼ -PR_CREATE ── open GitHub PR +wait_for_ci_gate ── wait for CI checks │ ▼ -CI_GATE ── wait for checks ──► CI_EVALUATOR - │ │ - │ ┌── pass ────────┤ - │ │ └── fail ──► ATTEMPT_CI_FIX ─┐ - │ │ (retry ≤N) │ - │ │ ◄────────────────────────────┘ - ▼ ▼ -HUMAN_REVIEW_GATE ── ⏸ wait for PR approval +ci_evaluator ─┬─ pass ──────────────────────────┐ + └─ fail → attempt_ci_fix (≤5x) ───┘ │ ▼ -MERGE ── merge PR ──► DONE ✓ +⏸ human_review_gate ── wait for PR approval + │ (changes_requested → implement_review → CI loop) + ▼ +complete_tasks → aggregate_epic_status → aggregate_feature_status → END ⏸ = human checkpoint (auto-approved when forge:yolo label is set) ``` +## Bug Ticket Lifecycle + +``` +Jira ticket created + labeled "forge:managed" (type: Bug) + │ + ▼ +triage_check ── validate bug report has required fields + │ + ▼ +⏸ triage_gate ── wait for reporter update if incomplete + │ + ▼ +analyze_bug ◄──┐ + │ │ reflect_rca (up to 3 reflection cycles) + └────────────┘ + │ + ▼ +⏸ rca_option_gate ── present fix options, user selects ">option N" + │ + ▼ +plan_bug_fix ── AI generates fix plan + │ + ▼ +⏸ plan_approval_gate + │ + ▼ +decompose_plan ── break into tasks + │ + ▼ +setup_workspace → implement → local_review → create_pr → CI → review + │ + ▼ +post_merge_summary → END +``` + ## Data Flow Summary ``` Inbound events: Jira/GitHub webhooks → FastAPI → Redis Streams State persistence: Redis (LangGraph AsyncRedisSaver, keyed by ticket) -LLM calls: Orchestrator nodes → Claude (Anthropic API / Vertex AI) -Code execution: IMPLEMENT node → Podman container → Deep Agents +LLM calls: Orchestrator nodes → Claude/Gemini (Anthropic API / Vertex AI) +Code execution: implement_task → Podman container → Deep Agents Outbound actions: Jira (comments, labels, transitions) GitHub (PRs, branches, reviews) Observability: Langfuse (LLM traces, workflow spans, costs) From a38f15c4525679b5f2681bc9c5f75ec22d105963 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Mon, 13 Jul 2026 21:13:15 +0300 Subject: [PATCH 03/20] =?UTF-8?q?docs:=20fix=20diagram=20alignment=20?= =?UTF-8?q?=E2=80=94=20all=20box=20lines=20at=20width=2075?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 311 ++++++++++++++++++++++--------------------- 1 file changed, 158 insertions(+), 153 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 149f94b4..a6469fc2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -6,195 +6,200 @@ ┌─────────────────────────────────────────────────────────────────────────┐ │ External Systems │ │ │ -│ ┌──────────┐ ┌──────────┐ ┌───────────┐ ┌────────────────┐ │ -│ │ Jira │ │ GitHub │ │ Deep │ │ Langfuse │ │ -│ │ │ │ │ │ Agents │ │ (Observability)│ │ -│ └────┬─────┘ └────┬─────┘ └───────────┘ └────────────────┘ │ -│ │ │ │ -└────────┼───────────────┼────────────────────────────────────────────────┘ - │ webhooks │ webhooks - ▼ ▼ +│ ┌──────────┐ ┌──────────┐ ┌─────────────┐ ┌──────────────────┐ │ +│ │ Jira │ │ GitHub │ │ Deep Agents │ │ Langfuse │ │ +│ │ │ │ │ │ │ │ (Observability) │ │ +│ └────┬─────┘ └────┬─────┘ └─────────────┘ └──────────────────┘ │ +│ │ │ │ +└────────┼──────────────┼─────────────────────────────────────────────────┘ + │ webhooks │ webhooks + v v ┌─────────────────────────────────────────────────────────────────────────┐ -│ FastAPI Server (:8000) │ +│ FastAPI Server (:8000) │ │ │ -│ POST /api/v1/webhooks/jira POST /api/v1/webhooks/github │ -│ GET /api/v1/health GET /metrics │ +│ POST /api/v1/webhooks/jira POST /api/v1/webhooks/github │ +│ GET /api/v1/health GET /metrics │ │ │ └────────────────────────────┬────────────────────────────────────────────┘ │ enqueue events - ▼ + v ┌─────────────────────────────────────────────────────────────────────────┐ -│ Redis │ +│ Redis │ │ │ -│ Streams: forge:events:jira State: AsyncRedisSaver │ -│ forge:events:github LangGraph checkpointing │ -│ Consumer group: forge-workers Per-ticket thread IDs (AISOS-123) │ +│ Streams: forge:events:jira State: AsyncRedisSaver │ +│ forge:events:github LangGraph checkpointing │ +│ Consumer group: forge-workers Per-ticket thread IDs (AISOS-123) │ │ │ └────────────────────────────┬────────────────────────────────────────────┘ │ dequeue + dispatch - ▼ + v ┌─────────────────────────────────────────────────────────────────────────┐ -│ WorkflowRouter │ +│ WorkflowRouter │ │ │ -│ Selects graph based on Jira issue type: │ -│ Feature/Story → FeatureWorkflow (StateGraph) │ -│ Bug → BugWorkflow (StateGraph) │ +│ Routes by Jira issue type: │ +│ Feature/Story --> FeatureWorkflow (StateGraph) │ +│ Bug --> BugWorkflow (StateGraph) │ │ │ -│ ┌──────────────────────────────┐ ┌────────────────────────────────┐ │ -│ │ Feature Workflow │ │ Bug Workflow │ │ -│ │ │ │ │ │ -│ │ generate_prd │ │ triage_check │ │ -│ │ → ⏸ prd_approval_gate │ │ → ⏸ triage_gate │ │ -│ │ generate_spec │ │ analyze_bug ◄─┐ │ │ -│ │ → ⏸ spec_approval_gate │ │ → reflect_rca ─┘ (≤3x) │ │ -│ │ decompose_epics │ │ → ⏸ rca_option_gate │ │ -│ │ → ⏸ plan_approval_gate │ │ plan_bug_fix │ │ -│ │ generate_tasks │ │ → ⏸ plan_approval_gate │ │ -│ │ → ⏸ task_approval_gate │ │ decompose_plan │ │ -│ │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ -│ │ task_router (per-repo) │ │ Shared implementation path: │ │ -│ │ setup_workspace │ │ (same nodes as Feature) │ │ -│ │ implement_task │ │ │ │ -│ │ local_review │ │ Post-merge: │ │ -│ │ update_documentation │ │ post_merge_summary → END │ │ -│ │ create_pr │ │ │ │ -│ │ teardown_workspace │ └────────────────────────────────┘ │ -│ │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ -│ │ wait_for_ci_gate │ │ -│ │ ci_evaluator ◄── ci_fix │ │ -│ │ ⏸ human_review_gate │ │ -│ │ complete_tasks │ │ -│ │ aggregate_epic_status │ │ -│ │ aggregate_feature_status │ │ -│ │ → END │ │ -│ └──────────────────────────────┘ │ +│ ┌──────────────────────────────┐ ┌──────────────────────────────┐ │ +│ │ Feature Workflow │ │ Bug Workflow │ │ +│ │ │ │ │ │ +│ │ generate_prd │ │ triage_check │ │ +│ │ >> prd_approval_gate │ │ >> triage_gate │ │ +│ │ generate_spec │ │ analyze_bug │ │ +│ │ >> spec_approval_gate │ │ <> reflect_rca (up to 3x) │ │ +│ │ decompose_epics │ │ >> rca_option_gate │ │ +│ │ >> plan_approval_gate │ │ plan_bug_fix │ │ +│ │ generate_tasks │ │ >> plan_approval_gate │ │ +│ │ >> task_approval_gate │ │ decompose_plan │ │ +│ │──────────────────────────────│ │──────────────────────────────│ │ +│ │ task_router (per-repo) │ │ Shared impl path: │ │ +│ │ setup_workspace │ │ (same nodes as Feature) │ │ +│ │ implement_task │ │ │ │ +│ │ local_review │ │ Post-merge: │ │ +│ │ update_documentation │ │ post_merge_summary --> END│ │ +│ │ create_pr │ │ │ │ +│ │ teardown_workspace │ └──────────────────────────────┘ │ +│ │──────────────────────────────│ │ +│ │ wait_for_ci_gate │ │ +│ │ ci_evaluator <-- ci_fix │ │ +│ │ >> human_review_gate │ │ +│ │ complete_tasks │ │ +│ │ aggregate_epic_status │ │ +│ │ aggregate_feature_status │ │ +│ │ --> END │ │ +│ └──────────────────────────────┘ │ │ │ -│ ⏸ = human checkpoint (auto-approved with forge:yolo label) │ +│ >> = human checkpoint (auto-approved with forge:yolo label) │ │ │ └───────────────┬─────────────────────────────┬───────────────────────────┘ │ │ - ▼ ▼ -┌──────────────────────────┐ ┌──────────────────────────────────────────┐ -│ LLM Backends │ │ Podman Container (ephemeral) │ -│ │ │ │ -│ Anthropic API (Claude) │ │ ┌─────────────────────────────────┐ │ -│ Vertex AI (Claude) │ │ │ Deep Agents + MCP tools │ │ -│ Vertex AI (Gemini) │ │ │ /task.json (task description) │ │ -│ │ │ │ /workspace (repo mounted) │ │ -│ Used by: PRD, spec, │ │ │ commits changes locally │ │ -│ plan, RCA, triage, │ │ └─────────────────────────────────┘ │ -│ code review, CI eval │ │ │ -└──────────────────────────┘ └──────────────────────────────────────────┘ + v v +┌────────────────────────────┐ ┌────────────────────────────────────────┐ +│ LLM Backends │ │ Podman Container (ephemeral) │ +│ │ │ │ +│ Anthropic API (Claude) │ │ ┌──────────────────────────────────┐ │ +│ Vertex AI (Claude) │ │ │ Deep Agents + MCP tools │ │ +│ Vertex AI (Gemini) │ │ │ /task.json (task description) │ │ +│ │ │ │ /workspace (repo mounted) │ │ +│ Used by: PRD, spec, │ │ │ commits changes locally │ │ +│ plan, RCA, triage, │ │ └──────────────────────────────────┘ │ +│ code review, CI eval │ │ │ +└────────────────────────────┘ └────────────────────────────────────────┘ ``` ## Feature Ticket Lifecycle ``` Jira ticket created + labeled "forge:managed" (type: Feature or Story) - │ - ▼ -generate_prd ── AI drafts Product Requirements Document - │ - ▼ -⏸ prd_approval_gate ── wait for approval (Jira comment or proposals PR) - │ - ▼ -generate_spec ── AI drafts Technical Specification - │ - ▼ -⏸ spec_approval_gate - │ - ▼ -decompose_epics ── AI breaks spec into Jira Epics - │ - ▼ -⏸ plan_approval_gate - │ - ▼ -generate_tasks ── AI creates implementation Tasks under Epics - │ - ▼ -⏸ task_approval_gate - │ - ▼ -task_router ── group tasks by target repo - │ - ▼ (per repo) -setup_workspace ── clone repo, create branch - │ - ▼ -implement_task ── Podman container + Deep Agents writes code - │ - ▼ -local_review ── AI reviews diff, fixes issues (≤2 passes) - │ - ▼ -update_documentation ── update stale docs - │ - ▼ -create_pr ── open GitHub PR - │ - ▼ -teardown_workspace ── cleanup (loop back if more repos) - │ - ▼ -wait_for_ci_gate ── wait for CI checks - │ - ▼ -ci_evaluator ─┬─ pass ──────────────────────────┐ - └─ fail → attempt_ci_fix (≤5x) ───┘ - │ - ▼ -⏸ human_review_gate ── wait for PR approval - │ (changes_requested → implement_review → CI loop) - ▼ -complete_tasks → aggregate_epic_status → aggregate_feature_status → END + | + v +generate_prd -- AI drafts Product Requirements Document + | + v +>> prd_approval_gate -- wait for approval (Jira comment or proposals PR) + | + v +generate_spec -- AI drafts Technical Specification + | + v +>> spec_approval_gate + | + v +decompose_epics -- AI breaks spec into Jira Epics + | + v +>> plan_approval_gate + | + v +generate_tasks -- AI creates implementation Tasks under Epics + | + v +>> task_approval_gate + | + v +task_router -- group tasks by target repo + | + v (per repo) +setup_workspace -- clone repo, create branch + | + v +implement_task -- Podman container + Deep Agents writes code + | + v +local_review -- AI reviews diff, fixes issues (up to 2 passes) + | + v +update_documentation -- update stale docs + | + v +create_pr -- open GitHub PR + | + v +teardown_workspace -- cleanup (loop back if more repos) + | + v +wait_for_ci_gate -- wait for CI checks + | + v +ci_evaluator --+-- pass ------+ + +-- fail ---+ | + | | + attempt_ci_fix | + (up to 5x) | + | | + +-----------+ | + | | + v v +>> human_review_gate -- wait for PR approval + | (changes_requested --> implement_review --> CI loop) + v +complete_tasks --> aggregate_epic_status --> aggregate_feature_status --> END -⏸ = human checkpoint (auto-approved when forge:yolo label is set) +>> = human checkpoint (auto-approved when forge:yolo label is set) ``` ## Bug Ticket Lifecycle ``` Jira ticket created + labeled "forge:managed" (type: Bug) - │ - ▼ -triage_check ── validate bug report has required fields - │ - ▼ -⏸ triage_gate ── wait for reporter update if incomplete - │ - ▼ -analyze_bug ◄──┐ - │ │ reflect_rca (up to 3 reflection cycles) - └────────────┘ - │ - ▼ -⏸ rca_option_gate ── present fix options, user selects ">option N" - │ - ▼ -plan_bug_fix ── AI generates fix plan - │ - ▼ -⏸ plan_approval_gate - │ - ▼ -decompose_plan ── break into tasks - │ - ▼ -setup_workspace → implement → local_review → create_pr → CI → review - │ - ▼ -post_merge_summary → END + | + v +triage_check -- validate bug report has required fields + | + v +>> triage_gate -- wait for reporter update if incomplete + | + v +analyze_bug <--+ + | | + +--- reflect_rca (up to 3 reflection cycles) + | + v +>> rca_option_gate -- present fix options, user selects ">option N" + | + v +plan_bug_fix -- AI generates fix plan + | + v +>> plan_approval_gate + | + v +decompose_plan -- break into tasks + | + v +setup_workspace --> implement --> local_review --> create_pr --> CI --> review + | + v +post_merge_summary --> END ``` ## Data Flow Summary ``` -Inbound events: Jira/GitHub webhooks → FastAPI → Redis Streams +Inbound events: Jira/GitHub webhooks --> FastAPI --> Redis Streams State persistence: Redis (LangGraph AsyncRedisSaver, keyed by ticket) -LLM calls: Orchestrator nodes → Claude/Gemini (Anthropic API / Vertex AI) -Code execution: implement_task → Podman container → Deep Agents +LLM calls: Orchestrator nodes --> Claude/Gemini (Anthropic / Vertex AI) +Code execution: implement_task --> Podman container --> Deep Agents Outbound actions: Jira (comments, labels, transitions) GitHub (PRs, branches, reviews) Observability: Langfuse (LLM traces, workflow spans, costs) From 9ce53bf2fea639ac30807f7fc698efc284d53e07 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Tue, 14 Jul 2026 11:33:57 +0300 Subject: [PATCH 04/20] docs: address review feedback on architecture diagram - Remove Deep Agents from External Systems (it's a library, not a service) - Show LLM Backends connected to both orchestrator and container - Clarify Deep Agents runs inside Podman containers - Add bidirectional LLM call arrows Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index a6469fc2..4cb46a32 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -6,10 +6,9 @@ ┌─────────────────────────────────────────────────────────────────────────┐ │ External Systems │ │ │ -│ ┌──────────┐ ┌──────────┐ ┌─────────────┐ ┌──────────────────┐ │ -│ │ Jira │ │ GitHub │ │ Deep Agents │ │ Langfuse │ │ -│ │ │ │ │ │ │ │ (Observability) │ │ -│ └────┬─────┘ └────┬─────┘ └─────────────┘ └──────────────────┘ │ +│ ┌──────────┐ ┌──────────┐ ┌──────────────────────────────────┐ │ +│ │ Jira │ │ GitHub │ │ Langfuse (Observability) │ │ +│ └────┬─────┘ └────┬─────┘ └──────────────────────────────────┘ │ │ │ │ │ └────────┼──────────────┼─────────────────────────────────────────────────┘ │ webhooks │ webhooks @@ -71,19 +70,21 @@ │ │ │ >> = human checkpoint (auto-approved with forge:yolo label) │ │ │ -└───────────────┬─────────────────────────────┬───────────────────────────┘ - │ │ - v v +└────────────────────────────┬────────────────────────────────────────────┘ + │ + ┌──────────────┼──────────────┐ + │ LLM calls │ │ LLM calls + v v v ┌────────────────────────────┐ ┌────────────────────────────────────────┐ │ LLM Backends │ │ Podman Container (ephemeral) │ │ │ │ │ │ Anthropic API (Claude) │ │ ┌──────────────────────────────────┐ │ -│ Vertex AI (Claude) │ │ │ Deep Agents + MCP tools │ │ +│ Vertex AI (Claude) │ │ │ Deep Agents (library) + MCP │ │ │ Vertex AI (Gemini) │ │ │ /task.json (task description) │ │ │ │ │ │ /workspace (repo mounted) │ │ -│ Used by: PRD, spec, │ │ │ commits changes locally │ │ -│ plan, RCA, triage, │ │ └──────────────────────────────────┘ │ -│ code review, CI eval │ │ │ +│ Called by orchestrator │ │ │ commits changes locally │ │ +│ nodes and container │ │ └──────────────────────────────────┘ │ +│ agents (bidirectional) │ │ │ └────────────────────────────┘ └────────────────────────────────────────┘ ``` @@ -199,7 +200,8 @@ post_merge_summary --> END Inbound events: Jira/GitHub webhooks --> FastAPI --> Redis Streams State persistence: Redis (LangGraph AsyncRedisSaver, keyed by ticket) LLM calls: Orchestrator nodes --> Claude/Gemini (Anthropic / Vertex AI) -Code execution: implement_task --> Podman container --> Deep Agents + Container agents --> Claude/Gemini (same backends) +Code execution: implement_task --> Podman container --> Deep Agents (library) Outbound actions: Jira (comments, labels, transitions) GitHub (PRs, branches, reviews) Observability: Langfuse (LLM traces, workflow spans, costs) From 537c3d91de821720c25fdf1795c784bc2c6fdced Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Tue, 14 Jul 2026 12:06:26 +0300 Subject: [PATCH 05/20] cosmetic changes Co-authored-by: Lukas Svaty --- docs/architecture.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 4cb46a32..8f6c9469 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -71,17 +71,17 @@ │ >> = human checkpoint (auto-approved with forge:yolo label) │ │ │ └────────────────────────────┬────────────────────────────────────────────┘ - │ + v ┌──────────────┼──────────────┐ - │ LLM calls │ │ LLM calls - v v v + │ LLM calls │ LLM calls + v v ┌────────────────────────────┐ ┌────────────────────────────────────────┐ │ LLM Backends │ │ Podman Container (ephemeral) │ │ │ │ │ │ Anthropic API (Claude) │ │ ┌──────────────────────────────────┐ │ │ Vertex AI (Claude) │ │ │ Deep Agents (library) + MCP │ │ │ Vertex AI (Gemini) │ │ │ /task.json (task description) │ │ -│ │ │ │ /workspace (repo mounted) │ │ +│ │<->│ │ /workspace (repo mounted) │ │ │ Called by orchestrator │ │ │ commits changes locally │ │ │ nodes and container │ │ └──────────────────────────────────┘ │ │ agents (bidirectional) │ │ │ From 2150db90a1fb897df9ca05bb7e76753fac41fb42 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Thu, 16 Jul 2026 17:12:39 +0300 Subject: [PATCH 06/20] Fix architecture diagram per PR review feedback - Convert all diagrams from ASCII art to mermaid (eshulman2) - Move Deep Agents inside Podman Container, not External Systems (StLuke) - Add bidirectional LLM arrows to both Router and Container (StLuke) - Add TaskTakeoverWorkflow (Task/Epic) to system overview and lifecycle (danchild) - Expand Bug workflow implementation path with correct node names - Data Flow Summary as markdown list instead of code block Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 336 +++++++++++++++++++------------------------ 1 file changed, 145 insertions(+), 191 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 8f6c9469..c5c03059 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -2,207 +2,161 @@ ## System Overview -``` -┌─────────────────────────────────────────────────────────────────────────┐ -│ External Systems │ -│ │ -│ ┌──────────┐ ┌──────────┐ ┌──────────────────────────────────┐ │ -│ │ Jira │ │ GitHub │ │ Langfuse (Observability) │ │ -│ └────┬─────┘ └────┬─────┘ └──────────────────────────────────┘ │ -│ │ │ │ -└────────┼──────────────┼─────────────────────────────────────────────────┘ - │ webhooks │ webhooks - v v -┌─────────────────────────────────────────────────────────────────────────┐ -│ FastAPI Server (:8000) │ -│ │ -│ POST /api/v1/webhooks/jira POST /api/v1/webhooks/github │ -│ GET /api/v1/health GET /metrics │ -│ │ -└────────────────────────────┬────────────────────────────────────────────┘ - │ enqueue events - v -┌─────────────────────────────────────────────────────────────────────────┐ -│ Redis │ -│ │ -│ Streams: forge:events:jira State: AsyncRedisSaver │ -│ forge:events:github LangGraph checkpointing │ -│ Consumer group: forge-workers Per-ticket thread IDs (AISOS-123) │ -│ │ -└────────────────────────────┬────────────────────────────────────────────┘ - │ dequeue + dispatch - v -┌─────────────────────────────────────────────────────────────────────────┐ -│ WorkflowRouter │ -│ │ -│ Routes by Jira issue type: │ -│ Feature/Story --> FeatureWorkflow (StateGraph) │ -│ Bug --> BugWorkflow (StateGraph) │ -│ │ -│ ┌──────────────────────────────┐ ┌──────────────────────────────┐ │ -│ │ Feature Workflow │ │ Bug Workflow │ │ -│ │ │ │ │ │ -│ │ generate_prd │ │ triage_check │ │ -│ │ >> prd_approval_gate │ │ >> triage_gate │ │ -│ │ generate_spec │ │ analyze_bug │ │ -│ │ >> spec_approval_gate │ │ <> reflect_rca (up to 3x) │ │ -│ │ decompose_epics │ │ >> rca_option_gate │ │ -│ │ >> plan_approval_gate │ │ plan_bug_fix │ │ -│ │ generate_tasks │ │ >> plan_approval_gate │ │ -│ │ >> task_approval_gate │ │ decompose_plan │ │ -│ │──────────────────────────────│ │──────────────────────────────│ │ -│ │ task_router (per-repo) │ │ Shared impl path: │ │ -│ │ setup_workspace │ │ (same nodes as Feature) │ │ -│ │ implement_task │ │ │ │ -│ │ local_review │ │ Post-merge: │ │ -│ │ update_documentation │ │ post_merge_summary --> END│ │ -│ │ create_pr │ │ │ │ -│ │ teardown_workspace │ └──────────────────────────────┘ │ -│ │──────────────────────────────│ │ -│ │ wait_for_ci_gate │ │ -│ │ ci_evaluator <-- ci_fix │ │ -│ │ >> human_review_gate │ │ -│ │ complete_tasks │ │ -│ │ aggregate_epic_status │ │ -│ │ aggregate_feature_status │ │ -│ │ --> END │ │ -│ └──────────────────────────────┘ │ -│ │ -│ >> = human checkpoint (auto-approved with forge:yolo label) │ -│ │ -└────────────────────────────┬────────────────────────────────────────────┘ - v - ┌──────────────┼──────────────┐ - │ LLM calls │ LLM calls - v v -┌────────────────────────────┐ ┌────────────────────────────────────────┐ -│ LLM Backends │ │ Podman Container (ephemeral) │ -│ │ │ │ -│ Anthropic API (Claude) │ │ ┌──────────────────────────────────┐ │ -│ Vertex AI (Claude) │ │ │ Deep Agents (library) + MCP │ │ -│ Vertex AI (Gemini) │ │ │ /task.json (task description) │ │ -│ │<->│ │ /workspace (repo mounted) │ │ -│ Called by orchestrator │ │ │ commits changes locally │ │ -│ nodes and container │ │ └──────────────────────────────────┘ │ -│ agents (bidirectional) │ │ │ -└────────────────────────────┘ └────────────────────────────────────────┘ +```mermaid +flowchart TD + subgraph External["External Systems"] + Jira + GitHub + Langfuse["Langfuse (Observability)"] + end + + subgraph API["FastAPI Server (:8000)"] + JiraWH["POST /webhooks/jira"] + GitHubWH["POST /webhooks/github"] + end + + subgraph Queue["Redis"] + Streams["Streams: forge:events:jira\nforge:events:github"] + State["AsyncRedisSaver\nLangGraph checkpointing"] + end + + subgraph Router["WorkflowRouter"] + Route{"Route by\nissue type"} + Feature["FeatureWorkflow\n(Feature/Story)"] + Bug["BugWorkflow\n(Bug)"] + Task["TaskTakeoverWorkflow\n(Task/Epic)"] + end + + subgraph Container["Podman Container (ephemeral)"] + Agent["Deep Agents + MCP\n/workspace (repo mounted)"] + end + + LLM["LLM Backends\nAnthropic API (Claude)\nVertex AI (Claude/Gemini)"] + + Jira -- webhooks --> JiraWH + GitHub -- webhooks --> GitHubWH + JiraWH --> Streams + GitHubWH --> Streams + Streams --> Route + Route --> Feature + Route --> Bug + Route --> Task + Feature --> Container + Bug --> Container + Task --> Container + Router <--> LLM + Container <--> LLM + Router --> Jira + Router --> GitHub + Router --> Langfuse ``` ## Feature Ticket Lifecycle -``` -Jira ticket created + labeled "forge:managed" (type: Feature or Story) - | - v -generate_prd -- AI drafts Product Requirements Document - | - v ->> prd_approval_gate -- wait for approval (Jira comment or proposals PR) - | - v -generate_spec -- AI drafts Technical Specification - | - v ->> spec_approval_gate - | - v -decompose_epics -- AI breaks spec into Jira Epics - | - v ->> plan_approval_gate - | - v -generate_tasks -- AI creates implementation Tasks under Epics - | - v ->> task_approval_gate - | - v -task_router -- group tasks by target repo - | - v (per repo) -setup_workspace -- clone repo, create branch - | - v -implement_task -- Podman container + Deep Agents writes code - | - v -local_review -- AI reviews diff, fixes issues (up to 2 passes) - | - v -update_documentation -- update stale docs - | - v -create_pr -- open GitHub PR - | - v -teardown_workspace -- cleanup (loop back if more repos) - | - v -wait_for_ci_gate -- wait for CI checks - | - v -ci_evaluator --+-- pass ------+ - +-- fail ---+ | - | | - attempt_ci_fix | - (up to 5x) | - | | - +-----------+ | - | | - v v ->> human_review_gate -- wait for PR approval - | (changes_requested --> implement_review --> CI loop) - v -complete_tasks --> aggregate_epic_status --> aggregate_feature_status --> END - ->> = human checkpoint (auto-approved when forge:yolo label is set) +```mermaid +flowchart TD + A["Jira ticket created\nforge:managed (Feature/Story)"] --> B[generate_prd] + B --> C[">> prd_approval_gate"] + C --> D[generate_spec] + D --> E[">> spec_approval_gate"] + E --> F[decompose_epics] + F --> G[">> plan_approval_gate"] + G --> H[generate_tasks] + H --> I[">> task_approval_gate"] + I --> J["task_router (per-repo)"] + + J --> K[setup_workspace] + K --> L["implement_task\n(Podman + Deep Agents)"] + L --> M[local_review] + M -->|"needs work (up to 2x)"| L + M --> N[update_documentation] + N --> O[create_pr] + O --> P["teardown_workspace"] + P -->|more repos| K + + P --> Q[wait_for_ci_gate] + Q --> R{ci_evaluator} + R -->|fail| S["attempt_ci_fix\n(up to 5x)"] + S --> Q + R -->|pass| T[">> human_review_gate"] + T -->|changes_requested| U[implement_review] + U --> Q + T -->|approved| V["complete_tasks\naggregate_epic_status\naggregate_feature_status\nEND"] ``` +`>>` = human checkpoint (auto-approved when `forge:yolo` label is set) + ## Bug Ticket Lifecycle +```mermaid +flowchart TD + A["Jira ticket created\nforge:managed (Bug)"] --> B[triage_check] + B -->|missing fields| C[">> triage_gate"] + C --> B + B -->|sufficient| D[analyze_bug] + D --> E["reflect_rca\n(up to 3 cycles)"] + E --> D + D --> F[">> rca_option_gate\nuser selects >option N"] + F --> G[plan_bug_fix] + G --> H[">> plan_approval_gate"] + H --> I[decompose_plan] + + I --> J[setup_workspace] + J --> K["implement_bug_fix\n(Podman + Deep Agents)"] + K --> L[local_review] + L -->|"needs work"| K + L --> M[update_documentation] + M --> N[create_pr] + N --> O[teardown_workspace] + O -->|more repos| J + + O --> P[wait_for_ci_gate] + P --> Q{ci_evaluator} + Q -->|fail| R["attempt_ci_fix"] + R --> P + Q -->|pass| S[">> human_review_gate"] + S -->|changes_requested| T[implement_review] + T --> P + S -->|merged| U[post_merge_summary] + U --> V[END] ``` -Jira ticket created + labeled "forge:managed" (type: Bug) - | - v -triage_check -- validate bug report has required fields - | - v ->> triage_gate -- wait for reporter update if incomplete - | - v -analyze_bug <--+ - | | - +--- reflect_rca (up to 3 reflection cycles) - | - v ->> rca_option_gate -- present fix options, user selects ">option N" - | - v -plan_bug_fix -- AI generates fix plan - | - v ->> plan_approval_gate - | - v -decompose_plan -- break into tasks - | - v -setup_workspace --> implement --> local_review --> create_pr --> CI --> review - | - v -post_merge_summary --> END + +## Task Ticket Lifecycle + +```mermaid +flowchart TD + A["Jira ticket created\nforge:managed (Task/Epic)"] --> B[triage_check] + B -->|missing context| C[">> triage_gate\nforge:task-triage-pending"] + C --> B + B -->|sufficient| D[generate_plan] + D --> E[">> task_plan_approval_gate\nforge:plan-pending"] + E -->|"? question"| F[answer_question] + F --> E + E -->|"! feedback"| D + E -->|approved| G[setup_workspace] + + G --> H["execute_task_changes\n(Podman + Deep Agents)"] + H --> I[qualitative_review] + I -->|"needs work (up to 2x)"| H + I -->|adequate| J[create_pr] + J --> K[teardown_workspace] + K -->|more repos| G + + K --> L[wait_for_ci_gate] + L --> M{ci_evaluator} + M -->|fail| N["attempt_ci_fix"] + N --> L + M -->|pass| O[">> human_review_gate"] + O -->|changes_requested| P[implement_review] + P --> L + O -->|merged| Q[complete_task_takeover] ``` ## Data Flow Summary -``` -Inbound events: Jira/GitHub webhooks --> FastAPI --> Redis Streams -State persistence: Redis (LangGraph AsyncRedisSaver, keyed by ticket) -LLM calls: Orchestrator nodes --> Claude/Gemini (Anthropic / Vertex AI) - Container agents --> Claude/Gemini (same backends) -Code execution: implement_task --> Podman container --> Deep Agents (library) -Outbound actions: Jira (comments, labels, transitions) - GitHub (PRs, branches, reviews) -Observability: Langfuse (LLM traces, workflow spans, costs) -``` +- **Inbound events:** Jira/GitHub webhooks --> FastAPI --> Redis Streams +- **State persistence:** Redis (LangGraph AsyncRedisSaver, keyed by ticket) +- **LLM calls:** Orchestrator nodes and container agents --> Claude/Gemini (Anthropic / Vertex AI), bidirectional +- **Code execution:** implement_task --> Podman container --> Deep Agents (library) +- **Outbound actions:** Jira (comments, labels, transitions), GitHub (PRs, branches, reviews) +- **Observability:** Langfuse (LLM traces, workflow spans, costs) From d47e2e4b75efdb7efcd419dfd5eb2b9040d8685d Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Sun, 19 Jul 2026 11:58:36 +0300 Subject: [PATCH 07/20] Add verbal explanations to architecture doc for newcomers - Add project intro paragraph explaining what Forge does - Add component descriptions for each layer in System Overview - Add context paragraphs before each workflow lifecycle diagram - All descriptions verified against current codebase Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/architecture.md b/docs/architecture.md index c5c03059..f175bddd 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,7 +1,18 @@ # Forge Architecture +Forge is an AI-powered SDLC orchestrator. It listens for Jira and GitHub events, routes them through LangGraph workflows, and drives implementation end-to-end — from requirements through code generation, CI repair, and human review. This page describes the major components and how they connect. + ## System Overview +Forge is built from five layers that form a pipeline from external events to code changes. + +- **External Systems** — Jira (ticket lifecycle), GitHub (PRs, CI, code review), and Langfuse (observability and cost tracking). Forge receives webhooks from Jira and GitHub and writes back to both throughout the workflow. +- **FastAPI Server** — A lightweight API layer that validates incoming webhooks and enqueues them as events. It also exposes health and Prometheus metrics endpoints. +- **Redis** — Serves two roles: an event bus (Redis Streams with consumer groups for reliable delivery) and a state store (LangGraph's `AsyncRedisSaver` checkpoints workflow state per ticket so workflows survive restarts). +- **WorkflowRouter** — The orchestration core. Incoming events are dispatched to one of three LangGraph `StateGraph` workflows based on Jira issue type: **FeatureWorkflow** (Feature/Story), **BugWorkflow** (Bug), or **TaskTakeoverWorkflow** (Task/Epic). Each workflow is a graph of nodes connected by conditional edges, with human approval gates at key decision points. +- **Podman Container** — Implementation runs in ephemeral rootless containers. Each container mounts the target repository, receives a task description, and uses Deep Agents (an AI coding library) with MCP tool access to make changes and commit them locally. The orchestrator handles pushing and PR creation after the container exits. +- **LLM Backends** — Claude and Gemini models are called bidirectionally: orchestrator nodes call them for planning and review, and container agents call them for code generation. Forge supports Anthropic's direct API and Vertex AI, selected by configuration. + ```mermaid flowchart TD subgraph External["External Systems"] @@ -53,6 +64,10 @@ flowchart TD ## Feature Ticket Lifecycle +The Feature workflow handles the largest scope of work. It takes a Jira Feature or Story from a one-line description through a full planning pipeline — PRD, technical spec, epic decomposition, and task breakdown — before any code is written. Each planning stage produces an artifact posted to Jira (or as a GitHub PR in the proposals repo) and pauses at a human approval gate. Reviewers can approve, request revisions with a `!` comment, or ask questions with `?` without advancing the workflow. + +Once all planning is approved, Forge groups tasks by target repository and implements them in parallel. Each task runs in its own Podman container. After implementation, Forge reviews the diff, updates documentation, and opens a PR. If CI fails, Forge analyzes the failure and attempts automated fixes (up to 5 times). When CI passes, the workflow pauses for human PR review. Review feedback triggers another implementation-CI cycle. After merge, Forge aggregates status up through tasks, epics, and the parent feature. + ```mermaid flowchart TD A["Jira ticket created\nforge:managed (Feature/Story)"] --> B[generate_prd] @@ -88,6 +103,10 @@ flowchart TD ## Bug Ticket Lifecycle +The Bug workflow starts with triage — Forge checks whether the ticket has enough context to investigate. If information is missing, it pauses and asks the reporter to fill in the gaps. Once the report is sufficient, Forge performs root cause analysis with up to three reflection cycles to refine its understanding. It then presents numbered fix options and waits for the user to select one with a `>option N` comment. + +After the user selects a fix approach, Forge generates a fix plan, pauses for approval, and decomposes it into implementation tasks. From there the workflow shares the same implementation path as the Feature workflow: container execution, local review, PR creation, CI repair loop, and human review. After the PR is merged, Forge posts a summary of the fix back to the Jira ticket. + ```mermaid flowchart TD A["Jira ticket created\nforge:managed (Bug)"] --> B[triage_check] @@ -123,6 +142,10 @@ flowchart TD ## Task Ticket Lifecycle +The Task workflow is the shortest path from ticket to PR. It handles standalone Jira Tasks and Epics that are already scoped enough to implement directly — no PRD, spec, or epic decomposition needed. Forge triages the ticket for sufficient context, generates an implementation plan, and pauses for approval. At the approval gate, reviewers can ask questions (`?`), request revisions (`!`), or approve to proceed. + +After approval, implementation follows the same container-based execution as the other workflows: Forge sets up a workspace, runs the changes in a Podman container with Deep Agents, reviews the output for quality (up to 2 retries), and opens a PR. The CI repair loop and human review gate work identically to the Feature and Bug workflows. + ```mermaid flowchart TD A["Jira ticket created\nforge:managed (Task/Epic)"] --> B[triage_check] From 4384012aa14fbcb750c78c812cd40f8c0d454cc0 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Sun, 19 Jul 2026 12:53:57 +0300 Subject: [PATCH 08/20] Clean up formatting: remove em dashes, use colons for component labels Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index f175bddd..f9e5b862 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,17 +1,17 @@ # Forge Architecture -Forge is an AI-powered SDLC orchestrator. It listens for Jira and GitHub events, routes them through LangGraph workflows, and drives implementation end-to-end — from requirements through code generation, CI repair, and human review. This page describes the major components and how they connect. +Forge is an AI-powered SDLC orchestrator. It listens for Jira and GitHub events, routes them through LangGraph workflows, and drives implementation end-to-end, from requirements through code generation, CI repair, and human review. This page describes the major components and how they connect. ## System Overview Forge is built from five layers that form a pipeline from external events to code changes. -- **External Systems** — Jira (ticket lifecycle), GitHub (PRs, CI, code review), and Langfuse (observability and cost tracking). Forge receives webhooks from Jira and GitHub and writes back to both throughout the workflow. -- **FastAPI Server** — A lightweight API layer that validates incoming webhooks and enqueues them as events. It also exposes health and Prometheus metrics endpoints. -- **Redis** — Serves two roles: an event bus (Redis Streams with consumer groups for reliable delivery) and a state store (LangGraph's `AsyncRedisSaver` checkpoints workflow state per ticket so workflows survive restarts). -- **WorkflowRouter** — The orchestration core. Incoming events are dispatched to one of three LangGraph `StateGraph` workflows based on Jira issue type: **FeatureWorkflow** (Feature/Story), **BugWorkflow** (Bug), or **TaskTakeoverWorkflow** (Task/Epic). Each workflow is a graph of nodes connected by conditional edges, with human approval gates at key decision points. -- **Podman Container** — Implementation runs in ephemeral rootless containers. Each container mounts the target repository, receives a task description, and uses Deep Agents (an AI coding library) with MCP tool access to make changes and commit them locally. The orchestrator handles pushing and PR creation after the container exits. -- **LLM Backends** — Claude and Gemini models are called bidirectionally: orchestrator nodes call them for planning and review, and container agents call them for code generation. Forge supports Anthropic's direct API and Vertex AI, selected by configuration. +- **External Systems:** Jira (ticket lifecycle), GitHub (PRs, CI, code review), and Langfuse (observability and cost tracking). Forge receives webhooks from Jira and GitHub and writes back to both throughout the workflow. +- **FastAPI Server:** A lightweight API layer that validates incoming webhooks and enqueues them as events. It also exposes health and Prometheus metrics endpoints. +- **Redis:** Serves two roles: an event bus (Redis Streams with consumer groups for reliable delivery) and a state store (LangGraph's `AsyncRedisSaver` checkpoints workflow state per ticket so workflows survive restarts). +- **WorkflowRouter:** The orchestration core. Incoming events are dispatched to one of three LangGraph `StateGraph` workflows based on Jira issue type: **FeatureWorkflow** (Feature/Story), **BugWorkflow** (Bug), or **TaskTakeoverWorkflow** (Task/Epic). Each workflow is a graph of nodes connected by conditional edges, with human approval gates at key decision points. +- **Podman Container:** Implementation runs in ephemeral rootless containers. Each container mounts the target repository, receives a task description, and uses Deep Agents (an AI coding library) with MCP tool access to make changes and commit them locally. The orchestrator handles pushing and PR creation after the container exits. +- **LLM Backends:** Claude and Gemini models are called bidirectionally. Orchestrator nodes call them for planning and review, and container agents call them for code generation. Forge supports Anthropic's direct API and Vertex AI, selected by configuration. ```mermaid flowchart TD @@ -64,7 +64,7 @@ flowchart TD ## Feature Ticket Lifecycle -The Feature workflow handles the largest scope of work. It takes a Jira Feature or Story from a one-line description through a full planning pipeline — PRD, technical spec, epic decomposition, and task breakdown — before any code is written. Each planning stage produces an artifact posted to Jira (or as a GitHub PR in the proposals repo) and pauses at a human approval gate. Reviewers can approve, request revisions with a `!` comment, or ask questions with `?` without advancing the workflow. +The Feature workflow handles the largest scope of work. It takes a Jira Feature or Story from a one-line description through a full planning pipeline (PRD, technical spec, epic decomposition, and task breakdown) before any code is written. Each planning stage produces an artifact posted to Jira (or as a GitHub PR in the proposals repo) and pauses at a human approval gate. Reviewers can approve, request revisions with a `!` comment, or ask questions with `?` without advancing the workflow. Once all planning is approved, Forge groups tasks by target repository and implements them in parallel. Each task runs in its own Podman container. After implementation, Forge reviews the diff, updates documentation, and opens a PR. If CI fails, Forge analyzes the failure and attempts automated fixes (up to 5 times). When CI passes, the workflow pauses for human PR review. Review feedback triggers another implementation-CI cycle. After merge, Forge aggregates status up through tasks, epics, and the parent feature. @@ -103,7 +103,7 @@ flowchart TD ## Bug Ticket Lifecycle -The Bug workflow starts with triage — Forge checks whether the ticket has enough context to investigate. If information is missing, it pauses and asks the reporter to fill in the gaps. Once the report is sufficient, Forge performs root cause analysis with up to three reflection cycles to refine its understanding. It then presents numbered fix options and waits for the user to select one with a `>option N` comment. +The Bug workflow starts with triage. Forge checks whether the ticket has enough context to investigate. If information is missing, it pauses and asks the reporter to fill in the gaps. Once the report is sufficient, Forge performs root cause analysis with up to three reflection cycles to refine its understanding. It then presents numbered fix options and waits for the user to select one with a `>option N` comment. After the user selects a fix approach, Forge generates a fix plan, pauses for approval, and decomposes it into implementation tasks. From there the workflow shares the same implementation path as the Feature workflow: container execution, local review, PR creation, CI repair loop, and human review. After the PR is merged, Forge posts a summary of the fix back to the Jira ticket. @@ -142,7 +142,7 @@ flowchart TD ## Task Ticket Lifecycle -The Task workflow is the shortest path from ticket to PR. It handles standalone Jira Tasks and Epics that are already scoped enough to implement directly — no PRD, spec, or epic decomposition needed. Forge triages the ticket for sufficient context, generates an implementation plan, and pauses for approval. At the approval gate, reviewers can ask questions (`?`), request revisions (`!`), or approve to proceed. +The Task workflow is the shortest path from ticket to PR. It handles standalone Jira Tasks and Epics that are already scoped enough to implement directly, without PRD, spec, or epic decomposition. Forge triages the ticket for sufficient context, generates an implementation plan, and pauses for approval. At the approval gate, reviewers can ask questions (`?`), request revisions (`!`), or approve to proceed. After approval, implementation follows the same container-based execution as the other workflows: Forge sets up a workspace, runs the changes in a Podman container with Deep Agents, reviews the output for quality (up to 2 retries), and opens a PR. The CI repair loop and human review gate work identically to the Feature and Bug workflows. From 59e6b5870a908e9eaef74e43f4a5ef8b0bc2e5af Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Sun, 19 Jul 2026 14:03:58 +0300 Subject: [PATCH 09/20] Remove inline code formatting from prose paragraphs Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index f9e5b862..7a670976 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -8,8 +8,8 @@ Forge is built from five layers that form a pipeline from external events to cod - **External Systems:** Jira (ticket lifecycle), GitHub (PRs, CI, code review), and Langfuse (observability and cost tracking). Forge receives webhooks from Jira and GitHub and writes back to both throughout the workflow. - **FastAPI Server:** A lightweight API layer that validates incoming webhooks and enqueues them as events. It also exposes health and Prometheus metrics endpoints. -- **Redis:** Serves two roles: an event bus (Redis Streams with consumer groups for reliable delivery) and a state store (LangGraph's `AsyncRedisSaver` checkpoints workflow state per ticket so workflows survive restarts). -- **WorkflowRouter:** The orchestration core. Incoming events are dispatched to one of three LangGraph `StateGraph` workflows based on Jira issue type: **FeatureWorkflow** (Feature/Story), **BugWorkflow** (Bug), or **TaskTakeoverWorkflow** (Task/Epic). Each workflow is a graph of nodes connected by conditional edges, with human approval gates at key decision points. +- **Redis:** Serves two roles: an event bus (Redis Streams with consumer groups for reliable delivery) and a state store (LangGraph's AsyncRedisSaver checkpoints workflow state per ticket so workflows survive restarts). +- **WorkflowRouter:** The orchestration core. Incoming events are dispatched to one of three LangGraph StateGraph workflows based on Jira issue type: **FeatureWorkflow** (Feature/Story), **BugWorkflow** (Bug), or **TaskTakeoverWorkflow** (Task/Epic). Each workflow is a graph of nodes connected by conditional edges, with human approval gates at key decision points. - **Podman Container:** Implementation runs in ephemeral rootless containers. Each container mounts the target repository, receives a task description, and uses Deep Agents (an AI coding library) with MCP tool access to make changes and commit them locally. The orchestrator handles pushing and PR creation after the container exits. - **LLM Backends:** Claude and Gemini models are called bidirectionally. Orchestrator nodes call them for planning and review, and container agents call them for code generation. Forge supports Anthropic's direct API and Vertex AI, selected by configuration. @@ -64,7 +64,7 @@ flowchart TD ## Feature Ticket Lifecycle -The Feature workflow handles the largest scope of work. It takes a Jira Feature or Story from a one-line description through a full planning pipeline (PRD, technical spec, epic decomposition, and task breakdown) before any code is written. Each planning stage produces an artifact posted to Jira (or as a GitHub PR in the proposals repo) and pauses at a human approval gate. Reviewers can approve, request revisions with a `!` comment, or ask questions with `?` without advancing the workflow. +The Feature workflow handles the largest scope of work. It takes a Jira Feature or Story from a one-line description through a full planning pipeline (PRD, technical spec, epic decomposition, and task breakdown) before any code is written. Each planning stage produces an artifact posted to Jira (or as a GitHub PR in the proposals repo) and pauses at a human approval gate. Reviewers can approve, request revisions with a "!" comment, or ask questions with "?" without advancing the workflow. Once all planning is approved, Forge groups tasks by target repository and implements them in parallel. Each task runs in its own Podman container. After implementation, Forge reviews the diff, updates documentation, and opens a PR. If CI fails, Forge analyzes the failure and attempts automated fixes (up to 5 times). When CI passes, the workflow pauses for human PR review. Review feedback triggers another implementation-CI cycle. After merge, Forge aggregates status up through tasks, epics, and the parent feature. @@ -99,11 +99,11 @@ flowchart TD T -->|approved| V["complete_tasks\naggregate_epic_status\naggregate_feature_status\nEND"] ``` -`>>` = human checkpoint (auto-approved when `forge:yolo` label is set) +">>" = human checkpoint (auto-approved when forge:yolo label is set) ## Bug Ticket Lifecycle -The Bug workflow starts with triage. Forge checks whether the ticket has enough context to investigate. If information is missing, it pauses and asks the reporter to fill in the gaps. Once the report is sufficient, Forge performs root cause analysis with up to three reflection cycles to refine its understanding. It then presents numbered fix options and waits for the user to select one with a `>option N` comment. +The Bug workflow starts with triage. Forge checks whether the ticket has enough context to investigate. If information is missing, it pauses and asks the reporter to fill in the gaps. Once the report is sufficient, Forge performs root cause analysis with up to three reflection cycles to refine its understanding. It then presents numbered fix options and waits for the user to select one with a ">option N" comment. After the user selects a fix approach, Forge generates a fix plan, pauses for approval, and decomposes it into implementation tasks. From there the workflow shares the same implementation path as the Feature workflow: container execution, local review, PR creation, CI repair loop, and human review. After the PR is merged, Forge posts a summary of the fix back to the Jira ticket. @@ -142,7 +142,7 @@ flowchart TD ## Task Ticket Lifecycle -The Task workflow is the shortest path from ticket to PR. It handles standalone Jira Tasks and Epics that are already scoped enough to implement directly, without PRD, spec, or epic decomposition. Forge triages the ticket for sufficient context, generates an implementation plan, and pauses for approval. At the approval gate, reviewers can ask questions (`?`), request revisions (`!`), or approve to proceed. +The Task workflow is the shortest path from ticket to PR. It handles standalone Jira Tasks and Epics that are already scoped enough to implement directly, without PRD, spec, or epic decomposition. Forge triages the ticket for sufficient context, generates an implementation plan, and pauses for approval. At the approval gate, reviewers can ask questions ("?"), request revisions ("!"), or approve to proceed. After approval, implementation follows the same container-based execution as the other workflows: Forge sets up a workspace, runs the changes in a Podman container with Deep Agents, reviews the output for quality (up to 2 retries), and opens a PR. The CI repair loop and human review gate work identically to the Feature and Bug workflows. From 481a927876863c87374d82a21dd14b201920d782 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Sun, 19 Jul 2026 14:23:23 +0300 Subject: [PATCH 10/20] Fix layer count: five to six in System Overview Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture.md b/docs/architecture.md index 7a670976..b161cadf 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -4,7 +4,7 @@ Forge is an AI-powered SDLC orchestrator. It listens for Jira and GitHub events, ## System Overview -Forge is built from five layers that form a pipeline from external events to code changes. +Forge is built from six layers that form a pipeline from external events to code changes. - **External Systems:** Jira (ticket lifecycle), GitHub (PRs, CI, code review), and Langfuse (observability and cost tracking). Forge receives webhooks from Jira and GitHub and writes back to both throughout the workflow. - **FastAPI Server:** A lightweight API layer that validates incoming webhooks and enqueues them as events. It also exposes health and Prometheus metrics endpoints. From 8c3a528447689fe64e0cc8d50a527523dccbe9ac Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Sun, 19 Jul 2026 20:14:51 +0300 Subject: [PATCH 11/20] Address PR review feedback on architecture diagram 1. Feature: change human_review_gate edge label from 'approved' to 'merged' 2. Bug: fix RCA flow to show conditional decision at reflect_rca instead of fork 3. Task: rename qualitative_review to run_qualitative_review (actual node name) 4. Fix checkpoint legend from HTML-escaped text to inline code 5. System overview: rename to Gateway/Workers to show gateway->redis->workers flow Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index b161cadf..f5919a31 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -21,7 +21,7 @@ flowchart TD Langfuse["Langfuse (Observability)"] end - subgraph API["FastAPI Server (:8000)"] + subgraph Gateway["FastAPI Gateway (:8000)"] JiraWH["POST /webhooks/jira"] GitHubWH["POST /webhooks/github"] end @@ -31,8 +31,8 @@ flowchart TD State["AsyncRedisSaver\nLangGraph checkpointing"] end - subgraph Router["WorkflowRouter"] - Route{"Route by\nissue type"} + subgraph Workers["Worker Processes (consumer group: forge-workers)"] + Router{"WorkflowRouter\nroute by issue type"} Feature["FeatureWorkflow\n(Feature/Story)"] Bug["BugWorkflow\n(Bug)"] Task["TaskTakeoverWorkflow\n(Task/Epic)"] @@ -48,18 +48,18 @@ flowchart TD GitHub -- webhooks --> GitHubWH JiraWH --> Streams GitHubWH --> Streams - Streams --> Route - Route --> Feature - Route --> Bug - Route --> Task + Streams --> Router + Router --> Feature + Router --> Bug + Router --> Task Feature --> Container Bug --> Container Task --> Container - Router <--> LLM + Workers <--> LLM Container <--> LLM - Router --> Jira - Router --> GitHub - Router --> Langfuse + Workers --> Jira + Workers --> GitHub + Workers --> Langfuse ``` ## Feature Ticket Lifecycle @@ -96,10 +96,10 @@ flowchart TD R -->|pass| T[">> human_review_gate"] T -->|changes_requested| U[implement_review] U --> Q - T -->|approved| V["complete_tasks\naggregate_epic_status\naggregate_feature_status\nEND"] + T -->|merged| V["complete_tasks\naggregate_epic_status\naggregate_feature_status\nEND"] ``` -">>" = human checkpoint (auto-approved when forge:yolo label is set) +`>>` = human checkpoint (auto-approved when forge:yolo label is set) ## Bug Ticket Lifecycle @@ -113,9 +113,9 @@ flowchart TD B -->|missing fields| C[">> triage_gate"] C --> B B -->|sufficient| D[analyze_bug] - D --> E["reflect_rca\n(up to 3 cycles)"] - E --> D - D --> F[">> rca_option_gate\nuser selects >option N"] + D --> E{reflect_rca} + E -->|"needs refinement (up to 3x)"| D + E -->|"analysis complete"| F[">> rca_option_gate\nuser selects >option N"] F --> G[plan_bug_fix] G --> H[">> plan_approval_gate"] H --> I[decompose_plan] @@ -159,7 +159,7 @@ flowchart TD E -->|approved| G[setup_workspace] G --> H["execute_task_changes\n(Podman + Deep Agents)"] - H --> I[qualitative_review] + H --> I[run_qualitative_review] I -->|"needs work (up to 2x)"| H I -->|adequate| J[create_pr] J --> K[teardown_workspace] From 2b77843783996b755c7a370dde3b94b3e15a59e2 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Sun, 19 Jul 2026 20:33:22 +0300 Subject: [PATCH 12/20] Fix review findings: label mismatch, legend placement, docs nav - Rename 'FastAPI Server' to 'FastAPI Gateway' in prose to match diagram - Move >> legend to end of page (applies to all three workflow diagrams) - Add architecture.md to zensical.toml nav under Developer Guide Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 6 +++--- zensical.toml | 1 + 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index f5919a31..142c29ef 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -7,7 +7,7 @@ Forge is an AI-powered SDLC orchestrator. It listens for Jira and GitHub events, Forge is built from six layers that form a pipeline from external events to code changes. - **External Systems:** Jira (ticket lifecycle), GitHub (PRs, CI, code review), and Langfuse (observability and cost tracking). Forge receives webhooks from Jira and GitHub and writes back to both throughout the workflow. -- **FastAPI Server:** A lightweight API layer that validates incoming webhooks and enqueues them as events. It also exposes health and Prometheus metrics endpoints. +- **FastAPI Gateway:** A lightweight API layer that validates incoming webhooks and enqueues them as events. It also exposes health and Prometheus metrics endpoints. - **Redis:** Serves two roles: an event bus (Redis Streams with consumer groups for reliable delivery) and a state store (LangGraph's AsyncRedisSaver checkpoints workflow state per ticket so workflows survive restarts). - **WorkflowRouter:** The orchestration core. Incoming events are dispatched to one of three LangGraph StateGraph workflows based on Jira issue type: **FeatureWorkflow** (Feature/Story), **BugWorkflow** (Bug), or **TaskTakeoverWorkflow** (Task/Epic). Each workflow is a graph of nodes connected by conditional edges, with human approval gates at key decision points. - **Podman Container:** Implementation runs in ephemeral rootless containers. Each container mounts the target repository, receives a task description, and uses Deep Agents (an AI coding library) with MCP tool access to make changes and commit them locally. The orchestrator handles pushing and PR creation after the container exits. @@ -99,8 +99,6 @@ flowchart TD T -->|merged| V["complete_tasks\naggregate_epic_status\naggregate_feature_status\nEND"] ``` -`>>` = human checkpoint (auto-approved when forge:yolo label is set) - ## Bug Ticket Lifecycle The Bug workflow starts with triage. Forge checks whether the ticket has enough context to investigate. If information is missing, it pauses and asks the reporter to fill in the gaps. Once the report is sufficient, Forge performs root cause analysis with up to three reflection cycles to refine its understanding. It then presents numbered fix options and waits for the user to select one with a ">option N" comment. @@ -175,6 +173,8 @@ flowchart TD O -->|merged| Q[complete_task_takeover] ``` +In all workflow diagrams above, `>>` marks a human checkpoint (auto-approved when the forge:yolo label is set). + ## Data Flow Summary - **Inbound events:** Jira/GitHub webhooks --> FastAPI --> Redis Streams diff --git a/zensical.toml b/zensical.toml index 692a4acb..40db45b0 100644 --- a/zensical.toml +++ b/zensical.toml @@ -18,6 +18,7 @@ nav = [ ]}, {"Developer Guide" = [ {"Overview" = "developer-guide.md"}, + {"Architecture" = "architecture.md"}, {"Local Setup" = "dev/setup.md"}, {"Testing" = "dev/testing.md"}, {"Contributing" = "dev/contributing.md"}, From b2777349a5fb0f2688ddc53a852e697291c05562 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Sun, 19 Jul 2026 20:40:18 +0300 Subject: [PATCH 13/20] Fix two factual errors found in code review 1. Feature diagram: local_review retries itself (self-loop), not implement_task. The actual graph routes local_review -> local_review or -> update_documentation. 2. Webhook paths: add missing /api/v1 prefix to match actual routes. Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 142c29ef..38b655fa 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -22,8 +22,8 @@ flowchart TD end subgraph Gateway["FastAPI Gateway (:8000)"] - JiraWH["POST /webhooks/jira"] - GitHubWH["POST /webhooks/github"] + JiraWH["POST /api/v1/webhooks/jira"] + GitHubWH["POST /api/v1/webhooks/github"] end subgraph Queue["Redis"] @@ -83,7 +83,7 @@ flowchart TD J --> K[setup_workspace] K --> L["implement_task\n(Podman + Deep Agents)"] L --> M[local_review] - M -->|"needs work (up to 2x)"| L + M -->|"mechanical fixes (up to 2x)"| M M --> N[update_documentation] N --> O[create_pr] O --> P["teardown_workspace"] From 2596a07f43daff14112790ff6e685c82615f28c9 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Mon, 20 Jul 2026 13:03:22 +0300 Subject: [PATCH 14/20] Add high-level pipeline diagram before detailed system overview Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/architecture.md b/docs/architecture.md index 38b655fa..035c8898 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -2,6 +2,21 @@ Forge is an AI-powered SDLC orchestrator. It listens for Jira and GitHub events, routes them through LangGraph workflows, and drives implementation end-to-end, from requirements through code generation, CI repair, and human review. This page describes the major components and how they connect. +## High-Level Pipeline + +At its core, Forge is a four-stage pipeline. External events flow in through an API gateway, queue in Redis, get processed by worker processes running LangGraph workflows, and produce code changes inside ephemeral containers. + +```mermaid +flowchart LR + A["Jira / GitHub\n(webhooks)"] --> B["FastAPI\nGateway"] + B --> C["Redis\n(Streams + State)"] + C --> D["Workers\n(LangGraph)"] + D --> E["Podman\nContainers"] + D <--> F["LLM\n(Claude / Gemini)"] + E <--> F + D --> A +``` + ## System Overview Forge is built from six layers that form a pipeline from external events to code changes. From 4aa367eee499b6f6b6e2b73290c257a02b0b216e Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Mon, 20 Jul 2026 13:17:46 +0300 Subject: [PATCH 15/20] Rewrite high-level architecture section with accurate component descriptions Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 035c8898..2b8bda99 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -2,13 +2,19 @@ Forge is an AI-powered SDLC orchestrator. It listens for Jira and GitHub events, routes them through LangGraph workflows, and drives implementation end-to-end, from requirements through code generation, CI repair, and human review. This page describes the major components and how they connect. -## High-Level Pipeline +## High-Level Architecture -At its core, Forge is a four-stage pipeline. External events flow in through an API gateway, queue in Redis, get processed by worker processes running LangGraph workflows, and produce code changes inside ephemeral containers. +Forge consists of three main services: the Gateway, the Worker, and Redis. + +- **Gateway (FastAPI)** accepts Jira and GitHub webhooks, validates signatures, deduplicates events, and publishes them to Redis Streams. It performs no workflow logic — its only job is ingestion. It also serves health, readiness, and Prometheus metrics endpoints. +- **Worker** consumes events from Redis Streams via consumer groups, resolves the target LangGraph workflow (Feature, Bug, or Task Takeover) based on ticket type, and drives execution through planning, implementation, CI repair, and human review stages. Implementation runs inside ephemeral Podman containers using Deep Agents. Multiple workers can run concurrently — Redis consumer groups distribute events across them, and each message is delivered to exactly one worker. +- **Redis** ties the system together. It serves as the event bus (Streams with consumer groups for reliable delivery), the workflow state store (LangGraph AsyncRedisSaver checkpoints per ticket for pause/resume and crash recovery), the retry queue (sorted set with exponential backoff), and supporting indexes (PR-to-ticket mapping, webhook deduplication). + +Because the Gateway and Workers communicate only through Redis, they can be deployed and scaled independently. ```mermaid flowchart LR - A["Jira / GitHub\n(webhooks)"] --> B["FastAPI\nGateway"] + A["Jira / GitHub\n(webhooks)"] --> B["Gateway\n(FastAPI)"] B --> C["Redis\n(Streams + State)"] C --> D["Workers\n(LangGraph)"] D --> E["Podman\nContainers"] From f26b5a15047ef7e0b5f63c5f74c34e9ab7a5cce7 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Mon, 20 Jul 2026 14:24:38 +0300 Subject: [PATCH 16/20] Address review feedback: legend placement, happy-path note, configurable CI limit - Move >> legend before the first diagram so readers see it before hitting >> prd_approval_gate - Add note that diagrams show the primary flow only (error-handling, revision loops omitted) - Change CI fix limit from "up to 5 times" to configurable with default - Task diagram already includes implement_review (added in prior commit) Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 2b8bda99..e33cc090 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -12,6 +12,8 @@ Forge consists of three main services: the Gateway, the Worker, and Redis. Because the Gateway and Workers communicate only through Redis, they can be deployed and scaled independently. +In the workflow diagrams below, `>>` marks a human checkpoint where the workflow pauses for approval, revision, or questions. These gates are auto-approved when the `forge:yolo` label is set. Diagrams show the primary flow only — error-handling, revision loops, and question-answering nodes are omitted for clarity. + ```mermaid flowchart LR A["Jira / GitHub\n(webhooks)"] --> B["Gateway\n(FastAPI)"] @@ -87,7 +89,7 @@ flowchart TD The Feature workflow handles the largest scope of work. It takes a Jira Feature or Story from a one-line description through a full planning pipeline (PRD, technical spec, epic decomposition, and task breakdown) before any code is written. Each planning stage produces an artifact posted to Jira (or as a GitHub PR in the proposals repo) and pauses at a human approval gate. Reviewers can approve, request revisions with a "!" comment, or ask questions with "?" without advancing the workflow. -Once all planning is approved, Forge groups tasks by target repository and implements them in parallel. Each task runs in its own Podman container. After implementation, Forge reviews the diff, updates documentation, and opens a PR. If CI fails, Forge analyzes the failure and attempts automated fixes (up to 5 times). When CI passes, the workflow pauses for human PR review. Review feedback triggers another implementation-CI cycle. After merge, Forge aggregates status up through tasks, epics, and the parent feature. +Once all planning is approved, Forge groups tasks by target repository and implements them in parallel. Each task runs in its own Podman container. After implementation, Forge reviews the diff, updates documentation, and opens a PR. If CI fails, Forge analyzes the failure and attempts automated fixes (up to 5 times by default, configurable via `CI_FIX_MAX_RETRIES`). When CI passes, the workflow pauses for human PR review. Review feedback triggers another implementation-CI cycle. After merge, Forge aggregates status up through tasks, epics, and the parent feature. ```mermaid flowchart TD @@ -112,7 +114,7 @@ flowchart TD P --> Q[wait_for_ci_gate] Q --> R{ci_evaluator} - R -->|fail| S["attempt_ci_fix\n(up to 5x)"] + R -->|fail| S["attempt_ci_fix\n(default 5x)"] S --> Q R -->|pass| T[">> human_review_gate"] T -->|changes_requested| U[implement_review] @@ -194,8 +196,6 @@ flowchart TD O -->|merged| Q[complete_task_takeover] ``` -In all workflow diagrams above, `>>` marks a human checkpoint (auto-approved when the forge:yolo label is set). - ## Data Flow Summary - **Inbound events:** Jira/GitHub webhooks --> FastAPI --> Redis Streams From 8f44f519b66cbaa3ad316753732e5e765a0cd980 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Mon, 20 Jul 2026 14:46:30 +0300 Subject: [PATCH 17/20] Cosmetic fixes Cosmetic fixes --- docs/architecture.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index e33cc090..7f6a6999 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -6,13 +6,13 @@ Forge is an AI-powered SDLC orchestrator. It listens for Jira and GitHub events, Forge consists of three main services: the Gateway, the Worker, and Redis. -- **Gateway (FastAPI)** accepts Jira and GitHub webhooks, validates signatures, deduplicates events, and publishes them to Redis Streams. It performs no workflow logic — its only job is ingestion. It also serves health, readiness, and Prometheus metrics endpoints. -- **Worker** consumes events from Redis Streams via consumer groups, resolves the target LangGraph workflow (Feature, Bug, or Task Takeover) based on ticket type, and drives execution through planning, implementation, CI repair, and human review stages. Implementation runs inside ephemeral Podman containers using Deep Agents. Multiple workers can run concurrently — Redis consumer groups distribute events across them, and each message is delivered to exactly one worker. +- **Gateway (FastAPI)** accepts Jira and GitHub webhooks, validates signatures, deduplicates events, and publishes them to Redis Streams. It performs no workflow logic; its only job is ingestion. It also serves health, readiness, and Prometheus metrics endpoints. +- **Worker** consumes events from Redis Streams via consumer groups, resolves the target LangGraph workflow (Feature, Bug, or Task Takeover) based on ticket type, and drives execution through planning, implementation, CI repair, and human review stages. Implementation runs inside ephemeral Podman containers using Deep Agents. Multiple workers can run concurrently, Redis consumer groups distribute events across them, and each message is delivered to exactly one worker. - **Redis** ties the system together. It serves as the event bus (Streams with consumer groups for reliable delivery), the workflow state store (LangGraph AsyncRedisSaver checkpoints per ticket for pause/resume and crash recovery), the retry queue (sorted set with exponential backoff), and supporting indexes (PR-to-ticket mapping, webhook deduplication). Because the Gateway and Workers communicate only through Redis, they can be deployed and scaled independently. -In the workflow diagrams below, `>>` marks a human checkpoint where the workflow pauses for approval, revision, or questions. These gates are auto-approved when the `forge:yolo` label is set. Diagrams show the primary flow only — error-handling, revision loops, and question-answering nodes are omitted for clarity. +In the workflow diagrams below, `>>` marks a human checkpoint where the workflow pauses for approval, revision, or questions. These gates are auto-approved when the `forge:yolo` label is set. Diagrams show the primary flow only; error-handling, revision loops, and question-answering nodes are omitted for clarity. ```mermaid flowchart LR @@ -126,7 +126,7 @@ flowchart TD The Bug workflow starts with triage. Forge checks whether the ticket has enough context to investigate. If information is missing, it pauses and asks the reporter to fill in the gaps. Once the report is sufficient, Forge performs root cause analysis with up to three reflection cycles to refine its understanding. It then presents numbered fix options and waits for the user to select one with a ">option N" comment. -After the user selects a fix approach, Forge generates a fix plan, pauses for approval, and decomposes it into implementation tasks. From there the workflow shares the same implementation path as the Feature workflow: container execution, local review, PR creation, CI repair loop, and human review. After the PR is merged, Forge posts a summary of the fix back to the Jira ticket. +After the user selects a fix approach, Forge generates a fix plan, pauses for approval, and decomposes it into implementation tasks. From there, the workflow shares the same implementation path as the Feature workflow: container execution, local review, PR creation, CI repair loop, and human review. After the PR is merged, Forge posts a summary of the fix back to the Jira ticket. ```mermaid flowchart TD @@ -163,7 +163,7 @@ flowchart TD ## Task Ticket Lifecycle -The Task workflow is the shortest path from ticket to PR. It handles standalone Jira Tasks and Epics that are already scoped enough to implement directly, without PRD, spec, or epic decomposition. Forge triages the ticket for sufficient context, generates an implementation plan, and pauses for approval. At the approval gate, reviewers can ask questions ("?"), request revisions ("!"), or approve to proceed. +The Task workflow is the shortest path from ticket to PR. It handles standalone Jira Tasks and Epics that are already scoped enough to implement directly, without PRD, spec, or epic decomposition. Forge triages the ticket for sufficient context, generates an implementation plan, and pauses for approval. At the approval gate, reviewers can ask questions ("?"), Request revisions ("!"), or approve to proceed. After approval, implementation follows the same container-based execution as the other workflows: Forge sets up a workspace, runs the changes in a Podman container with Deep Agents, reviews the output for quality (up to 2 retries), and opens a PR. The CI repair loop and human review gate work identically to the Feature and Bug workflows. From 0c596649d2770f270998f86df0422785b1255c45 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Tue, 21 Jul 2026 16:56:08 +0300 Subject: [PATCH 18/20] Address architecture review: add missing views, fix correctness claims Addresses all feedback from PR review: - Fix "exactly one worker" to at-least-once delivery guarantee - Qualify horizontal scaling with asyncio.Lock limitation - Add Scope/Goals/Non-goals/Audience section - Add Runtime and Deployment Topology (process topology, Podman host requirement, scaling constraints, container lifecycle, required connectivity, Redis data loss impact) - Add State/Events/Concurrency Model (at-least-once delivery, idempotency gaps, PEL reclaim gap, per-ticket ordering, checkpoint keying, checkpoint-to-side-effect consistency) - Add Failure and Recovery Model (per-component failure modes, retry budget, crash recovery, forge:retry, partial multi-repo, approval gate timeout) - Add Security and Trust Boundaries (webhook auth, credential scoping, container isolation with gaps, prompt injection, MCP permissions, auditability) - Add Quality Attributes (availability, throughput, latency, durability, auditability, known bottlenecks) - Add Key Architectural Decisions with rationale (Redis Streams, LangGraph, Podman, workflow separation, human gates) - Add Known Limitations section (8 items) - Simplify lifecycle diagrams to stable high-level step names - Link to detailed workflow guides for node-level flows Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 522 +++++++++++++++++++++++++++++++++---------- 1 file changed, 410 insertions(+), 112 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 7f6a6999..6f91331b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,18 +1,30 @@ # Forge Architecture -Forge is an AI-powered SDLC orchestrator. It listens for Jira and GitHub events, routes them through LangGraph workflows, and drives implementation end-to-end, from requirements through code generation, CI repair, and human review. This page describes the major components and how they connect. +## Scope and Audience -## High-Level Architecture +This document is the primary architecture reference for Forge, an AI-powered SDLC orchestrator. It covers system structure, runtime topology, state management, failure modes, security boundaries, and key design decisions. -Forge consists of three main services: the Gateway, the Worker, and Redis. +**Audience:** Maintainers, operators, and contributors who need to understand how Forge works, why it is built this way, and what constraints govern its operation. -- **Gateway (FastAPI)** accepts Jira and GitHub webhooks, validates signatures, deduplicates events, and publishes them to Redis Streams. It performs no workflow logic; its only job is ingestion. It also serves health, readiness, and Prometheus metrics endpoints. -- **Worker** consumes events from Redis Streams via consumer groups, resolves the target LangGraph workflow (Feature, Bug, or Task Takeover) based on ticket type, and drives execution through planning, implementation, CI repair, and human review stages. Implementation runs inside ephemeral Podman containers using Deep Agents. Multiple workers can run concurrently, Redis consumer groups distribute events across them, and each message is delivered to exactly one worker. -- **Redis** ties the system together. It serves as the event bus (Streams with consumer groups for reliable delivery), the workflow state store (LangGraph AsyncRedisSaver checkpoints per ticket for pause/resume and crash recovery), the retry queue (sorted set with exponential backoff), and supporting indexes (PR-to-ticket mapping, webhook deduplication). +**Goals:** -Because the Gateway and Workers communicate only through Redis, they can be deployed and scaled independently. +- Explain the runtime topology and deployment constraints +- Document state ownership, consistency guarantees, and failure recovery +- Define security and trust boundaries, especially around AI-generated code execution +- Record key architectural decisions with rationale and trade-offs +- Provide stable, high-level lifecycle views that link to detailed workflow guides -In the workflow diagrams below, `>>` marks a human checkpoint where the workflow pauses for approval, revision, or questions. These gates are auto-approved when the `forge:yolo` label is set. Diagrams show the primary flow only; error-handling, revision loops, and question-answering nodes are omitted for clarity. +**Non-goals:** + +- Node-level workflow documentation (see the [Feature](workflows/feature.md), [Bug](workflows/bug.md), and [Task](workflows/task.md) workflow guides) +- Operational runbooks or deployment procedures +- API reference (see the OpenAPI spec at `/docs` when the gateway is running) + +--- + +## System Context and External Actors + +Forge sits between project management (Jira), source control (GitHub), and LLM providers, orchestrating work from ticket creation through merged PR. ```mermaid flowchart LR @@ -25,16 +37,17 @@ flowchart LR D --> A ``` -## System Overview +**External actors:** + +- **Jira** — Source of ticket lifecycle events. Forge reads ticket metadata and writes comments, labels, and transitions back. Webhooks deliver issue and comment events. +- **GitHub** — Source of PR, CI, and code review events. Forge creates branches, pushes commits, opens PRs, and responds to review feedback. Webhooks deliver PR, check suite, and review events. +- **LLM providers** — Anthropic (direct API) and Google Vertex AI (Claude and Gemini models). Called by both orchestrator nodes (planning, review) and container agents (code generation). +- **Langfuse** — Observability platform for LLM call tracing, workflow spans, and cost tracking. Optional; disabled when credentials are not configured. +- **Human reviewers** — Approve or revise artifacts at defined gates. The workflow pauses indefinitely at each gate until a human responds. -Forge is built from six layers that form a pipeline from external events to code changes. +--- -- **External Systems:** Jira (ticket lifecycle), GitHub (PRs, CI, code review), and Langfuse (observability and cost tracking). Forge receives webhooks from Jira and GitHub and writes back to both throughout the workflow. -- **FastAPI Gateway:** A lightweight API layer that validates incoming webhooks and enqueues them as events. It also exposes health and Prometheus metrics endpoints. -- **Redis:** Serves two roles: an event bus (Redis Streams with consumer groups for reliable delivery) and a state store (LangGraph's AsyncRedisSaver checkpoints workflow state per ticket so workflows survive restarts). -- **WorkflowRouter:** The orchestration core. Incoming events are dispatched to one of three LangGraph StateGraph workflows based on Jira issue type: **FeatureWorkflow** (Feature/Story), **BugWorkflow** (Bug), or **TaskTakeoverWorkflow** (Task/Epic). Each workflow is a graph of nodes connected by conditional edges, with human approval gates at key decision points. -- **Podman Container:** Implementation runs in ephemeral rootless containers. Each container mounts the target repository, receives a task description, and uses Deep Agents (an AI coding library) with MCP tool access to make changes and commit them locally. The orchestrator handles pushing and PR creation after the container exits. -- **LLM Backends:** Claude and Gemini models are called bidirectionally. Orchestrator nodes call them for planning and review, and container agents call them for code generation. Forge supports Anthropic's direct API and Vertex AI, selected by configuration. +## Component Responsibilities ```mermaid flowchart TD @@ -85,122 +98,407 @@ flowchart TD Workers --> Langfuse ``` -## Feature Ticket Lifecycle +**Gateway (FastAPI)** — Accepts Jira and GitHub webhooks over HTTPS, validates HMAC-SHA256 signatures (when secrets are configured), and publishes events to Redis Streams. It performs no workflow logic. It also serves health, readiness, and Prometheus metrics endpoints. Middleware includes CORS and correlation ID propagation. A `DeduplicationService` exists in the codebase but is not yet wired into the webhook routes (see [Known Limitations](#known-limitations)). + +**Worker** — Consumes events from Redis Streams via the `forge-workers` consumer group. Each worker generates a unique consumer ID (`worker-{uuid}`). The `WorkflowRouter` resolves the target LangGraph workflow (Feature, Bug, or Task Takeover) based on Jira issue type and drives execution through planning, implementation, CI repair, and human review stages. Implementation runs inside ephemeral Podman containers using Deep Agents. + +**Redis** — Serves multiple roles: + +- *Event bus* — Redis Streams (`forge:events:jira`, `forge:events:github`) with the `forge-workers` consumer group for message distribution +- *Workflow state store* — LangGraph `AsyncRedisSaver` checkpoints workflow state per ticket (keyed by ticket key, e.g. `AISOS-123`) +- *Retry queue* — Sorted set (`forge:retry:queue`) with exponential backoff +- *Dead-letter queue* — List (`forge:retry:dlq`) for messages that exhaust retries +- *Supporting indexes* — PR-to-ticket mapping hash (`forge:state:pr_index`), webhook deduplication keys (`forge:dedup:*`, 24h TTL) + +**Podman Container** — Ephemeral rootless containers that execute implementation tasks. Each container receives the target repository mounted at `/workspace` (read-write), a task file at `/task.json` (read-only), and LLM credentials as environment variables. The container runs Deep Agents with MCP tool access to make changes and commit them locally. The orchestrator handles pushing and PR creation after the container exits. + +**LLM Backends** — Claude and Gemini models are called bidirectionally. Orchestrator nodes call them for planning and review; container agents call them for code generation. Two backends are supported: Anthropic's direct API (via `ANTHROPIC_API_KEY`) and Google Vertex AI (via service account credentials). The backend is selected automatically based on which credentials are configured. Request-level rate limiting uses token bucket algorithms (Anthropic: 0.5 req/s burst 5; Jira: 1.5 req/s burst 10; GitHub: 1.0 req/s burst 20). + +--- + +## Runtime and Deployment Topology + +### Process Topology + +Forge runs as two process types plus Redis: + +- **Gateway process** — A single FastAPI/Uvicorn process. Stateless; can be load-balanced if needed. No special host requirements beyond network access to Redis. +- **Worker process(es)** — One or more `OrchestratorWorker` processes. Each creates its own Redis consumer group member. Workers **must run on a host with Podman installed** and permission to spawn rootless containers. Each worker can run up to 20 concurrent tasks (configurable via `QUEUE_MAX_CONCURRENT_TASKS`), bounded by an `asyncio.Semaphore`. +- **Redis** — Single-instance Redis server. No Sentinel or Cluster configuration exists; HA must be provided externally if required. + +### Scaling Constraints + +Gateway and Worker communicate only through Redis and can be deployed on separate hosts. However, horizontal Worker scaling has a **known limitation**: per-ticket event serialization uses an in-process `asyncio.Lock`, not a distributed lock. If two Worker processes receive events for the same ticket concurrently, they will process them in parallel with no coordination. See [Known Limitations](#known-limitations) for details. + +### Container Lifecycle + +- Containers are named `forge-{ticket_key}-{short_uuid}` and are removed after execution by default +- Setting `FORGE_CONTAINER_KEEP=true` preserves failed containers for debugging +- Workspace directories are cleaned up via `podman unshare rm -rf` (to handle root-owned files), with `shutil.rmtree` as a fallback +- Stale workspaces older than 24 hours are eligible for cleanup via `cleanup_stale_workspaces()` + +### Required Connectivity + +- Workers --> Redis (event consumption, checkpoint read/write, retry queue) +- Gateway --> Redis (event publishing) +- Workers --> Jira API (comments, labels, transitions) +- Workers --> GitHub API (branches, commits, PRs, reviews) +- Workers --> LLM provider (Anthropic API or Vertex AI) +- Workers --> Langfuse (optional, trace ingestion) +- Containers --> LLM provider (code generation) +- Containers --> Langfuse (optional, in-container tracing) +- Containers --> internet (for dependency installation; network mode configurable via `FORGE_SANDBOX_NETWORK`, default `slirp4netns`) + +### What Is Lost if Redis Is Lost + +Redis holds both ephemeral queues and durable state. If Redis data is lost: + +- All in-flight events in the streams are lost (unprocessed webhooks) +- All workflow checkpoint state is lost (paused workflows cannot resume) +- The retry queue and dead-letter queue contents are lost +- PR-to-ticket index mappings are lost (webhook routing for PR events will fail until rebuilt) +- Deduplication keys are lost (duplicate event processing may occur temporarily) + +Operators should configure Redis persistence (RDB snapshots, AOF, or both) appropriate to their durability requirements. Forge does not configure Redis persistence itself. + +--- + +## State, Events, and Concurrency Model + +### Delivery Guarantee: At-Least-Once + +Forge uses **at-least-once** event processing. Redis consumer groups assign each stream entry to one consumer initially, but messages are acknowledged (`XACK`) only **after** successful processing. If a worker crashes between completing processing and sending the acknowledgement, the message remains in the Pending Entries List (PEL) and can be redelivered. + +The system does **not** provide exactly-once semantics. There is no transaction or Lua script wrapping handler execution and acknowledgement atomically. + +### Idempotency + +A `DeduplicationService` using Redis `SETNX` with 24-hour TTL keys (`forge:dedup:{event_id}`) is implemented but **not yet wired into the webhook routes**. Currently, duplicate webhooks from Jira or GitHub will be processed as separate events. + +Within workflows, certain operations have natural idempotency properties: + +- Branch creation — `git push` to an existing branch with the same content is a no-op +- PR creation — the workflow checks for existing PRs before creating new ones +- Label operations — setting a label that already exists is idempotent in both Jira and GitHub + +However, Jira comment posting and some GitHub operations are **not** idempotent. A duplicated event could result in duplicate comments. + +### Pending Entry Reclaim + +There is **no automated PEL reclaim mechanism**. The codebase does not use `XPENDING`, `XCLAIM`, or `XAUTOCLAIM`. If a worker crashes mid-processing before enqueuing the message for retry, the message remains in the PEL indefinitely. Recovery requires manual intervention or restarting the consumer group. This is a [known limitation](#known-limitations). + +### Per-Ticket Ordering + +Within a single worker process, events for the same ticket are serialized by an `asyncio.Lock` per ticket key. This ensures that concurrent events for ticket `AISOS-123` within one process are handled sequentially. + +Across multiple worker processes, **no distributed coordination exists**. Redis consumer groups distribute messages round-robin across consumers. If events for the same ticket are assigned to different workers, they will execute concurrently. This can lead to checkpoint conflicts or race conditions on Jira/GitHub side effects. + +**Current safe configuration:** Run a single Worker process per deployment, or accept that concurrent same-ticket processing may occur across workers. The in-process lock handles the common case (multiple events arriving in quick succession to the same consumer). + +### Checkpoint Model + +LangGraph workflow state is persisted via `AsyncRedisSaver` in Redis. The checkpoint lifecycle: + +- **Thread ID** = Jira ticket key (e.g., `AISOS-123`). One ticket maps to exactly one workflow instance and one checkpoint thread. +- **Writes** happen automatically after each LangGraph node completes. +- **Reads** happen when a new event arrives for an existing ticket — the workflow resumes from the last checkpoint. +- **Deletion** — `clear_checkpoint(thread_id)` removes all checkpoint data for a ticket. +- **Redis key pattern** — `checkpoint:{thread_id}:{...}` + +### Consistency Between Checkpoints and Side Effects + +Checkpoint writes and external side effects (Jira comments, GitHub PRs) are **not transactional**. A node may successfully post a comment to Jira but crash before its checkpoint is written, causing the comment to be posted again on retry. Similarly, a checkpoint may be written after the node function returns but before all side effects complete. + +The ordering between the Jira and GitHub event streams is **independent**. Events are consumed from both streams concurrently, and there is no ordering guarantee across streams. + +--- + +## Failure and Recovery Model + +### Component Failure Modes + +**Gateway failure** — Incoming webhooks are dropped. Jira and GitHub will retry webhook delivery according to their own retry policies (Jira retries up to 10 times with backoff; GitHub retries based on recent failure rate). No data is lost in Redis. + +**Worker failure (crash)** — In-flight messages remain in the Redis PEL. The workflow checkpoint reflects the last completed node. On restart, the worker begins consuming new messages. PEL messages require manual intervention (see [Known Limitations](#known-limitations)). Retry-eligible messages that were already enqueued in the retry sorted set will be picked up by any running worker. + +**Redis failure** — Complete system outage. Gateway cannot enqueue events; workers cannot consume, checkpoint, or retry. All in-memory state (streams, checkpoints, retry queue, DLQ, indexes) is at risk unless Redis persistence is configured. See [What Is Lost if Redis Is Lost](#what-is-lost-if-redis-is-lost). + +**LLM provider failure** — Orchestrator nodes and container agents fail. The retry mechanism handles transient LLM errors. Persistent failures exhaust retries and move the event to the dead-letter queue. The `forge:blocked` label is applied to the Jira ticket for visibility. + +**Jira/GitHub API failure** — Webhook-triggered nodes fail. Rate limiting (token bucket) prevents overloading. API errors during workflow execution are handled by the retry mechanism. + +**Container failure** — Non-zero exit codes are captured by the orchestrator. If `FORGE_CONTAINER_KEEP=true`, failed containers are preserved for debugging. The workflow node reports the failure, and the retry mechanism determines whether to re-attempt. + +### Retry Budget and Backoff + +- **Max retry attempts:** 3 +- **Backoff:** Exponential — 30s initial delay, 2x multiplier, capped at 3600s (1 hour) +- **Retry queue:** Redis sorted set (`forge:retry:queue`) with score = next retry timestamp, polled every 10 seconds +- **Dead-letter queue:** Redis list (`forge:retry:dlq`). Messages that exhaust all retries are moved here. Manual requeue via `requeue_dead_letter()` resets the attempt counter. + +### Crash Recovery + +- **Automatic:** Workers restart and consume new messages. Retry-queue entries are picked up by any running worker. LangGraph workflows resume from the last checkpoint when a new event arrives. +- **Manual:** PEL entries from crashed workers require manual `XCLAIM` or consumer group reset. DLQ entries require manual requeue or investigation. + +### Blocked Workflows and `forge:retry` + +When a workflow enters an error state, the `forge:blocked` label is applied to the Jira ticket. Adding the `forge:retry` label triggers re-entry at the failed step using the last checkpoint. This works at any workflow stage, including approval gates (where it regenerates the artifact). + +### Partial Multi-Repository Execution + +For tickets that span multiple repositories, the task router groups tasks by repo and processes them either sequentially or in parallel (up to 5 concurrent repos via LangGraph's `Send` API). If execution fails for one repo, that repo's error is recorded but other repos continue. The `aggregate_parallel_results()` node merges results and errors from all branches. + +### Approval Gate Timeout + +**There is no timeout on human approval gates.** When a workflow reaches a pause gate (`>> gate`), it persists `is_paused=True` in the checkpoint and waits indefinitely for a resume event (Jira comment or label change). If approval never arrives, the workflow remains paused in Redis checkpoint state. There is no automatic escalation or expiration. + +--- + +## Security and Trust Boundaries -The Feature workflow handles the largest scope of work. It takes a Jira Feature or Story from a one-line description through a full planning pipeline (PRD, technical spec, epic decomposition, and task breakdown) before any code is written. Each planning stage produces an artifact posted to Jira (or as a GitHub PR in the proposals repo) and pauses at a human approval gate. Reviewers can approve, request revisions with a "!" comment, or ask questions with "?" without advancing the workflow. +### Webhook Authentication -Once all planning is approved, Forge groups tasks by target repository and implements them in parallel. Each task runs in its own Podman container. After implementation, Forge reviews the diff, updates documentation, and opens a PR. If CI fails, Forge analyzes the failure and attempts automated fixes (up to 5 times by default, configurable via `CI_FIX_MAX_RETRIES`). When CI passes, the workflow pauses for human PR review. Review feedback triggers another implementation-CI cycle. After merge, Forge aggregates status up through tasks, epics, and the parent feature. +Both Jira and GitHub webhook endpoints validate HMAC-SHA256 signatures using `hmac.compare_digest()` (constant-time comparison). However, signature validation is **conditional**: it only runs when the corresponding secret is configured (`JIRA_WEBHOOK_SECRET`, `GITHUB_WEBHOOK_SECRET`). If secrets are not set, the endpoint accepts unsigned payloads without warning. + +**Recommendation for production:** Always configure webhook secrets. Without them, any network actor that can reach the gateway can inject events. + +### Credentials in the Worker + +Worker processes require: + +- Redis connection credentials (`REDIS_URL`) +- Jira API token (`JIRA_API_TOKEN`) — used for reading/writing tickets, comments, labels, transitions +- GitHub App credentials (`GITHUB_APP_ID`, `GITHUB_PRIVATE_KEY`) or personal access token — used for repository operations, PR management, and webhook validation +- LLM provider credentials — either `ANTHROPIC_API_KEY` (direct API) or Google service account credentials for Vertex AI +- Langfuse credentials (optional) — `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` + +### Credentials in the Container + +Implementation containers receive a **subset** of credentials as environment variables: + +- LLM credentials (API key or Vertex AI service account, depending on backend) +- Git user identity (`GIT_USER_NAME`, `GIT_USER_EMAIL`) +- Langfuse credentials (when enabled) + +Containers do **not** receive Jira credentials, GitHub tokens, or Redis credentials. All Jira/GitHub operations are performed by the orchestrator after the container exits. + +### Container Isolation + +- **Rootless Podman** — Containers run without root privileges on the host +- **Network** — Configurable via `FORGE_SANDBOX_NETWORK` (default: `slirp4netns`, providing NAT-based isolation) +- **Resource limits** — Memory: 4GB (configurable), CPU: 2 cores (configurable), Timeout: 30 minutes (configurable) +- **Filesystem** — Workspace mounted read-write at `/workspace` (necessary for code changes); task file mounted read-only at `/task.json` + +**Not currently enforced:** + +- `--security-opt no-new-privileges` is not set +- `--cap-drop ALL` is not applied (relies on Podman rootless defaults) +- No explicit seccomp profile +- Root filesystem is not read-only (`--read-only` not set) +- No AppArmor or SELinux enforcement beyond the `:Z` mount relabeling flag + +### Prompt Injection Boundaries + +Forge processes externally-supplied text from Jira tickets, GitHub PR descriptions, code review comments, and repository content. This text is passed to LLM prompts. The system relies on: + +- **Structured prompt templates** — System prompts are loaded from versioned templates in `src/forge/prompts/v1/`, not dynamically constructed from user input +- **Repository guardrails** — The workspace manager searches cloned repos for `CONSTITUTION.md`, `AGENTS.md`, and `CLAUDE.md` files and injects them as system context, giving repo owners control over agent behavior +- **Human gates** — All generated artifacts (PRDs, specs, plans, code) must pass human approval before merging, providing a boundary against manipulated output +- **Container isolation** — Code execution happens in ephemeral containers with limited credentials, bounding the blast radius of a compromised agent + +### MCP Tool Permissions + +Container agents use Deep Agents with MCP tool access. The agent can read and write files within `/workspace`, run shell commands, and use MCP-configured tools. It cannot access host filesystems outside the mount, Jira/GitHub APIs directly, or Redis. + +### Secret Redaction and Auditability + +- LLM interactions are traced via Langfuse (when enabled), providing an audit trail of all AI-generated content +- Container stdout/stderr is captured in worker logs +- Jira comments and GitHub PR descriptions provide a human-readable record of all workflow artifacts +- Credential values are passed as environment variables, not written to files or logs + +--- + +## Quality Attributes and Operational Characteristics + +### Availability + +- Gateway and Worker are stateless process types (state lives in Redis). Either can restart without data loss. +- Redis is a single point of failure. Redis downtime causes complete system outage. HA must be provided externally. +- Human approval gates have no timeout. Workflows can remain paused indefinitely without consuming resources beyond Redis checkpoint storage. + +### Throughput and Concurrency + +- Each Worker process handles up to 20 concurrent tasks (`QUEUE_MAX_CONCURRENT_TASKS`), bounded by an `asyncio.Semaphore`. +- The consumer reads up to 10 messages per `XREADGROUP` call with a 5-second block timeout. +- Multi-repo tasks can run up to 5 concurrent repository implementations via LangGraph `Send` fan-out. +- LLM rate limiting (Anthropic: 0.5 req/s burst 5) is the primary throughput bottleneck for planning-heavy workflows. + +### Per-Ticket Latency + +End-to-end latency is dominated by: + +- LLM response time (seconds to minutes per node, depending on prompt complexity) +- Human approval gates (minutes to days, unbounded) +- Container execution (30-minute default timeout per task) +- CI pipeline execution (external, typically 5-30 minutes) + +A fully autonomous (`forge:yolo`) simple task with fast CI can complete in 10-20 minutes. A multi-epic feature with human review at each gate can take days. + +### Durability + +- Workflow state is durable to Redis checkpoint writes. State survives Worker restarts. +- Event durability depends on Redis persistence configuration (operator responsibility). +- External side effects (Jira comments, GitHub PRs) are durable once written to those platforms. +- Container-local changes are lost if the container exits before the orchestrator pushes them. The orchestrator pushes immediately after successful container exit. + +### Auditability + +- Every workflow artifact (PRD, spec, plan, tasks) is posted as a Jira comment or GitHub PR, creating a human-readable audit trail +- LLM calls are traced in Langfuse with session ID = ticket key, including token counts and costs +- OpenTelemetry tracing is available (configurable via `OTLP_ENDPOINT`) +- Worker logs include correlation IDs for request tracing + +### Known Bottlenecks + +- **LLM rate limits** — The Anthropic rate limiter (0.5 req/s) serializes LLM calls across all concurrent tasks within a worker +- **Single Redis** — All state, queuing, and coordination goes through one Redis instance with no sharding +- **Container spawn overhead** — Podman container creation, repo cloning, and dependency installation add minutes per task + +--- + +## Key Architectural Decisions + +### Redis Streams for Event Bus + +**Decision:** Use Redis Streams with consumer groups instead of a dedicated message broker (RabbitMQ, Kafka). + +**Rationale:** Redis already serves as the checkpoint store and index. Using it for event queuing eliminates an additional infrastructure dependency. Consumer groups provide the needed delivery semantics (at-least-once, consumer-level distribution). The tradeoff is that Redis Streams lack built-in dead-letter queues, durable replay, and cross-datacenter replication — these are acceptable given Forge's current scale. + +### LangGraph for Workflow Orchestration + +**Decision:** Use LangGraph `StateGraph` with `AsyncRedisSaver` checkpointing instead of a traditional workflow engine (Temporal, Airflow). + +**Rationale:** LangGraph provides native support for LLM-driven decision nodes, conditional routing, and checkpointed pause/resume. The Python-native graph definition allows workflow logic to live alongside the orchestration code. The tradeoff is a less mature ecosystem and fewer operational tools compared to established workflow engines. + +### Host-Level Podman for Code Execution + +**Decision:** Run implementation tasks in Podman containers on the Worker host instead of using Kubernetes jobs, remote VMs, or in-process execution. + +**Rationale:** Rootless Podman provides process isolation, filesystem isolation, and resource limits without requiring a Kubernetes cluster. Workers must run on Podman-capable hosts, which constrains deployment options but simplifies the container lifecycle (local spawn, local cleanup). The tradeoff is that horizontal scaling requires Podman on every Worker host. + +### Workflow Separation by Issue Type + +**Decision:** Maintain three separate LangGraph workflow definitions (Feature, Bug, Task Takeover) rather than a single parameterized workflow. + +**Rationale:** Each workflow type has fundamentally different planning stages (PRD/spec/epic decomposition for Features, triage/RCA for Bugs, direct planning for Tasks). Separating them keeps each graph readable and independently modifiable. The shared implementation/CI/review tail is implemented as common nodes used by all three workflows. The tradeoff is some duplication in graph wiring. + +### Human Approval Gates + +**Decision:** Pause workflows at defined gates and wait indefinitely for human approval, rather than proceeding autonomously by default. + +**Rationale:** Forge generates and executes code in real repositories. Human review at planning stages (PRD, spec, plan, tasks) and at code review catches errors before they reach production. The `forge:yolo` label provides an opt-in escape hatch for tickets where autonomous operation is acceptable. The tradeoff is increased latency and human involvement for every ticket. + +--- + +## Known Limitations + +- **No PEL reclaim** — If a worker crashes mid-processing, unacknowledged messages remain in the Redis PEL indefinitely. No automated `XCLAIM`/`XAUTOCLAIM` mechanism exists. Recovery requires manual intervention. +- **No distributed per-ticket lock** — Per-ticket event serialization uses an in-process `asyncio.Lock`. Multiple Worker processes can process events for the same ticket concurrently, potentially causing checkpoint conflicts or duplicate side effects. +- **Webhook deduplication not wired** — A `DeduplicationService` exists but is not connected to the webhook routes. Duplicate webhooks are processed as separate events. +- **Webhook signature validation is optional** — If `JIRA_WEBHOOK_SECRET` or `GITHUB_WEBHOOK_SECRET` is not configured, the corresponding endpoint accepts unsigned payloads. +- **No approval gate timeout** — Workflows paused at human gates wait indefinitely. There is no automatic escalation, expiration, or notification for stale gates. +- **Single Redis dependency** — No Sentinel, Cluster, or HA configuration. Redis is a single point of failure. +- **Container security hardening gaps** — No `--cap-drop ALL`, `--no-new-privileges`, `--read-only` root filesystem, or explicit seccomp profile. Relies on Podman rootless defaults. +- **No cross-stream ordering** — Events from the Jira and GitHub streams are consumed independently. No ordering guarantee exists across streams for the same ticket. + +--- + +## Workflow Lifecycles + +The diagrams below show the high-level lifecycle for each workflow type. `>>` marks human approval gates where the workflow pauses until a reviewer approves, requests revisions (`!` comment), or asks questions (`?` comment). Gates are auto-approved when the `forge:yolo` label is set. + +These diagrams show the primary flow at a stable abstraction level. Error handling, revision loops, and question-answering branches are omitted. For detailed node-level flows with all branches, see the individual workflow guides: + +- [Feature Workflow Guide](workflows/feature.md) +- [Bug Workflow Guide](workflows/bug.md) +- [Task Workflow Guide](workflows/task.md) + +### Feature Lifecycle + +A Feature or Story goes through a full planning pipeline before implementation. Each planning artifact is reviewed and approved by a human before proceeding. ```mermaid flowchart TD - A["Jira ticket created\nforge:managed (Feature/Story)"] --> B[generate_prd] - B --> C[">> prd_approval_gate"] - C --> D[generate_spec] - D --> E[">> spec_approval_gate"] - E --> F[decompose_epics] - F --> G[">> plan_approval_gate"] - G --> H[generate_tasks] - H --> I[">> task_approval_gate"] - I --> J["task_router (per-repo)"] - - J --> K[setup_workspace] - K --> L["implement_task\n(Podman + Deep Agents)"] - L --> M[local_review] - M -->|"mechanical fixes (up to 2x)"| M - M --> N[update_documentation] - N --> O[create_pr] - O --> P["teardown_workspace"] - P -->|more repos| K - - P --> Q[wait_for_ci_gate] - Q --> R{ci_evaluator} - R -->|fail| S["attempt_ci_fix\n(default 5x)"] - S --> Q - R -->|pass| T[">> human_review_gate"] - T -->|changes_requested| U[implement_review] - U --> Q - T -->|merged| V["complete_tasks\naggregate_epic_status\naggregate_feature_status\nEND"] + A["Ticket created\n(Feature/Story)"] --> B["Generate PRD"] + B --> C[">> PRD approval"] + C --> D["Generate technical spec"] + D --> E[">> Spec approval"] + E --> F["Decompose into epics"] + F --> G[">> Plan approval"] + G --> H["Generate tasks"] + H --> I[">> Task approval"] + I --> J["Route tasks by repo"] + + J --> K["Implement in container"] + K --> L["Review and open PR"] + L --> M["CI repair loop"] + M --> N[">> Human code review"] + N -->|merged| O["Aggregate status\nComplete"] ``` -## Bug Ticket Lifecycle - -The Bug workflow starts with triage. Forge checks whether the ticket has enough context to investigate. If information is missing, it pauses and asks the reporter to fill in the gaps. Once the report is sufficient, Forge performs root cause analysis with up to three reflection cycles to refine its understanding. It then presents numbered fix options and waits for the user to select one with a ">option N" comment. +### Bug Lifecycle -After the user selects a fix approach, Forge generates a fix plan, pauses for approval, and decomposes it into implementation tasks. From there, the workflow shares the same implementation path as the Feature workflow: container execution, local review, PR creation, CI repair loop, and human review. After the PR is merged, Forge posts a summary of the fix back to the Jira ticket. +A Bug goes through triage and root cause analysis before implementation. The reporter may be asked to fill in missing context, and the user selects a fix approach from multiple options. ```mermaid flowchart TD - A["Jira ticket created\nforge:managed (Bug)"] --> B[triage_check] - B -->|missing fields| C[">> triage_gate"] + A["Ticket created\n(Bug)"] --> B["Triage check"] + B -->|"missing info"| C[">> Ask reporter"] C --> B - B -->|sufficient| D[analyze_bug] - D --> E{reflect_rca} - E -->|"needs refinement (up to 3x)"| D - E -->|"analysis complete"| F[">> rca_option_gate\nuser selects >option N"] - F --> G[plan_bug_fix] - G --> H[">> plan_approval_gate"] - H --> I[decompose_plan] - - I --> J[setup_workspace] - J --> K["implement_bug_fix\n(Podman + Deep Agents)"] - K --> L[local_review] - L -->|"needs work"| K - L --> M[update_documentation] - M --> N[create_pr] - N --> O[teardown_workspace] - O -->|more repos| J - - O --> P[wait_for_ci_gate] - P --> Q{ci_evaluator} - Q -->|fail| R["attempt_ci_fix"] - R --> P - Q -->|pass| S[">> human_review_gate"] - S -->|changes_requested| T[implement_review] - T --> P - S -->|merged| U[post_merge_summary] - U --> V[END] + B -->|"sufficient"| D["Root cause analysis"] + D --> E[">> Present fix options\n(user selects >option N)"] + E --> F["Generate fix plan"] + F --> G[">> Plan approval"] + + G --> H["Implement in container"] + H --> I["Review and open PR"] + I --> J["CI repair loop"] + J --> K[">> Human code review"] + K -->|merged| L["Post-merge summary\nComplete"] ``` -## Task Ticket Lifecycle +### Task Lifecycle -The Task workflow is the shortest path from ticket to PR. It handles standalone Jira Tasks and Epics that are already scoped enough to implement directly, without PRD, spec, or epic decomposition. Forge triages the ticket for sufficient context, generates an implementation plan, and pauses for approval. At the approval gate, reviewers can ask questions ("?"), Request revisions ("!"), or approve to proceed. - -After approval, implementation follows the same container-based execution as the other workflows: Forge sets up a workspace, runs the changes in a Podman container with Deep Agents, reviews the output for quality (up to 2 retries), and opens a PR. The CI repair loop and human review gate work identically to the Feature and Bug workflows. +A Task or Epic goes directly from triage to implementation planning, skipping PRD, spec, and epic decomposition. ```mermaid flowchart TD - A["Jira ticket created\nforge:managed (Task/Epic)"] --> B[triage_check] - B -->|missing context| C[">> triage_gate\nforge:task-triage-pending"] + A["Ticket created\n(Task/Epic)"] --> B["Triage check"] + B -->|"missing context"| C[">> Ask for details"] C --> B - B -->|sufficient| D[generate_plan] - D --> E[">> task_plan_approval_gate\nforge:plan-pending"] - E -->|"? question"| F[answer_question] - F --> E - E -->|"! feedback"| D - E -->|approved| G[setup_workspace] - - G --> H["execute_task_changes\n(Podman + Deep Agents)"] - H --> I[run_qualitative_review] - I -->|"needs work (up to 2x)"| H - I -->|adequate| J[create_pr] - J --> K[teardown_workspace] - K -->|more repos| G - - K --> L[wait_for_ci_gate] - L --> M{ci_evaluator} - M -->|fail| N["attempt_ci_fix"] - N --> L - M -->|pass| O[">> human_review_gate"] - O -->|changes_requested| P[implement_review] - P --> L - O -->|merged| Q[complete_task_takeover] + B -->|"sufficient"| D["Generate plan"] + D --> E[">> Plan approval"] + + E --> F["Implement in container"] + F --> G["Review and open PR"] + G --> H["CI repair loop"] + H --> I[">> Human code review"] + I -->|merged| J["Complete"] ``` +### Diagram Maintenance + +The lifecycle diagrams above use stable, high-level step names rather than concrete graph node names. The source of truth for node-level workflow details is the graph definition code in `src/forge/orchestrator/`. When graph node names or edges change, the detailed workflow guides should be updated; these high-level diagrams should only change when the overall lifecycle structure changes. + +--- + ## Data Flow Summary -- **Inbound events:** Jira/GitHub webhooks --> FastAPI --> Redis Streams -- **State persistence:** Redis (LangGraph AsyncRedisSaver, keyed by ticket) -- **LLM calls:** Orchestrator nodes and container agents --> Claude/Gemini (Anthropic / Vertex AI), bidirectional -- **Code execution:** implement_task --> Podman container --> Deep Agents (library) -- **Outbound actions:** Jira (comments, labels, transitions), GitHub (PRs, branches, reviews) -- **Observability:** Langfuse (LLM traces, workflow spans, costs) +- **Inbound events:** Jira/GitHub webhooks --> FastAPI gateway --> Redis Streams +- **State persistence:** Redis (`AsyncRedisSaver`, keyed by Jira ticket key) +- **LLM calls:** Orchestrator nodes and container agents --> Claude/Gemini (Anthropic direct API or Vertex AI) +- **Code execution:** Workflow node --> Podman container --> Deep Agents with MCP tools +- **Outbound actions:** Jira (comments, labels, transitions), GitHub (branches, commits, PRs, reviews) +- **Observability:** Langfuse (LLM traces, workflow spans, costs), OpenTelemetry (configurable) From 1ad089a1f3ba8550fcabbf109149664575955ad3 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Wed, 22 Jul 2026 10:58:36 +0300 Subject: [PATCH 19/20] Split architecture.md into structured parts, fix broken links and formatting - Fix broken workflow links (workflows/*.md -> guide/*-workflow.md) - Replace em-dash separators with colons for cleaner formatting - Convert Known Limitations anchor links to plain text - Fix "What Is Lost if Redis Is Lost" cross-reference - Split monolithic doc into architecture/overview.md, internals.md, reference.md - Add architecture/index.md as landing page - Update zensical.toml navigation for new directory structure Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture.md | 504 --------------------------------- docs/architecture/index.md | 13 + docs/architecture/internals.md | 197 +++++++++++++ docs/architecture/overview.md | 111 ++++++++ docs/architecture/reference.md | 176 ++++++++++++ zensical.toml | 7 +- 6 files changed, 503 insertions(+), 505 deletions(-) delete mode 100644 docs/architecture.md create mode 100644 docs/architecture/index.md create mode 100644 docs/architecture/internals.md create mode 100644 docs/architecture/overview.md create mode 100644 docs/architecture/reference.md diff --git a/docs/architecture.md b/docs/architecture.md deleted file mode 100644 index 6f91331b..00000000 --- a/docs/architecture.md +++ /dev/null @@ -1,504 +0,0 @@ -# Forge Architecture - -## Scope and Audience - -This document is the primary architecture reference for Forge, an AI-powered SDLC orchestrator. It covers system structure, runtime topology, state management, failure modes, security boundaries, and key design decisions. - -**Audience:** Maintainers, operators, and contributors who need to understand how Forge works, why it is built this way, and what constraints govern its operation. - -**Goals:** - -- Explain the runtime topology and deployment constraints -- Document state ownership, consistency guarantees, and failure recovery -- Define security and trust boundaries, especially around AI-generated code execution -- Record key architectural decisions with rationale and trade-offs -- Provide stable, high-level lifecycle views that link to detailed workflow guides - -**Non-goals:** - -- Node-level workflow documentation (see the [Feature](workflows/feature.md), [Bug](workflows/bug.md), and [Task](workflows/task.md) workflow guides) -- Operational runbooks or deployment procedures -- API reference (see the OpenAPI spec at `/docs` when the gateway is running) - ---- - -## System Context and External Actors - -Forge sits between project management (Jira), source control (GitHub), and LLM providers, orchestrating work from ticket creation through merged PR. - -```mermaid -flowchart LR - A["Jira / GitHub\n(webhooks)"] --> B["Gateway\n(FastAPI)"] - B --> C["Redis\n(Streams + State)"] - C --> D["Workers\n(LangGraph)"] - D --> E["Podman\nContainers"] - D <--> F["LLM\n(Claude / Gemini)"] - E <--> F - D --> A -``` - -**External actors:** - -- **Jira** — Source of ticket lifecycle events. Forge reads ticket metadata and writes comments, labels, and transitions back. Webhooks deliver issue and comment events. -- **GitHub** — Source of PR, CI, and code review events. Forge creates branches, pushes commits, opens PRs, and responds to review feedback. Webhooks deliver PR, check suite, and review events. -- **LLM providers** — Anthropic (direct API) and Google Vertex AI (Claude and Gemini models). Called by both orchestrator nodes (planning, review) and container agents (code generation). -- **Langfuse** — Observability platform for LLM call tracing, workflow spans, and cost tracking. Optional; disabled when credentials are not configured. -- **Human reviewers** — Approve or revise artifacts at defined gates. The workflow pauses indefinitely at each gate until a human responds. - ---- - -## Component Responsibilities - -```mermaid -flowchart TD - subgraph External["External Systems"] - Jira - GitHub - Langfuse["Langfuse (Observability)"] - end - - subgraph Gateway["FastAPI Gateway (:8000)"] - JiraWH["POST /api/v1/webhooks/jira"] - GitHubWH["POST /api/v1/webhooks/github"] - end - - subgraph Queue["Redis"] - Streams["Streams: forge:events:jira\nforge:events:github"] - State["AsyncRedisSaver\nLangGraph checkpointing"] - end - - subgraph Workers["Worker Processes (consumer group: forge-workers)"] - Router{"WorkflowRouter\nroute by issue type"} - Feature["FeatureWorkflow\n(Feature/Story)"] - Bug["BugWorkflow\n(Bug)"] - Task["TaskTakeoverWorkflow\n(Task/Epic)"] - end - - subgraph Container["Podman Container (ephemeral)"] - Agent["Deep Agents + MCP\n/workspace (repo mounted)"] - end - - LLM["LLM Backends\nAnthropic API (Claude)\nVertex AI (Claude/Gemini)"] - - Jira -- webhooks --> JiraWH - GitHub -- webhooks --> GitHubWH - JiraWH --> Streams - GitHubWH --> Streams - Streams --> Router - Router --> Feature - Router --> Bug - Router --> Task - Feature --> Container - Bug --> Container - Task --> Container - Workers <--> LLM - Container <--> LLM - Workers --> Jira - Workers --> GitHub - Workers --> Langfuse -``` - -**Gateway (FastAPI)** — Accepts Jira and GitHub webhooks over HTTPS, validates HMAC-SHA256 signatures (when secrets are configured), and publishes events to Redis Streams. It performs no workflow logic. It also serves health, readiness, and Prometheus metrics endpoints. Middleware includes CORS and correlation ID propagation. A `DeduplicationService` exists in the codebase but is not yet wired into the webhook routes (see [Known Limitations](#known-limitations)). - -**Worker** — Consumes events from Redis Streams via the `forge-workers` consumer group. Each worker generates a unique consumer ID (`worker-{uuid}`). The `WorkflowRouter` resolves the target LangGraph workflow (Feature, Bug, or Task Takeover) based on Jira issue type and drives execution through planning, implementation, CI repair, and human review stages. Implementation runs inside ephemeral Podman containers using Deep Agents. - -**Redis** — Serves multiple roles: - -- *Event bus* — Redis Streams (`forge:events:jira`, `forge:events:github`) with the `forge-workers` consumer group for message distribution -- *Workflow state store* — LangGraph `AsyncRedisSaver` checkpoints workflow state per ticket (keyed by ticket key, e.g. `AISOS-123`) -- *Retry queue* — Sorted set (`forge:retry:queue`) with exponential backoff -- *Dead-letter queue* — List (`forge:retry:dlq`) for messages that exhaust retries -- *Supporting indexes* — PR-to-ticket mapping hash (`forge:state:pr_index`), webhook deduplication keys (`forge:dedup:*`, 24h TTL) - -**Podman Container** — Ephemeral rootless containers that execute implementation tasks. Each container receives the target repository mounted at `/workspace` (read-write), a task file at `/task.json` (read-only), and LLM credentials as environment variables. The container runs Deep Agents with MCP tool access to make changes and commit them locally. The orchestrator handles pushing and PR creation after the container exits. - -**LLM Backends** — Claude and Gemini models are called bidirectionally. Orchestrator nodes call them for planning and review; container agents call them for code generation. Two backends are supported: Anthropic's direct API (via `ANTHROPIC_API_KEY`) and Google Vertex AI (via service account credentials). The backend is selected automatically based on which credentials are configured. Request-level rate limiting uses token bucket algorithms (Anthropic: 0.5 req/s burst 5; Jira: 1.5 req/s burst 10; GitHub: 1.0 req/s burst 20). - ---- - -## Runtime and Deployment Topology - -### Process Topology - -Forge runs as two process types plus Redis: - -- **Gateway process** — A single FastAPI/Uvicorn process. Stateless; can be load-balanced if needed. No special host requirements beyond network access to Redis. -- **Worker process(es)** — One or more `OrchestratorWorker` processes. Each creates its own Redis consumer group member. Workers **must run on a host with Podman installed** and permission to spawn rootless containers. Each worker can run up to 20 concurrent tasks (configurable via `QUEUE_MAX_CONCURRENT_TASKS`), bounded by an `asyncio.Semaphore`. -- **Redis** — Single-instance Redis server. No Sentinel or Cluster configuration exists; HA must be provided externally if required. - -### Scaling Constraints - -Gateway and Worker communicate only through Redis and can be deployed on separate hosts. However, horizontal Worker scaling has a **known limitation**: per-ticket event serialization uses an in-process `asyncio.Lock`, not a distributed lock. If two Worker processes receive events for the same ticket concurrently, they will process them in parallel with no coordination. See [Known Limitations](#known-limitations) for details. - -### Container Lifecycle - -- Containers are named `forge-{ticket_key}-{short_uuid}` and are removed after execution by default -- Setting `FORGE_CONTAINER_KEEP=true` preserves failed containers for debugging -- Workspace directories are cleaned up via `podman unshare rm -rf` (to handle root-owned files), with `shutil.rmtree` as a fallback -- Stale workspaces older than 24 hours are eligible for cleanup via `cleanup_stale_workspaces()` - -### Required Connectivity - -- Workers --> Redis (event consumption, checkpoint read/write, retry queue) -- Gateway --> Redis (event publishing) -- Workers --> Jira API (comments, labels, transitions) -- Workers --> GitHub API (branches, commits, PRs, reviews) -- Workers --> LLM provider (Anthropic API or Vertex AI) -- Workers --> Langfuse (optional, trace ingestion) -- Containers --> LLM provider (code generation) -- Containers --> Langfuse (optional, in-container tracing) -- Containers --> internet (for dependency installation; network mode configurable via `FORGE_SANDBOX_NETWORK`, default `slirp4netns`) - -### What Is Lost if Redis Is Lost - -Redis holds both ephemeral queues and durable state. If Redis data is lost: - -- All in-flight events in the streams are lost (unprocessed webhooks) -- All workflow checkpoint state is lost (paused workflows cannot resume) -- The retry queue and dead-letter queue contents are lost -- PR-to-ticket index mappings are lost (webhook routing for PR events will fail until rebuilt) -- Deduplication keys are lost (duplicate event processing may occur temporarily) - -Operators should configure Redis persistence (RDB snapshots, AOF, or both) appropriate to their durability requirements. Forge does not configure Redis persistence itself. - ---- - -## State, Events, and Concurrency Model - -### Delivery Guarantee: At-Least-Once - -Forge uses **at-least-once** event processing. Redis consumer groups assign each stream entry to one consumer initially, but messages are acknowledged (`XACK`) only **after** successful processing. If a worker crashes between completing processing and sending the acknowledgement, the message remains in the Pending Entries List (PEL) and can be redelivered. - -The system does **not** provide exactly-once semantics. There is no transaction or Lua script wrapping handler execution and acknowledgement atomically. - -### Idempotency - -A `DeduplicationService` using Redis `SETNX` with 24-hour TTL keys (`forge:dedup:{event_id}`) is implemented but **not yet wired into the webhook routes**. Currently, duplicate webhooks from Jira or GitHub will be processed as separate events. - -Within workflows, certain operations have natural idempotency properties: - -- Branch creation — `git push` to an existing branch with the same content is a no-op -- PR creation — the workflow checks for existing PRs before creating new ones -- Label operations — setting a label that already exists is idempotent in both Jira and GitHub - -However, Jira comment posting and some GitHub operations are **not** idempotent. A duplicated event could result in duplicate comments. - -### Pending Entry Reclaim - -There is **no automated PEL reclaim mechanism**. The codebase does not use `XPENDING`, `XCLAIM`, or `XAUTOCLAIM`. If a worker crashes mid-processing before enqueuing the message for retry, the message remains in the PEL indefinitely. Recovery requires manual intervention or restarting the consumer group. This is a [known limitation](#known-limitations). - -### Per-Ticket Ordering - -Within a single worker process, events for the same ticket are serialized by an `asyncio.Lock` per ticket key. This ensures that concurrent events for ticket `AISOS-123` within one process are handled sequentially. - -Across multiple worker processes, **no distributed coordination exists**. Redis consumer groups distribute messages round-robin across consumers. If events for the same ticket are assigned to different workers, they will execute concurrently. This can lead to checkpoint conflicts or race conditions on Jira/GitHub side effects. - -**Current safe configuration:** Run a single Worker process per deployment, or accept that concurrent same-ticket processing may occur across workers. The in-process lock handles the common case (multiple events arriving in quick succession to the same consumer). - -### Checkpoint Model - -LangGraph workflow state is persisted via `AsyncRedisSaver` in Redis. The checkpoint lifecycle: - -- **Thread ID** = Jira ticket key (e.g., `AISOS-123`). One ticket maps to exactly one workflow instance and one checkpoint thread. -- **Writes** happen automatically after each LangGraph node completes. -- **Reads** happen when a new event arrives for an existing ticket — the workflow resumes from the last checkpoint. -- **Deletion** — `clear_checkpoint(thread_id)` removes all checkpoint data for a ticket. -- **Redis key pattern** — `checkpoint:{thread_id}:{...}` - -### Consistency Between Checkpoints and Side Effects - -Checkpoint writes and external side effects (Jira comments, GitHub PRs) are **not transactional**. A node may successfully post a comment to Jira but crash before its checkpoint is written, causing the comment to be posted again on retry. Similarly, a checkpoint may be written after the node function returns but before all side effects complete. - -The ordering between the Jira and GitHub event streams is **independent**. Events are consumed from both streams concurrently, and there is no ordering guarantee across streams. - ---- - -## Failure and Recovery Model - -### Component Failure Modes - -**Gateway failure** — Incoming webhooks are dropped. Jira and GitHub will retry webhook delivery according to their own retry policies (Jira retries up to 10 times with backoff; GitHub retries based on recent failure rate). No data is lost in Redis. - -**Worker failure (crash)** — In-flight messages remain in the Redis PEL. The workflow checkpoint reflects the last completed node. On restart, the worker begins consuming new messages. PEL messages require manual intervention (see [Known Limitations](#known-limitations)). Retry-eligible messages that were already enqueued in the retry sorted set will be picked up by any running worker. - -**Redis failure** — Complete system outage. Gateway cannot enqueue events; workers cannot consume, checkpoint, or retry. All in-memory state (streams, checkpoints, retry queue, DLQ, indexes) is at risk unless Redis persistence is configured. See [What Is Lost if Redis Is Lost](#what-is-lost-if-redis-is-lost). - -**LLM provider failure** — Orchestrator nodes and container agents fail. The retry mechanism handles transient LLM errors. Persistent failures exhaust retries and move the event to the dead-letter queue. The `forge:blocked` label is applied to the Jira ticket for visibility. - -**Jira/GitHub API failure** — Webhook-triggered nodes fail. Rate limiting (token bucket) prevents overloading. API errors during workflow execution are handled by the retry mechanism. - -**Container failure** — Non-zero exit codes are captured by the orchestrator. If `FORGE_CONTAINER_KEEP=true`, failed containers are preserved for debugging. The workflow node reports the failure, and the retry mechanism determines whether to re-attempt. - -### Retry Budget and Backoff - -- **Max retry attempts:** 3 -- **Backoff:** Exponential — 30s initial delay, 2x multiplier, capped at 3600s (1 hour) -- **Retry queue:** Redis sorted set (`forge:retry:queue`) with score = next retry timestamp, polled every 10 seconds -- **Dead-letter queue:** Redis list (`forge:retry:dlq`). Messages that exhaust all retries are moved here. Manual requeue via `requeue_dead_letter()` resets the attempt counter. - -### Crash Recovery - -- **Automatic:** Workers restart and consume new messages. Retry-queue entries are picked up by any running worker. LangGraph workflows resume from the last checkpoint when a new event arrives. -- **Manual:** PEL entries from crashed workers require manual `XCLAIM` or consumer group reset. DLQ entries require manual requeue or investigation. - -### Blocked Workflows and `forge:retry` - -When a workflow enters an error state, the `forge:blocked` label is applied to the Jira ticket. Adding the `forge:retry` label triggers re-entry at the failed step using the last checkpoint. This works at any workflow stage, including approval gates (where it regenerates the artifact). - -### Partial Multi-Repository Execution - -For tickets that span multiple repositories, the task router groups tasks by repo and processes them either sequentially or in parallel (up to 5 concurrent repos via LangGraph's `Send` API). If execution fails for one repo, that repo's error is recorded but other repos continue. The `aggregate_parallel_results()` node merges results and errors from all branches. - -### Approval Gate Timeout - -**There is no timeout on human approval gates.** When a workflow reaches a pause gate (`>> gate`), it persists `is_paused=True` in the checkpoint and waits indefinitely for a resume event (Jira comment or label change). If approval never arrives, the workflow remains paused in Redis checkpoint state. There is no automatic escalation or expiration. - ---- - -## Security and Trust Boundaries - -### Webhook Authentication - -Both Jira and GitHub webhook endpoints validate HMAC-SHA256 signatures using `hmac.compare_digest()` (constant-time comparison). However, signature validation is **conditional**: it only runs when the corresponding secret is configured (`JIRA_WEBHOOK_SECRET`, `GITHUB_WEBHOOK_SECRET`). If secrets are not set, the endpoint accepts unsigned payloads without warning. - -**Recommendation for production:** Always configure webhook secrets. Without them, any network actor that can reach the gateway can inject events. - -### Credentials in the Worker - -Worker processes require: - -- Redis connection credentials (`REDIS_URL`) -- Jira API token (`JIRA_API_TOKEN`) — used for reading/writing tickets, comments, labels, transitions -- GitHub App credentials (`GITHUB_APP_ID`, `GITHUB_PRIVATE_KEY`) or personal access token — used for repository operations, PR management, and webhook validation -- LLM provider credentials — either `ANTHROPIC_API_KEY` (direct API) or Google service account credentials for Vertex AI -- Langfuse credentials (optional) — `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` - -### Credentials in the Container - -Implementation containers receive a **subset** of credentials as environment variables: - -- LLM credentials (API key or Vertex AI service account, depending on backend) -- Git user identity (`GIT_USER_NAME`, `GIT_USER_EMAIL`) -- Langfuse credentials (when enabled) - -Containers do **not** receive Jira credentials, GitHub tokens, or Redis credentials. All Jira/GitHub operations are performed by the orchestrator after the container exits. - -### Container Isolation - -- **Rootless Podman** — Containers run without root privileges on the host -- **Network** — Configurable via `FORGE_SANDBOX_NETWORK` (default: `slirp4netns`, providing NAT-based isolation) -- **Resource limits** — Memory: 4GB (configurable), CPU: 2 cores (configurable), Timeout: 30 minutes (configurable) -- **Filesystem** — Workspace mounted read-write at `/workspace` (necessary for code changes); task file mounted read-only at `/task.json` - -**Not currently enforced:** - -- `--security-opt no-new-privileges` is not set -- `--cap-drop ALL` is not applied (relies on Podman rootless defaults) -- No explicit seccomp profile -- Root filesystem is not read-only (`--read-only` not set) -- No AppArmor or SELinux enforcement beyond the `:Z` mount relabeling flag - -### Prompt Injection Boundaries - -Forge processes externally-supplied text from Jira tickets, GitHub PR descriptions, code review comments, and repository content. This text is passed to LLM prompts. The system relies on: - -- **Structured prompt templates** — System prompts are loaded from versioned templates in `src/forge/prompts/v1/`, not dynamically constructed from user input -- **Repository guardrails** — The workspace manager searches cloned repos for `CONSTITUTION.md`, `AGENTS.md`, and `CLAUDE.md` files and injects them as system context, giving repo owners control over agent behavior -- **Human gates** — All generated artifacts (PRDs, specs, plans, code) must pass human approval before merging, providing a boundary against manipulated output -- **Container isolation** — Code execution happens in ephemeral containers with limited credentials, bounding the blast radius of a compromised agent - -### MCP Tool Permissions - -Container agents use Deep Agents with MCP tool access. The agent can read and write files within `/workspace`, run shell commands, and use MCP-configured tools. It cannot access host filesystems outside the mount, Jira/GitHub APIs directly, or Redis. - -### Secret Redaction and Auditability - -- LLM interactions are traced via Langfuse (when enabled), providing an audit trail of all AI-generated content -- Container stdout/stderr is captured in worker logs -- Jira comments and GitHub PR descriptions provide a human-readable record of all workflow artifacts -- Credential values are passed as environment variables, not written to files or logs - ---- - -## Quality Attributes and Operational Characteristics - -### Availability - -- Gateway and Worker are stateless process types (state lives in Redis). Either can restart without data loss. -- Redis is a single point of failure. Redis downtime causes complete system outage. HA must be provided externally. -- Human approval gates have no timeout. Workflows can remain paused indefinitely without consuming resources beyond Redis checkpoint storage. - -### Throughput and Concurrency - -- Each Worker process handles up to 20 concurrent tasks (`QUEUE_MAX_CONCURRENT_TASKS`), bounded by an `asyncio.Semaphore`. -- The consumer reads up to 10 messages per `XREADGROUP` call with a 5-second block timeout. -- Multi-repo tasks can run up to 5 concurrent repository implementations via LangGraph `Send` fan-out. -- LLM rate limiting (Anthropic: 0.5 req/s burst 5) is the primary throughput bottleneck for planning-heavy workflows. - -### Per-Ticket Latency - -End-to-end latency is dominated by: - -- LLM response time (seconds to minutes per node, depending on prompt complexity) -- Human approval gates (minutes to days, unbounded) -- Container execution (30-minute default timeout per task) -- CI pipeline execution (external, typically 5-30 minutes) - -A fully autonomous (`forge:yolo`) simple task with fast CI can complete in 10-20 minutes. A multi-epic feature with human review at each gate can take days. - -### Durability - -- Workflow state is durable to Redis checkpoint writes. State survives Worker restarts. -- Event durability depends on Redis persistence configuration (operator responsibility). -- External side effects (Jira comments, GitHub PRs) are durable once written to those platforms. -- Container-local changes are lost if the container exits before the orchestrator pushes them. The orchestrator pushes immediately after successful container exit. - -### Auditability - -- Every workflow artifact (PRD, spec, plan, tasks) is posted as a Jira comment or GitHub PR, creating a human-readable audit trail -- LLM calls are traced in Langfuse with session ID = ticket key, including token counts and costs -- OpenTelemetry tracing is available (configurable via `OTLP_ENDPOINT`) -- Worker logs include correlation IDs for request tracing - -### Known Bottlenecks - -- **LLM rate limits** — The Anthropic rate limiter (0.5 req/s) serializes LLM calls across all concurrent tasks within a worker -- **Single Redis** — All state, queuing, and coordination goes through one Redis instance with no sharding -- **Container spawn overhead** — Podman container creation, repo cloning, and dependency installation add minutes per task - ---- - -## Key Architectural Decisions - -### Redis Streams for Event Bus - -**Decision:** Use Redis Streams with consumer groups instead of a dedicated message broker (RabbitMQ, Kafka). - -**Rationale:** Redis already serves as the checkpoint store and index. Using it for event queuing eliminates an additional infrastructure dependency. Consumer groups provide the needed delivery semantics (at-least-once, consumer-level distribution). The tradeoff is that Redis Streams lack built-in dead-letter queues, durable replay, and cross-datacenter replication — these are acceptable given Forge's current scale. - -### LangGraph for Workflow Orchestration - -**Decision:** Use LangGraph `StateGraph` with `AsyncRedisSaver` checkpointing instead of a traditional workflow engine (Temporal, Airflow). - -**Rationale:** LangGraph provides native support for LLM-driven decision nodes, conditional routing, and checkpointed pause/resume. The Python-native graph definition allows workflow logic to live alongside the orchestration code. The tradeoff is a less mature ecosystem and fewer operational tools compared to established workflow engines. - -### Host-Level Podman for Code Execution - -**Decision:** Run implementation tasks in Podman containers on the Worker host instead of using Kubernetes jobs, remote VMs, or in-process execution. - -**Rationale:** Rootless Podman provides process isolation, filesystem isolation, and resource limits without requiring a Kubernetes cluster. Workers must run on Podman-capable hosts, which constrains deployment options but simplifies the container lifecycle (local spawn, local cleanup). The tradeoff is that horizontal scaling requires Podman on every Worker host. - -### Workflow Separation by Issue Type - -**Decision:** Maintain three separate LangGraph workflow definitions (Feature, Bug, Task Takeover) rather than a single parameterized workflow. - -**Rationale:** Each workflow type has fundamentally different planning stages (PRD/spec/epic decomposition for Features, triage/RCA for Bugs, direct planning for Tasks). Separating them keeps each graph readable and independently modifiable. The shared implementation/CI/review tail is implemented as common nodes used by all three workflows. The tradeoff is some duplication in graph wiring. - -### Human Approval Gates - -**Decision:** Pause workflows at defined gates and wait indefinitely for human approval, rather than proceeding autonomously by default. - -**Rationale:** Forge generates and executes code in real repositories. Human review at planning stages (PRD, spec, plan, tasks) and at code review catches errors before they reach production. The `forge:yolo` label provides an opt-in escape hatch for tickets where autonomous operation is acceptable. The tradeoff is increased latency and human involvement for every ticket. - ---- - -## Known Limitations - -- **No PEL reclaim** — If a worker crashes mid-processing, unacknowledged messages remain in the Redis PEL indefinitely. No automated `XCLAIM`/`XAUTOCLAIM` mechanism exists. Recovery requires manual intervention. -- **No distributed per-ticket lock** — Per-ticket event serialization uses an in-process `asyncio.Lock`. Multiple Worker processes can process events for the same ticket concurrently, potentially causing checkpoint conflicts or duplicate side effects. -- **Webhook deduplication not wired** — A `DeduplicationService` exists but is not connected to the webhook routes. Duplicate webhooks are processed as separate events. -- **Webhook signature validation is optional** — If `JIRA_WEBHOOK_SECRET` or `GITHUB_WEBHOOK_SECRET` is not configured, the corresponding endpoint accepts unsigned payloads. -- **No approval gate timeout** — Workflows paused at human gates wait indefinitely. There is no automatic escalation, expiration, or notification for stale gates. -- **Single Redis dependency** — No Sentinel, Cluster, or HA configuration. Redis is a single point of failure. -- **Container security hardening gaps** — No `--cap-drop ALL`, `--no-new-privileges`, `--read-only` root filesystem, or explicit seccomp profile. Relies on Podman rootless defaults. -- **No cross-stream ordering** — Events from the Jira and GitHub streams are consumed independently. No ordering guarantee exists across streams for the same ticket. - ---- - -## Workflow Lifecycles - -The diagrams below show the high-level lifecycle for each workflow type. `>>` marks human approval gates where the workflow pauses until a reviewer approves, requests revisions (`!` comment), or asks questions (`?` comment). Gates are auto-approved when the `forge:yolo` label is set. - -These diagrams show the primary flow at a stable abstraction level. Error handling, revision loops, and question-answering branches are omitted. For detailed node-level flows with all branches, see the individual workflow guides: - -- [Feature Workflow Guide](workflows/feature.md) -- [Bug Workflow Guide](workflows/bug.md) -- [Task Workflow Guide](workflows/task.md) - -### Feature Lifecycle - -A Feature or Story goes through a full planning pipeline before implementation. Each planning artifact is reviewed and approved by a human before proceeding. - -```mermaid -flowchart TD - A["Ticket created\n(Feature/Story)"] --> B["Generate PRD"] - B --> C[">> PRD approval"] - C --> D["Generate technical spec"] - D --> E[">> Spec approval"] - E --> F["Decompose into epics"] - F --> G[">> Plan approval"] - G --> H["Generate tasks"] - H --> I[">> Task approval"] - I --> J["Route tasks by repo"] - - J --> K["Implement in container"] - K --> L["Review and open PR"] - L --> M["CI repair loop"] - M --> N[">> Human code review"] - N -->|merged| O["Aggregate status\nComplete"] -``` - -### Bug Lifecycle - -A Bug goes through triage and root cause analysis before implementation. The reporter may be asked to fill in missing context, and the user selects a fix approach from multiple options. - -```mermaid -flowchart TD - A["Ticket created\n(Bug)"] --> B["Triage check"] - B -->|"missing info"| C[">> Ask reporter"] - C --> B - B -->|"sufficient"| D["Root cause analysis"] - D --> E[">> Present fix options\n(user selects >option N)"] - E --> F["Generate fix plan"] - F --> G[">> Plan approval"] - - G --> H["Implement in container"] - H --> I["Review and open PR"] - I --> J["CI repair loop"] - J --> K[">> Human code review"] - K -->|merged| L["Post-merge summary\nComplete"] -``` - -### Task Lifecycle - -A Task or Epic goes directly from triage to implementation planning, skipping PRD, spec, and epic decomposition. - -```mermaid -flowchart TD - A["Ticket created\n(Task/Epic)"] --> B["Triage check"] - B -->|"missing context"| C[">> Ask for details"] - C --> B - B -->|"sufficient"| D["Generate plan"] - D --> E[">> Plan approval"] - - E --> F["Implement in container"] - F --> G["Review and open PR"] - G --> H["CI repair loop"] - H --> I[">> Human code review"] - I -->|merged| J["Complete"] -``` - -### Diagram Maintenance - -The lifecycle diagrams above use stable, high-level step names rather than concrete graph node names. The source of truth for node-level workflow details is the graph definition code in `src/forge/orchestrator/`. When graph node names or edges change, the detailed workflow guides should be updated; these high-level diagrams should only change when the overall lifecycle structure changes. - ---- - -## Data Flow Summary - -- **Inbound events:** Jira/GitHub webhooks --> FastAPI gateway --> Redis Streams -- **State persistence:** Redis (`AsyncRedisSaver`, keyed by Jira ticket key) -- **LLM calls:** Orchestrator nodes and container agents --> Claude/Gemini (Anthropic direct API or Vertex AI) -- **Code execution:** Workflow node --> Podman container --> Deep Agents with MCP tools -- **Outbound actions:** Jira (comments, labels, transitions), GitHub (branches, commits, PRs, reviews) -- **Observability:** Langfuse (LLM traces, workflow spans, costs), OpenTelemetry (configurable) diff --git a/docs/architecture/index.md b/docs/architecture/index.md new file mode 100644 index 00000000..ca0a8e15 --- /dev/null +++ b/docs/architecture/index.md @@ -0,0 +1,13 @@ +# Forge Architecture + +This is the primary architecture reference for Forge, an AI-powered SDLC orchestrator. It covers system structure, runtime topology, state management, failure modes, security boundaries, and key design decisions. + +**Audience:** Maintainers, operators, and contributors who need to understand how Forge works, why it is built this way, and what constraints govern its operation. + +The architecture documentation is organized into three parts: + +| Part | Contents | +|------|----------| +| [System & Components](overview.md) | Scope, system context, external actors, component responsibilities | +| [Internals](internals.md) | Runtime topology, state and concurrency model, failure recovery, security boundaries | +| [Reference](reference.md) | Quality attributes, architectural decisions, known limitations, workflow lifecycles | diff --git a/docs/architecture/internals.md b/docs/architecture/internals.md new file mode 100644 index 00000000..731267b4 --- /dev/null +++ b/docs/architecture/internals.md @@ -0,0 +1,197 @@ +# Internals + +## Runtime and Deployment Topology + +### Process Topology + +Forge runs as two process types plus Redis: + +- **Gateway process**: A single FastAPI/Uvicorn process. Stateless; can be load-balanced if needed. No special host requirements beyond network access to Redis. +- **Worker process(es)**: One or more `OrchestratorWorker` processes. Each creates its own Redis consumer group member. Workers **must run on a host with Podman installed** and permission to spawn rootless containers. Each worker can run up to 20 concurrent tasks (configurable via `QUEUE_MAX_CONCURRENT_TASKS`), bounded by an `asyncio.Semaphore`. +- **Redis**: Single-instance Redis server. No Sentinel or Cluster configuration exists; HA must be provided externally if required. + +### Scaling Constraints + +Gateway and Worker communicate only through Redis and can be deployed on separate hosts. However, horizontal Worker scaling has a **known limitation**: per-ticket event serialization uses an in-process `asyncio.Lock`, not a distributed lock. If two Worker processes receive events for the same ticket concurrently, they will process them in parallel with no coordination. See Known Limitations in the [Reference](reference.md#known-limitations) section for details. + +### Container Lifecycle + +- Containers are named `forge-{ticket_key}-{short_uuid}` and are removed after execution by default +- Setting `FORGE_CONTAINER_KEEP=true` preserves failed containers for debugging +- Workspace directories are cleaned up via `podman unshare rm -rf` (to handle root-owned files), with `shutil.rmtree` as a fallback +- Stale workspaces older than 24 hours are eligible for cleanup via `cleanup_stale_workspaces()` + +### Required Connectivity + +- Workers --> Redis (event consumption, checkpoint read/write, retry queue) +- Gateway --> Redis (event publishing) +- Workers --> Jira API (comments, labels, transitions) +- Workers --> GitHub API (branches, commits, PRs, reviews) +- Workers --> LLM provider (Anthropic API or Vertex AI) +- Workers --> Langfuse (optional, trace ingestion) +- Containers --> LLM provider (code generation) +- Containers --> Langfuse (optional, in-container tracing) +- Containers --> internet (for dependency installation; network mode configurable via `FORGE_SANDBOX_NETWORK`, default `slirp4netns`) + +### What Is Lost if Redis Is Lost + +Redis holds both ephemeral queues and durable state. If Redis data is lost: + +- All in-flight events in the streams are lost (unprocessed webhooks) +- All workflow checkpoint state is lost (paused workflows cannot resume) +- The retry queue and dead-letter queue contents are lost +- PR-to-ticket index mappings are lost (webhook routing for PR events will fail until rebuilt) +- Deduplication keys are lost (duplicate event processing may occur temporarily) + +Operators should configure Redis persistence (RDB snapshots, AOF, or both) appropriate to their durability requirements. Forge does not configure Redis persistence itself. + +## State, Events, and Concurrency Model + +### Delivery Guarantee: At-Least-Once + +Forge uses **at-least-once** event processing. Redis consumer groups assign each stream entry to one consumer initially, but messages are acknowledged (`XACK`) only **after** successful processing. If a worker crashes between completing processing and sending the acknowledgement, the message remains in the Pending Entries List (PEL) and can be redelivered. + +The system does **not** provide exactly-once semantics. There is no transaction or Lua script wrapping handler execution and acknowledgement atomically. + +### Idempotency + +A `DeduplicationService` using Redis `SETNX` with 24-hour TTL keys (`forge:dedup:{event_id}`) is implemented but **not yet wired into the webhook routes**. Currently, duplicate webhooks from Jira or GitHub will be processed as separate events. + +Within workflows, certain operations have natural idempotency properties: + +- Branch creation: `git push` to an existing branch with the same content is a no-op +- PR creation: the workflow checks for existing PRs before creating new ones +- Label operations: setting a label that already exists is idempotent in both Jira and GitHub + +However, Jira comment posting and some GitHub operations are **not** idempotent. A duplicated event could result in duplicate comments. + +### Pending Entry Reclaim + +There is **no automated PEL reclaim mechanism**. The codebase does not use `XPENDING`, `XCLAIM`, or `XAUTOCLAIM`. If a worker crashes mid-processing before enqueuing the message for retry, the message remains in the PEL indefinitely. Recovery requires manual intervention or restarting the consumer group. This is a known limitation (see [Reference](reference.md#known-limitations)). + +### Per-Ticket Ordering + +Within a single worker process, events for the same ticket are serialized by an `asyncio.Lock` per ticket key. This ensures that concurrent events for ticket `AISOS-123` within one process are handled sequentially. + +Across multiple worker processes, **no distributed coordination exists**. Redis consumer groups distribute messages round-robin across consumers. If events for the same ticket are assigned to different workers, they will execute concurrently. This can lead to checkpoint conflicts or race conditions on Jira/GitHub side effects. + +**Current safe configuration:** Run a single Worker process per deployment, or accept that concurrent same-ticket processing may occur across workers. The in-process lock handles the common case (multiple events arriving in quick succession to the same consumer). + +### Checkpoint Model + +LangGraph workflow state is persisted via `AsyncRedisSaver` in Redis. The checkpoint lifecycle: + +- **Thread ID** = Jira ticket key (e.g., `AISOS-123`). One ticket maps to exactly one workflow instance and one checkpoint thread. +- **Writes** happen automatically after each LangGraph node completes. +- **Reads** happen when a new event arrives for an existing ticket: the workflow resumes from the last checkpoint. +- **Deletion**: `clear_checkpoint(thread_id)` removes all checkpoint data for a ticket. +- **Redis key pattern**: `checkpoint:{thread_id}:{...}` + +### Consistency Between Checkpoints and Side Effects + +Checkpoint writes and external side effects (Jira comments, GitHub PRs) are **not transactional**. A node may successfully post a comment to Jira but crash before its checkpoint is written, causing the comment to be posted again on retry. Similarly, a checkpoint may be written after the node function returns but before all side effects complete. + +The ordering between the Jira and GitHub event streams is **independent**. Events are consumed from both streams concurrently, and there is no ordering guarantee across streams. + +## Failure and Recovery Model + +### Component Failure Modes + +**Gateway failure**: Incoming webhooks are dropped. Jira and GitHub will retry webhook delivery according to their own retry policies (Jira retries up to 10 times with backoff; GitHub retries based on recent failure rate). No data is lost in Redis. + +**Worker failure (crash)**: In-flight messages remain in the Redis PEL. The workflow checkpoint reflects the last completed node. On restart, the worker begins consuming new messages. PEL messages require manual intervention (see Known Limitations in the [Reference](reference.md#known-limitations) section). Retry-eligible messages that were already enqueued in the retry sorted set will be picked up by any running worker. + +**Redis failure**: Complete system outage. Gateway cannot enqueue events; workers cannot consume, checkpoint, or retry. All in-memory state (streams, checkpoints, retry queue, DLQ, indexes) is at risk unless Redis persistence is configured. See [What Is Lost if Redis Is Lost](#what-is-lost-if-redis-is-lost). + +**LLM provider failure**: Orchestrator nodes and container agents fail. The retry mechanism handles transient LLM errors. Persistent failures exhaust retries and move the event to the dead-letter queue. The `forge:blocked` label is applied to the Jira ticket for visibility. + +**Jira/GitHub API failure**: Webhook-triggered nodes fail. Rate limiting (token bucket) prevents overloading. API errors during workflow execution are handled by the retry mechanism. + +**Container failure**: Non-zero exit codes are captured by the orchestrator. If `FORGE_CONTAINER_KEEP=true`, failed containers are preserved for debugging. The workflow node reports the failure, and the retry mechanism determines whether to re-attempt. + +### Retry Budget and Backoff + +- **Max retry attempts:** 3 +- **Backoff:** Exponential: 30s initial delay, 2x multiplier, capped at 3600s (1 hour) +- **Retry queue:** Redis sorted set (`forge:retry:queue`) with score = next retry timestamp, polled every 10 seconds +- **Dead-letter queue:** Redis list (`forge:retry:dlq`). Messages that exhaust all retries are moved here. Manual requeue via `requeue_dead_letter()` resets the attempt counter. + +### Crash Recovery + +- **Automatic:** Workers restart and consume new messages. Retry-queue entries are picked up by any running worker. LangGraph workflows resume from the last checkpoint when a new event arrives. +- **Manual:** PEL entries from crashed workers require manual `XCLAIM` or consumer group reset. DLQ entries require manual requeue or investigation. + +### Blocked Workflows and `forge:retry` + +When a workflow enters an error state, the `forge:blocked` label is applied to the Jira ticket. Adding the `forge:retry` label triggers re-entry at the failed step using the last checkpoint. This works at any workflow stage, including approval gates (where it regenerates the artifact). + +### Partial Multi-Repository Execution + +For tickets that span multiple repositories, the task router groups tasks by repo and processes them either sequentially or in parallel (up to 5 concurrent repos via LangGraph's `Send` API). If execution fails for one repo, that repo's error is recorded but other repos continue. The `aggregate_parallel_results()` node merges results and errors from all branches. + +### Approval Gate Timeout + +**There is no timeout on human approval gates.** When a workflow reaches a pause gate (`>> gate`), it persists `is_paused=True` in the checkpoint and waits indefinitely for a resume event (Jira comment or label change). If approval never arrives, the workflow remains paused in Redis checkpoint state. There is no automatic escalation or expiration. + +## Security and Trust Boundaries + +### Webhook Authentication + +Both Jira and GitHub webhook endpoints validate HMAC-SHA256 signatures using `hmac.compare_digest()` (constant-time comparison). However, signature validation is **conditional**: it only runs when the corresponding secret is configured (`JIRA_WEBHOOK_SECRET`, `GITHUB_WEBHOOK_SECRET`). If secrets are not set, the endpoint accepts unsigned payloads without warning. + +**Recommendation for production:** Always configure webhook secrets. Without them, any network actor that can reach the gateway can inject events. + +### Credentials in the Worker + +Worker processes require: + +- Redis connection credentials (`REDIS_URL`) +- Jira API token (`JIRA_API_TOKEN`): used for reading/writing tickets, comments, labels, transitions +- GitHub App credentials (`GITHUB_APP_ID`, `GITHUB_PRIVATE_KEY`) or personal access token: used for repository operations, PR management, and webhook validation +- LLM provider credentials: either `ANTHROPIC_API_KEY` (direct API) or Google service account credentials for Vertex AI +- Langfuse credentials (optional): `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` + +### Credentials in the Container + +Implementation containers receive a **subset** of credentials as environment variables: + +- LLM credentials (API key or Vertex AI service account, depending on backend) +- Git user identity (`GIT_USER_NAME`, `GIT_USER_EMAIL`) +- Langfuse credentials (when enabled) + +Containers do **not** receive Jira credentials, GitHub tokens, or Redis credentials. All Jira/GitHub operations are performed by the orchestrator after the container exits. + +### Container Isolation + +- **Rootless Podman**: Containers run without root privileges on the host +- **Network**: Configurable via `FORGE_SANDBOX_NETWORK` (default: `slirp4netns`, providing NAT-based isolation) +- **Resource limits**: Memory: 4GB (configurable), CPU: 2 cores (configurable), Timeout: 30 minutes (configurable) +- **Filesystem**: Workspace mounted read-write at `/workspace` (necessary for code changes); task file mounted read-only at `/task.json` + +**Not currently enforced:** + +- `--security-opt no-new-privileges` is not set +- `--cap-drop ALL` is not applied (relies on Podman rootless defaults) +- No explicit seccomp profile +- Root filesystem is not read-only (`--read-only` not set) +- No AppArmor or SELinux enforcement beyond the `:Z` mount relabeling flag + +### Prompt Injection Boundaries + +Forge processes externally-supplied text from Jira tickets, GitHub PR descriptions, code review comments, and repository content. This text is passed to LLM prompts. The system relies on: + +- **Structured prompt templates**: System prompts are loaded from versioned templates in `src/forge/prompts/v1/`, not dynamically constructed from user input +- **Repository guardrails**: The workspace manager searches cloned repos for `CONSTITUTION.md`, `AGENTS.md`, and `CLAUDE.md` files and injects them as system context, giving repo owners control over agent behavior +- **Human gates**: All generated artifacts (PRDs, specs, plans, code) must pass human approval before merging, providing a boundary against manipulated output +- **Container isolation**: Code execution happens in ephemeral containers with limited credentials, bounding the blast radius of a compromised agent + +### MCP Tool Permissions + +Container agents use Deep Agents with MCP tool access. The agent can read and write files within `/workspace`, run shell commands, and use MCP-configured tools. It cannot access host filesystems outside the mount, Jira/GitHub APIs directly, or Redis. + +### Secret Redaction and Auditability + +- LLM interactions are traced via Langfuse (when enabled), providing an audit trail of all AI-generated content +- Container stdout/stderr is captured in worker logs +- Jira comments and GitHub PR descriptions provide a human-readable record of all workflow artifacts +- Credential values are passed as environment variables, not written to files or logs diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md new file mode 100644 index 00000000..6b00b906 --- /dev/null +++ b/docs/architecture/overview.md @@ -0,0 +1,111 @@ +# System & Components + +## Scope and Audience + +This document is the primary architecture reference for Forge, an AI-powered SDLC orchestrator. It covers system structure, runtime topology, state management, failure modes, security boundaries, and key design decisions. + +**Audience:** Maintainers, operators, and contributors who need to understand how Forge works, why it is built this way, and what constraints govern its operation. + +**Goals:** + +- Explain the runtime topology and deployment constraints +- Document state ownership, consistency guarantees, and failure recovery +- Define security and trust boundaries, especially around AI-generated code execution +- Record key architectural decisions with rationale and trade-offs +- Provide stable, high-level lifecycle views that link to detailed workflow guides + +**Non-goals:** + +- Node-level workflow documentation (see the [Feature](../guide/feature-workflow.md), [Bug](../guide/bug-workflow.md), and [Task](../guide/task-workflow.md) workflow guides) +- Operational runbooks or deployment procedures +- API reference (see the OpenAPI spec at `/docs` when the gateway is running) + +## System Context and External Actors + +Forge sits between project management (Jira), source control (GitHub), and LLM providers, orchestrating work from ticket creation through merged PR. + +```mermaid +flowchart LR + A["Jira / GitHub\n(webhooks)"] --> B["Gateway\n(FastAPI)"] + B --> C["Redis\n(Streams + State)"] + C --> D["Workers\n(LangGraph)"] + D --> E["Podman\nContainers"] + D <--> F["LLM\n(Claude / Gemini)"] + E <--> F + D --> A +``` + +**External actors:** + +- **Jira**: Source of ticket lifecycle events. Forge reads ticket metadata and writes comments, labels, and transitions back. Webhooks deliver issue and comment events. +- **GitHub**: Source of PR, CI, and code review events. Forge creates branches, pushes commits, opens PRs, and responds to review feedback. Webhooks deliver PR, check suite, and review events. +- **LLM providers**: Anthropic (direct API) and Google Vertex AI (Claude and Gemini models). Called by both orchestrator nodes (planning, review) and container agents (code generation). +- **Langfuse**: Observability platform for LLM call tracing, workflow spans, and cost tracking. Optional; disabled when credentials are not configured. +- **Human reviewers**: Approve or revise artifacts at defined gates. The workflow pauses indefinitely at each gate until a human responds. + +## Component Responsibilities + +```mermaid +flowchart TD + subgraph External["External Systems"] + Jira + GitHub + Langfuse["Langfuse (Observability)"] + end + + subgraph Gateway["FastAPI Gateway (:8000)"] + JiraWH["POST /api/v1/webhooks/jira"] + GitHubWH["POST /api/v1/webhooks/github"] + end + + subgraph Queue["Redis"] + Streams["Streams: forge:events:jira\nforge:events:github"] + State["AsyncRedisSaver\nLangGraph checkpointing"] + end + + subgraph Workers["Worker Processes (consumer group: forge-workers)"] + Router{"WorkflowRouter\nroute by issue type"} + Feature["FeatureWorkflow\n(Feature/Story)"] + Bug["BugWorkflow\n(Bug)"] + Task["TaskTakeoverWorkflow\n(Task/Epic)"] + end + + subgraph Container["Podman Container (ephemeral)"] + Agent["Deep Agents + MCP\n/workspace (repo mounted)"] + end + + LLM["LLM Backends\nAnthropic API (Claude)\nVertex AI (Claude/Gemini)"] + + Jira -- webhooks --> JiraWH + GitHub -- webhooks --> GitHubWH + JiraWH --> Streams + GitHubWH --> Streams + Streams --> Router + Router --> Feature + Router --> Bug + Router --> Task + Feature --> Container + Bug --> Container + Task --> Container + Workers <--> LLM + Container <--> LLM + Workers --> Jira + Workers --> GitHub + Workers --> Langfuse +``` + +**Gateway (FastAPI)**: Accepts Jira and GitHub webhooks over HTTPS, validates HMAC-SHA256 signatures (when secrets are configured), and publishes events to Redis Streams. It performs no workflow logic. It also serves health, readiness, and Prometheus metrics endpoints. Middleware includes CORS and correlation ID propagation. A `DeduplicationService` exists in the codebase but is not yet wired into the webhook routes (see Known Limitations in the [Reference](reference.md#known-limitations) section). + +**Worker**: Consumes events from Redis Streams via the `forge-workers` consumer group. Each worker generates a unique consumer ID (`worker-{uuid}`). The `WorkflowRouter` resolves the target LangGraph workflow (Feature, Bug, or Task Takeover) based on Jira issue type and drives execution through planning, implementation, CI repair, and human review stages. Implementation runs inside ephemeral Podman containers using Deep Agents. + +**Redis**: Serves multiple roles: + +- *Event bus*: Redis Streams (`forge:events:jira`, `forge:events:github`) with the `forge-workers` consumer group for message distribution +- *Workflow state store*: LangGraph `AsyncRedisSaver` checkpoints workflow state per ticket (keyed by ticket key, e.g. `AISOS-123`) +- *Retry queue*: Sorted set (`forge:retry:queue`) with exponential backoff +- *Dead-letter queue*: List (`forge:retry:dlq`) for messages that exhaust retries +- *Supporting indexes*: PR-to-ticket mapping hash (`forge:state:pr_index`), webhook deduplication keys (`forge:dedup:*`, 24h TTL) + +**Podman Container**: Ephemeral rootless containers that execute implementation tasks. Each container receives the target repository mounted at `/workspace` (read-write), a task file at `/task.json` (read-only), and LLM credentials as environment variables. The container runs Deep Agents with MCP tool access to make changes and commit them locally. The orchestrator handles pushing and PR creation after the container exits. + +**LLM Backends**: Claude and Gemini models are called bidirectionally. Orchestrator nodes call them for planning and review; container agents call them for code generation. Two backends are supported: Anthropic's direct API (via `ANTHROPIC_API_KEY`) and Google Vertex AI (via service account credentials). The backend is selected automatically based on which credentials are configured. Request-level rate limiting uses token bucket algorithms (Anthropic: 0.5 req/s burst 5; Jira: 1.5 req/s burst 10; GitHub: 1.0 req/s burst 20). diff --git a/docs/architecture/reference.md b/docs/architecture/reference.md new file mode 100644 index 00000000..48c9eb8b --- /dev/null +++ b/docs/architecture/reference.md @@ -0,0 +1,176 @@ +# Reference + +## Quality Attributes and Operational Characteristics + +### Availability + +- Gateway and Worker are stateless process types (state lives in Redis). Either can restart without data loss. +- Redis is a single point of failure. Redis downtime causes complete system outage. HA must be provided externally. +- Human approval gates have no timeout. Workflows can remain paused indefinitely without consuming resources beyond Redis checkpoint storage. + +### Throughput and Concurrency + +- Each Worker process handles up to 20 concurrent tasks (`QUEUE_MAX_CONCURRENT_TASKS`), bounded by an `asyncio.Semaphore`. +- The consumer reads up to 10 messages per `XREADGROUP` call with a 5-second block timeout. +- Multi-repo tasks can run up to 5 concurrent repository implementations via LangGraph `Send` fan-out. +- LLM rate limiting (Anthropic: 0.5 req/s burst 5) is the primary throughput bottleneck for planning-heavy workflows. + +### Per-Ticket Latency + +End-to-end latency is dominated by: + +- LLM response time (seconds to minutes per node, depending on prompt complexity) +- Human approval gates (minutes to days, unbounded) +- Container execution (30-minute default timeout per task) +- CI pipeline execution (external, typically 5-30 minutes) + +A fully autonomous (`forge:yolo`) simple task with fast CI can complete in 10-20 minutes. A multi-epic feature with human review at each gate can take days. + +### Durability + +- Workflow state is durable to Redis checkpoint writes. State survives Worker restarts. +- Event durability depends on Redis persistence configuration (operator responsibility). +- External side effects (Jira comments, GitHub PRs) are durable once written to those platforms. +- Container-local changes are lost if the container exits before the orchestrator pushes them. The orchestrator pushes immediately after successful container exit. + +### Auditability + +- Every workflow artifact (PRD, spec, plan, tasks) is posted as a Jira comment or GitHub PR, creating a human-readable audit trail +- LLM calls are traced in Langfuse with session ID = ticket key, including token counts and costs +- OpenTelemetry tracing is available (configurable via `OTLP_ENDPOINT`) +- Worker logs include correlation IDs for request tracing + +### Known Bottlenecks + +- **LLM rate limits**: The Anthropic rate limiter (0.5 req/s) serializes LLM calls across all concurrent tasks within a worker +- **Single Redis**: All state, queuing, and coordination goes through one Redis instance with no sharding +- **Container spawn overhead**: Podman container creation, repo cloning, and dependency installation add minutes per task + +## Key Architectural Decisions + +### Redis Streams for Event Bus + +**Decision:** Use Redis Streams with consumer groups instead of a dedicated message broker (RabbitMQ, Kafka). + +**Rationale:** Redis already serves as the checkpoint store and index. Using it for event queuing eliminates an additional infrastructure dependency. Consumer groups provide the needed delivery semantics (at-least-once, consumer-level distribution). The tradeoff is that Redis Streams lack built-in dead-letter queues, durable replay, and cross-datacenter replication; these are acceptable given Forge's current scale. + +### LangGraph for Workflow Orchestration + +**Decision:** Use LangGraph `StateGraph` with `AsyncRedisSaver` checkpointing instead of a traditional workflow engine (Temporal, Airflow). + +**Rationale:** LangGraph provides native support for LLM-driven decision nodes, conditional routing, and checkpointed pause/resume. The Python-native graph definition allows workflow logic to live alongside the orchestration code. The tradeoff is a less mature ecosystem and fewer operational tools compared to established workflow engines. + +### Host-Level Podman for Code Execution + +**Decision:** Run implementation tasks in Podman containers on the Worker host instead of using Kubernetes jobs, remote VMs, or in-process execution. + +**Rationale:** Rootless Podman provides process isolation, filesystem isolation, and resource limits without requiring a Kubernetes cluster. Workers must run on Podman-capable hosts, which constrains deployment options but simplifies the container lifecycle (local spawn, local cleanup). The tradeoff is that horizontal scaling requires Podman on every Worker host. + +### Workflow Separation by Issue Type + +**Decision:** Maintain three separate LangGraph workflow definitions (Feature, Bug, Task Takeover) rather than a single parameterized workflow. + +**Rationale:** Each workflow type has fundamentally different planning stages (PRD/spec/epic decomposition for Features, triage/RCA for Bugs, direct planning for Tasks). Separating them keeps each graph readable and independently modifiable. The shared implementation/CI/review tail is implemented as common nodes used by all three workflows. The tradeoff is some duplication in graph wiring. + +### Human Approval Gates + +**Decision:** Pause workflows at defined gates and wait indefinitely for human approval, rather than proceeding autonomously by default. + +**Rationale:** Forge generates and executes code in real repositories. Human review at planning stages (PRD, spec, plan, tasks) and at code review catches errors before they reach production. The `forge:yolo` label provides an opt-in escape hatch for tickets where autonomous operation is acceptable. The tradeoff is increased latency and human involvement for every ticket. + +## Known Limitations + +- **No PEL reclaim**: If a worker crashes mid-processing, unacknowledged messages remain in the Redis PEL indefinitely. No automated `XCLAIM`/`XAUTOCLAIM` mechanism exists. Recovery requires manual intervention. +- **No distributed per-ticket lock**: Per-ticket event serialization uses an in-process `asyncio.Lock`. Multiple Worker processes can process events for the same ticket concurrently, potentially causing checkpoint conflicts or duplicate side effects. +- **Webhook deduplication not wired**: A `DeduplicationService` exists but is not connected to the webhook routes. Duplicate webhooks are processed as separate events. +- **Webhook signature validation is optional**: If `JIRA_WEBHOOK_SECRET` or `GITHUB_WEBHOOK_SECRET` is not configured, the corresponding endpoint accepts unsigned payloads. +- **No approval gate timeout**: Workflows paused at human gates wait indefinitely. There is no automatic escalation, expiration, or notification for stale gates. +- **Single Redis dependency**: No Sentinel, Cluster, or HA configuration. Redis is a single point of failure. +- **Container security hardening gaps**: No `--cap-drop ALL`, `--no-new-privileges`, `--read-only` root filesystem, or explicit seccomp profile. Relies on Podman rootless defaults. +- **No cross-stream ordering**: Events from the Jira and GitHub streams are consumed independently. No ordering guarantee exists across streams for the same ticket. + +## Workflow Lifecycles + +The diagrams below show the high-level lifecycle for each workflow type. `>>` marks human approval gates where the workflow pauses until a reviewer approves, requests revisions (`!` comment), or asks questions (`?` comment). Gates are auto-approved when the `forge:yolo` label is set. + +These diagrams show the primary flow at a stable abstraction level. Error handling, revision loops, and question-answering branches are omitted. For detailed node-level flows with all branches, see the individual workflow guides: + +- [Feature Workflow Guide](../guide/feature-workflow.md) +- [Bug Workflow Guide](../guide/bug-workflow.md) +- [Task Workflow Guide](../guide/task-workflow.md) + +### Feature Lifecycle + +A Feature or Story goes through a full planning pipeline before implementation. Each planning artifact is reviewed and approved by a human before proceeding. + +```mermaid +flowchart TD + A["Ticket created\n(Feature/Story)"] --> B["Generate PRD"] + B --> C[">> PRD approval"] + C --> D["Generate technical spec"] + D --> E[">> Spec approval"] + E --> F["Decompose into epics"] + F --> G[">> Plan approval"] + G --> H["Generate tasks"] + H --> I[">> Task approval"] + I --> J["Route tasks by repo"] + + J --> K["Implement in container"] + K --> L["Review and open PR"] + L --> M["CI repair loop"] + M --> N[">> Human code review"] + N -->|merged| O["Aggregate status\nComplete"] +``` + +### Bug Lifecycle + +A Bug goes through triage and root cause analysis before implementation. The reporter may be asked to fill in missing context, and the user selects a fix approach from multiple options. + +```mermaid +flowchart TD + A["Ticket created\n(Bug)"] --> B["Triage check"] + B -->|"missing info"| C[">> Ask reporter"] + C --> B + B -->|"sufficient"| D["Root cause analysis"] + D --> E[">> Present fix options\n(user selects >option N)"] + E --> F["Generate fix plan"] + F --> G[">> Plan approval"] + + G --> H["Implement in container"] + H --> I["Review and open PR"] + I --> J["CI repair loop"] + J --> K[">> Human code review"] + K -->|merged| L["Post-merge summary\nComplete"] +``` + +### Task Lifecycle + +A Task or Epic goes directly from triage to implementation planning, skipping PRD, spec, and epic decomposition. + +```mermaid +flowchart TD + A["Ticket created\n(Task/Epic)"] --> B["Triage check"] + B -->|"missing context"| C[">> Ask for details"] + C --> B + B -->|"sufficient"| D["Generate plan"] + D --> E[">> Plan approval"] + + E --> F["Implement in container"] + F --> G["Review and open PR"] + G --> H["CI repair loop"] + H --> I[">> Human code review"] + I -->|merged| J["Complete"] +``` + +### Diagram Maintenance + +The lifecycle diagrams above use stable, high-level step names rather than concrete graph node names. The source of truth for node-level workflow details is the graph definition code in `src/forge/orchestrator/`. When graph node names or edges change, the detailed workflow guides should be updated; these high-level diagrams should only change when the overall lifecycle structure changes. + +## Data Flow Summary + +- **Inbound events:** Jira/GitHub webhooks --> FastAPI gateway --> Redis Streams +- **State persistence:** Redis (`AsyncRedisSaver`, keyed by Jira ticket key) +- **LLM calls:** Orchestrator nodes and container agents --> Claude/Gemini (Anthropic direct API or Vertex AI) +- **Code execution:** Workflow node --> Podman container --> Deep Agents with MCP tools +- **Outbound actions:** Jira (comments, labels, transitions), GitHub (branches, commits, PRs, reviews) +- **Observability:** Langfuse (LLM traces, workflow spans, costs), OpenTelemetry (configurable) diff --git a/zensical.toml b/zensical.toml index 40db45b0..ea9cbdb1 100644 --- a/zensical.toml +++ b/zensical.toml @@ -18,7 +18,12 @@ nav = [ ]}, {"Developer Guide" = [ {"Overview" = "developer-guide.md"}, - {"Architecture" = "architecture.md"}, + {"Architecture" = [ + {"Overview" = "architecture/index.md"}, + {"System & Components" = "architecture/overview.md"}, + {"Internals" = "architecture/internals.md"}, + {"Reference" = "architecture/reference.md"}, + ]}, {"Local Setup" = "dev/setup.md"}, {"Testing" = "dev/testing.md"}, {"Contributing" = "dev/contributing.md"}, From d6e0897fc31941d6473da67d3e563d6ceb5d7a85 Mon Sep 17 00:00:00 2001 From: Eran Kuris Date: Wed, 22 Jul 2026 11:44:10 +0300 Subject: [PATCH 20/20] Trim architecture docs for readability Cut ~50% of content by removing operational details (Redis key patterns, retry backoff specifics, container cleanup procedures, PEL reclaim details, connectivity lists, quality attributes) that belong in ops runbooks. Kept all mermaid diagrams, architectural decisions, known limitations, and workflow lifecycles. Compressed component descriptions and failure modes into scannable tables. 247 lines total, down from ~500. Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/architecture/index.md | 12 +- docs/architecture/internals.md | 209 ++++++--------------------------- docs/architecture/overview.md | 48 ++------ docs/architecture/reference.md | 109 +++-------------- 4 files changed, 64 insertions(+), 314 deletions(-) diff --git a/docs/architecture/index.md b/docs/architecture/index.md index ca0a8e15..98cad0d6 100644 --- a/docs/architecture/index.md +++ b/docs/architecture/index.md @@ -1,13 +1,11 @@ # Forge Architecture -This is the primary architecture reference for Forge, an AI-powered SDLC orchestrator. It covers system structure, runtime topology, state management, failure modes, security boundaries, and key design decisions. +Architecture reference for Forge, an AI-powered SDLC orchestrator. Covers system structure, runtime topology, state management, failure modes, security boundaries, and key design decisions. -**Audience:** Maintainers, operators, and contributors who need to understand how Forge works, why it is built this way, and what constraints govern its operation. - -The architecture documentation is organized into three parts: +For workflow details, see the [Feature](../guide/feature-workflow.md), [Bug](../guide/bug-workflow.md), and [Task](../guide/task-workflow.md) guides. For API reference, see the OpenAPI spec at `/docs` when the gateway is running. | Part | Contents | |------|----------| -| [System & Components](overview.md) | Scope, system context, external actors, component responsibilities | -| [Internals](internals.md) | Runtime topology, state and concurrency model, failure recovery, security boundaries | -| [Reference](reference.md) | Quality attributes, architectural decisions, known limitations, workflow lifecycles | +| [System & Components](overview.md) | System context, external actors, component responsibilities | +| [Internals](internals.md) | Runtime topology, state and concurrency, failure recovery, security | +| [Reference](reference.md) | Architectural decisions, known limitations, workflow lifecycles | diff --git a/docs/architecture/internals.md b/docs/architecture/internals.md index 731267b4..f349d88d 100644 --- a/docs/architecture/internals.md +++ b/docs/architecture/internals.md @@ -1,197 +1,56 @@ # Internals -## Runtime and Deployment Topology - -### Process Topology +## Runtime Topology Forge runs as two process types plus Redis: -- **Gateway process**: A single FastAPI/Uvicorn process. Stateless; can be load-balanced if needed. No special host requirements beyond network access to Redis. -- **Worker process(es)**: One or more `OrchestratorWorker` processes. Each creates its own Redis consumer group member. Workers **must run on a host with Podman installed** and permission to spawn rootless containers. Each worker can run up to 20 concurrent tasks (configurable via `QUEUE_MAX_CONCURRENT_TASKS`), bounded by an `asyncio.Semaphore`. -- **Redis**: Single-instance Redis server. No Sentinel or Cluster configuration exists; HA must be provided externally if required. - -### Scaling Constraints - -Gateway and Worker communicate only through Redis and can be deployed on separate hosts. However, horizontal Worker scaling has a **known limitation**: per-ticket event serialization uses an in-process `asyncio.Lock`, not a distributed lock. If two Worker processes receive events for the same ticket concurrently, they will process them in parallel with no coordination. See Known Limitations in the [Reference](reference.md#known-limitations) section for details. - -### Container Lifecycle - -- Containers are named `forge-{ticket_key}-{short_uuid}` and are removed after execution by default -- Setting `FORGE_CONTAINER_KEEP=true` preserves failed containers for debugging -- Workspace directories are cleaned up via `podman unshare rm -rf` (to handle root-owned files), with `shutil.rmtree` as a fallback -- Stale workspaces older than 24 hours are eligible for cleanup via `cleanup_stale_workspaces()` - -### Required Connectivity - -- Workers --> Redis (event consumption, checkpoint read/write, retry queue) -- Gateway --> Redis (event publishing) -- Workers --> Jira API (comments, labels, transitions) -- Workers --> GitHub API (branches, commits, PRs, reviews) -- Workers --> LLM provider (Anthropic API or Vertex AI) -- Workers --> Langfuse (optional, trace ingestion) -- Containers --> LLM provider (code generation) -- Containers --> Langfuse (optional, in-container tracing) -- Containers --> internet (for dependency installation; network mode configurable via `FORGE_SANDBOX_NETWORK`, default `slirp4netns`) - -### What Is Lost if Redis Is Lost - -Redis holds both ephemeral queues and durable state. If Redis data is lost: - -- All in-flight events in the streams are lost (unprocessed webhooks) -- All workflow checkpoint state is lost (paused workflows cannot resume) -- The retry queue and dead-letter queue contents are lost -- PR-to-ticket index mappings are lost (webhook routing for PR events will fail until rebuilt) -- Deduplication keys are lost (duplicate event processing may occur temporarily) - -Operators should configure Redis persistence (RDB snapshots, AOF, or both) appropriate to their durability requirements. Forge does not configure Redis persistence itself. - -## State, Events, and Concurrency Model - -### Delivery Guarantee: At-Least-Once - -Forge uses **at-least-once** event processing. Redis consumer groups assign each stream entry to one consumer initially, but messages are acknowledged (`XACK`) only **after** successful processing. If a worker crashes between completing processing and sending the acknowledgement, the message remains in the Pending Entries List (PEL) and can be redelivered. - -The system does **not** provide exactly-once semantics. There is no transaction or Lua script wrapping handler execution and acknowledgement atomically. - -### Idempotency - -A `DeduplicationService` using Redis `SETNX` with 24-hour TTL keys (`forge:dedup:{event_id}`) is implemented but **not yet wired into the webhook routes**. Currently, duplicate webhooks from Jira or GitHub will be processed as separate events. - -Within workflows, certain operations have natural idempotency properties: - -- Branch creation: `git push` to an existing branch with the same content is a no-op -- PR creation: the workflow checks for existing PRs before creating new ones -- Label operations: setting a label that already exists is idempotent in both Jira and GitHub - -However, Jira comment posting and some GitHub operations are **not** idempotent. A duplicated event could result in duplicate comments. - -### Pending Entry Reclaim - -There is **no automated PEL reclaim mechanism**. The codebase does not use `XPENDING`, `XCLAIM`, or `XAUTOCLAIM`. If a worker crashes mid-processing before enqueuing the message for retry, the message remains in the PEL indefinitely. Recovery requires manual intervention or restarting the consumer group. This is a known limitation (see [Reference](reference.md#known-limitations)). - -### Per-Ticket Ordering - -Within a single worker process, events for the same ticket are serialized by an `asyncio.Lock` per ticket key. This ensures that concurrent events for ticket `AISOS-123` within one process are handled sequentially. - -Across multiple worker processes, **no distributed coordination exists**. Redis consumer groups distribute messages round-robin across consumers. If events for the same ticket are assigned to different workers, they will execute concurrently. This can lead to checkpoint conflicts or race conditions on Jira/GitHub side effects. - -**Current safe configuration:** Run a single Worker process per deployment, or accept that concurrent same-ticket processing may occur across workers. The in-process lock handles the common case (multiple events arriving in quick succession to the same consumer). - -### Checkpoint Model - -LangGraph workflow state is persisted via `AsyncRedisSaver` in Redis. The checkpoint lifecycle: - -- **Thread ID** = Jira ticket key (e.g., `AISOS-123`). One ticket maps to exactly one workflow instance and one checkpoint thread. -- **Writes** happen automatically after each LangGraph node completes. -- **Reads** happen when a new event arrives for an existing ticket: the workflow resumes from the last checkpoint. -- **Deletion**: `clear_checkpoint(thread_id)` removes all checkpoint data for a ticket. -- **Redis key pattern**: `checkpoint:{thread_id}:{...}` - -### Consistency Between Checkpoints and Side Effects - -Checkpoint writes and external side effects (Jira comments, GitHub PRs) are **not transactional**. A node may successfully post a comment to Jira but crash before its checkpoint is written, causing the comment to be posted again on retry. Similarly, a checkpoint may be written after the node function returns but before all side effects complete. - -The ordering between the Jira and GitHub event streams is **independent**. Events are consumed from both streams concurrently, and there is no ordering guarantee across streams. - -## Failure and Recovery Model - -### Component Failure Modes - -**Gateway failure**: Incoming webhooks are dropped. Jira and GitHub will retry webhook delivery according to their own retry policies (Jira retries up to 10 times with backoff; GitHub retries based on recent failure rate). No data is lost in Redis. - -**Worker failure (crash)**: In-flight messages remain in the Redis PEL. The workflow checkpoint reflects the last completed node. On restart, the worker begins consuming new messages. PEL messages require manual intervention (see Known Limitations in the [Reference](reference.md#known-limitations) section). Retry-eligible messages that were already enqueued in the retry sorted set will be picked up by any running worker. - -**Redis failure**: Complete system outage. Gateway cannot enqueue events; workers cannot consume, checkpoint, or retry. All in-memory state (streams, checkpoints, retry queue, DLQ, indexes) is at risk unless Redis persistence is configured. See [What Is Lost if Redis Is Lost](#what-is-lost-if-redis-is-lost). - -**LLM provider failure**: Orchestrator nodes and container agents fail. The retry mechanism handles transient LLM errors. Persistent failures exhaust retries and move the event to the dead-letter queue. The `forge:blocked` label is applied to the Jira ticket for visibility. - -**Jira/GitHub API failure**: Webhook-triggered nodes fail. Rate limiting (token bucket) prevents overloading. API errors during workflow execution are handled by the retry mechanism. - -**Container failure**: Non-zero exit codes are captured by the orchestrator. If `FORGE_CONTAINER_KEEP=true`, failed containers are preserved for debugging. The workflow node reports the failure, and the retry mechanism determines whether to re-attempt. - -### Retry Budget and Backoff - -- **Max retry attempts:** 3 -- **Backoff:** Exponential: 30s initial delay, 2x multiplier, capped at 3600s (1 hour) -- **Retry queue:** Redis sorted set (`forge:retry:queue`) with score = next retry timestamp, polled every 10 seconds -- **Dead-letter queue:** Redis list (`forge:retry:dlq`). Messages that exhaust all retries are moved here. Manual requeue via `requeue_dead_letter()` resets the attempt counter. - -### Crash Recovery - -- **Automatic:** Workers restart and consume new messages. Retry-queue entries are picked up by any running worker. LangGraph workflows resume from the last checkpoint when a new event arrives. -- **Manual:** PEL entries from crashed workers require manual `XCLAIM` or consumer group reset. DLQ entries require manual requeue or investigation. - -### Blocked Workflows and `forge:retry` - -When a workflow enters an error state, the `forge:blocked` label is applied to the Jira ticket. Adding the `forge:retry` label triggers re-entry at the failed step using the last checkpoint. This works at any workflow stage, including approval gates (where it regenerates the artifact). - -### Partial Multi-Repository Execution - -For tickets that span multiple repositories, the task router groups tasks by repo and processes them either sequentially or in parallel (up to 5 concurrent repos via LangGraph's `Send` API). If execution fails for one repo, that repo's error is recorded but other repos continue. The `aggregate_parallel_results()` node merges results and errors from all branches. - -### Approval Gate Timeout - -**There is no timeout on human approval gates.** When a workflow reaches a pause gate (`>> gate`), it persists `is_paused=True` in the checkpoint and waits indefinitely for a resume event (Jira comment or label change). If approval never arrives, the workflow remains paused in Redis checkpoint state. There is no automatic escalation or expiration. - -## Security and Trust Boundaries - -### Webhook Authentication - -Both Jira and GitHub webhook endpoints validate HMAC-SHA256 signatures using `hmac.compare_digest()` (constant-time comparison). However, signature validation is **conditional**: it only runs when the corresponding secret is configured (`JIRA_WEBHOOK_SECRET`, `GITHUB_WEBHOOK_SECRET`). If secrets are not set, the endpoint accepts unsigned payloads without warning. - -**Recommendation for production:** Always configure webhook secrets. Without them, any network actor that can reach the gateway can inject events. - -### Credentials in the Worker +- **Gateway**: Single FastAPI/Uvicorn process. Stateless; can be load-balanced. +- **Worker(s)**: One or more `OrchestratorWorker` processes. Each joins the Redis consumer group. **Must run on a host with Podman installed.** Each worker handles up to 20 concurrent tasks (configurable via `QUEUE_MAX_CONCURRENT_TASKS`). +- **Redis**: Single-instance server. No built-in HA; must be provided externally if required. -Worker processes require: +Gateway and Worker communicate only through Redis and can be deployed on separate hosts. Horizontal Worker scaling has a limitation: per-ticket event serialization uses an in-process `asyncio.Lock`, not a distributed lock (see [Known Limitations](reference.md#known-limitations)). -- Redis connection credentials (`REDIS_URL`) -- Jira API token (`JIRA_API_TOKEN`): used for reading/writing tickets, comments, labels, transitions -- GitHub App credentials (`GITHUB_APP_ID`, `GITHUB_PRIVATE_KEY`) or personal access token: used for repository operations, PR management, and webhook validation -- LLM provider credentials: either `ANTHROPIC_API_KEY` (direct API) or Google service account credentials for Vertex AI -- Langfuse credentials (optional): `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` +## State and Event Processing -### Credentials in the Container +**Delivery guarantee:** At-least-once. Messages are acknowledged (`XACK`) only after successful processing. The system does not provide exactly-once semantics. -Implementation containers receive a **subset** of credentials as environment variables: +**Checkpointing:** LangGraph workflow state is persisted via `AsyncRedisSaver`, keyed by Jira ticket key (e.g., `AISOS-123`). Checkpoints are written after each graph node completes. When a new event arrives for an existing ticket, the workflow resumes from its last checkpoint. -- LLM credentials (API key or Vertex AI service account, depending on backend) -- Git user identity (`GIT_USER_NAME`, `GIT_USER_EMAIL`) -- Langfuse credentials (when enabled) +**Idempotency:** A `DeduplicationService` exists but is not yet wired into the webhook routes. Branch creation and label operations are naturally idempotent; Jira comment posting is not. -Containers do **not** receive Jira credentials, GitHub tokens, or Redis credentials. All Jira/GitHub operations are performed by the orchestrator after the container exits. +**Consistency caveat:** Checkpoint writes and external side effects (Jira comments, GitHub PRs) are not transactional. A crash between a side effect and its checkpoint write can cause duplicate actions on retry. -### Container Isolation +## Failure and Recovery -- **Rootless Podman**: Containers run without root privileges on the host -- **Network**: Configurable via `FORGE_SANDBOX_NETWORK` (default: `slirp4netns`, providing NAT-based isolation) -- **Resource limits**: Memory: 4GB (configurable), CPU: 2 cores (configurable), Timeout: 30 minutes (configurable) -- **Filesystem**: Workspace mounted read-write at `/workspace` (necessary for code changes); task file mounted read-only at `/task.json` +| Component | Failure impact | Recovery | +|-----------|---------------|----------| +| Gateway | Incoming webhooks dropped | Jira/GitHub retry delivery per their own policies | +| Worker | In-flight messages stay in Redis PEL | Restart consumes new messages; PEL requires manual `XCLAIM` | +| Redis | Complete system outage; all state at risk | Configure Redis persistence (RDB/AOF) externally | +| LLM provider | Planning/code generation fails | Retried up to 3 times, then moved to dead-letter queue | +| Container | Non-zero exit captured by orchestrator | Retry mechanism determines re-attempt | -**Not currently enforced:** +**Retry policy:** Up to 3 attempts with exponential backoff (30s initial, 2x multiplier, capped at 1 hour). Failed messages go to a dead-letter queue for manual investigation. -- `--security-opt no-new-privileges` is not set -- `--cap-drop ALL` is not applied (relies on Podman rootless defaults) -- No explicit seccomp profile -- Root filesystem is not read-only (`--read-only` not set) -- No AppArmor or SELinux enforcement beyond the `:Z` mount relabeling flag +**Blocked workflows:** The `forge:blocked` label is applied to Jira tickets in error state. Adding `forge:retry` triggers re-entry at the failed step. -### Prompt Injection Boundaries +**Approval gates:** Workflows pause indefinitely at human review gates. There is no automatic timeout or escalation. -Forge processes externally-supplied text from Jira tickets, GitHub PR descriptions, code review comments, and repository content. This text is passed to LLM prompts. The system relies on: +## Security Boundaries -- **Structured prompt templates**: System prompts are loaded from versioned templates in `src/forge/prompts/v1/`, not dynamically constructed from user input -- **Repository guardrails**: The workspace manager searches cloned repos for `CONSTITUTION.md`, `AGENTS.md`, and `CLAUDE.md` files and injects them as system context, giving repo owners control over agent behavior -- **Human gates**: All generated artifacts (PRDs, specs, plans, code) must pass human approval before merging, providing a boundary against manipulated output -- **Container isolation**: Code execution happens in ephemeral containers with limited credentials, bounding the blast radius of a compromised agent +**Webhook authentication:** HMAC-SHA256 validation via `hmac.compare_digest()`. Validation is conditional: it only runs when secrets are configured (`JIRA_WEBHOOK_SECRET`, `GITHUB_WEBHOOK_SECRET`). **Always configure secrets in production.** -### MCP Tool Permissions +**Credential distribution:** -Container agents use Deep Agents with MCP tool access. The agent can read and write files within `/workspace`, run shell commands, and use MCP-configured tools. It cannot access host filesystems outside the mount, Jira/GitHub APIs directly, or Redis. +| Credential | Worker | Container | +|------------|--------|-----------| +| Redis | Yes | No | +| Jira API token | Yes | No | +| GitHub App credentials | Yes | No | +| LLM provider (API key or Vertex AI) | Yes | Yes | +| Langfuse | Yes | Yes (when enabled) | +| Git identity | No | Yes | -### Secret Redaction and Auditability +Containers do not receive Jira, GitHub, or Redis credentials. All external platform operations are performed by the orchestrator after the container exits. -- LLM interactions are traced via Langfuse (when enabled), providing an audit trail of all AI-generated content -- Container stdout/stderr is captured in worker logs -- Jira comments and GitHub PR descriptions provide a human-readable record of all workflow artifacts -- Credential values are passed as environment variables, not written to files or logs +**Container isolation:** Rootless Podman with configurable network mode (`slirp4netns` default), memory limit (4GB), CPU limit (2 cores), and 30-minute timeout. Workspace mounted read-write at `/workspace`; task file read-only at `/task.json`. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 6b00b906..8d135775 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -1,26 +1,6 @@ # System & Components -## Scope and Audience - -This document is the primary architecture reference for Forge, an AI-powered SDLC orchestrator. It covers system structure, runtime topology, state management, failure modes, security boundaries, and key design decisions. - -**Audience:** Maintainers, operators, and contributors who need to understand how Forge works, why it is built this way, and what constraints govern its operation. - -**Goals:** - -- Explain the runtime topology and deployment constraints -- Document state ownership, consistency guarantees, and failure recovery -- Define security and trust boundaries, especially around AI-generated code execution -- Record key architectural decisions with rationale and trade-offs -- Provide stable, high-level lifecycle views that link to detailed workflow guides - -**Non-goals:** - -- Node-level workflow documentation (see the [Feature](../guide/feature-workflow.md), [Bug](../guide/bug-workflow.md), and [Task](../guide/task-workflow.md) workflow guides) -- Operational runbooks or deployment procedures -- API reference (see the OpenAPI spec at `/docs` when the gateway is running) - -## System Context and External Actors +## System Context Forge sits between project management (Jira), source control (GitHub), and LLM providers, orchestrating work from ticket creation through merged PR. @@ -37,11 +17,11 @@ flowchart LR **External actors:** -- **Jira**: Source of ticket lifecycle events. Forge reads ticket metadata and writes comments, labels, and transitions back. Webhooks deliver issue and comment events. -- **GitHub**: Source of PR, CI, and code review events. Forge creates branches, pushes commits, opens PRs, and responds to review feedback. Webhooks deliver PR, check suite, and review events. -- **LLM providers**: Anthropic (direct API) and Google Vertex AI (Claude and Gemini models). Called by both orchestrator nodes (planning, review) and container agents (code generation). -- **Langfuse**: Observability platform for LLM call tracing, workflow spans, and cost tracking. Optional; disabled when credentials are not configured. -- **Human reviewers**: Approve or revise artifacts at defined gates. The workflow pauses indefinitely at each gate until a human responds. +- **Jira**: Source of ticket lifecycle events (issue and comment webhooks) +- **GitHub**: Source of PR, CI, and code review events (PR, check suite, and review webhooks) +- **LLM providers**: Anthropic (direct API) and Google Vertex AI (Claude and Gemini models) +- **Langfuse**: Optional observability for LLM call tracing and cost tracking +- **Human reviewers**: Approve or revise artifacts at defined workflow gates ## Component Responsibilities @@ -94,18 +74,12 @@ flowchart TD Workers --> Langfuse ``` -**Gateway (FastAPI)**: Accepts Jira and GitHub webhooks over HTTPS, validates HMAC-SHA256 signatures (when secrets are configured), and publishes events to Redis Streams. It performs no workflow logic. It also serves health, readiness, and Prometheus metrics endpoints. Middleware includes CORS and correlation ID propagation. A `DeduplicationService` exists in the codebase but is not yet wired into the webhook routes (see Known Limitations in the [Reference](reference.md#known-limitations) section). - -**Worker**: Consumes events from Redis Streams via the `forge-workers` consumer group. Each worker generates a unique consumer ID (`worker-{uuid}`). The `WorkflowRouter` resolves the target LangGraph workflow (Feature, Bug, or Task Takeover) based on Jira issue type and drives execution through planning, implementation, CI repair, and human review stages. Implementation runs inside ephemeral Podman containers using Deep Agents. +**Gateway (FastAPI)**: Accepts webhooks over HTTPS, validates HMAC-SHA256 signatures, and publishes events to Redis Streams. Performs no workflow logic. -**Redis**: Serves multiple roles: +**Worker**: Consumes events from Redis Streams via the `forge-workers` consumer group. The `WorkflowRouter` resolves the target LangGraph workflow (Feature, Bug, or Task Takeover) based on Jira issue type and drives execution through planning, implementation, CI repair, and human review stages. -- *Event bus*: Redis Streams (`forge:events:jira`, `forge:events:github`) with the `forge-workers` consumer group for message distribution -- *Workflow state store*: LangGraph `AsyncRedisSaver` checkpoints workflow state per ticket (keyed by ticket key, e.g. `AISOS-123`) -- *Retry queue*: Sorted set (`forge:retry:queue`) with exponential backoff -- *Dead-letter queue*: List (`forge:retry:dlq`) for messages that exhaust retries -- *Supporting indexes*: PR-to-ticket mapping hash (`forge:state:pr_index`), webhook deduplication keys (`forge:dedup:*`, 24h TTL) +**Redis**: Event bus (Redis Streams), workflow state store (LangGraph `AsyncRedisSaver` checkpoints per ticket), retry queue, dead-letter queue, and supporting indexes (PR-to-ticket mapping, deduplication keys). -**Podman Container**: Ephemeral rootless containers that execute implementation tasks. Each container receives the target repository mounted at `/workspace` (read-write), a task file at `/task.json` (read-only), and LLM credentials as environment variables. The container runs Deep Agents with MCP tool access to make changes and commit them locally. The orchestrator handles pushing and PR creation after the container exits. +**Podman Container**: Ephemeral rootless containers that execute implementation tasks. Each container receives the repo at `/workspace` (read-write), a task file at `/task.json` (read-only), and LLM credentials. Runs Deep Agents with MCP tool access. The orchestrator handles pushing and PR creation after the container exits. -**LLM Backends**: Claude and Gemini models are called bidirectionally. Orchestrator nodes call them for planning and review; container agents call them for code generation. Two backends are supported: Anthropic's direct API (via `ANTHROPIC_API_KEY`) and Google Vertex AI (via service account credentials). The backend is selected automatically based on which credentials are configured. Request-level rate limiting uses token bucket algorithms (Anthropic: 0.5 req/s burst 5; Jira: 1.5 req/s burst 10; GitHub: 1.0 req/s burst 20). +**LLM Backends**: Claude and Gemini models called by both orchestrator nodes (planning, review) and container agents (code generation). Supports Anthropic direct API and Google Vertex AI, selected automatically based on configured credentials. diff --git a/docs/architecture/reference.md b/docs/architecture/reference.md index 48c9eb8b..983fb39e 100644 --- a/docs/architecture/reference.md +++ b/docs/architecture/reference.md @@ -1,108 +1,44 @@ # Reference -## Quality Attributes and Operational Characteristics - -### Availability - -- Gateway and Worker are stateless process types (state lives in Redis). Either can restart without data loss. -- Redis is a single point of failure. Redis downtime causes complete system outage. HA must be provided externally. -- Human approval gates have no timeout. Workflows can remain paused indefinitely without consuming resources beyond Redis checkpoint storage. - -### Throughput and Concurrency - -- Each Worker process handles up to 20 concurrent tasks (`QUEUE_MAX_CONCURRENT_TASKS`), bounded by an `asyncio.Semaphore`. -- The consumer reads up to 10 messages per `XREADGROUP` call with a 5-second block timeout. -- Multi-repo tasks can run up to 5 concurrent repository implementations via LangGraph `Send` fan-out. -- LLM rate limiting (Anthropic: 0.5 req/s burst 5) is the primary throughput bottleneck for planning-heavy workflows. - -### Per-Ticket Latency - -End-to-end latency is dominated by: - -- LLM response time (seconds to minutes per node, depending on prompt complexity) -- Human approval gates (minutes to days, unbounded) -- Container execution (30-minute default timeout per task) -- CI pipeline execution (external, typically 5-30 minutes) - -A fully autonomous (`forge:yolo`) simple task with fast CI can complete in 10-20 minutes. A multi-epic feature with human review at each gate can take days. - -### Durability - -- Workflow state is durable to Redis checkpoint writes. State survives Worker restarts. -- Event durability depends on Redis persistence configuration (operator responsibility). -- External side effects (Jira comments, GitHub PRs) are durable once written to those platforms. -- Container-local changes are lost if the container exits before the orchestrator pushes them. The orchestrator pushes immediately after successful container exit. - -### Auditability - -- Every workflow artifact (PRD, spec, plan, tasks) is posted as a Jira comment or GitHub PR, creating a human-readable audit trail -- LLM calls are traced in Langfuse with session ID = ticket key, including token counts and costs -- OpenTelemetry tracing is available (configurable via `OTLP_ENDPOINT`) -- Worker logs include correlation IDs for request tracing - -### Known Bottlenecks - -- **LLM rate limits**: The Anthropic rate limiter (0.5 req/s) serializes LLM calls across all concurrent tasks within a worker -- **Single Redis**: All state, queuing, and coordination goes through one Redis instance with no sharding -- **Container spawn overhead**: Podman container creation, repo cloning, and dependency installation add minutes per task - ## Key Architectural Decisions ### Redis Streams for Event Bus -**Decision:** Use Redis Streams with consumer groups instead of a dedicated message broker (RabbitMQ, Kafka). - -**Rationale:** Redis already serves as the checkpoint store and index. Using it for event queuing eliminates an additional infrastructure dependency. Consumer groups provide the needed delivery semantics (at-least-once, consumer-level distribution). The tradeoff is that Redis Streams lack built-in dead-letter queues, durable replay, and cross-datacenter replication; these are acceptable given Forge's current scale. +Use Redis Streams with consumer groups instead of a dedicated message broker (RabbitMQ, Kafka). Redis already serves as the checkpoint store, so reusing it for event queuing eliminates an infrastructure dependency. The tradeoff: no built-in dead-letter queues or cross-datacenter replication. ### LangGraph for Workflow Orchestration -**Decision:** Use LangGraph `StateGraph` with `AsyncRedisSaver` checkpointing instead of a traditional workflow engine (Temporal, Airflow). - -**Rationale:** LangGraph provides native support for LLM-driven decision nodes, conditional routing, and checkpointed pause/resume. The Python-native graph definition allows workflow logic to live alongside the orchestration code. The tradeoff is a less mature ecosystem and fewer operational tools compared to established workflow engines. +Use LangGraph `StateGraph` with `AsyncRedisSaver` checkpointing instead of Temporal or Airflow. LangGraph provides native LLM-driven decision nodes, conditional routing, and checkpointed pause/resume. The tradeoff: a less mature ecosystem with fewer operational tools. ### Host-Level Podman for Code Execution -**Decision:** Run implementation tasks in Podman containers on the Worker host instead of using Kubernetes jobs, remote VMs, or in-process execution. - -**Rationale:** Rootless Podman provides process isolation, filesystem isolation, and resource limits without requiring a Kubernetes cluster. Workers must run on Podman-capable hosts, which constrains deployment options but simplifies the container lifecycle (local spawn, local cleanup). The tradeoff is that horizontal scaling requires Podman on every Worker host. +Run implementation tasks in rootless Podman containers on the Worker host instead of Kubernetes jobs or remote VMs. This simplifies the container lifecycle but requires Podman on every Worker host. ### Workflow Separation by Issue Type -**Decision:** Maintain three separate LangGraph workflow definitions (Feature, Bug, Task Takeover) rather than a single parameterized workflow. - -**Rationale:** Each workflow type has fundamentally different planning stages (PRD/spec/epic decomposition for Features, triage/RCA for Bugs, direct planning for Tasks). Separating them keeps each graph readable and independently modifiable. The shared implementation/CI/review tail is implemented as common nodes used by all three workflows. The tradeoff is some duplication in graph wiring. +Three separate LangGraph workflow definitions (Feature, Bug, Task Takeover) rather than one parameterized workflow. Each has fundamentally different planning stages. Shared implementation/CI/review nodes are reused across all three. ### Human Approval Gates -**Decision:** Pause workflows at defined gates and wait indefinitely for human approval, rather than proceeding autonomously by default. - -**Rationale:** Forge generates and executes code in real repositories. Human review at planning stages (PRD, spec, plan, tasks) and at code review catches errors before they reach production. The `forge:yolo` label provides an opt-in escape hatch for tickets where autonomous operation is acceptable. The tradeoff is increased latency and human involvement for every ticket. +Workflows pause at defined gates and wait indefinitely for human approval. The `forge:yolo` label provides an opt-in escape hatch for autonomous operation. The tradeoff: increased latency for every ticket. ## Known Limitations -- **No PEL reclaim**: If a worker crashes mid-processing, unacknowledged messages remain in the Redis PEL indefinitely. No automated `XCLAIM`/`XAUTOCLAIM` mechanism exists. Recovery requires manual intervention. -- **No distributed per-ticket lock**: Per-ticket event serialization uses an in-process `asyncio.Lock`. Multiple Worker processes can process events for the same ticket concurrently, potentially causing checkpoint conflicts or duplicate side effects. -- **Webhook deduplication not wired**: A `DeduplicationService` exists but is not connected to the webhook routes. Duplicate webhooks are processed as separate events. -- **Webhook signature validation is optional**: If `JIRA_WEBHOOK_SECRET` or `GITHUB_WEBHOOK_SECRET` is not configured, the corresponding endpoint accepts unsigned payloads. -- **No approval gate timeout**: Workflows paused at human gates wait indefinitely. There is no automatic escalation, expiration, or notification for stale gates. -- **Single Redis dependency**: No Sentinel, Cluster, or HA configuration. Redis is a single point of failure. -- **Container security hardening gaps**: No `--cap-drop ALL`, `--no-new-privileges`, `--read-only` root filesystem, or explicit seccomp profile. Relies on Podman rootless defaults. -- **No cross-stream ordering**: Events from the Jira and GitHub streams are consumed independently. No ordering guarantee exists across streams for the same ticket. +- **No PEL reclaim**: Unacknowledged messages from crashed workers remain in Redis PEL indefinitely. Recovery requires manual `XCLAIM`. +- **No distributed per-ticket lock**: Multiple workers can process events for the same ticket concurrently, causing potential checkpoint conflicts. +- **Webhook deduplication not wired**: `DeduplicationService` exists but is not connected to webhook routes. +- **Webhook signature validation is optional**: Endpoints accept unsigned payloads when secrets are not configured. +- **No approval gate timeout**: Paused workflows wait indefinitely with no escalation. +- **Single Redis dependency**: No Sentinel, Cluster, or HA. Redis is a single point of failure. +- **Container security hardening gaps**: No `--cap-drop ALL`, `--no-new-privileges`, or `--read-only` root filesystem. +- **No cross-stream ordering**: Jira and GitHub streams are consumed independently with no ordering guarantee. ## Workflow Lifecycles -The diagrams below show the high-level lifecycle for each workflow type. `>>` marks human approval gates where the workflow pauses until a reviewer approves, requests revisions (`!` comment), or asks questions (`?` comment). Gates are auto-approved when the `forge:yolo` label is set. - -These diagrams show the primary flow at a stable abstraction level. Error handling, revision loops, and question-answering branches are omitted. For detailed node-level flows with all branches, see the individual workflow guides: - -- [Feature Workflow Guide](../guide/feature-workflow.md) -- [Bug Workflow Guide](../guide/bug-workflow.md) -- [Task Workflow Guide](../guide/task-workflow.md) +`>>` marks human approval gates. Gates are auto-approved when the `forge:yolo` label is set. For detailed node-level flows, see the [Feature](../guide/feature-workflow.md), [Bug](../guide/bug-workflow.md), and [Task](../guide/task-workflow.md) workflow guides. ### Feature Lifecycle -A Feature or Story goes through a full planning pipeline before implementation. Each planning artifact is reviewed and approved by a human before proceeding. - ```mermaid flowchart TD A["Ticket created\n(Feature/Story)"] --> B["Generate PRD"] @@ -124,8 +60,6 @@ flowchart TD ### Bug Lifecycle -A Bug goes through triage and root cause analysis before implementation. The reporter may be asked to fill in missing context, and the user selects a fix approach from multiple options. - ```mermaid flowchart TD A["Ticket created\n(Bug)"] --> B["Triage check"] @@ -145,8 +79,6 @@ flowchart TD ### Task Lifecycle -A Task or Epic goes directly from triage to implementation planning, skipping PRD, spec, and epic decomposition. - ```mermaid flowchart TD A["Ticket created\n(Task/Epic)"] --> B["Triage check"] @@ -161,16 +93,3 @@ flowchart TD H --> I[">> Human code review"] I -->|merged| J["Complete"] ``` - -### Diagram Maintenance - -The lifecycle diagrams above use stable, high-level step names rather than concrete graph node names. The source of truth for node-level workflow details is the graph definition code in `src/forge/orchestrator/`. When graph node names or edges change, the detailed workflow guides should be updated; these high-level diagrams should only change when the overall lifecycle structure changes. - -## Data Flow Summary - -- **Inbound events:** Jira/GitHub webhooks --> FastAPI gateway --> Redis Streams -- **State persistence:** Redis (`AsyncRedisSaver`, keyed by Jira ticket key) -- **LLM calls:** Orchestrator nodes and container agents --> Claude/Gemini (Anthropic direct API or Vertex AI) -- **Code execution:** Workflow node --> Podman container --> Deep Agents with MCP tools -- **Outbound actions:** Jira (comments, labels, transitions), GitHub (branches, commits, PRs, reviews) -- **Observability:** Langfuse (LLM traces, workflow spans, costs), OpenTelemetry (configurable)