Technical Specification

Implementation Reference

This document describes the actual current state of the GRC-AI-Automation project as of Day 7. It supersedes any contradictions in PROBLEM.md or SOLUTION.md, which were written before implementation began.

Project Summary

AI-augmented vendor onboarding pipeline for GRC. Vendors pass through four sequential review stages before being onboarded or rejected. Stages 2 (Legal), 3 (Security), 4 (Financial) use RAG-powered LLM analysis. Stages 1 (Use Case) are human-driven intake forms. Every decision and AI output is written to a persistent audit log.

Stack: FastAPI · Gunicorn (UvicornWorker) · SQLAlchemy / SQLite · ChromaDB · sentence-transformers · litellm

Divergences from PROBLEM.md / SOLUTION.md

The following items were planned but not implemented in the MVP:

Planned	Actual
Parallel Security + Financial review after Legal	Parallel: NDA → Legal → Security → Financial
Separate NDA gates for Security and Financial	Single NDA gate
Decision endpoints trigger state machine transitions	Decisions are persisted but do NOT advance vendor status
`WorkflowService` fully wired (all 8 methods)	Only `trigger_legal_review`, `confirm_nda`, `trigger_security_review` are live
Integration with SharePoint, Jira, Slack, DocuSign	Not implemented (post-MVP)
Role-based access controls per review team	Not implemented (post-MVP)
Encryption at rest / in transit	Not implemented (post-MVP)

Repo Structure

GRC-AI-Automation/
├── backend/
│   ├── main.py                          # FastAPI app + lifespan (KB seeding)
│   ├── gunicorn.conf.py                 # 2 workers, UvicornWorker, on_starting hook
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── pytest.ini                       # asyncio_mode=auto, testpaths=tests
│   ├── mock_data/                       # Plain-text mock vendor documents (Day 7)
│   │   ├── acme_analytics_privacy_policy.txt
│   │   ├── dataflow_privacy_policy.txt
│   │   └── securevault_security_questionnaire.txt
│   ├── api/routes/
│   │   ├── vendors.py                   # CRUD + confirm-nda (complete-onboarding/reject → 501)
│   │   ├── documents.py                 # Upload, extract, chunk, embed
│   │   ├── reviews.py                   # List/get reviews, trigger AI, submit-form (→ 501)
│   │   ├── decisions.py                 # Record/list decisions (no state machine transition)
│   │   └── dev.py                       # POST /dev/seed (Day 7)
│   ├── core/
│   │   ├── config.py                    # Pydantic BaseSettings
│   │   ├── database.py                  # SQLAlchemy engine + SessionLocal + get_db
│   │   └── models.py                    # ORM models + 5 enums
│   ├── schemas/
│   │   ├── vendor.py                    # VendorCreate, VendorRead, VendorList
│   │   ├── document.py                  # DocumentRead
│   │   ├── review.py                    # ReviewRead
│   │   ├── decision.py                  # DecisionCreate, DecisionRead
│   │   └── forms.py                     # UseCaseFormInput, FinancialRiskFormInput
│   ├── services/
│   │   ├── workflow.py                  # WorkflowService (partially implemented — see below)
│   │   ├── llm/client.py                # LLMClient (litellm wrapper)
│   │   ├── document/
│   │   │   ├── extractor.py             # PDF (pdfplumber) / DOCX (python-docx) / TXT → raw_text
│   │   │   └── chunker.py               # RecursiveCharacterTextSplitter (512 chars, 64 overlap)
│   │   ├── rag/
│   │   │   ├── embedder.py              # sentence-transformers wrapper (lazy-init)
│   │   │   ├── store.py                 # ChromaDB wrapper (PersistentClient or HttpClient)
│   │   │   └── retriever.py             # query → newline-delimited chunk string
│   │   ├── knowledge_base/
│   │   │   ├── loader.py                # Seed kb_legal + kb_security on startup (no-op if exists)
│   │   │   ├── legal_kb.py              # GDPR / PIPEDA / CCPA / DPA entries
│   │   │   └── security_kb.py           # NIST CSF / SOC 2 / ISO 27001 entries
│   │   ├── legal/analyzer.py            # Stage 2 AI module (6-domain RAG + LLM)
│   │   └── security/analyzer.py         # Stage 3 AI module (6-domain RAG + LLM + risk scoring)
│   └── tests/                           # ~151 tests across 20 modules (all passing)
├── frontend/
│   ├── README.md                        # "Vite + React + TypeScript. Implemented in Day 6."
│   ├── dist/                            # Built output (index.html + assets)
│   └── node_modules/                    # npm dependencies installed
├── docker-compose.yml
├── .env.example
└── README.md

Data Models

Location: backend/core/models.py

Enums

class VendorStatus(str, enum.Enum):
    INTAKE → USE_CASE_REVIEW → USE_CASE_APPROVED
    → LEGAL_REVIEW → LEGAL_APPROVED
    → SECURITY_REVIEW → SECURITY_APPROVED
    → FINANCIAL_REVIEW → FINANCIAL_APPROVED
    → ONBOARDED | REJECTED

class DocumentStage: USE_CASE | LEGAL | SECURITY | FINANCIAL
class ReviewType:    AI_ANALYSIS | HUMAN_FORM
class ReviewStatus:  PENDING | IN_PROGRESS | COMPLETE | ERROR
class DecisionAction: APPROVE | REJECT | APPROVE_WITH_CONDITIONS

Tables

Table	Key columns
`vendors`	id, name, website, description, status, created_at
`documents`	id, vendor_id, stage, doc_type, filename, raw_text, chroma_collection_id, uploaded_at
`reviews`	id, vendor_id, stage, review_type, status, ai_output (JSON), form_input (JSON), triggered_at, completed_at
`decisions`	id, review_id, actor, action, rationale, conditions (JSON), decided_at
`audit_logs`	id, vendor_id, event_type, actor, payload (JSON), timestamp

Cascade deletes flow from vendors down through all child tables.

Workflow State Machine

INTAKE
  → USE_CASE_REVIEW        Stage 1 — human form (submit-form → 501, not implemented)
  → USE_CASE_APPROVED
  → LEGAL_REVIEW           Stage 2 — AI + RAG (trigger_legal_review ✅)
  → LEGAL_APPROVED
  → NDA_PENDING            (skipped in current state machine — confirm-nda goes direct to SECURITY_REVIEW)
  → SECURITY_REVIEW        Stage 3 — AI + RAG (trigger_security_review ✅, NDA gate enforced)
  → SECURITY_APPROVED
  → FINANCIAL_REVIEW       Stage 4 — human form (submit-form → 501, not implemented)
  → FINANCIAL_APPROVED
  → ONBOARDED              (complete-onboarding → 501, not implemented)
  / REJECTED               (reject → 501, not implemented)

NDA_PENDING note: The NDA_PENDING enum value exists in the model but the confirm_nda transition skips it — it goes directly from LEGAL_APPROVED to SECURITY_REVIEW. The SOLUTION.md described LEGAL_APPROVED → NDA_PENDING → SECURITY_REVIEW but the route only checks for LEGAL_APPROVED.

API Endpoints

Vendors

Method	Path	Status	Notes
`POST`	`/vendors/`	✅	Creates vendor in INTAKE; no review auto-created
`GET`	`/vendors/`	✅	Paginated (skip, limit)
`GET`	`/vendors/{id}`	✅
`POST`	`/vendors/{id}/confirm-nda`	✅	LEGAL_APPROVED → SECURITY_REVIEW; calls WorkflowService.confirm_nda
`POST`	`/vendors/{id}/complete-onboarding`	❌ 501	Not implemented
`POST`	`/vendors/{id}/reject`	❌ 501	Not implemented

Documents

Method	Path	Status	Notes
`POST`	`/vendors/{id}/documents`	✅	Extracts text, chunks, embeds into ChromaDB; creates Document record
`GET`	`/vendors/{id}/documents`	✅
`GET`	`/documents/{id}`	✅

Reviews

Method	Path	Status	Notes
`GET`	`/vendors/{id}/reviews`	✅
`GET`	`/reviews/{id}`	✅
`POST`	`/reviews/{id}/trigger`	✅ (LEGAL/SECURITY)	Dispatches to WorkflowService; returns 501 for other stages
`POST`	`/reviews/{id}/submit-form`	❌ 501	Not implemented

Decisions

Method	Path	Status	Notes
`POST`	`/reviews/{id}/decisions`	✅ (partial)	Persists decision; does NOT advance vendor.status
`GET`	`/reviews/{id}/decisions`	✅

Dev / System

Method	Path	Status	Notes
`POST`	`/dev/seed`	✅	Wipes all data; seeds 3 vendors with docs + pending reviews
`GET`	`/health`	✅	Returns `{"status": "ok"}`

WorkflowService

Location: backend/services/workflow.py

Method	Status	Notes
`create_vendor_and_intake`	❌ `NotImplementedError`	Vendor creation is done inline in the route
`submit_use_case_form`	❌ `NotImplementedError`
`trigger_legal_review(review_id, doc_id)`	✅	Sets IN_PROGRESS → runs LegalAnalyzer → COMPLETE or ERROR; writes audit log
`submit_legal_decision`	❌ `NotImplementedError`
`confirm_nda(vendor_id)`	✅	LEGAL_APPROVED → SECURITY_REVIEW; writes audit log
`trigger_security_review(review_id, doc_id)`	✅	Enforces NDA gate (vendor.status == SECURITY_REVIEW); runs SecurityAnalyzer; writes audit log
`submit_security_decision`	❌ `NotImplementedError`
`submit_financial_form`	❌ `NotImplementedError`
`complete_onboarding`	❌ `NotImplementedError`
`reject_vendor`	❌ `NotImplementedError`

AI Analysis Modules

Stage 2 — LegalAnalyzer

Location: backend/services/legal/analyzer.py

Retrieval domains (6): data_privacy · data_security · data_subject_rights · processor_obligations · retention_deletion · cross_border_transfers

Per domain: 3 chunks from kb_legal + 3 chunks from vendor_{id}_LEGAL_{doc_id} → single LLM call → JSON output

Output: LegalAnalysisResult dataclass

regulation_findings: list[{regulation, article, status, finding, evidence}]
overall_risk: "low" | "medium" | "high" | "critical"
recommendation: "approve" | "approve_with_conditions" | "reject"
summary: str
conditions: list[str]

Stage 3 — SecurityAnalyzer

Location: backend/services/security/analyzer.py

Retrieval domains (6): access_control · data_protection · incident_response · vulnerability_management · business_continuity · supply_chain

Per domain: 3 chunks from kb_security + 3 chunks from vendor_{id}_SECURITY_{doc_id} → single LLM call → JSON output with risk_score (1–5)

Output: SecurityAnalysisResult dataclass

control_findings: list[{domain, framework, control_id, status, finding, evidence, risk_score}]
overall_risk: "low" | "medium" | "high" | "critical"
recommendation: "approve" | "approve_with_conditions" | "reject"
summary: str
conditions: list[str]
risk_score: float  # mean of per-domain risk_scores

LLM Client

Location: backend/services/llm/client.py

class LLMClient:
    async def complete(self, system: str, user: str) -> str
    async def complete_with_json_output(self, system: str, user: str) -> dict
    # Strips ```json``` markdown fences before parsing; raises on invalid JSON

Uses litellm.acompletion. Model string is "{LLM_PROVIDER}/{LLM_MODEL}" (e.g., "anthropic/claude-sonnet-4-6").

RAG Layer

Location: backend/services/rag/

Embedder     → sentence-transformers (all-MiniLM-L6-v2, lazy-init, normalized output)
VectorStore  → ChromaDB PersistentClient (local) or HttpClient (Docker, CHROMA_HOST set)
Retriever    → retrieve(query, collection, n=5) → "\n\n---\n\n".join(chunks)

Collection naming:

kb_legal — regulatory knowledge base (seeded on startup)
kb_security — security control framework (seeded on startup)
vendor_{vendor_id}_{stage}_{doc_id} — per-document vendor chunks

Knowledge Base

Location: backend/services/knowledge_base/

Seeded once at app startup via KnowledgeBaseLoader().seed_if_empty() in the FastAPI lifespan handler.

Collection	Frameworks covered
`kb_legal`	GDPR (Art. 5, 28, 32), PIPEDA (Principles 4.1, 4.7, 4.9), CCPA/CPPA, DPA/SCC
`kb_security`	NIST CSF, SOC 2 Type II, ISO 27001, pen-test standards, vulnerability management

Configuration

Location: backend/core/config.py

Variable	Default	Description
`DATABASE_URL`	`sqlite:///./vendor_onboarding.db`	SQLite for dev; use named volume path in Docker
`LLM_PROVIDER`	`anthropic`	`anthropic` \| `openai` \| `openrouter`
`LLM_MODEL`	`claude-sonnet-4-6`	Passed to litellm
`LLM_PROVIDER_API_KEY`	—	Required
`CHROMA_PERSIST_DIR`	`./chroma_data`	Local embedded mode only
`CHROMA_HOST`	(empty)	Set to `chromadb` in Docker to use HttpClient
`CHROMA_PORT`	`8000`	ChromaDB server port
`EMBEDDING_MODEL`	`all-MiniLM-L6-v2`	Any sentence-transformers model

Computed properties:

chroma_use_server → bool(CHROMA_HOST)
llm_model_string → f"{LLM_PROVIDER}/{LLM_MODEL}"

Docker Compose

Location: docker-compose.yml

Service	Image / Config	Exposed port
`api`	Gunicorn + UvicornWorker, 2 workers	8000
`chromadb`	chromadb v0.6.0	8001 (host) ← 8000 (container)

Named volumes: chroma_data (ChromaDB persistence), sqlite_data (SQLite persistence). Optional GPU support on api service (count: 1).

Testing

Location: backend/tests/ Run: cd backend && .venv/bin/python -m pytest tests/ -v

All tests use in-memory SQLite (StaticPool) with per-test schema create/drop. External services (ChromaDB, LLM API) are patched or stubbed.

Module	Tests	Covers
`test_config.py`	6	`llm_model_string`, `chroma_use_server`
`test_models.py`	11	ORM creation, FK relationships, cascade deletes
`test_schemas.py`	15	Pydantic validation
`test_api_vendors.py`	16	Vendor CRUD, pagination, 404s, health check
`test_api_documents.py`	10	Upload, extraction, chunking, chroma_collection_id
`test_llm_client.py`	7	JSON parsing, markdown fence stripping
`test_extractor.py`	6	PDF/DOCX/TXT extraction
`test_chunker.py`	7	Text splitting, metadata, chunk_index
`test_embedder.py`	6	Shape, lazy loading, normalization
`test_vector_store.py`	7	Upsert, query, collection_exists
`test_retriever.py`	5	Separator joining, edge cases
`test_kb_loader.py`	5	Skip-if-exists, seed-when-absent
`test_legal_analyzer.py`	—	Stage 2 analysis logic
`test_security_analyzer.py`	—	Stage 3 analysis logic
`test_workflow_legal.py`	—	trigger_legal_review + audit log
`test_workflow_security.py`	—	trigger_security_review + NDA gate
`test_api_reviews.py`	—	Review trigger endpoint
`test_api_seed.py`	—	/dev/seed endpoint (Day 7)
Total	~151	All passing

Key test patterns:

os.environ["DATABASE_URL"] = "sqlite://" set at top of conftest.py before any imports
KnowledgeBaseLoader.seed_if_empty is a no-op stub in tests — no mock needed
LegalAnalyzer.analyze patched as services.workflow.LegalAnalyzer.analyze
WorkflowService.trigger_legal_review patched as api.routes.reviews.WorkflowService.trigger_legal_review
seed_client fixture stubs VectorStore.upsert_chunks to avoid live ChromaDB dependency

Mock Data (Day 7)

Location: backend/mock_data/ Seeded via: POST /dev/seed

Vendor	Document	Compliance Profile
Acme Analytics Inc.	`acme_analytics_privacy_policy.txt`	Clean GDPR/PIPEDA compliance, SOC 2 Type II
DataFlow Inc.	`dataflow_privacy_policy.txt`	Gaps: no international transfer mechanism, no GDPR Art. 28 DPA language
SecureVault Ltd.	`securevault_security_questionnaire.txt`	ISO 27001 certified; gaps: no recent pen-test, no IR notification SLA

Acme and DataFlow seed with a LEGAL stage document + AI_ANALYSIS review. SecureVault seeds with a SECURITY stage document + AI_ANALYSIS review.

Seed response:

{
  "message": "Seeded 3 vendors with documents and pending reviews.",
  "vendors": [
    {"id": 1, "name": "Acme Analytics Inc.", "document_id": 1, "review_id": 1},
    ...
  ]
}

Quickstart

git clone git@github.com:AlexPerrin/GRC-AI-Automation.git
cd GRC-AI-Automation
cp .env.example .env        # fill in LLM_PROVIDER_API_KEY
docker compose up --build
# Seed demo data:
curl -X POST http://localhost:8000/dev/seed
# API docs:
open http://localhost:8000/docs

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Technical Specification

Implementation Reference

Project Summary

Divergences from PROBLEM.md / SOLUTION.md

Repo Structure

Data Models

Enums

Tables

Workflow State Machine

API Endpoints

Vendors

Documents

Reviews

Decisions

Dev / System

WorkflowService

AI Analysis Modules

Stage 2 — LegalAnalyzer

Stage 3 — SecurityAnalyzer

LLM Client

RAG Layer

Knowledge Base

Configuration

Docker Compose

Testing

Mock Data (Day 7)

Quickstart

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally