Unified databases (PRO-1618): adopt the client contract (no type, context[] body, four-key query, llm_prompt verbatim)

[SohamRatnaparkhi]

Stacked on #11. Brings the plugin onto the PRO-1618 unified API contract (the hydradb-application #1625 query response plus the stacked ingest-contract PR). Everything below applies to a unified database only; a split database keeps every byte it had, and that is now pinned by goldens cut from the pre-contract code.

Unified: what changed

Query (POST /query)

• No type is sent (absent is the unified default; knowledge/memory are 400).
• follow_forceful_relations goes with the request, from the new config knob followForcefulRelations (default true, env HYDRADB_FOLLOW_FORCEFUL_RELATIONS).
• The four-key response (chunks, graph, forceful_relations, llm_prompt) is recognised by shape (graph and forceful_relations arrays and a string llm_prompt), never by a flag, and parsed into unified.chunks[] (contextId, chunkId, score, content, enrichment (a plain string), enrichmentKind (the declared context_category, kept even when there is no enrichment), temporal[]), unified.graph[] (origin, pathSummary, triplets), unified.forcefulRelations[] (via{from,to}, chunk with the same fields) and unified.llmPrompt. A body that still names the bucket relations is not read as unified (no fallback); recallUnified refuses it with a named error that lands in the recall's errors.
• The injected <hydradb-context> block is llm_prompt verbatim: markdown (## Results with ### 1., ## Forceful relations with ### R1., ## Related facts with [P1], ## Temporal facts, ## Sources), numbered for citation as [1], [R1], [P1]. It is injected whole: only secret redaction applies, never truncation or summarising, and the maxContextChars budget applies to split databases only (see "No compaction" below). This also removes the undefined buildSection call that Unified databases (PRO-1618): layout-aware recall and ingest, items[] body #11 left on the unified render path.
• query text output renders the structured fields in the server's markdown layout: ## Results with ### 1. per chunk (**Relevance:**, **Category:**, content, **Enrichment:**, **Temporal:**), ## Forceful relations with the server's guide line and ### R1. / **Linked from:**, and ## Related facts with [P1] paths; last-recall reports unifiedCount, unifiedGraphPathCount, unifiedForcefulRelationCount.
• A split-mode call that the server refuses and the client retries as unified is reported as the unified result it is, not pushed through the MEMORY/KNOWLEDGE template.

Ingest (POST /context/ingest; every write: turn capture, session upsert, ingest --note, ingest --session, workspace sync)

• JSON body with list key context (never items), plus request-level enrich/upsert/instructions when set.
• Item fields per the contract: text or conversation, context_id, title, enrich, instructions (was custom_instructions), happened_at as YYYY-MM-DD (cut from the file mtime), attributes, custom_attributes.
• is_markdown and user_name are not item fields in the contract, so a text item carries them inside custom_attributes instead of as invented top-level fields; a conversation's attribution stays the per-turn name. Note: the server build I checked still accepts both at top level, so this is a contract-strictness choice and is a one-line flip in memoryToItem if you would rather send them.
• The 202 is parsed (results[].source_id is the context id): ingest --note prints it, ingest --session --json returns contextIds. A 202 that names a refused item (status: "failed") raises, the way a split database's 4xx does, so the workspace sync never records the file as synced and turn/session capture never advance their hashes for content the server did not store.

Other endpoints

• POST /context/list, GET /context/relations, DELETE /context send no type on a unified database. POST /databases keeps type: "unified".

Split: unchanged

The multipart ingest, type on every call, the legacy normaliser and the MEMORY/KNOWLEDGE block are untouched. conformance/golden/split-normalized.golden.json and split-context-block.golden.txt were generated from the PR #11 code before any edit, and the final tree reproduces them byte-for-byte (the check regenerates and diffs them).

Files

• scripts/lib/hydra/index.mjs: no type on unified query/list/relations/delete; context list key; follow_forceful_relations; request-level ingest defaults.
• scripts/lib/hydra-client.mjs: isUnifiedQueryResponse, normalizeUnifiedResponse, EMPTY_UNIFIED_RECALL, parseUnifiedIngestResponse, toHappenedAt; memoryToItem on the contract's names; addItems sends context, returns the parsed 202 and raises on refused items.
• scripts/lib/context-format.mjs: buildUnifiedContextString (llm_prompt verbatim) and buildUnifiedStructuredString (markdown: content, enrichment, category, temporal, forceful relations, graph); replaces the undefined buildSection.
• scripts/plugin.mjs: forceful-relations pass-through, unified fallback in split mode, unified counts in last-recall, context ids from the 202, structured text render, followForcefulRelations in doctor.
• scripts/lib/config.mjs, config.example.json, .hydradb-plugin.json.example: followForcefulRelations.
• conformance/fixtures.mjs (new): the split and unified query fixtures.
• conformance/unified-query-envelope.json (new): the real unified /query envelope from the server's handler test, read end to end by wire test 10e.
• conformance/tests.mjs, scripts/check.mjs: tests 10/10a/10b/10c/10d/10e/11/12/12b/12c/13x/14 rewritten or added (20 to 28 wire tests), unified and split goldens (3 to 6).
• conformance/golden/: query-unified.shape.json (new), split-*.golden.* (new), doctor.shape.json (adds followForcefulRelations).
• Docs: hydradb-api-info/unified-databases.md (rewritten to the contract), docs/usage.md, README.md, conformance/README.md, skills query, search, auto-recall, hydradb-context, last-recall, setup.

Checks

• npm run check: 7 scripts, 6 JSON files, 16 conformance vectors, 29 HTTP-level wire tests, 6 goldens. Pass.
• npm run pack:dry-run: pass.
• CLI smoke against a stub unified server (query, query --json, ingest --note, user-prompt-submit then last-recall, stop, ingest --session --json): zero calls carried type, all three ingests were JSON context[] bodies, the injected block contained llm_prompt verbatim.
• Review round 1 (Greptile, commit fab86ba): a 202 that refuses an item now raises instead of returning (test 12c: the workspace sync leaves the file untracked, so it is retried next run); the structured unified render includes each chunk's temporal facts; doctor text output prints followForcefulRelations.
• Contract revision (commit 27a0f3d): response root key relations is now forceful_relations (plugin name forcefulRelations), graph[].origin (query_path | chunk_relation) is carried, the forceful-relations guide line is carried, and a wire test pins the unified envelope meta (no tenant_id/sub_tenant_id/source_type; nothing on the unified path reads meta).
• Final chunk shape (commit 58699e1): chunks[].enrichment is a plain string (was {text, kind}) and enrichment_kind is its sibling, on forceful_relations[].chunk too; the plugin exposes them as enrichment and enrichmentKind. llm_prompt is markdown, so the fixture prompt, the structured render, the docs and the skills moved off the old === CONTEXT === / === GRAPH === layout. The query-unified shape golden now lists enrichment and enrichmentKind (on chunks and forceful-relation chunks); both split goldens are unchanged.
• No compaction (commit 715472e, product decision "remove compaction from query response in llm_prompt/context"): the unified llm_prompt is injected whole, past maxContextChars (the budget now applies only to the split MEMORY/KNOWLEDGE sections, so a split block is byte-identical); the structured render drops truncateText on content (700), enrichment (280) and temporal (280); the unified normaliser drops its length caps on content and enrichment (1200), temporal and path summaries (400), graph triplet names/predicates/contexts/temporal details, and context/chunk/via ids. Secret redaction is kept. New wire test 10d-2 pins it with a 20k-character llm_prompt and long fields (29 wire tests); both split goldens are unchanged.
• Ingest 202 sanitising (commit 9e85ec0, OpenHack P4 finding): parseUnifiedIngestResponse now redacts, strips terminal control sequences from, and bounds every server-provided string (context ids, title, status, error, error_code, message) before ingest --note prints it, ingest --session --json returns it, or a refusal error carries it; commit 45b5ce6 (Greptile finding) strips control characters before redacting, so a key split by one cannot slip past the secret patterns. Test 12b covers both.

Host size limit

Claude Code caps a hook's additionalContext at 10,000 characters, with no setting to raise it. Over the cap, Claude Code saves the block to a file in the session directory and shows Claude the file path plus a preview of the first 2,000 characters. The plugin does not truncate to fit; a unified block that long reaches Claude through that file. docs/usage.md says so.

Not touched: the shared four-repo CONTRACT.md (the PRO-1298 client contract; its header forbids editing one repo's copy alone) and the hydradb-api-info/* pages that document split-only v1 endpoints.

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no outstanding previous finding or newly introduced actionable defect remains.

Summary

This PR adopts the PRO-1618 unified HydraDB client contract while preserving byte-compatible behavior for split-layout databases. Greptile automatically discovered a related ticket that helped explain the purpose of this PR: unified databases remove the public knowledge/memory fork while keeping storage layout fixed at database creation.

• Sends unified query, context-management, and ingest requests without a type discriminator.
• Parses the four-key unified query response and injects the server-provided llm_prompt verbatim after secret redaction.
• Migrates unified ingestion to context[], parses per-item 202 results, and rejects refused writes before local sync or capture state advances.
• Adds forceful-relation configuration, structured unified rendering, CLI reporting, contract fixtures, and split-regression goldens.
• The changes since the previous review correctly reverse sanitization order so control characters cannot split a secret before redaction.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    P[Plugin command or hook] --> L{Database layout}
    L -- Split --> S[Legacy typed request]
    S --> N[Legacy normalization]
    N --> B[Bounded MEMORY / KNOWLEDGE block]
    L -- Unified --> U[Request without type]
    U --> Q[Four-key query response]
    Q --> R[Unified normalization]
    R --> V[Redacted llm_prompt verbatim]
    V --> C[Claude Code additionalContext]
    L -- Unified ingest --> I[POST context/ingest with context array]
    I --> T{Per-item 202 status}
    T -- Queued --> A[Return context IDs and advance state]
    T -- Failed --> E[Raise error and preserve retry state]

Loading

_{Reviews (7) · Last reviewed commit: "fix(unified): strip control characters b..."}

[openhack-agent]

OpenHack Summary

Security review of Unified databases (PRO-1618): adopt the client contract (no type, context[] body, four-key query, llm_prompt verbatim). 25 changed files; 1 finding at or above the low reporting threshold.

0   0   0   1

Confidence Score: 4/5

Review the findings below before merging.

Security merge-readiness rubric: 1 = critical, 2 = high, 3 = medium, 4 = low, 5 = no reportable findings. This score reflects scan findings, not a guarantee of correctness or complete coverage.

Files Needing Attention: scripts/lib/hydra-client.mjs

Important Files Changed

• .hydradb-plugin.json.example (modified)
• README.md (modified)
• config.example.json (modified)
• conformance/README.md (modified)
• conformance/fixtures.mjs (added)
• conformance/golden/doctor.shape.json (modified)
• conformance/golden/query-unified.shape.json (added)
• conformance/golden/split-context-block.golden.txt (added)
• conformance/golden/split-normalized.golden.json (added)
• conformance/tests.mjs (modified)
• conformance/unified-query-envelope.json (added)
• docs/usage.md (modified)
• hydradb-api-info/unified-databases.md (modified)
• scripts/check.mjs (modified)
• scripts/lib/config.mjs (modified)
• scripts/lib/context-format.mjs (modified)
• scripts/lib/hydra-client.mjs (modified)
• scripts/lib/hydra/index.mjs (modified)
• scripts/plugin.mjs (modified)
• skills/auto-recall/SKILL.md (modified)
• skills/hydradb-context/SKILL.md (modified)
• skills/last-recall/SKILL.md (modified)
• skills/query/SKILL.md (modified)
• skills/search/SKILL.md (modified)
• skills/setup/SKILL.md (modified)

Prompt To Fix With AI

Review the findings for https://github.com/hydra-db/hydradb-claude-code/pull/13 at commit 58699e12f08a8ec0ecfdc04aa64f168221670063. Verify each finding against the current code before fixing it. Preserve unrelated changes and run focused regression tests.

### Issue 1: [P4] Unsanitized unified-ingest response data is echoed to terminal and JSON output
Vulnerability type: CWE-116: Improper Encoding or Escaping of Output
scripts/lib/hydra-client.mjs:687

For unified ingestion, the new parser copies `results[].source_id` directly into `contextIds`, and the new CLI paths expose those values. `/hydradb:ingest --note` appends the first context ID to stdout; `ingest --session --json` returns all context IDs; and failed 202 responses are converted into errors containing server-provided error text. Unlike recall normalization, these values are not passed through `redactSecrets` or `trimText`, and terminal output is not protected against ANSI/OSC control sequences. A malicious or compromised backend can therefore inject deceptive terminal output or bypass the plugin's redaction layer. The pre-existing raw HTTP error-body path is not treated as part of this regression; the new regression is the successful 202 and newly surfaced per-item response data.

Recommendation: Treat every unified-ingest response field as untrusted. In `parseUnifiedIngestResponse`, apply `redactSecrets` and bounded `trimText` to context IDs, titles, statuses, error codes, error text, and the response message, using appropriate caps. Before writing any server-derived value to terminal stdout or stderr, remove ANSI/OSC and other terminal control sequences or use a terminal-safe escaping helper. Apply equivalent bounded sanitization before placing values in session JSON, while preserving structured error data separately if needed. Add tests covering successful `source_id`, failed-item `error`, and session JSON values containing control sequences and secret-shaped tokens.

Last reviewed commit: 58699e1 · View review on OpenHack

---

TIP: Mention @openhack-agent in a PR comment to request a review or ask a question. Use @openhack-agent fix all for every finding, or @openhack-agent fix unresolved threads for open review threads only.

[openhack-agent]

OpenHack reviewed this commit. See the OpenHack Summary for the confidence score and fix actions.

[openhack-agent]

OpenHack reviewed this commit. See the OpenHack Summary for the confidence score and fix actions.