Unified databases (PRO-1618): layout-aware recall and ingest, items[] body - #11
SohamRatnaparkhi wants to merge 11 commits into
Conversation
…tems[] body, type on create - the wrapper reads the database's layout once (GET /databases details[]), sends items[] as the ingest body and type on database create; the vendored SDK predates all three, so they go over a small raw v2 path with the same envelope unwrap and error translation - HydraClient routes every write through items[] on a unified database (memoryToItem / appKnowledgeToItem) and recalls it as one ranked list - searchMode accepts unified/auto, ingestionMode accepts unified; on a unified database the layout wins, on a split one nothing changes - context block gains a CONTEXT section for unified recall; check.mjs pins the item mapping; docs and the setup skill describe the behaviour Signed-off-by: SohamRatnaparkhi <soham.ratnaparkhi@gmail.com>
… skill Signed-off-by: SohamRatnaparkhi <soham.ratnaparkhi@gmail.com>
0d34b42 to
43a0734
Compare
Greptile SummaryThe PR adds layout-aware support for unified HydraDB databases while retaining split-database behavior.
Confidence Score: 5/5The PR appears safe to merge because the previously reported unified-operation serializer failure is addressed and no blocking residual failure remains. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| scripts/lib/hydra-client.mjs | Adds layout detection, unified item conversion, raw-operation routing, and refusal-based recovery; the previously reported SDK serializer rejection is addressed. |
| scripts/lib/hydra/index.mjs | Adds envelope-compatible raw v2 paths for unified query, ingest, list, relations, delete, database creation, and layout discovery. |
| scripts/plugin.mjs | Resolves recall mode from immutable database layout and renders unified results as one context collection. |
| conformance/tests.mjs | Expands wire and mapping coverage for unified operations, retries, refusal classification, and field preservation. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Load configured database] --> B[GET /databases]
B --> C{Layout}
C -->|Unified| D[Raw v2 unified operations]
D --> E[Recall one ranked context list]
D --> F[Ingest items array]
D --> G[Delete with type unified]
C -->|Split or probe failure| H[Existing SDK memory and knowledge paths]
H --> I{Unified-layout refusal?}
I -->|Yes| D
I -->|No| J[Propagate translated error]
Reviews (9): Last reviewed commit: "docs(contract): adopt the canonical unio..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
⏚ Security Scan Complete✅ No vulnerabilities detected! This pull request has been scanned and no security issues were found. |
… serializers; a refused default retries as unified Greptile on #11: the vendored SDK validates the type enum before sending, so kind 'unified' threw client-side and never reached HydraDB. Those calls are now hand-built over the raw v2 path (already snake_case, the plugin's seam), the delete keeps its envelope for per-id classification, and when a probe could not tell and the server names the unified rule the client pins unified and retries once. Four wire tests cover the paths. Signed-off-by: SohamRatnaparkhi <soham.ratnaparkhi@gmail.com>
⏚ Security Scan Complete✅ No vulnerabilities detected! This pull request has been scanned and no security issues were found. |
tushar-hydradb
left a comment
tushar-hydradb
left a comment
There was a problem hiding this comment.
uploadKnowledge is the one write with no recovery
hydra-client.mjs:631-647 falls straight through to context.ingest({kind:"knowledge"}) with no try/catch, while recallMemories, recallKnowledge, addMemories, deleteMemories and deleteKnowledge all route through _retryAsUnified
so on a unified database with ingestionMode auto, one flaky GET /databases makes isUnified() read false, the ingest hits resolveWriteCorpus first and 400s, and it stays broken for the process while every memory lane file in the same sync recovers. conformance test 13 covers this scenario for addMemories and not for this
appKnowledgeToItem maps fields nothing emits and drops the ones that exist
:405-423 reads record.tenant_metadata and record.app_metadata, and buildKnowledgeItem at workspace-sync.mjs:242-263 emits metadata and additional_metadata. source, description, url and timestamp arent mapped at all
so a workspace file synced through the knowledge lane on unified arrives as bare text plus a context_id, with its hydradb://workspace// url gone. the same file on split keeps all of it
memoryToItem drops is_markdown and user_name
:373-401. buildMemoryItems sets both on every chunk. so a synced CLAUDE.md chunk loses its markdown flag, which changes how it chunks and renders, and its attribution, silently
smaller
- _retryAsUnified at :485-492 pins _unifiedPromise = true before calling retry(), so a retry that fails for an unrelated reason leaves the process pinned to unified for its lifetime
- appKnowledgeToItem can emit text: "" at :407, and server validation is per item but all or nothing for the request, so one empty record 400s the whole batch where split tolerated it
- the /unified database/i regex at :487 doesnt match all of our own refusals. ingestUnifiedItemsRequiredMsg reads "this database is unified", which that pattern misses. im asking for a stable error code on hydradb-application 870, then this can be a code check
- hydradb-api-info/unified-databases.md shows title in the example items[] body, and the server has no field for it. thats our swagger's fault, im fixing it there, but the doc here shouldnt ship promising it
…ducer emits uploadKnowledge was the only layout-sensitive write with no try/catch, so one flaky GET /databases pinned the process to split and every knowledge write 400d for the rest of its life while the memory lane in the same sync recovered - uploadKnowledge routes through _retryAsUnified like the other five calls - appKnowledgeToItem reads metadata/additional_metadata, the names buildKnowledgeItem actually emits, and carries source, description, url and timestamp instead of dropping them - memoryToItem refuses is_markdown, and user_name on a text item, rather than dropping them silently, items[] has no home for either - _retryAsUnified pins unified only once the retry has actually succeeded - an empty knowledge record is skipped with a warning, one of them would 400 the whole batch where the split lane just stored it - the refusal is read off the error code first, and the regex fallback now covers both wordings the server uses conformance goes from 13 to 18 wire tests Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfafUvF9MmUELFvmt5qFhs Signed-off-by: tushar-hydradb <tushar@hydradb.com>
is_markdown and user_name-on-a-text-item now fail loudly instead of being dropped, and the workspace sync sets both on every memory chunk, so a markdown file through the memory lane on a unified database will error until the server accepts is_markdown Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfafUvF9MmUELFvmt5qFhs Signed-off-by: tushar-hydradb <tushar@hydradb.com>
⏚ Security Scan Complete✅ No vulnerabilities detected! This pull request has been scanned and no security issues were found. |
…E_UNSUPPORTED MemoryItem has always had IsMarkdown and UserName, only IngestItem was missing them, so there was never anything to refuse. memoryToItem maps both instead of throwing, and a conversation keeps its attribution on the per-turn name, which is what the server reads first the error code is CORPUS_TYPE_UNSUPPORTED, and it names the family rather than the member, so it covers knowledge/memory on a unified database, unified on a split one, and all on an ingest. only the first is ours to retry, so the two siblings are excluded first, the code narrows, and the message decides. the message stays as the fallback for a server that sends no code, since the server now treats that text as a contract conformance goes to 19 wire tests Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfafUvF9MmUELFvmt5qFhs Signed-off-by: tushar-hydradb <tushar@hydradb.com>
⏚ Security Scan Complete✅ No vulnerabilities detected! This pull request has been scanned and no security issues were found. |
…ude it the code is on the syntax refusal too (invalid type "momory": must be ...), so matching on the code alone would have retried a typo as unified. the exclusion now covers all three siblings, and `invalid type` catches both the syntax refusal and `all` on an ingest the envelope repeats the code at detail.error_code, so read that as well as error.code Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfafUvF9MmUELFvmt5qFhs Signed-off-by: tushar-hydradb <tushar@hydradb.com>
⏚ Security Scan Complete✅ No vulnerabilities detected! This pull request has been scanned and no security issues were found. |
one code now covers five refusals and they do not all point the same way. ingestItemsNotKnowledgeMsg picked up CORPUS_TYPE_UNSUPPORTED in the same batch, and it is a caller error rather than a layout answer, so reading it as one would have retried as unified and pinned a SPLIT database to unified for the life of the process. neither client can send items with type=knowledge today, the items branch never sets a type, but the pin is the kind of damage worth excluding on principle the all-on-ingest advice is layout-aware now and says "This database is unified, so send 'unified'" inside a refusal that is not ours, so the exclude-before-code ordering is what stops that sentence being read as a layout answer. pinned by a test rather than left to hold by accident Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfafUvF9MmUELFvmt5qFhs Signed-off-by: tushar-hydradb <tushar@hydradb.com>
|
|
pushed 5 commits fixing the review. additive, signed off, npm run check green with 19 wire tests against 13 on the base
on the refusal detection: CORPUS_TYPE_UNSUPPORTED covers five refusals pointing in different directions, so the code names a class and the message picks the branch. matching the code alone would have retried a split database's refusal as unified. the sibling strings are excluded by name with tests merge after 870, this sends is_markdown and user_name that a pre-870 server drops silently btw 61b13e1's message describes the is_markdown workaround that dd2c8ca reversed. final tree is right, only the log reads oddly |
…tract the server now has TestCorpusRefusalWordingIsAClientContract asserting these strings; this is the half that asserts we still read them in the right direction. one table, six refusals verbatim from corpus.go, context.go and corpus_type.go, two of them retry-as-unified and four not replaces the scattered one-off assertions, which covered the same ground but did not make the shape of the contract legible. also pins the context_category refusal, which carries its own code and so can never reach this branch Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfafUvF9MmUELFvmt5qFhs Signed-off-by: tushar-hydradb <tushar@hydradb.com>
tushar-hydradb
left a comment
tushar-hydradb
left a comment
There was a problem hiding this comment.
approving. uploadKnowledge recovers like the rest, the field mapping reads what the producer actually emits, and is_markdown maps instead of throwing so markdown workspace sync works on unified
merge after 870
an agent tool propagates a failed call, so a message reaches the model, and a client that recovers branches on the path. both are interface, and changing either needs a test rather than a copy edit records the two parts that were counter-intuitive when we hit them. an error code is not automatically the discriminator: CORPUS_TYPE_UNSUPPORTED spans six refusals pointing in different directions, so the message is load-bearing precisely because the code is not sufficient. and a value that varies per request cannot be matched on, which is why the operation path and the request url stay separate four instances named so it reads as a pattern, each pointing at the test that holds it. writing it down now rather than after a fifth, since the case for documenting it gets weaker every time someone else finds one first CONTRACT.md is the shared four-repo text, so this same block goes to hydradb-cli and hydradb-mcp as their own PRs. it is committed verbatim identical here and in openclaw-hydradb Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfafUvF9MmUELFvmt5qFhs Signed-off-by: tushar-hydradb <tushar@hydradb.com>
this stack forked the shared file in two directions. mcp and cli added "## 0. Client groups", the subgraph and byog rows and rule 7; the two plugins added the errors-are-interface note, and the unified vocabulary row landed in mcp and openclaw but not cli and claude-code. neither pair could see the other, which is the exact failure the file header warns about the union is the mcp copy plus the errors note, because mcp already carried the unified row and everything else. verified as a true superset: the only lines it drops are the two it supersedes, the un-extended rule 4 and the pre-groups header built once as a single artifact and copied into both repos rather than edited twice, then diffed every pair. all three files are byte-identical at e020286d7ce6c2a809558ad749ee0566. the same bytes go to hydradb-cli and hydradb-mcp Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfafUvF9MmUELFvmt5qFhs Signed-off-by: tushar-hydradb <tushar@hydradb.com>
tushar-hydradb
left a comment
tushar-hydradb
left a comment
There was a problem hiding this comment.
approving again, the contract.md sync dismissed the last one
all four client copies are byte identical now, 258 lines, same md5
⏚ Security Scan Complete✅ No vulnerabilities detected! This pull request has been scanned and no security issues were found. |
2 participants
What
Client side of PRO-1618 (hydradb-application #870 / #899, cortex-ingestion #620). A database created with
type: "unified"keeps knowledge and memory in one corpus; on it the server accepts onlytype=unified(the default) and refusesknowledge/memory. No new API: the same v2 endpoints, one moretypevalue.scripts/lib/hydra/index.mjs) reads the configured database's layout once fromGET /databases(details[].type), sendsitems[]as the JSON ingest body, andtypeon database create. The vendored SDK predates all three, so they go over a small raw v2 path with the same envelope unwrap and error translation; they move back onto the SDK once it is regenerated.HydraClientroutes every write (turn capture, session upsert,/hydradb-remember, workspace sync, app-knowledge upload) throughitems[]on a unified database (memoryToItem,appKnowledgeToItem) and recalls it as one ranked list (recallUnified); deletes sendunifiedthere.searchModeacceptsunified/auto,ingestionModeacceptsunified. On a unified database the layout wins over the mode knobs; on a split database nothing changes.CONTEXTsection for unified recall;recall --jsoncarriessearchMode: "unified"and aunifiedresult.docs/usage.md,hydradb-api-info/unified-databases.md, the setup skill.Compatibility
Nothing changes for a split database (every database created before): the probe answers
split, every wire body is identical, and a failed probe also reads as split.Checks
npm run check: syntax, JSON, recall normalisation, hook output, config defaults, vendored SDK drift, 16 conformance vectors, 9 wire tests, 3 golden shapes, plus the new item-mapping assertions. All pass.Do not merge until hydradb-application #870 is on staging.