perf(graph-db): store each code edge as one relation row - #2277
Conversation
A code edge was an edge entity carrying its JSON record plus two relations (symbol -> edge, edge -> symbol), each with a locator node and a native edge. It is now one relation from its source symbol to its target symbol, kind `CodeEdge.<kind>`, whose native edge carries the edge record. Readers take the payload from the fan-out row: adjacency, key walks, degrees, edges-among, the evidence traversal, and the catalog warm read no edge entity and walk one hop instead of two. Graph-db relation payloads now live only on the native edge. Code-graph records are stored as compact text: structural JSON tokens and 64-hex digests become short marked codes; free-form values stay JSON. A superseded-format rebuild now discards only sealed generations whose receipt names an older graph format, so a generation sealed before the staging container is replaced keeps serving. Graph format 4 -> 5, projector v9 -> v10, same typed rebuild. Refs #1103
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 31b2130c00
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| decode_graph_properties( | ||
| locator | ||
| .properties | ||
| edge.properties |
There was a problem hiding this comment.
When a format-5 store has a semantically corrupted native edge whose namespace, projection, or RELATION_KIND_PROPERTY disagrees with its locator, keyed relation reads now decode and return that edge's payload without checking those owner scalars; only the edge type is compared with the locator kind. Since this commit moves the sole payload copy from the validated locator to the edge, db.relation can silently attribute data to the wrong relation/projection instead of returning GraphDbError::Corrupt; validate the edge owner scalars against the locator before accepting edge.properties.
AGENTS.md reference: AGENTS.md:L189-L190
Useful? React with 👍 / 👎.
1 participant
Refs #1103. Continues #2257 and #2268, rebased over #2271 (f1f15ed9's sealed-segment row source; code edges are emitted through its windowed spill). This change stores each code edge as one relation row and code-graph records as compact text. The graph store shrinks 33–50% at the same base. Graph-evidence reads drop 20–60%, and every retrieval response is unchanged except one sampled prefix, explained below.
What changed
(a) One row per code edge. Before, a code edge was:
edge:<digest>entity carrying the JSONCanonicalRelationEdgeV1record, plusCodeRelationSource.<kind>(symbol → edge) andCodeRelationTarget(edge → symbol), each stored as a locator node and a native edge.Now it is one relation
edge:<digest>from its source symbol to its target symbol:CodeEdge.<kind>, so the store still filters kinds.Graph-db relation payloads now live only on the native edge; the locator's second copy is gone.
(b) Compact records. Every code-graph record (symbol, edge, file, import) is stored as compact text:
"symbol.v1.sha256:) becomeU+0001and a code.U+0002and base64url.serde_jsonescapes every control character, so neither marker can occur in the input, and decoding restores the exact JSON.Readers migrated. No reader treats an edge as an entity any more.
rgover every crate found edge-as-entity reads only intracedecay-code-index's graph projection. Each reader now takes the payload from the fan-out row:relation_keys(the #2224 key walk)semantic_neighbors(callers, callees, context related symbols, edge-kind counts)degreesCodeRelationTargetedges_amongoutgoing_relation_targetsthen an entity decodeoutgoing_relationsreader.rs)by_qualified_name, receipts, and dashboard edge views consume these reader APIs; none of them addressed edge entities directly.Format revisions. Graph-db format 4 → 5 and
CODE_GRAPH_PROJECTOR_REVISIONv9 → v10, with the same typedFormatSupersededrebuild (delete, then republish from source; no conversion).Rebuild race fix. An upgrade-in-place run found a real race in the #2257 rebuild. With a projector bump, the new generation was sealed before the staging container was opened. The superseded-format deletion then removed the whole
.sealedroot, including that fresh artifact. The first run served from memory, and the next restart had to repair. Sealed receipts now recordgraph_format, and the rebuild discards only sealed generations that do not name the current format;.staging-*seals in flight are left to their builder. After the fix, the upgraded profile keeps its sealed store (111,109,420 B) across a restart with zero repairs.Bytes
Per symbol (j768 has 98,305 symbols and 0 edges): 1,687 → 1,130 B of graph store per symbol.
Code-graph manifest of a 3-function, 3-call fixture:
Per code edge (estimate: graph store minus symbols at the j768 per-symbol cost, divided by edges):
Per store per source byte. Base is master
402b14456band after is this branch at that base (rebased ontoeedc714bf5, a contracts-only change, before merge). Both use the perf CLI, a fresh isolated profile, and one capped daemon: 6 GB, or 12 GB with 4 workers for the clone. Source isgraph_statistics.source_total_bytes.Clone peak memory fell from 10.60 GB to 8.60–9.05 GB. Time to
readywas 500 s on master and 539 s / 725 s on two runs of this branch under a machine load of 60–72. The graph was ready at about 300 s on master and 285 s on the second branch run, so the slow run reflects load variance, not the graph build.Identical results
This uses the #2185 fixture (
search,context,callers,calleesper symbol) against same-base master. Per-request fields and the profile-keyed generation id are normalized. Cost receipts are compared separately, because they are meant to improve.The one rsbuild difference is
contextforcreateRsbuild. Itsrelated_symbolslist is a truncated prefix of callers and callees in store order (context_related_relation_budgetcaps it at 16–64 rows, by design "just a prefix"). Master's prefix order came from two different relation-id orders: callees insource:<sha(edge)>order and callers intarget:<sha(edge)>order. With one row per edge, both directions followedge:<digest>order. No single row order can reproduce two unrelated hash orders, so this sample differs. Both orders are arbitrary and deterministic, and the untruncated neighbor sets are the same. The stale comment on that budget is updated.Cost receipts (fixture totals for callers and callees). Before → after, as reads / adjacency queries / rows / bytes hydrated:
#2224 receipt. On starship
handlecallees (depth 1, no dispatch), all 12 pages return the same 112 rows (identical row digest), with 11graph_sealedpoint reads per page on both sides. Adjacency drops from 2 queries / 226 rows to 1 query / 113 rows per page, andbytes_hydrateddrops by about 15%.Upgrade in place. A master-built j768 profile was restarted on this binary. It reached
readyin 22 s with statusfresh, and republished its sealed store (111,109,420 B, byte-identical in size to a fresh index) in that first run. It gave 80/80 identical responses after warm-up (WARM=1). A second restart served that store with no repair. The code-index artifacts are byte-identical.Tests: fail on master, pass here
To check that these fail on master, master's graph-db and code-index
srcwere checked out over this commit, and the tests below were run against them.graph_manifest_stores_each_code_edge_as_one_row_within_the_byte_budget(code_index_suite): builds a real parser-backed generation. It asserts 5 entities, 6 relations (3CodeEdge.callsand 3 file-symbol), record bytes ≤ 2,200, and thatedges_amongreturns alpha→beta, alpha→gamma, and beta→gamma. Master: 8 entities, 9 relations, 4,259 bytes (fails).superseded_format_store_is_rebuilt_fresh_keeping_only_current_sealed_generations(graph_db_suite): a format-4 store with a format-4 keyed row, one old sealed generation, and one sealed generation that names the current format. After open, the store is fresh, the old generation is gone, and the current one is kept. Master: opens format 4 as current (fails).single_row_edge_reads_back_its_payload_through_keys_and_fanouts(graph_db_suite): a raw format-5 store where one native edge row carries the payload and the locator carries a literal key and compact identities.relation,outgoing_relations, andoutgoing_relation_idsall return the literal relation with its payload. Master: refuses format 5 (fails).persisted_scalar_identity_mismatch_is_corrupt_on_point_read: now format 5 (fails on master).resident_accounting(the perf(code-index): build code-graph rows from sealed segments #2271 test): its row count is re-pinned from (4,201, 5,100) to (3,001, 3,900), because each of the fixture's 1,200 edges is now one relation row. Its peak-memory budget still holds.graph_projection_publication: an import assertion that parsed the raw record JSON is replaced by the existing reader assertion,external_type_import_candidatesreturning the import.Checks
On the final rebased tree, graph-db and code-index were rerun after the last rebase. The other suites ran on this diff at
402b14456b. All use--profile perf.code-index-runtime failed 4 scheduler timeouts (
Elapsed) in one run under a machine load of about 65; an isolated rerun passed 516/516.cargo clippy -p tracedecay-graph-db -p tracedecay-code-index --all-targets --features tracedecay-graph-db/test-helpers -- -D warningsis clean, andcargo fmt --all -- --checkis clean. Thetracedecay-mcpchange is a doc comment only.Remaining multiple of source
j768 is 98k one-line functions, so its per-symbol cost dominates. What remains:
metadata.occurrence), and several digests per record remain even in compact form.ENTITY_IDplus a 63-byte key.