Skip to content

Avoid loading unused columns during card JSON assembly - #6092

Draft
christse wants to merge 1 commit into
mainfrom
codex/narrow-card-json-reads
Draft

christse wants to merge 1 commit into
mainfrom
codex/narrow-card-json-reads

Conversation

@christse

Copy link
Copy Markdown
Contributor

Background and Goal

Card JSON assembly still performs broad index reads: individual card reads and linked-card/file expansion select i.* plus every rendered HTML format and markdown. Healthy card assembly does not use most of those columns, but Postgres transfers them and the driver parses the JSONB values before Node discards them. Root search results already have a narrower projection; this change targets the follow-up entry reads.

Add an internal dataOnly projection and use it in cardDocument() and the batched lookups in loadLinks(). Full reads remain the default for other callers. Query predicates, graph traversal, authorization, response formatting, and the public API are unchanged. No migrations or card-definition changes.

Where to start

  • packages/runtime-common/index-query-engine.ts: entryColumns() selects the columns needed for JSON assembly. Healthy instances omit their search documents and rendered output; file rows retain search attributes. Both retain dependencies, timestamps, and screenshots. Effective source/render errors retain their error document, title source, and last-known-good isolated HTML.
  • packages/runtime-common/realm-index-query-engine.ts: opt the card-document read and the existing linked-card/file batches into that projection.

Handoff

The implementation is deliberately small: 35 added production lines across two files, plus focused regression coverage. The new database tests compare full and narrow results using multi-megabyte unused fields, cover canonical URLs/aliases, production and working tables, missing/deleted rows, file metadata, screenshots, and source/render error precedence. The existing linked-card integration test now also checks that its actual database results omit unused fields.

Locally, all 50 tests in index-entry-projection-test and prerender-html-split-test pass. pnpm lint passes in runtime-common and realm-server. A separate SQLite query smoke check passed for all four entry methods and both table pairs; this does not substitute for the host/browser suite.

Before taking this out of draft:

  • Run the updated load-links-batching-test and existing card/file endpoint and host tests in a healthy test environment. The local services required for browser integration returned 502, so that integration test was not run here.
  • Profile representative search/card workloads before and after, including concurrent readers. This patch reduces avoidable database transfer and allocation; it does not eliminate necessary pristine-document parsing, graph assembly, or final JSON serialization, and does not yet establish a production latency improvement or resolution of the 502/503 incidents.

@github-actions

github-actions Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Host Test Results

    1 files      1 suites   2h 6m 8s ⏱️
4 739 tests 4 725 ✅ 14 💤 0 ❌
4 754 runs  4 740 ✅ 14 💤 0 ❌

Results for commit 088b3ce.

Realm Server Test Results

    1 files    215 suites   1h 14m 52s ⏱️
2 815 tests 2 815 ✅ 0 💤 0 ❌
2 854 runs  2 854 ✅ 0 💤 0 ❌

Results for commit 088b3ce.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant