Skip to content

Commit aa56857

Browse files
feat: send the flow bundle as one xz document per side (#171)
* feat: send the flow bundle as one xz document per side Large repos got no flows or merge risk: crate's bundle was 24.6 MB of gzip, over the service's 4.3 MB limit, because every fact went four times. 149,315 of its 149,316 analysis edges are call-graph edges, nested relations repeat the top level, and the methods index and the graph's nodes are one table. - `side_document()` writes each commit as one document, `head.json` and `base.json`, in place of the two analyses and two call graphs. It holds a row per method or graph node, calls by caller with their lines, the component tree, and only the analysis edges the calls do not imply. The service's `bundle.expand_side` reads it back into every field a feature reads. - `_compress` is xz -9e instead of gzip: its window spans the bundle, so the base side costs little next to a head that repeats most of it. The briefing request shares it. - A changed file over 300 KB is sent at neither commit (the same limit as source). On crate the bundle is 1.42 MB. Needs CodeBoarding/licensing-aws#56 deployed first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: send only the analysis edges that are no call The service reads every call already, so the analysis edges that repeat one were rebuilt on its side from leaf membership and then corrected with hidden_edges. Without them crate's guide holds the same content and its peak memory drops from 1.8 to 1.3 GB, so extra_edges is now just the analysis edges that are no call, and the derivation and hidden_edges go. call_graph() no longer builds what side_document dropped: `callable`, and each site's file and column. It returns each call's lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: rename closed_features to remote_features and make it readable The script calls the service's review-guide and architecture endpoints; nothing about it is closed. scripts/remote_features.py replaces scripts/closed_features.py, and the action step, problem_history's import and the tests follow. The public input closed_features_source keeps its name, since renaming it would break workflows that set it. The rewrite uses descriptive names and short functions: side_document is now a _Table of _Row objects plus _calls, _components and _analysis_edges, and collect() reads changed files through _changed_contents. The module docstring says what a request sends, why it is packed as side documents under xz, and each size limit. The side documents, collect() output and request body are byte-identical to before on 42 cases, crate's head and base with its real call graph among them, and call_graph() reads two real pickles the same. One fix: a parent's members were left out when they equalled its leaves' members, but the service rebuilds a parent from its direct children. A parent holding fewer methods than a child read back with the child's extra methods. It now compares with its children's members. No analysis in the 42 cases hit it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: read the side documents' analyses through the installed engine remote_features.py parsed the legacy analysis.json itself: methods_index, each component's file_methods and every relation's all_edges. It now reads an analysis only through the pinned release's parse_unified_analysis, the same engine that unpickles the call graph beside it. The review analyses both commits with that release, so both are always in the format it reads, and a format change in Core needs no change here. The check that a methods_index key is its file and qualified name goes: the keys are built from Core's method entries. InstalledCoreTests runs in the core-compatibility job against the pinned release. tests/fixtures/analysis holds one analysis as Core 0.15.5 and Core main (9cf50fb9, after #571) wrote it. Each release reads only its own, and every fixture it reads must give the same side document. Checked: on CodeBoarding's own analysis (two baseline syncs, 153 files, 22 components) and this repo's, the side document is byte-identical to the one built from the raw document, and Core main reading the v1 rewrite gives the same bytes as 0.15.5 reading the legacy original. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: check requests against both of the service's size limits before sending The service refuses a request on two sizes: what is sent (4.3 MB of xz, the Function URL's limit) and what it decompresses to (256 MiB of JSON for a feature, 32 MiB for a briefing). The Action checked only the first, so a briefing of 32-43 MiB of JSON, which xz takes under 4.3 MB, went out whole, was refused, and went out again on every sync. _encode now enforces both in one step: it measures the JSON, which is cheap, then compresses once. The limits sit together and name the service's constants they follow. The briefing leaves findings out against the JSON limit first, largest first, counted per entry, and only recompresses when the compressed size is still over, aimed by the ratio it just measured, at most 3 compressions in all. It used to recompress once per dropped finding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf: compress each feature request once, one at a time The guide and architecture each compressed their own body in parallel, nearly the same bytes twice, and architecture compressed again after dropping source.json. At preset 9 | EXTREME each compressor takes about 700 MB. Both features can send the body without source, so it is compressed once and carries why source is missing, which only architecture reads. Architecture's body with source is compressed only when there is source; when it does not fit, the shared bytes stand in. A run compresses once without source and at most twice with it, and never two at a time. The JSON check needs no ratio guess: the 256 MiB limit is 62 times the 4.3 MB one, so it already turns away whatever could only fit at better than 62:1, and no tighter cut-off holds for every request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: build the review diff from Core's parsed analysis, not the document load_analysis already parsed both sides with the pinned release's parse_unified_analysis and rolled relations up with its project_relations_to_level, but the diff then read members, file hashes and method hashes from the raw document: file_methods as {file_path, methods} groups and methods_index. Core #571's "v1" document has neither, so once the Action pins a release with it, the review diff fails with "'str' object has no attribute 'get'". The diff now reads only what Core parsed: each component's file_methods, the root's file index and Core's projected relations at every level. The review analyses both commits with the pinned release, so both sides are always in the format it reads, and a format change in Core needs no change here. The core-compatibility job rendered the committed .codeboarding/analysis.json, which after a pin moves can still be in the format of the release before, which the new one refuses. It now runs InstalledCoreTests: tests/fixtures/ analysis holds one analysis as Core 0.15.5 and Core main (9cf50fb9) wrote it, each release reads only its own, and every fixture the pinned one reads must give the same view. Checked on CodeBoarding's own analysis (two baseline syncs, 22 components, depth 1 and 2) and this repo's: output and summary are byte-identical to main's script on the legacy documents with 0.15.5, and to that again on their v1 rewrites with Core main, where main's script fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 5334187 commit aa56857

7 files changed

Lines changed: 1103 additions & 538 deletions

File tree

‎.github/workflows/test.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,8 @@ jobs:
4242
python -m pip install --disable-pip-version-check "$requirement"
4343
- name: Check the provider table against the installed release
4444
run: python -m unittest tests.test_provider_table_drift -v
45+
- name: Build side documents with the installed release
46+
run: python -m unittest tests.test_remote_features.InstalledCoreTests -v
4547
- name: Install sync artifacts with the installed CodeBoarding release
4648
shell: bash
4749
env:

‎action.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -736,8 +736,8 @@ runs:
736736

737737
# After the artifact holds the analysis and before it is uploaded: each feature is one more
738738
# file in it. Uses production by default; an empty URL disables it. It never fails the run.
739-
- name: Run closed features
740-
id: closed_features
739+
- name: Run remote features
740+
id: remote_features
741741
if: steps.guard.outputs.skip != 'true' && steps.guard.outputs.mode == 'review' && inputs.review_guide_url != ''
742742
continue-on-error: true
743743
shell: bash
@@ -777,7 +777,7 @@ runs:
777777
REQUESTY_BASE_URL: ${{ inputs.requesty_base_url }}
778778
VERCEL_API_KEY: ${{ inputs.vercel_api_key }}
779779
VERCEL_BASE_URL: ${{ inputs.vercel_base_url }}
780-
run: python3 "$GITHUB_ACTION_PATH/scripts/closed_features.py"
780+
run: python3 "$GITHUB_ACTION_PATH/scripts/remote_features.py"
781781

782782
- name: Upload review artifact
783783
id: upload_review_artifact_dotcom

0 commit comments

Comments
 (0)