Skip to content

[Push] Store file indexes under target relationship directories - #413

Open
adamziel wants to merge 2 commits into
adamziel/advance-previous-local-indexfrom
adamziel/remote-index-state-directory
Open

[Push] Store file indexes under target relationship directories#413
adamziel wants to merge 2 commits into
adamziel/advance-previous-local-indexfrom
adamziel/remote-index-state-directory

Conversation

@adamziel

@adamziel adamziel commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

This puts each target URL in its own remote-<hash>/ directory, so files-pull, files-diff, and files-push share one relationship-local index without colliding with another target.

The hash uses the target URL after removing URL user-info, SECRET_KEY, and the site-export-api endpoint alias. It does not include the local tree. A caller which uses another local document root must use another --state-dir.

The pull files now use .remote-index.jsonl, .remote-index.next.jsonl, .local-index.jsonl, pull-plan.jsonl, and pull-index-updates.wal. WAL records carry explicit remote and local fields; replay updates the remote projection first, then the local projection, and clears the WAL only after both replacements complete.

The relationship directory owns the file indexes and push state. Old index filenames are removed rather than kept as aliases.

Stacked on #406.

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Pull pipeline performance — large-directory

Site: large-directory · 2,000+ plus targeted file-transfer scenarios files · 10,000 posts · 25,000 postmeta · PHP 8.5.9

Stage PR trunk Δ Status Details
playground-sqlite-db-pull 7.23 s 7.15 s ⚪ +85 ms (+1.2%) condition=db-pull in PHP.wasm
runtime=php.wasm 8.3
wp_mysql_parser=enabled
mode=lexer
native_lexer=verified
native_token_stream=WP_MySQL_Native_Token_Stream
native_token_count=18
native_parser=selected
trunk: condition=db-pull in PHP.wasm
runtime=php.wasm 8.3
wp_mysql_parser=enabled
mode=lexer
native_lexer=verified
native_token_stream=WP_MySQL_Native_Token_Stream
native_token_count=18
native_parser=selected
playground-sqlite-db-apply 2.86 s 2.87 s ⚪ -5 ms (-0.2%) condition=db-apply to SQLite in PHP.wasm
runtime=php.wasm 8.3
wp_mysql_parser=enabled
mode=parser
native_lexer=verified
native_token_stream=WP_MySQL_Native_Token_Stream
native_token_count=18
native_parser=verified
native_ast=WP_MySQL_Native_Parser_Node
sqlite_driver_parser=verified
trunk: condition=db-apply to SQLite in PHP.wasm
runtime=php.wasm 8.3
wp_mysql_parser=enabled
mode=parser
native_lexer=verified
native_token_stream=WP_MySQL_Native_Token_Stream
native_token_count=18
native_parser=verified
native_ast=WP_MySQL_Native_Parser_Node
sqlite_driver_parser=verified
Total 10.10 s 10.02 s ⚪ +80 ms (+0.8%)

Numbers carry runner noise; treat single-run deltas as directional, not authoritative.

📈 Trunk performance history — commit-by-commit timeline.

@adamziel
adamziel force-pushed the adamziel/remote-index-state-directory branch 3 times, most recently from f3084d9 to 25e0c48 Compare July 30, 2026 10:27
@adamziel
adamziel force-pushed the adamziel/remote-index-state-directory branch from 25e0c48 to 77106b6 Compare July 30, 2026 10:36
@adamziel
adamziel force-pushed the adamziel/advance-previous-local-index branch 6 times, most recently from 99b3cd6 to 63f989a Compare July 31, 2026 02:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant