Skip to content

A big room's first sync takes minutes instead of hours - #122

Open
tiredunc wants to merge 4 commits into
n-delic:mainfrom
tiredunc:perf/room-first-sync
Open

tiredunc wants to merge 4 commits into
n-delic:mainfrom
tiredunc:perf/room-first-sync

Conversation

@tiredunc

Copy link
Copy Markdown

After the joining fix, a guest's first sync of a big room still took hours and sat at 0% of code. Two causes, measured on a PS2 project (18,800 functions; the host sends 5,568 matches, 3,185 best attempts and 60,570 notes). (1) The drafter made machine drafts while the host's names and matches were still arriving: it fought the sync for the database (names applied 5 times slower, 2 s each instead of 0.4 s) and every draft was made again after. It now waits until the room has caught up; a project that isn't in a room is unchanged. (2) Each match, best attempt and note was its own commit, and each commit waits for the disk (13 ms here): about 250 s. They are now saved 500 at a time in one transaction each: 7 s. A transaction inside another now joins it (it used to be an error, so nothing relied on that). If a batch fails it is all undone and the next sync does it again. Tested: new tests for the nested transaction and for the drafter waiting; full suite passes except two tests that also fail on main (usage-limit time parsing, tray GTK check). Ran live on that project: the sync after joining went from hours to under 3 minutes. Known limit: a room whose first sync keeps failing keeps its drafts waiting.

Drafting while the host's names and matches arrived fought the sync for the database (names applied 5 times slower) and made every draft again afterwards.
… 16,000

Each match, best attempt and note was its own commit, and each commit waits for the disk (13 ms here): about 250 s on a big room. They are now saved 500 at a time in one transaction each (7 s), and a transaction inside another joins it.
@github-actions github-actions Bot added the enhancement New feature or request label Oct 26, 2026
@ensapra

ensapra commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

How does it compare with #121?

@tiredunc

Copy link
Copy Markdown
Author

They address different parts of the same slowdown. I split them so the straightforward index change could be reviewed separately.

#121 makes finding existing data faster. Each incoming match or note was causing a search through a whole database table. The indexes let it go straight to the relevant records.

#122 saves incoming matches, best attempts and notes in groups of 500 instead of committing each one separately. It also makes background drafting wait until the initial sync finishes. Before that, drafting was competing with the sync for database access and producing drafts that needed to be regenerated once the host’s data arrived.

With only #121, the lookups get faster, but those separate commits and competing background work remain. The two PRs work together.

On the large project I tested, the full sync after joining went from hours to under three minutes with both fixes. I haven’t measured a full sync with #121 alone, so that timing is for the combined result.

Retain both lookup-index regressions from main and the nested-transaction regression from this branch.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants