fix: capture reliability — job/write lane split, network-idle precondition, and realistic budgets

[Seanathon]

Investigating the two links that failed (twenty.com, stigg.io) turned up two independent bugs. Both are reproduced with numbers.

1. Adding a link blocked on the previous capture

Jobs ran on the same promise chain as SQLite writes, so a capture held the write lane for its whole duration — Chrome launch, page load, and a ~30s LLM round-trip, none of which write anything. addItemSkill's one-row INSERT queued behind all of it.

Measured on the real server, four links added half a second apart:

	before	after
add #1	60ms	59ms
add #2	42,360ms	54ms
add #3	41,845ms	55ms
add #4	41,342ms	59ms

The two constraints collapsed into that one chain were never the same constraint: concurrency 1 for jobs bounds memory (two Chromiums OOM the LXC, NFR-1); single-writer for SQLite bounds write interleaving, and every such write is sub-millisecond. Split into two lanes. Jobs still run strictly one at a time — verified max concurrent processing stayed 1 across four queued adds, no leaked Chrome.

This revises AD6, which documented the job queue as being the SQLite single-writer guard. Architecture doc updated. worker.test.ts's "no double serializer" case is deleted rather than weakened, since it asserted exactly the shared lane that caused this.

Note the smaller-looking fix is the unsafe one: addItemSkill awaits its write before firing the job, and that ordering guarantees the row exists before the job mutates it. Simply not awaiting would leave the job's status UPDATE hitting a missing row.

Separate lanes do open a window the shared lane had closed — capture and enrichment read the item, await the LLM, then write the full row back, so a mid-flight edit would be silently reverted. Both call sites now re-read inside a single enqueued write op. Covered by a test that patches notes during the LLM call and asserts they survive.

2. "Timed out." — the reported failure

Capture navigated with waitUntil: 'networkidle2', making a quiet network a precondition for getting an item at all. Any page holding a connection open (analytics, chat widget, video preload, websocket) never satisfies it, and a slow link does the same to an ordinary page.

Reproduced under CDP throttling: twenty.com rejected at 30755ms with TimeoutError, which cleanErrorReason maps to 'timed out' and the card renders capitalized as "Timed out." — the exact reported string, and it fires well before the job budget, matching "before actual timeout was triggered".

browser.ts's renderPageText had already learned this and carries a comment saying so; the screenshot path never got the same treatment.

Now navigates on domcontentloaded and waits for the network to settle separately and best-effort — if it expires we shoot a beat early rather than lose the item. Screenshots verified fully rendered (fonts, images, product mockups).

3. Budgets sized for what this app is

Splitting the settle out exposed the nav budget, and 30s alone still wasn't enough for a heavy page on a slow link. board-oss is single-tenant and self-hosted: nobody is queued behind you competing for the worker, so a capture that takes two minutes and succeeds beats one that fails fast and leaves you re-adding the link by hand. These are ceilings for a wedged job, not latency targets.

	before	after
navigation	30s	120s
network settle	—	30s (new, best-effort)
og:image	8s	30s
snapshot	45s	180s
CLI agent	120s	300s
capture job	180s	600s

The job budget must exceed the sum of its parts or it silently truncates them; config.test.ts asserts that relationship now instead of a magic number.

Raising the job ceiling also made one previously-survivable gap dangerous: captureLibrary's direct fetch had no timeout at all, so a hanging server would have stranded an item in processing for 10 minutes and held the serial lane. Longer budgets are only safe for bounded waits — it now has its own 60s ceiling.

Same throttle that killed twenty.com at 30.9s now captures it in 78s.

Verification

• 595 tests + typecheck green
• End-to-end against a copy of the real DB: four concurrent adds, all done, 21 enriched fields and a screenshot asset each
• Throttled captures of both reported URLs now succeed
• Live readable path re-checked (listmonk 872ms, GitHub repo 1448ms)

🤖 Generated with Claude Code

https://claude.ai/code/session_01C83mrW9X8zBLY1sSZRgdCa