server: /branches/merge is a synchronous long-running POST — a client timeout orphans the merge, and a concurrent retry double-applies it (duplicate rows past @key)

[FlorianSeemann]

Summary

POST /branches/merge executes a potentially long-running operation (cost scales with the target table — #384) behind a single synchronous HTTP request. Two consequences on v0.8.1:

1. Orphaned server-side work. When the merge outlives the client's read timeout, the client gives up but the server keeps executing. The client cannot tell whether the merge applied, is still running, or failed — there is no job handle to poll and no way to re-attach.
2. A retry races the still-running first request and BOTH apply. Standard HTTP clients retry timed-out requests. We observed two executions of the same merge race each other: both computed the same insert-diff from the same pre-merge base, and both committed. The target branch ended up with duplicate rows for the same @key slug — the uniqueness the schema promises was violated by the write path itself.

Real-world impact (production, v0.8.1)

Graph: ~1.7M nodes incl. a 750k-row Vector(3072) table; storage on self-hosted MinIO (Railway); server on an 8 GB host. During a period of storage I/O pressure, merges slowed past the ETL client's 60 s timeout; the client's timeout-retry then double-applied a connector merge. The full fan-out of two source records (15 nodes) landed twice on main. Every subsequent merge touching those slugs failed 409 unique_violation, wedging the hourly ingestion pipeline until the rows were manually deleted and re-ingested. A sibling symptom under the same conditions: an abandoned first merge colliding with the next leg's merge as 500 storage: Concurrent modification: table version N already exists.

Ask

Preferably both:

• Async job API: POST /branches/merge returns 202 { job_id } immediately; GET /branches/merge/{job_id} reports running | merged | conflict | failed. No orphaned work, clients can wait arbitrarily long, and a re-POST of the same (source, target) while a job is running returns the existing job instead of starting a second one.
• Serialization + at-most-once per graph: even with the sync API, two in-flight merges of the same source into the same target must not both apply. Serializing merges per target (and/or making the uniqueness check transactional with the merge commit) would turn the retry into already_up_to_date instead of a duplicate insert.

Related: #384 (merge cost is what makes timeouts likely), #282 (Concurrent modification on merge), companion issue on idempotency keys for retried writes.

Workaround we use today

Client-side: never retry /branches/merge (single attempt, long timeout), and route bulk ETL merges bucket-direct via the CLI from a single-writer CI lane — which gives up the server's actor governance, so it's a workaround, not a fix.