Add a change feed: GET /changes, SSE stream, and list_changes - #10
Merged
Conversation
Agents coordinating across the fleet can only be woken by a background process. Polling list_docs per workspace and diffing snapshots over-fetches, holds client state, and cannot see hard deletes at all. This adds an event log with a resumable cursor so consumers learn what changed without asking repeatedly. - events table, appended in the same transaction as the mutation it describes, so a rollback cannot orphan an event - GET /changes, cursor-paged and filterable by kind - GET /changes/stream, Server-Sent Events; no new dependency, since the cursor makes reconnection stateless and removes WebSocket's main advantage - list_changes MCP tool mirroring the REST endpoint The cursor is the opaque form <workspace>:<kinds>:<xact_id>:<id>. Three review rounds went into that shape: - a bare bigserial id is not safe, because Postgres allocates sequence values outside commit order: a lower id committing after a higher one has been read is excluded forever - gating on xact_id while paging on id is also not safe, because the two orderings are independent - a cursor must be bound to the scope that produced it, or carrying one across a workspace or kind filter silently skips events Reads therefore order by (xact_id, id) and gate on xact_id < pg_snapshot_xmin(pg_current_snapshot()), which makes the visible set closed downward: nothing can appear beneath a cursor already issued. Scope mismatch is rejected rather than silently restarted. Cascading folio deletes emit one event per child via a CTE, so the mutated set and the recorded set are identical by construction rather than by two statements agreeing. Tested against a real Postgres via scripts/test.sh - bare go test skips the 37 DB-backed tests and still reports ok. The concurrency test was verified to fail against id-only ordering before the fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agents coordinating across the fleet can only be woken by a background process. Polling
list_docsper workspace and diffing snapshots over-fetches, holds client state, and cannot see hard deletes at all. This adds an event log with a resumable cursor.eventstable, appended in the same transaction as the mutation it describes, so a rollback cannot orphan an eventGET /changes— cursor-paged, filterable by kindGET /changes/stream— Server-Sent Events, emittingid:per frame and honouringLast-Event-IDlist_changesMCP tool mirroring the REST endpointThe cursor took six review rounds
It is the opaque form
workspace:kindshash:xact_id:id. Three designs were wrong before this one, each caught by adversarial review:bigserialidxact_id, page onidReads order by
(xact_id, id)and gate onxact_id < pg_snapshot_xmin(pg_current_snapshot()), which keeps the visible set closed downward — nothing can appear beneath a cursor already issued. Scope mismatch is rejected rather than silently restarted. The kinds component is a length-prefixed, full-width sha256, injective by construction.The cursor is deliberately unsigned: RLS enforces row visibility, so forging one changes where you resume, not what you can see. It is a resumption hint, not a capability.
Cascading folio deletes emit one event per child via a CTE, so the mutated set and the recorded set are identical by construction rather than by two statements agreeing.
Testing
Verified against a real Postgres via
scripts/test.sh— barego test ./...skips the 37 DB-backed tests and still printsok, which is how the first round shipped with no coverage of the new code.The concurrency test was checked to fail against id-only ordering before the fix (
skipped event B! got 0 events), on a copied tree.TestKindsHash_Injectiveuses the reviewer's exact newline counterexample.