Skip to content

fix: self-heal claims/earnings indexes (#886) + accepted signal filter (#873) - #896

Merged
biwasxyz merged 2 commits into
mainfrom
fix/schema-selfheal-and-accepted-filter
Jul 29, 2026
Merged

fix: self-heal claims/earnings indexes (#886) + accepted signal filter (#873)#896
biwasxyz merged 2 commits into
mainfrom
fix/schema-selfheal-and-accepted-filter

Conversation

@biwasxyz

Copy link
Copy Markdown
Contributor

Two independent fixes, one commit each — separable if you'd rather land them apart.

Closes #886
Closes #873


#886 — self-heal claims/earnings hot-path indexes

The issue asks to move both indexes into SCHEMA_SQL. That would brick DO construction, so this does something different.

SCHEMA_SQL runs before any migration, and neither target exists yet on a fresh DB:

Index Depends on Created by
idx_beat_claims_status_beat_address beat_claims table migration 8 (schema.ts:418)
idx_earnings_unpaid_leaderboard earnings.voided_at migration 9 (schema.ts:440)

A base-schema index on either would throw no such table / no such column and brick the constructor before a migration could repair it — the same trap already documented for idx_signals_quality_score at schema.ts:152-156. That is very likely why the original TODO hedged with "once /schema-health confirms."

Instead: a post-migration self-heal pass (SELF_HEAL_POST_MIGRATION_INDEXES_SQL) that runs after the migration block on every cold start.

  • Deliberately not gated on appliedVersion. That is the entire point — a version-gated migration that fails once is never retried, because the counter advances regardless. That is how the April incident (391.5B rows read, $391.52) silently dropped the signals composites and turned hot reads into full-table scans.
  • Each statement is individually try/caught, so a genuinely absent table can never brick construction.
  • /schema-health now reports missing_post_migration_indexes separately from the signals set, so a drift report still tells you which layer drifted. healthy folds in both.

Tests pin the two hand-maintained lists (SQL vs. EXPECTED_*) together so they cannot drift, and assert every statement is idempotent.

#873accepted status filter

approvedbrief_included on compile means neither exact status alone yields the durable accepted set, and an item silently drops out of a status=approved query once compiled. Adds a virtual status=accepted expanding to s.status IN ('approved', 'brief_included').

  • SIGNAL_LIST_STATUS_FILTERS is a separate constant, not an addition to SIGNAL_STATUSES — the latter backs the SignalStatus type (types.ts:210) and the review-transition write path (news-do.ts:1764), and accepted is never a stored state.
  • Counts unaffected: querySignalCountRows iterates concrete statuses and takes no single status filter.
  • Explicit IN list (not an inequality) so SQLite still uses idx_signals_status_created.

Covers all three acceptance cases from the issue, plus the 400 path.

Verification

  • tsc --noEmit clean
  • vitest run516 passed / 49 files, no new lint errors
  • The two commits were split from a single working state and verified to reconstruct it byte-identically

Not included

The MCP news_list_signals status enum lives in aibtc-mcp-server, so accepted is not selectable over MCP until that schema is updated separately.

biwasxyz added 2 commits July 29, 2026 14:35
…ons (#886)

The April billing incident (391.5B DO rows read, $391.52) was caused by
hot-path composite indexes silently going missing in production: a
version-gated migration that fails once is never retried, because the
version counter advances regardless. The signals composites were fixed by
moving them into the always-re-applied SCHEMA_SQL; the claims/earnings
indexes were left as a TODO and still carried the same risk.

They cannot follow the signals fix into SCHEMA_SQL. That block runs before
any migration, and neither target exists yet on a fresh DB — beat_claims is
created by migration 8, and earnings.voided_at is added by migration 9. A
base-schema index on either would throw "no such table" / "no such column"
and brick DO construction before a migration could repair it. That is the
same trap already documented for idx_signals_quality_score.

Re-apply them after the migration block instead. The pass is deliberately
NOT gated on appliedVersion — running on every cold start is what gives the
self-heal property. Each statement is individually guarded so a genuinely
absent table can never brick the constructor.

- SELF_HEAL_POST_MIGRATION_INDEXES_SQL covers
  idx_beat_claims_status_beat_address and idx_earnings_unpaid_leaderboard
- /schema-health reports missing_post_migration_indexes separately from the
  signals set, so a drift report still says which layer drifted
- tests pin the two hand-maintained lists together and assert every
  statement is idempotent
…_included (#873)

The public lifecycle moves an editorially accepted signal from `approved` to
`brief_included` when a brief compiles, so callers cannot query the durable
accepted set with one filter — and the same item silently drops out of a
`status=approved` query once it is compiled, making retrospective checks
unstable.

Add a virtual `status=accepted` list filter that expands to
`s.status IN ('approved', 'brief_included')`. Existing exact-lifecycle
filters are untouched.

SIGNAL_LIST_STATUS_FILTERS is deliberately a separate constant rather than an
addition to SIGNAL_STATUSES: the latter backs both the SignalStatus type and
the review-transition write path, and `accepted` is not a state a signal can
ever be stored in. Signal counts are unaffected — querySignalCountRows
iterates concrete statuses and never takes a single status filter.

The expansion uses an explicit IN list so SQLite can still use
idx_signals_status_created instead of falling back to a scan.

Note: the MCP `news_list_signals` status enum lives in aibtc-mcp-server, so
`accepted` is not selectable over MCP until that schema is updated separately.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
agent-news 191b143 Jul 29 2026, 08:52 AM

@github-actions

Copy link
Copy Markdown
Contributor

Preview deployed: https://agent-news-staging.hosting-962.workers.dev

This preview uses sample data — beats, signals, and streaks are seeded automatically.

@arc0btc arc0btc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two focused, independently-shippable fixes — both come with the "why not the obvious approach" reasoning already written into the code, which makes this easy to review.

What works well:

  • #886 correctly identifies that the issue's own suggested fix (move the indexes into SCHEMA_SQL) would brick DO construction on a fresh database, since beat_claims (migration 8) and earnings.voided_at (migration 9) don't exist yet at SCHEMA_SQL time. Re-applying post-migration, un-gated on appliedVersion, mirrors the exact mechanism that caused the April incident this fixes (a migration that silently fails once and is never retried because the version counter still advances). Each statement is individually try/caught so it degrades to a logged no-op rather than a crash.
  • #873's SIGNAL_LIST_STATUS_FILTERS as a separate constant from SIGNAL_STATUSES is the right call — keeping accepted out of the type that backs the review-transition write path avoids a virtual, non-storable status ever leaking into a place that expects a real one.
  • The explicit IN (?, ?) expansion for the accepted filter (rather than a != or NOT IN) preserves idx_signals_status_created usability — good performance awareness, called out directly in the comment.
  • Test coverage is thorough on both sides: the schema-migration tests pin the SQL statement list against the EXPECTED_* name list so they can't silently drift, and assert every statement is idempotent (IF NOT EXISTS) since the pass isn't version-gated. The signals test covers the compile-transition scenario (approved + brief_included both present, rejected excluded) plus the 400 path advertising accepted as valid.
  • /schema-health reporting missing_post_migration_indexes separately from missing_signals_indexes is a good diagnostic choice — a future drift report will say which layer failed instead of just "something's missing."

[question] this.ctx.storage.sql.exec(stmt) catches and logs on failure (news-do.ts:257-261) but doesn't distinguish "table doesn't exist yet" (expected/benign on an old DO that hasn't run migration 8/9) from a genuine unexpected failure (e.g. a typo in the SQL). Does the existing signals-index self-heal (the idx_signals_quality_score precedent this mirrors) have the same blind spot, or does it check the error type before swallowing? If it's already accepted for that path, no change needed here — just flagging since a real bug in the CREATE INDEX statement would be silently invisible except in logs, and /schema-health would keep reporting it as unhealthy without surfacing why.

[nit] The PR body says "391.5B DO rows read, $391.52" for the April incident — worth double-checking that figure made it into a durable doc (postmortem/runbook) somewhere outside this commit message, since commit messages are easy to lose track of when searching for incident history later.

Code quality notes: Nothing to add beyond the above — the two changes are appropriately separated (constants vs. schema vs. routing), no dead code, and the self-heal pattern reuses the existing signals-index approach rather than inventing a new one.

Operational context: We don't run this DO directly, but the failure mode described (version-gated migration silently not retried, hot-path index missing, full-table scans) is a pattern worth remembering generally — it's the kind of thing that stays invisible until a billing spike, exactly as happened here in April.

Approving — all five checklist dimensions covered, tests are solid, and the design correctly avoids the bricking trap the linked issue's suggested approach would have hit.

@biwasxyz
biwasxyz merged commit ac9af00 into main Jul 29, 2026
7 checks passed
@biwasxyz
biwasxyz deleted the fix/schema-selfheal-and-accepted-filter branch July 29, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants