Skip to content

feat(artifacts): add UserArtifactsIndex and backfill the keys it needs - #982

Merged
philmerrell merged 1 commit into
developfrom
feature/artifact-user-index-backfill
Sep 6, 2026
Merged

feat(artifacts): add UserArtifactsIndex and backfill the keys it needs#982
philmerrell merged 1 commit into
developfrom
feature/artifact-user-index-backfill

Conversation

@philmerrell

Copy link
Copy Markdown
Contributor

Groundwork for serving the library's user-wide listing from an index instead of a base-table Query. Nothing reads it yet — the query switch ships separately, for the deploy-ordering reason below.

Why an index at all

The base table is already partitioned by user, so this was never about reachability — list_for_user can Query PK=USER#{uid} today. It's about what that Query has to read. HEAD and version rows share the partition, so it spans roughly 3× the rows it returns, then filters and date-sorts them in memory. That in-memory sort is also what makes it unpaginable: the page boundary would fall in the wrong place.

Only HEAD rows carry GSI2PK, so the index holds one row per artifact, already newest-first.

The backfill is the load-bearing half

UserArtifactsIndex is sparse. A HEAD row without GSI2PK isn't stale in it — it's absent from it forever, silently. The writer began stamping the keys on 2026-09-04, so every row written before that has neither attribute. Switching the query without backfilling would drop every older artifact from its owner's library with no error anywhere.

backfill_artifact_user_index_keys.py stamps them: dry-run by default, idempotent (attribute_not_exists(GSI2PK), so it also yields to the writer), and it never resurrects a row deleted mid-run (attribute_exists(SK)).

It reads updated_at to build GSI2SK and never assigns it — that attribute is embedded in both GSI sort keys and is writer-owned, the same restraint rename observes. A HEAD row lacking updated_at is reported by name rather than stamped with a fabricated timestamp that would sort it wrongly forever.

Already run on dev

dry run : HEAD rows=22 already=0 stamped=22 skipped=0 failed=0
--apply : HEAD rows=22 already=0 stamped=22 skipped=0 failed=0
re-run  : HEAD rows=22 already=22 stamped=0 skipped=0 failed=0

Verified independently afterwards: 0 rows missing GSI2PK, 0 version rows wrongly stamped, and GSI1SK == GSI2SK on all 22. That last one is the useful check — GSI1SK is the writer's own ARTIFACT#{updated_at}#{aid}, so the agreement validates the backfill's format against production-written data rather than against my reading of the code.

Deploy notes

  • Adding this is one UpdateTable, and only one GSI may be added per UpdateTable. The committed gsi-inventory.json gains exactly one line, so no release split needed.
  • CFN UPDATE_COMPLETE ≠ index ACTIVE — DynamoDB backfills asynchronously. This is why the query switch is a separate PR: platform.yml and backend.yml share a concurrency group but their order isn't enforced, so shipping both together risks app-api querying an index that doesn't exist yet.
  • Prod needs the script run too, before the query switch reaches it. It's safe to run before or after the index exists — adding the attributes to a row causes DynamoDB to index it either way; running first just means the index is complete the moment it reports ACTIVE.

Tests

Infrastructure 784 passed. Backend 186 passed across the artifact suites, including 10 new for the script — weighted toward what it refuses to do (fabricate a sort key, touch updated_at, stamp version rows, resurrect a deleted row), since those are the silent failures.

🤖 Generated with Claude Code

Groundwork for serving the library's user-wide listing from an index
instead of a base-table Query. Nothing reads it yet — the query switch
ships separately, for the deploy-ordering reason below.

## Why an index at all

The base table is already partitioned by user, so this was never about
reachability: `list_for_user` can Query PK=USER#{uid} today. It is about
what that Query has to read. HEAD and version rows share the partition,
so it spans roughly 3x the rows it returns, then filters and date-sorts
them in memory — which is also what makes it unpaginable, since the page
boundary would fall in the wrong place. Only HEAD rows carry GSI2PK, so
the index holds one row per artifact, already newest-first.

## The backfill is the load-bearing half

`UserArtifactsIndex` is sparse. A HEAD row without GSI2PK is not stale
in it — it is absent from it forever, silently. The writer began
stamping the keys on 2026-09-04; every row written before that has
neither attribute, so switching the query without backfilling would drop
every older artifact from its owner's library with no error anywhere.

`backfill_artifact_user_index_keys.py` stamps them. Dry-run by default,
idempotent (`attribute_not_exists(GSI2PK)`, so it also yields to the
writer), and it never resurrects a row deleted mid-run
(`attribute_exists(SK)`).

It reads `updated_at` to build GSI2SK and never assigns it — that
attribute is embedded in both GSI sort keys and is writer-owned, the
same restraint `rename` observes. A HEAD row lacking `updated_at` is
reported by name rather than stamped with a fabricated timestamp that
would sort it wrongly forever.

Run on dev: 22 HEAD rows, 22 stamped, 0 skipped, 0 failed; re-run
reported 22 already-stamped, 0 stamped. Verified independently
afterwards: 0 rows missing GSI2PK, 0 version rows wrongly stamped, and
GSI1SK == GSI2SK on all 22 — GSI1SK is the *writer's* own
`ARTIFACT#{updated_at}#{aid}`, so that agreement checks the backfill's
format against production-written data rather than against a reading of
the code.

## Deploy notes

Adding this is one `UpdateTable`, and only one GSI may be added per
`UpdateTable` — the committed `gsi-inventory.json` gains exactly one
line here, so no release split is needed.

CFN reporting UPDATE_COMPLETE does NOT mean the index is ACTIVE;
DynamoDB backfills it asynchronously. That is why the query switch is a
separate PR: platform.yml and backend.yml share a concurrency group but
their order is not enforced, so shipping both together risks app-api
querying an index that does not exist yet.

Infrastructure: 784 passed. Backend: 186 passed across the artifact
suites, including 10 new tests for the script.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@philmerrell
philmerrell merged commit 26913c0 into develop Sep 6, 2026
4 checks passed
@philmerrell philmerrell mentioned this pull request Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant