Skip to content

Keep the blocks an archive's index was read from, so a worker started again reads nothing - #153

Merged
alekswebnet merged 1 commit into
mainfrom
feat/keep-index-blocks
Nov 7, 2026
Merged

alekswebnet merged 1 commit into
mainfrom
feat/keep-index-blocks

Conversation

@alekswebnet

Copy link
Copy Markdown
Contributor

Closes #136.

The blocks an archive's index was read from lived only in the Service Worker's memory. Chromium stops an idle worker after about thirty seconds, so the usual revisit found that cache empty. Before the worker could answer anything, it read the archive's tail and head again, one request after the other.

What changes

  • Keeping the blocks. At the end of each index for a whole reader over HTTP, the worker writes the blocks that reader's small reads went through into the package's cache. That covers the index, content.json, and whatever the plan and the check read; the player names no file. The blocks are stored with the validator and size they were read under (keepIndexBlocks, ChunkStore.setIndexBlocks, blocksReadBy).
    • The write happens after the reply, so the load does not wait for it.
    • It is written again only when a load read a block beyond those already kept.
    • A reader whose frame has booted keeps nothing more: what it reads after that is the runtime's, not the index's.
  • Putting them back. openReader puts the blocks back into the block cache before zip.js opens the archive, when the validator and size still match (seedIndexBlocks, seedBlocks, BlockCache.seed). zip.js then reads the index from memory, as a reader rebuilt after attach-libraries already does.
  • Only with a validator, an ETag or Last-Modified the page can read. Without one, an archive republished at the same URL keeps its package id, and old blocks would send zip.js to the old offsets. Such a host reads the index again, as now.
  • Never the authority. A copy that is missing, cut short, or holds a block of the wrong length for its place is ignored, and the bytes come from the network. The copy goes with the package's cache when that is evicted. It is stored at <package>/index-blocks, beside the entries, not among their chunks.
  • The /counted/ test host gives an ETag when a test asks with &etag=1.
  • Docs: the rule in design/serving.md (Small reads over HTTP go through a block cache…) and its line in AGENTS.md. docs/setup.md now asks a host on another origin to expose ETag (or Last-Modified).

Checked

  • Timings on the demo's quiz, from a host paced to 1 MB/s with 100 ms added to every request, with the worker stopped through CDP between two loads:

    Player Requests on the reload Ready
    main 4: probe, tail, head, one more block 0.64 s
    this branch, with ETag 1: the probe 0.21 s
    this branch, no validator 4 0.65 s

    First loads are unchanged: five requests, ready in 2.3 to 2.6 s.

  • tests/browser-emulated/worker-stopped.test.ts stops the worker through CDP and counts the reload's requests at /counted/. With an ETag, the probe is the only request; without the change, bytes=0-3378 follows it, and the test fails. Without a validator, the index is read again.

  • Unit tests:

    • BlockCache hands over a key's blocks and puts them back, and leaves out a block of the wrong length or one it already holds.
    • The chunk store's record reads back whole, and a truncated one reads as none.
    • range-http-handle: a worker started again costs nothing with the blocks put back, and the tail and the head without them.
  • Suites: the player's suite passes, 604 tests across the unit, browser and emulated projects. pnpm test passes across the workspace (embed 81, skills 73, verify 27, demo 21, and the rest).

Not done, as the issue says: the warm's whole entries already have this gap for an archive republished without a validator, and nothing clears a package's cache when its archive changes.

Release: in the player's next minor.

The block cache under small reads over HTTP lives in the Service
Worker's memory, and Chromium stops an idle worker after about thirty
seconds, so a reload read the archive's tail and head again before the
worker answered anything. At the end of each index of a whole reader
over HTTP, the worker now writes the blocks that reader's small reads
went through into the package's cache, with the validator and size they
were read under, and openReader puts them back before zip.js opens the
archive. Only with a validator; a copy that does not match or does not
read back whole is read around.

On the demo's quiz, from a host paced to 1 MB/s with 100 ms on every
request, a reload after the worker stopped is the probe alone, ready at
0.21 s, where it took four requests and 0.64 s.

Closes #136.
@alekswebnet
alekswebnet merged commit a82e4a8 into main Nov 7, 2026
9 checks passed
@alekswebnet
alekswebnet deleted the feat/keep-index-blocks branch November 8, 2026 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Once the browser stops the worker, a reload reads the index again: keep its blocks with the package

1 participant