Skip to content

Make an idle tick ask only whether anything changed - #5

Merged
jeffwall-curlewlabs merged 2 commits into
mainfrom
feat/idle-tick-cache
Oct 5, 2026
Merged

jeffwall-curlewlabs merged 2 commits into
mainfrom
feat/idle-tick-cache

Conversation

@jeffwall-curlewlabs

Copy link
Copy Markdown
Contributor

Closes #4. Stacked on #3: it builds on #3's clock and is based on its branch, so this diff shows only the change below. When #3 merges, GitHub retargets this to main.

Why

Every tick downloaded the full text of every workflow file on the default branch and parsed each with yq, though the files change a few times a day and a clock ticks every few minutes. On a repository with 52 workflow files that is about 1.3 MB and 52 yq runs per tick, per host.

What changes

default_branch.py replaces the all-text read, for the clock and for missed:

  • Every read asks for the id of the default branch's .github/workflows tree. That id changes only when a workflow file changes.
  • When the tree was read before, its entries and each file's parse come from a cache keyed by content id, and nothing else is fetched. No yq process starts.
  • When it changed, the entries are listed by the tree's id, so they describe exactly that tree. Only files whose blob is new are fetched, in one batched query, and parsed.
  • The cache is only a cache. Every key is a content id, so it is never stale.
    • A cached text is re-hashed against its blob id on every read, and anything unreadable is a miss.
    • Deleting the cache costs one full read and changes no decision.
    • Entries unused for 30 days are pruned.
    • It lives under $XDG_CACHE_HOME/github-cron-trigger.
  • Parses are keyed by blob id and PARSE_FORMAT, not yq's version. The yq floor exists so supported releases read workflows alike, and an idle tick should start no process.

Checked

  • Unit tests, with canned GraphQL answers and real yq:

    • an idle read is one query and no parse;
    • a changed file is the only one fetched and parsed;
    • a lost cache costs a full read with the same result;
    • corrupted texts, listings and parses are misses;
    • pruning keeps what is in use.
    • 157 tests pass, and ruff, strict mypy and actionlint are clean.
  • The live check now reads this repository twice and requires the second read to be one query with no parse.

  • Measured against a real repository with 52 workflow files:

    Read Queries Downloaded Time yq
    Cold 4 1.3 MB 3.1 s runs
    Idle 1 79 bytes 0.6 s none

    The cache held 1.9 MB.

The clock remembered what it had sent in a local state file, which tied
delivery to one host: while that host slept every slot waited for GitHub's
backstop, and a second host would have sent every slot again.

A clock now claims a slot before sending it, by creating a ref under
refs/github-cron-trigger-clock/ in the served repository. Ref creation is
atomic, so of any number of clocks ticking at once exactly one sends each
slot. A failed send releases its claim, so the next tick on any host
retries. A line with no marks is baselined; a workflow that stops being
enrolled loses its marks, so enrolling it again baselines it again; each
line keeps its two newest marks. Marks of a workflow file that could not be
read this tick are left alone.

`--state` is gone: a host keeps nothing. Without `--send` a tick is a dry
run that reports what it would do and writes nothing.

The ledger's ref primitives (create with a note, delete, list) are shared
with the marks. CI's live check runs two clocks at once against a workflow
enrolled for it, clock-check.yml, and checks each slot is sent once.

Closes #2.
Every tick downloaded the full text of every workflow file on the default
branch and parsed each with yq, though workflow files change a few times a
day and a clock ticks every few minutes. For a repository with 52 workflow
files that is about 1.3 MB and 52 yq runs per tick.

A tick now asks for the id of the default branch's `.github/workflows`
tree. When the tree is one it has read before, everything comes from a
cache keyed by content id - the tree's entries, each file's parse - and
nothing else is fetched or parsed. When it changed, only the files whose
blob is new are fetched, in one batched query, and parsed. Cached texts are
checked against their git blob id on every read, and anything unreadable is
a miss, so the cache can never be stale and deleting it changes no
decision. Entries unused for 30 days are pruned.

Measured on that 52-file repository: a cold read is 4 queries, 1.3 MB and
3.1 s; an idle read is 1 query, 79 bytes, 0.6 s and no yq.

Closes #4.
@jeffwall-curlewlabs

Copy link
Copy Markdown
Contributor Author

Merged with main's branch protection removed for the merge and restored straight after, on the maintainer's call. One required check, test (ubuntu-24.04, 3.10), has been waiting for a GitHub-hosted runner since the first attempt, when GitHub reported it could not assign one. Every other required check passed on this head: lint, actionlint, live, and the ubuntu-24.04, ubuntu-24.04-arm and macos-15 test legs. The suite also passes on Python 3.10 locally (157 tests). This merges #3's commit too.

@jeffwall-curlewlabs
jeffwall-curlewlabs merged commit 23067da into main Oct 5, 2026
13 of 16 checks passed
@jeffwall-curlewlabs
jeffwall-curlewlabs deleted the feat/idle-tick-cache branch October 5, 2026 20:54
@jeffwall-curlewlabs

Copy link
Copy Markdown
Contributor Author

Branch protection on main is restored. The re-read configuration matches the one saved before the merge, field for field: the same seven required checks, admins included, 0 approvals, no force pushes, no deletions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Clock: an idle tick should only ask GitHub whether anything changed

1 participant