Skip to content

show the rollback happening, with no key and no network - #60

Merged
hyj28 merged 3 commits into
mainfrom
rollback-demo
Sep 24, 2026
Merged

hyj28 merged 3 commits into
mainfrom
rollback-demo

Conversation

@hyj28

@hyj28 hyj28 commented Sep 24, 2026

Copy link
Copy Markdown
Owner

The quick start ended at uv run pytest. That proves 1277 tests pass and demonstrates nothing about the claim on the README's first line — that the agent checkpoints its work and rolls back when it drifts. This adds a program a visitor runs in 0.1 seconds, with no provider key and no network, that walks a run from checkpoint through drift detection to rollback and recovery.

The step function is a scripted stand-in, and the program says so in its header and its docstring. A demo that read as a real agent run would be the kind of claim this repository spends its commit messages arguing against.

Review found the demo was true only under thresholds it never disclosed

It ran with loop_window=2, loop_repetitions=2 against library defaults of 6 and 3. Restoring the defaults made the same trajectory demonstrate the opposite of the claim:

checkpoint: checkpoint-2 (accepted, logical step 4)   <- the DRIFTED workspace, accepted
coarse trigger: action_loop
rollback: restored checkpoint-0 (initial, logical step 0)   <- healthy work discarded
RuntimeError: demo invariant failed: checkpoint restoration was incomplete
EXIT=1

My first fix instruction pointed at the wrong knob. It had the four 99 thresholds disclosed as "deliberately disabled" — but those detectors cannot fire on this trajectory at all (every step sets changed_paths, none sets error, commands_run is 0, reward is None), so replacing them with defaults gives byte-identical output. The disclosure advertised a suppression that changed nothing while the one load-bearing retune went unmentioned.

The fix is not a better disclosure. HeuristicConfig() is now constructed with no arguments, and the trajectory is staged long enough for the default detector to fire. A reader who copies the configuration gets the behaviour they just watched.

That timeline is also the more interesting one. The drifted state is checkpointed as accepted at logical step 8 — and the run still recovers, because the six-step lookback makes that checkpoint ineligible and selection falls back to the healthy one at step 4. The trigger line prints the lookback so the skipped checkpoint reads as the mechanism working rather than as a bug.

Every line of stdout used to be narration

rollback: restored checkpoint-1 was printed from Checkpoint metadata, not from observed state. A probe that made the workspace swap a no-op and gutted the demo's own restored = (...) check produced stdout matching the expected literal exactly, exit 0 — so acceptance was guarded only by an assertion living inside the artifact under test.

The retried step now reads the workspace and prints what it found:

retry observes: progress.txt='healthy 4', detour.txt exists=False

Re-running that same probe against this version fails on run.stdout == EXPECTED_STDOUT itself. Verified by mutating a scratch copy with PYTHONPATH forced, checking the import path resolved to the copy first.

Also from review

  • No fine judge is configured, so every coarse trigger rolls back unreviewed — now stated in the header, because the printed shape is otherwise indistinguishable from a reviewed two-tier run.
  • TimelineCheckpointStore.__init__ dropped the base's keyword-only ignore, so ignore=("*.log",) raised TypeError where DirectoryCheckpointStore accepts it. The other three overrides match their bases exactly.
  • The test now asserts the caller's cwd stays empty and no driftlock-rollback-demo-* directory survives. Both were previously established only by running it by hand.
  • uv.lock still said requires-python = ">=3.11" while pyproject.toml has required >=3.13 since before this branch, so uv lock --check failed on main. Repaired in its own commit so the commit beside it is only the example.

Front page and community files

The quick start now ends at the demo. The repo layout stops omitting integrations/lhtb/ — the adapter the published results came from, referenced three times on the page without ever saying where it lives.

CODE_OF_CONDUCT.md was the one item GitHub's community checklist still marked missing, and it appears in the sidebar beside License and Security. It adopts the Contributor Covenant by reference rather than pasting boilerplate into a repository whose every other document is specific, and adds the part that is: review here is adversarial by design, which is a standard applied to work and never to people. Reports route through GitHub, so no personal contact address is published.

docs/assets/driftlock-social.svg is the 1280×640 source for the social preview card. GitHub has no API for that image, so the PNG is uploaded by hand; the source lives here so it can be regenerated.

Gates

pytest 1278 passed, 3 skipped (baseline 1277 / 3). ruff check . and ruff format --check . exit 0. No existing test modified, deleted, skipped or mocked.

Run gate: five scenarios driven by someone who had not seen the code — normal run, determinism (two stdouts byte-identical), cwd independence (external scratch dir, left empty), no credentials (env -u across every provider variable), cleanliness (no leftover temp dir). All matched.

What this does not do

It does not re-run the experiment, and the demo is not evidence about model behaviour — only about the runner's control flow under a planted stall. That is stated in the program itself, not only here.

The lock still said requires-python = ">=3.11" while pyproject.toml has required
">=3.13" since before this branch, so `uv lock --check` failed on main and every
ordinary `uv run` dirtied the working tree. Dropping typing-extensions follows
from the narrowed range: it was gated on python_full_version < '3.13' and cannot
be selected by any supported interpreter, so resolution is unchanged.

Pre-existing drift, repaired on its own so the commit beside it is only the example.
The quick start ended at `uv run pytest`, which proves 1277 tests pass and
demonstrates nothing about the claim on the README's first line. This adds a
program a visitor can run in 0.1s, with no provider key and no network, that
walks a run from checkpoint through drift detection to rollback and recovery.

The step function is a scripted stand-in and the program says so in its header
and its docstring. A demo that read as a real agent run would be the kind of
claim this repository spends its commit messages arguing against.

Three things review changed, and the first two are the reason this is worth
committing at all.

The demo originally ran with loop_window=2 and loop_repetitions=2 against
library defaults of 6 and 3, and nothing said so. Restoring the defaults made
the same trajectory demonstrate the opposite of the claim: the periodic
checkpointer snapshots the drifted workspace as "accepted", rollback selection
walks back past it to the initial checkpoint, the healthy work is lost, and the
run exits 1. It now constructs HeuristicConfig() with no arguments at all and
stages a trajectory long enough for the default detector, so a reader who copies
the configuration gets the behaviour they just watched.

That timeline is also the more honest one: the drifted state IS checkpointed as
"accepted" at logical step 8, and the run still recovers, because the six-step
lookback makes that checkpoint ineligible and selection falls back to the
healthy one at step 4. The trigger line prints the lookback, so the skipped
checkpoint reads as the mechanism working rather than as a bug.

Every line of stdout used to be narration: "rollback: restored checkpoint-1"
was printed from checkpoint metadata, not from observed state. A probe that
made the workspace swap a no-op and gutted the demo's own invariant check
produced stdout matching the expected literal exactly, exit 0. The retried step
now reads the workspace and prints what it found, so the literal carries
evidence. Re-running that same probe against this version fails on the stdout
comparison itself.

Also: the runner has no fine judge here, so every coarse trigger rolls back
unreviewed — stated in the header, because the printed shape is otherwise
indistinguishable from a reviewed two-tier run. TimelineCheckpointStore.__init__
now accepts the base's keyword-only `ignore` instead of dropping it. The test
asserts the caller's cwd stays empty and no demo temp directory survives, both
of which were previously established only by running it by hand.
The quick start now ends at something that demonstrates the claim rather than at
a test count, and the repo layout stops omitting `integrations/lhtb/` — the
adapter the published results actually came from, which the page references
three times without ever saying where it lives.

CODE_OF_CONDUCT.md was the one thing GitHub's community checklist still marked
missing, and it shows in the sidebar beside License and Security. It adopts the
Contributor Covenant by reference rather than pasting 130 lines of boilerplate
into a repository whose every other document is specific, and adds the part that
is specific: review here is adversarial by design, which is a standard applied
to work and never to people. Reports go through GitHub rather than an email
address, so no personal contact is published.

docs/assets/driftlock-social.svg is the 1280x640 source for the repository's
social preview card. GitHub has no API for setting that image, so the PNG is
uploaded by hand; keeping the source here means it can be regenerated.
@hyj28
hyj28 merged commit b89c5f0 into main Sep 24, 2026
2 checks passed
@hyj28
hyj28 deleted the rollback-demo branch September 24, 2026 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant