Skip to content

state: a rehydrated run sees its data in the order the live run built it (#871) - #878

Merged
Masterplanner25 merged 1 commit into
mainfrom
fix/871-rehydrated-key-order
Sep 24, 2026
Merged

Masterplanner25 merged 1 commit into
mainfrom
fix/871-rehydrated-key-order

Conversation

@Masterplanner25

Copy link
Copy Markdown
Owner

Closes #871.

The defect

The graph snapshot and both stores persisted state with sort_keys=True, so a
map a program built as {web, code, data} came back {code, data, web} after a
cross-process resume:

in-process : {"web": 1, "code": 2, "data": 3}
rehydrated : {"code": 2, "data": 3, "web": 1}
control    : {"web": 1, "code": 2, "data": 3}   # same program, no resume

Same content, different string — so anything derived from it differed across a
restart: a draft, a hash, an @exactly_once key. That last one is how the
reporting project found it, an idempotent publish re-firing on a rehydrated
replay of an identical draft.

JSON objects preserve order and json.load round-trips it, so nothing but the
sort was destroying it.

Where it was, and where it stays

Six sort_keys sites, three kinds:

Kind Sites Change
Persisted state graph snapshot, local store, SQLite columns sort removed
Identity compute_action_id, bytecode cache key unchanged
Display nodus graph output, dependency graph file unchanged

The distinction is reported vs compared. Persisted state is read back and
handed to a program, so it must come back as it went in. An identity is hashed,
and there two equal maps built in different orders have to produce one answer.

The regression test asserts both directions — removing the sort where it is
load-bearing has to fail too. Without that half this reads as "sorting is always
wrong", which is how the next person removes the wrong one.

Scope: one site or four?

The graph snapshot alone fixes the reported symptom — step results and state
cells both round-trip after that one line. I fixed the stores as well, because
the store keeps a second copy of workflow_state and nodus workflow inspect prints it, so a user was seeing sorted keys while the program saw
insertion order. Measured before and after:

before:  store workflow_state: {"cfg": {"apple": 2.0, "mango": 3.0, "zebra": 1.0}}
after:   store workflow_state: {"cfg": {"zebra": 1.0, "apple": 2.0, "mango": 3.0}}

_state_json names the rule once rather than splitting guest data from
bookkeeping field by field: claim and wait are mostly bookkeeping, but wait
carries the guest's payload, and a hand-maintained "may reorder / may not" list
is the kind that drifts. Nothing there is content-addressed, compared as a
string, or indexed on, so there is nothing to trade away.

No canonical mode added to std:json

That was the issue's alternative fix. With order round-tripping it is not needed
here: @exactly_once already canonicalises internally, so a key derived from a
step result is now stable across a restart without the caller doing anything.
Said so in the changelog rather than leaving it as an open question.

Tests

tests/test_rehydrated_key_order.py, 8 tests. Three per-decision neuters,
restoring from saved byte copies:

BASELINE (fixed tree)                          8 passed
graph snapshot sorts again (the 5.14 rule)     RED   3 failed
store serialization sorts again                RED   2 failed
the effect action id STOPS canonicalising      RED   1 failed
RESTORED                                       8 passed

The third is the inverse direction, and it is the one that matters for the next
reader.

Two test-authoring bugs of mine, both caught by a test failing for the wrong
reason
— worth naming because both would have produced a green test that
checked nothing:

  • The first store assertion read the repo-root store and took list_runs()[-1]
    — a stranger's record among ~400. It looks its own run up by id now, and the
    store-level check builds its stores in a tempdir.
  • Parametrising the harness by NODUS_WORKFLOW_STORE_BACKEND does not work:
    the variable is read when the default runner is built, and that runner is
    cached per process, so setting it in setUp after another test built one
    silently reads a store it is not writing to. The tests passed alone and failed
    in the file. The snapshot round trip is backend-independent, so one run covers
    it; the store serialization is covered directly instead.

Checks

🤖 Generated with Claude Code

… it (#871)

The graph snapshot and both stores persisted state with sort_keys=True, so a
map a program built as {web, code, data} came back {code, data, web} after a
cross-process resume. Same content, different string -- so anything derived
from it differed across a restart: a draft, a hash, an @exactly_once key. That
last one is how it was found, an idempotent publish re-firing on a rehydrated
replay of an identical draft.

JSON objects preserve order and json.load round-trips it, so nothing but the
sort was destroying it. Step results and state cells both round-trip now, on
both backends.

CANONICAL FORM IS UNCHANGED WHERE AN IDENTITY IS COMPUTED. compute_action_id
and the bytecode cache key still hash sorted payloads and must: there two equal
maps built in different orders have to produce one answer. The line is
REPORTED vs COMPARED -- persisted state is read back and handed to a program,
an identity is hashed -- and the test asserts both directions, so removing the
sort where it IS load-bearing fails too. Without that half this reads as
"sorting is always wrong", which is how the next person removes the wrong one.

One rule for the whole record rather than a guest-data field list: `_state_json`
names it once. `claim` and `wait` are mostly bookkeeping, but `wait` carries the
guest's payload, and a hand-maintained split between "may reorder" and "may not"
is the kind that drifts. Nothing there is content-addressed, compared as a
string, or indexed on.

No canonical mode added to std:json. It was the alternative fix and is not
needed for this: @exactly_once already canonicalises internally, so a key
derived from a step result is stable across a restart without the caller doing
anything.

Two test-authoring notes, both found by the tests failing for the wrong reason:

- The first store assertion read the repo-root store and took list_runs()[-1],
  which is a stranger's record among ~400. It looks up its own run by id now,
  and the store-level check builds its stores in a tempdir instead.
- Parametrising the harness by NODUS_WORKFLOW_STORE_BACKEND does not work:
  the variable is read when the default runner is BUILT and that runner is
  cached per process, so setting it in setUp after another test built one
  silently reads a store it is not writing to. Passed alone, failed in the
  file. The snapshot round trip is backend-independent, so one run covers it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Masterplanner25
Masterplanner25 merged commit aaa879b into main Sep 24, 2026
6 checks passed
@Masterplanner25
Masterplanner25 deleted the fix/871-rehydrated-key-order branch September 24, 2026 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG-871: rehydrated step results come back key-sorted, so strings derived from them differ after a restart

1 participant