Skip to content

fix(runtime): order workflow compensation by completion instant, not timestamp text - #1464

Open
doublewhy wants to merge 13 commits into
devfrom
1344-compensation-order
Open

doublewhy wants to merge 13 commits into
devfrom
1344-compensation-order

Conversation

@doublewhy

@doublewhy doublewhy commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

specs/formal/workflows/compensation.md requires reverse completion order, but maybe_apply_compensation sorted completed steps by timestamp text. isoformat() drops a whole second's fraction, so ...:01.001000Z sorted before ...:01Z; offsets and identical stamps misordered too. The fix sorts by the instant parsed with the history contract checks' parser (diagnostics._parse_timestamp, offset-less read as UTC), oldest first, then reverses.

For the 2026-10-25 review, SEM-204 gets its own policy phase owning the compensation module and passes governance. #1510 stays open for the other ^SEM- requirements.

Requirement UIDs

  • SEM-204 (Workflow Compensation Semantics), ACTIVE, via the new workflow-compensation-semantics phase.

Related Issues

Closes #1462

ADR Impact

  • No ADR required.

Changes

Done-when / required action Change Fails without it Lane / evidence
cancel_workflow regression: completions straddling a whole second, reverse order expected; fails on dev maybe_apply_compensation test_issue_1462_compensation_order.py, dev's source on the merged tree: 6 of 10 fail ['a', 'b'] == ['b', 'a'] shards, run 40232579083
Same for reconcile_workflow_timeouts; offset-less, offsets, identical, +00:00/Z; contract checks before and after same same module, 2 triggers × 5 forms same
Run compensation/timeout modules, default and -m integration — — 353 passed
Republish the captures captures evidence checkers canonical / checks, same run
Review: resolve the SEM-204 route in #1510, set RAES_REQUIREMENT_UID=SEM-204, pass governance phase, ownership, record before: 2 ownership mismatches, missing IMPLEMENTS and TESTS; test_issue_1462_policy_ownership.py: 6 of 14 fail without the phase, the governance case without the links test shards; local governance exit 0
Review: keep the parsed-instant/reverse-history fix; run its regressions and required CI source unchanged row 1 same run

Test Plan

  • Unit tests pass
  • Integration tests pass if applicable
  • Completion command passes
  • No coverage regression

pytest, both #1462 modules and six modules that cancel or time out workflows: 353 passed, none integration-marked. Stash failures ran on the merged tree. Existing governance/policy modules: 375 passed. RAES_REQUIREMENT_UID=SEM-204 check_requirement_governance.py --base-rev origin/dev --require-governance exits 0. With SEM-204 set, make policy, verify-fast-feedback, verify-changed and contracts passed; so did lint and docs-local.

Captures 884a8e85 (specification) and 3fc3ccf9 (formal): collect_research_evidence.py on digest fa973567, then research_capture record. Production pipeline over 6 artifacts: 38 of 42 stages passed (4 preregistered non-executions), load-bearing 10 of 10. replay_case on 10 retained cases, 4 production CLI cases, 14 participant functions: 20 of 20 observations matched. Both evidence checkers pass.

CI run 40232579083 on 50916dcb passed every job, including integration and all four shards (24 of 24 #1462 cases). Sonar: 0 new issues, new-code coverage 100.0%.

Ground Control Checks

  • Repository policy command passes
  • Pre-push code review and test-quality review completed, or not run for this lane

make policy with SEM-204: 17 of 17 stages. Self-review only.

Traceability

  • IMPLEMENTS: implementations/python/packages/raes_runtime/control_plane_workflows.py
  • TESTS: implementations/python/tests/test_issue_1462_compensation_order.py, implementations/python/tests/test_issue_1462_policy_ownership.py

Checklist

  • Code follows the project coding standards
  • FM level classified if semantic change: not applicable; no rule, state machine or contract changes. No executable or runtime adoption is claimed.
  • Published contract schemas regenerated if models changed: no model or schema changes
  • PR title is a Conventional Commit (release-please derives the version and CHANGELOG.md from it)
  • Architectural docs updated if applicable: not applicable

…timestamp text

Producers format step completion times with isoformat(), which drops the
fraction on a whole second, so a later completion stamped
...:01.001000Z sorted before an earlier one stamped ...:01Z and
compensation ran in completion order. Sort completed steps by the parsed
instant.
…ring compensation

The compensation sort used the strict parse_timestamp, which raises on a
timestamp without an offset. The workflow history contract checks admit
such a value and read it as UTC, so cancelling or timing out a workflow
with an admitted offset-less completion raised instead of compensating.
Sort with the parser those checks use (diagnostics._parse_timestamp).

The regression test now seeds every instant in the past, so the history
stays monotonic after the cancellation or timeout. It covers both
compensation triggers and three admitted timestamp forms.
… order

sorted(..., reverse=True) is stable, so completions with equal instants
kept their history order and were compensated in the order they ran.
The workflow history contract checks reject only decreasing timestamps,
so in an admitted history the history order of equal instants is their
completion order. Sort ascending and reverse the list in place, so that
equal instants are also compensated in reverse completion order.

The reversal is an explicit list.reverse() call. SonarCloud reported
sorted(...)[::-1] as python:S7511 ("Remove this redundant subscript
reversal"). The reversal is not redundant, and sorted(..., reverse=True)
is not equivalent here: it keeps equal keys in their original order.

The regression test gains an identical-instant pair and a pair that
writes one instant two ways (+00:00, then Z). It now asserts the order
of compensation_workflow_started events, the execution order that
specs/formal/workflows/compensation.md requires.
Collected on dev 348c490 plus this branch with blitz collect_research_evidence.py (production pipeline over 6 pinned artifacts, 10 retained cases via replay_case, 4 production CLI cases, 14 participant test functions), then recorded with python -m tools.research_capture record.
@doublewhy
doublewhy force-pushed the 1344-compensation-order branch from 421ed21 to 818d221 Compare October 14, 2026 01:44
@doublewhy
doublewhy marked this pull request as ready for review October 14, 2026 02:16
@Brad-Edwards

Copy link
Copy Markdown
Collaborator

Recommendation: do not merge as is.

Reviewed head 818d221339011a31293147645b5dcacbec7d51bb, targeting dev.

  1. The PR identifies SEM-204 as governing the changed compensation semantics but leaves requirement governance skipped because its phase ownership excludes control_plane_workflows.py (Requirement ownership roots do not cover the files of DSL-117, DSL-120, DSL-121, ACT-624, SEM-223 and SEM-204 #1510). Current repo instructions require the requirement UID on UID-free branches; a known ownership failure needs an authorized policy resolution, not omission of the UID.

Required action:

Coverage and outstanding work: #1462's whole-second, naive-as-UTC, differing-offset and equal-instant cases are covered for cancellation and timeouts, with before/after contract validation and captures. Reusing the admission timestamp parser is correct. No source defect found; the merge blocker is the skipped applicable governance check.

Verification limits: Complete snapshot and green required status contexts; those contexts do not establish that skipped requirement governance passed.

…e locator

The Cambridge mirror's dist directory now holds Isabelle2026 only, so
the locked Isabelle2025-2_linux.tar.gz answers 404 there and
canonical / generic-tool-local-inputs fails exact verification, which
skips the checks, integration and proof jobs. The official release
locator serves the same bytes; the fetch still verifies the unchanged
lock size and SHA-256.
… module

SEM-204 reached semantics-expansion only through the shared ^SEM- pattern,
whose ownership roots exclude raes_runtime/control_plane_workflows.py, so
requirement governance for the #1462 compensation fix reported an ownership
mismatch (#1510). Add a workflow-compensation-semantics phase listed before
semantics-expansion, with the same reference-implementations prerequisite.
It keeps the shared semantic roots SEM-204 already reached, without the
paths named there for SEM-218 and SEM-233, and adds the compensation module
and the carried #1474 locator doc. The shared phase's roots and the other
#1510 requirements are unchanged.

SEM-204's record gains the IMPLEMENTS and TESTS links governance requires.
The new policy test pins the route, its bounds and SEM-223's unchanged
phase, and runs SEM-204 governance over the delivery files.
They pin the 2026-10-14 implementation digest; the merged source gets a
fresh capture.
Collected on dev cf19e2d plus this branch (implementation digest
fa973567) with blitz collect_research_evidence.py: production pipeline
over 6 pinned artifacts (38 of 42 stages passed, 4 preregistered
non-executions), 10 retained cases via replay_case, 4 production CLI
cases and 14 participant test functions (18 passed). Recorded with
python -m tools.research_capture record.
@doublewhy

Copy link
Copy Markdown
Contributor Author

Addressed the 2026-10-25 review at 50916dcb:

@doublewhy

Copy link
Copy Markdown
Contributor Author

This branch conflicts with dev again after the Quality refactor merges, in control_plane_workflows.py. The fix is not on dev: compensation is still ordered by the raw timestamp string (sorted(..., key=lambda event: event.timestamp, reverse=True), line 49), so #1462 still reproduces. Do you want this as a standalone fix now, or will you fold it into the runtime refactor? If it is folded in, I'll close this PR and leave #1462 open.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants