Skip to content

Saved observation sets are trusted as written: an aggregate that contradicts its own inventory builds a bundle that verifies #38

Open
Open
Saved observation sets are trusted as written: an aggregate that contradicts its own inventory builds a bundle that verifies#38

Description

@drevendev
drevendev
opened on Sep 30, 2026

Found by the review of 2026-09-30. It is about the offline input path (devostasis build and evaluate over a saved observation set, and the vector runner), not about the live collection, which derives everything from what it collected.

1. An aggregate that contradicts its own inventory is trusted, and the bundle verifies

normalize.derive computes each aggregate from its inventory, but returns early when the aggregate is already present (if observation_id in obs: return), and every saved set that observe writes already carries all of them. So on build an aggregate is never checked against the inventory it claims to be derived from.

Reproduced on the example bundle's own observations (examples/sample-bundle/observations.json): the issue inventory shows one stale open issue and forge.issues.stale_open_count_30d says 1. Changing only the aggregate to 30:

after derive(), forged aggregate: 30 (recomputed from the inventory would be the honest value)
clutter honest: LIGHT | clutter forged: HEAVY
forged build: bundle written; verify problems: none (verified)
snapshot clutter band: ['HEAVY']

The bundle carries both facts in observations.json, the inventory that says one and the aggregate that says thirty, and devostasis verify accepts it: verification binds snapshot.json to the digest of observations.json (OBSERVATIONS_DIGEST_MISMATCH) but never asks whether the aggregates inside are the ones their inventories produce. A verified bundle says HEAVY while its own evidence says LIGHT.

2. The evaluators accept values the observation contract admits but no collector writes

Reproduced at the direct evaluator boundary (saved sets and vectors reach it):

  • counts are coerced by as_int: a Clutter stale count of -5, 0.9 or "0" is read as an integer, so a vector expecting CLEAN / AVAILABLE / EXACT passes over them;
  • a revision record whose historical_contribution is VERIFY_FAIL while its history_state is PASS_ONLY_OBSERVED was counted as a pass. The review PR now treats that as a defect (REVISION_RECORD_INCONSISTENT), and a current verdict outside the canonical vocabulary as UNKNOWN; the count coercion is still open.

3. The receipt/observation coherence audit, and a caveat for its repair

PV-AUDIT-RECEIPT-OBS-CONSISTENCY-001 (research run 404, REPAIR REQUIRED, RCPT-OBS-01..08, Drive only until now) asks that a saved set's receipt and observations agree before evaluation: every observation requested and matching per_key status and freshness, returned_keys equal to the value-bearing envelopes, no requested key vanishing.

As stated, that invariant would reject every saved set this implementation writes. The collector finalizes the receipt in GitHubAdapter.collect (obs.finalize_receipt(receipt)), and normalize.derive adds the aggregates afterwards. In the example bundle, 27 of the 36 observations are derived or configuration envelopes the receipt does not list, by construction:

observations 36 per_key 9 requested 9
observations not in per_key: 27
of which derived/config: 27

So the repair has to decide which envelopes the receipt speaks for. The coherent reading is: the receipt binds the collected envelopes (the nine inventories), and the derived ones are bound by recomputation from those inventories, which is finding 1. Finalizing the receipt after derivation instead would change every receipt, which is identity-bearing (#19), so it is the more expensive of the two.

Proposed repair

  1. On loading a saved set (build, evaluate), recompute every aggregate normalize.derive knows from the inventories present and fail closed on a stored aggregate that differs in status, value or coverage, before any evaluation or write; an aggregate whose inventory is absent is kept and marked as not recomputable.
  2. The same recomputation as a verification step over observations.json when the member is present. The 308 stored bundles are honest observe outputs, so this should move none of them; measure that before adding the step.
  3. Admit counts as non-negative integers (not booleans, floats or strings) at the observation boundary, so as_int never has to guess.
  4. Implement run 404's invariant over the collected envelopes, and write down in docs/spec/observations.md which envelopes the receipt speaks for.

It belongs to the admission package R020-W01/W04 of the 0.2.0 candidate. No threshold, band or identity change.

🤖 Generated with Claude Code

Activity

  1. andy-zen-dev commented on Oct 24, 2026

    @andy-zen-dev
    andy-zen-dev
    on Oct 24, 2026 – with ChatGPT Codex Connector

    Exact-head admission finding: malformed CI completeness flags are treated as complete (2026-10-24)

    Repository/source reviewed: PR #81 / release candidate #75 HEAD a4761daa9fa87026ea106a3d722e7299b3de3b8b, src/devostasis/revision_history.py blob e7c871965c2f783ae6991d6d2c6d20596e81daa7, _series_records(); also _evaluate_revisions() in src/devostasis/vitals/integrity.py and Observation.__post_init__() in src/devostasis/observations.py. I am adding this to existing #38 (saved-input admission) rather than filing a duplicate Issue. This is independent of #83's carried-array problem.

    The candidate current-inventory repair only calls the series incomplete if a known coverage field is identical to False:

    if series.status == AVAILABLE and isinstance(series.coverage, dict) and any(
    series.coverage.get(key) is False
    for key in ("complete", "runs_complete", "attempts_complete", "suites_complete")
):
    problems.append("REVISION_SERIES_COVERAGE_INCOMPLETE")

    An explicit {"complete": 0}, {"complete": "false"}, {"complete": null}, {"attempts_complete": []} or even non-object coverage: [] is not diagnosed. Observation.__post_init__ checks envelope status/value type but not coverage-flag types; ObservationSet.add() does not strengthen that validation. Thus the AVAILABLE series can return inventory_complete=True with malformed completeness claims. reconcile() then treats a carried revision missing from the current inventory as branch-departed; _evaluate_revisions() can proceed without the reconciled.problems fail-closed gate. A previous failure plus an apparently complete inventory of four passing revisions is the important false-favorable regression shape to test, not an assertion that a full end-to-end reproduction has already passed.

    Executed narrow oracle (Python 3.13): 13 cases around the literal exact-source predicate: valid omitted/empty coverage, all-true flags, and Boolean-false flags behaved as currently intended; 8/8 malformed examples were treated as COMPLETE by the existing predicate while a type-aware candidate rejected them. This is a source-faithful isolated test, not the actual Devostasis pytest, historical replay, or upstream CI.

    Scoped implementation/acceptance: For supplied known completeness flags, require type(value) is bool (not truthiness/isinstance(value, bool) on integer 0); non-Boolean supplied values and non-object coverage should generate an explicit malformed-coverage diagnostic preventing a completeness claim. Preserve absent/empty coverage's existing compatibility pending any separately accepted contract decision; preserve genuine False as INCOMPLETE, genuine True as true; avoid touching thresholds, history lineage, or band rules. Add parameterized _series_records -> reconcile -> integrity.evaluate tests for these values and a carried VERIFY_FAIL absent from four current VERIFY_PASS revisions; require UNKNOWN/no favorable exact band for malformed input, and run full pytest + four-generation historical bundle replay on the fixing HEAD.

    GitHub effect: finding only; no patch/PR/merge or CI pass claimed. Current connector permissions for andy-zen-dev: pull=true, push=false, triage=false, maintain=false, admin=false.

    Prepared implementation + chain regression handoff (2026-10-24)

    Exact target: PR #81 HEAD a4761daa9fa87026ea106a3d722e7299b3de3b8b; src/devostasis/revision_history.py blob e7c871965c2f783ae6991d6d2c6d20596e81daa7; tests/test_revision_history.py blob ca112cedb6b2544c286406fb3bd42bed56c4b7a3.

    Prepared a focused two-file patch for the existing issue. No policy threshold changes and no new research-case IDs. It replaces only the AVAILABLE series coverage admission check with:

    if series.status == AVAILABLE:
    coverage = series.coverage
    keys = ("complete", "runs_complete", "attempts_complete", "suites_complete")
    if coverage is not None and not isinstance(coverage, dict):
        problems.append("REVISION_SERIES_COVERAGE_INVALID")
    elif isinstance(coverage, dict):
        for key in keys:
            if key in coverage and type(coverage[key]) is not bool:
                problems.append(f"REVISION_SERIES_COVERAGE_INVALID:{key}")
        if any(key in coverage and coverage[key] is False for key in keys):
            problems.append("REVISION_SERIES_COVERAGE_INCOMPLETE")

    Integration regression (13 parametrized scenarios) appended in the prepared patch to tests/test_revision_history.py: use _chain(store) to persist the preceding VERIFY_FAIL for revision r; construct the next saved AVAILABLE revision series with _passing([1,2,3,5]) (r absent) and overwrite only coverage. For the eight malformed variants ({"complete":0}, {"complete":"false"}, {"complete":null}, {"attempts_complete":[]}, {"runs_complete":1}, {"suites_complete":"true"}, [], "invalid"), assert the carried r remains FAILURE_OBSERVED, the Integrity band is None, evaluation status is UNKNOWN, and an explicit REVISION_SERIES_COVERAGE_INVALID diagnostic exists. Four literal False flags must retain the prior failed revision and emit REVISION_SERIES_COVERAGE_INCOMPLETE. Positive control: literal {"complete":true} may prove r departed and omit it. No branch/source behavior change is intended for absent/empty coverage.

    Verification performed: git apply --check and git apply PASS against a synthetic workspace containing the exact fetched hunk contexts at the expected file lines; appended regression Python syntax compiled; 12/12 isolated tests of the exact replacement condition PASS. This is not git apply against an actual checkout, and no real Devostasis pytest or historical replay ran. Two upstream exact-head GitHub Actions runs remain action_required, not PASS. Candidate patch SHA-256: d036bc761b71808753dff89b1110a1fd933daf0b2214639343dffbcf87cc4f1c (4,830 bytes, held outside GitHub for integration).

    Next acceptance: apply this scoped change on a focused feature branch targeting release/0.6.0, execute the new tests plus full offline pytest/history replay, and independently review the changed HEAD before merge. This update records a prepared handoff only; no commit/PR/merge is claimed.

  2. andy-zen-dev commented on Oct 24, 2026

    @andy-zen-dev
    andy-zen-dev
    on Oct 24, 2026 – with ChatGPT Codex Connector

    2026-10-24 — executable implementation handoff for malformed revision-series coverage (#38)

    Scope: one fail-closed input-admission repair, not a Vital threshold/contract reinterpretation. Verified upstream source context: PR #81 head a4761daa9fa87026ea106a3d722e7299b3de3b8b; src/devostasis/revision_history.py blob e7c871965c2f783ae6991d6d2c6d20596e81daa7; _series_records() lines 310–343. The previously filed #38 finding remains open. This is a concrete follow-up to comment #6379736684, not a duplicate Issue.

    Current code admits an AVAILABLE revision inventory as complete unless a known coverage flag is identically False; malformed explicit values like {"complete":0}, {"complete":null}, {"attempts_complete":[]} and non-object coverage:[] slip through. An absent carried failed revision may then be treated as departed.

    Ready replacement for the three-line coverage predicate at lines 340–342:

        if series.status == AVAILABLE:
        coverage = series.coverage
        keys = ("complete", "runs_complete", "attempts_complete", "suites_complete")
        if coverage is not None and not isinstance(coverage, dict):
            problems.append("REVISION_SERIES_COVERAGE_MALFORMED")
        elif isinstance(coverage, dict):
            if any(key in coverage and type(coverage[key]) is not bool for key in keys):
                problems.append("REVISION_SERIES_COVERAGE_MALFORMED")
            elif any(coverage.get(key) is False for key in keys):
                problems.append("REVISION_SERIES_COVERAGE_INCOMPLETE")

    The type check is intentionally type(value) is bool: 0 and 1 cannot masquerade as boolean proof. Legacy None / empty coverage remains accepted, genuine False remains incomplete, and valid True remains acceptable. A malformed field plus False still fails closed as MALFORMED. This is not authority to change the accepted completeness contract.

    Prepared regression shape (20 parameterized pytest instances):

    • 8 malformed inputs: [], "false", 0, {"complete":null}, {"complete":0}, {"runs_complete":"false"}, {"attempts_complete":[]}, {"suites_complete":{}}; actual _series_records() must emit REVISION_SERIES_COVERAGE_MALFORMED and inventory_complete=False.
    • 4 compatibility controls: None, {}, {"complete":true}, and both runs_complete/attempts_complete:true.
    • 4 Boolean-false controls, one for each supported key, must give REVISION_SERIES_COVERAGE_INCOMPLETE.
    • 4 real bundle-chain cases: stored B1 contains revision r with VERIFY_FAIL; B2 presents four passing revisions without r and explicit malformed coverage. Assert B2 retains r: FAILURE_OBSERVED, Integrity stays UNKNOWN with the malformed-coverage diagnostic, and committed B2 verify_dir succeeds without contradicting its own evidence.

    Verification boundary: an isolated predicate oracle identified 8/8 malformed cases currently escaping the existing check; the proposed guard rejects those 8/8, and seven valid/Boolean-false controls match intended behavior. The test module (86 lines) passes Python 3.13 syntax compilation; a generated unified diff containing this replacement and the test module passes git apply --check and compilation on an exact-context fragment mirror only, not a real Devostasis checkout. Real package pytest, historical replay, exact-HEAD hosted CI and independent later-head review are NOT_RUN / NOT_PASS. No GitHub code mutation or merge occurred because the connected account has pull=true, push=false.

    Before integration, document the new diagnostic in docs/spec/integrity-ci.md, run full offline suite and historical bundle checks on the fixing branch, and preserve distinct accepted research semantics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions