Build cross-tool orchestration and failure fixture suites #13

Outcome

Create the suite-level conformance fixtures required to prove that flow can plan, execute, validate, explain, interrupt, retry, and resume work across independently released renderflow, optiflow, and aniflow providers.

Flow owns orchestration fixtures and expected cross-tool state transitions. Raw format corpora remain in the provider repositories.

Motivation

Apache Tika-style parser corpora are valuable because they turn rare format behavior, malformed inputs, and regressions into durable tests. This repository needs the equivalent for its own product boundary.

The corpus must be versioned, deterministic, redistribution-safe, inspectable, and generator-first. It must not become an unreviewed dump of large or copyrighted binaries, and its coverage manifest must never claim that finite fixtures prove support for every possible file.

Cross-suite rules

• Keep fixture generation and assertions in the owning repository.
• Use public CLI/API and versioned artifacts for cross-tool tests; never import sibling source trees.
• Prefer small source recipes plus deterministic generators over committed large binaries.
• Record seed, generator/tool version, parameters, expected digest, media type, validity class, license/provenance, and expected behavior.
• Partition fixtures into valid, boundary, malformed, unsupported, and resource-stress classes.
• Malformed fixtures must have intentional corruption recipes and named expected diagnostics.
• Resource-stress cases must be safely bounded in normal CI; expensive/nightly tiers must be explicit.
• Never include secrets, personal data, proprietary samples, malware, weaponized polyglots, or uncontrolled decompression bombs.
• Fuzz-found regressions must be minimized, assigned a stable ID, documented, and promoted into the deterministic corpus.
• Tests must validate typed outcomes, artifacts, digests, cleanup, and source immutability—not merely process exit or file existence.
• Keep generated files reproducible and add a drift check; CI must never rewrite canonical fixtures.

Required fixture families

Representative workflows

• Static publication/comic: immutable source → renderflow artifacts → optiflow inspection/optimization opportunity → validation and ship-readiness evidence.
• Temporal release: immutable time-based source → aniflow temporal processing/analysis → renderflow derivatives → optiflow inspection where semantically appropriate.
• Minimal single-provider and multi-provider graphs to isolate orchestration behavior.

Contract and compatibility cases

• Compatible, unavailable, missing, stale, future, and mismatched provider capability/contract versions.
• Valid and invalid provider locks, manifests, execution reports, provenance, digests, output declarations, and artifact sets.
• Partial outputs, invalid outputs, extra outputs, missing outputs, path escapes, symlinks, conflicting artifact identity, and provider success with failed validation.
• Upgrade/compatibility fixtures for every released Flow run/checkpoint/plan schema that policy promises to consume.

State-machine and authority cases

• Cancellation before start, during a provider, and between steps.
• Timeout, nonzero exit, signal termination, output-limit failure, host interruption, and restart.
• Resume after completed work, interrupted work, invalid checkpoint, changed inputs, changed provider lock, and changed plan.
• Retryable versus terminal failure; idempotency; no duplicate consequential side effects.
• Explicit refusal when mutation, upload, publication, signing, or another consequential action lacks authority.
• Redacted diagnostics and explanations that identify the failing boundary without leaking argv, environment, secrets, or private content.

Provider-fixture boundary

• Consume small immutable releases or pinned test artifacts from provider repositories where practical.
• Supply hermetic fake providers that implement public contracts for orchestration-only failures.
• Never reproduce renderflow parsing, optiflow inspection, or aniflow temporal algorithms inside flow.
• Verify at least one clean-room run using independently built/released provider artifacts rather than sibling workspaces.

Deliverables

• Stable workflow/fixture IDs and a machine-readable scenario manifest.
• Expected state-transition traces, plans, locks, reports, artifacts, digests, diagnostics, and explanations.
• Deterministic hermetic provider kit for orchestration failures.
• PR and scheduled/release tiers with resource budgets.
• Compatibility matrix covering supported provider and Flow contract versions.
• A contribution path for promoting production regressions into minimized scenarios.

Acceptance criteria

• Static-publication and temporal-release positive fixtures pass end to end.
• Negative fixtures cover unavailable capability, partial/invalid output, provider-version mismatch, corrupt artifact, and insufficient validation evidence.
• Flow never accepts provider exit or file existence as proof of success.
• Interruption and resume preserve completed work without duplicating side effects.
• Changed inputs, plans, provider locks, or artifacts invalidate unsafe resume.
• Authority boundaries are explicit and tested for consequential actions.
• Explanations are typed, actionable, and redacted.
• Provider repositories remain independently owned and source-decoupled.
• Clean-room tests use immutable provider artifacts.
• Fixture generation and expected evidence are deterministic and drift-checked.
• Current validation, contract, package, documentation, and release checks remain green.

Dependencies and scheduling

• This is a bounded child of suite epic [EPIC] Finish the flow suite and prove production-ready orchestration #11 and should be linked there.
• Reconcile with flow Implement a versioned cross-holon orchestration vertical slice #3 and [Publication] Orchestrate exhaustive comic finalization and multilingual release workflows #10: the corpus supplies reusable evidence and must not become a competing orchestration implementation epic.
• Provider-local corpus issues are dependencies for real raw-file coverage; hermetic fake-provider work can proceed independently.
• Complete before the final post-roadmap audit so that audit can verify behavior from repeatable evidence.

Bounded execution plan

Dependency scan recorded 2026-09-20. Reverify live state before each implementation branch.

Accepted foundations

• #7 — federated extension contract.
• #23 — deterministic resolution and hermetic execution seam.
• #26 — stdio framing and transcript validation.
• renderflow#366 — golden artifact corpus foundation.
• optiflow#51 — safe extension/provider SDK.
• aniflow#29 — hermetic provider-conformance kit.

Ordered child checkpoints

• #28 — freeze the orchestration scenario manifest and fixture contract.
• #29 — build the hermetic orchestration provider kit. Completed through merged PR [FLO-13.2c] Complete artifact adversaries, compositions, and provider-kit evidence #60 and green default-branch CI.
• #30 — prove compatibility, artifact, and provider-failure scenarios. Depends on [FLO-13.1] Freeze the orchestration scenario manifest and fixture contract #28, [FLO-13.2] Build the hermetic orchestration provider kit #29, and [FLO-3.2] Specify external process transport and provider trust boundaries #25.
• #31 — prove interruption, retry, resume, and authority state transitions. Depends on [FLO-13.2] Build the hermetic orchestration provider kit #29, [FLO-13.3] Prove compatibility, artifact, and provider-failure scenarios #30, and the durable plan/run/checkpoint foundation from Implement a versioned cross-holon orchestration vertical slice #3.
• #32 — prove the static publication workflow with released providers.
• #33 — prove the temporal release workflow with released providers.
• #34 — add clean-room compatibility, CI tiers, and regression promotion; reconcile final evidence.

After the shared #28–#31 foundation is accepted, #32 and #33 may proceed independently when their exact provider artifacts are available. #34 is the final integration gate.

Active external dependencies

• #25 — complete external-process, artifact, integrity, and authority boundaries.
• #3 — durable multi-provider plan/run/checkpoint/retry/resume behavior.
• renderflow#412 and renderflow#413 — rendering corpus and synthetic comic fixture pack.
• optiflow#65 — deterministic file/media corpus.
• aniflow#24 — deterministic temporal-media corpus.

Provider corpus issues own raw-format depth. A Flow real-provider checkpoint may proceed once the exact required immutable fixture subset and public contract artifacts exist; it must not claim coverage beyond that subset. #10 consumes the static orchestration evidence and is not a prerequisite for this fixture program.

At the planning snapshot, no open pull requests were observed in Flow, Renderflow, Optiflow, or Aniflow.

Non-goals

• Owning provider algorithms or raw-format corpora.
• Source-tree imports across repositories.
• Performing real consequential publication/signing in ordinary CI.
• Treating one happy-path demo as suite completeness.