Skip to content

Harden task validation and repair across the remaining FlyDSL tasks - #129

Merged
irvineoy merged 5 commits into
mainfrom
codex/harden-remaining-task-runtimes
Oct 28, 2026
Merged

irvineoy merged 5 commits into
mainfrom
codex/harden-remaining-task-runtimes

Conversation

@irvineoy

@irvineoy irvineoy commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

This change protects evaluator state across the remaining FlyDSL task runtimes and makes task maintenance fail closed. It also repairs the 39 packages that failed or warned in the initial qualification of the retained tasks. Original scored cases, seeds, numerical gates, warmups and sample counts are preserved.

  • Follow task-local candidate helper imports and reject known evaluator-state access and new packages/extensions that shadow protected Python modules. Freeze candidate sources and captured action evidence during semantic review.
  • Add a repair-only quality-loop mode, selected-warning repair and bounded attempts. Repairs retain the original task/case/timing contract, preserve failed proposals and need a fresh validator PASS plus an independent review resolving every retained finding. They do not promote an optimizer result or report a speedup.
  • Validate measured outputs, every contributing sample in the repaired Event tasks, changed-input replays and poisoned fresh single-call graphs. Keep task-owned checks self-contained and use the canonical benchmark helper for the shared capture interface.
  • Fix operator/reference defects and missing public dtype, stride, routing, scale-buffer and output contracts. Sage uses residual FP4 components; the FP8 MoE uses an explicit protected staged baseline with the independent PyTorch oracle unchanged. These repairs change implementation and benchmark identity. Remeasure both arms; historical scores are not comparable, and this PR claims no optimization gain.

Validation:

  • CPU: python3 -m pytest -q tests — 19,196 passed, 195 skipped, 35 subtests passed. After the final unscored MoE boundary additions, focused checks passed again: 97 direct controls and 52 task-contract/history regressions. make check-perf-helpers also passed. The Linux process-cleanup test now handles a process disappearing while its /proc status is being read; its full test module passed (45 tests), followed by 20 repeated runs of both real subprocess cleanup cases (40 checks). This is a test-only race fix; task and evaluator runtime sources are unchanged.
  • GPU: All 39 repaired task packages have framework-finalized PASS reports. Qualifications use MI355X/gfx950 and a pinned Docker runtime. Finalized reports, source/runtime identities, raw action results and earlier WARN/FAIL evidence are retained outside the task packages. Reports span immutable qualification snapshots; this is not a claim that every task ran under one Git snapshot.
  • Sage: the formally qualified final implementation also passed 70 input checks at the unchanged 0.25 normalized-max-error gate; the worst observed value was 0.0642947. Extra FP4 compute/storage is part of the repaired baseline, not a same-cost accuracy or speedup claim.

No agent optimization campaign was run for these repairs. GPU qualification on other architectures is not claimed.

The source checks are structural controls, not a Python security sandbox. Submitted candidates still need their own full correctness/performance evaluation and independent semantic review.

irvineoy and others added 4 commits October 27, 2026 21:21
PR 127 hardened 37 FlyDSL task runtimes against candidates that reach
evaluator state: importing the materialized timing helper, mutating
imported modules such as torch.cuda.Event, reading frame or function
globals, routing to sys through os or builtins, or disabling the
PyTorch dispatch audit. The other 74 runtimes had no import closure at
all, so a candidate could fabricate speedups there.

Port the reviewed candidate_source_closure and
check_candidate_control_plane bodies unchanged into those 74 copies,
using the same check_dependencies edits PR 127 applied to the matching
pre-image: the softmax_kernel form for the newer structure and the
merge_state/rmsnorm2d form for the older one. 60 copies are now
byte-identical to an already hardened runtime.

The bundled flydsl_compat/meta.py in six flydsl2flydsl tasks walked
frame.f_back for MLIR locations, so their shipped baselines failed the
frame check. Select the same frame with traceback.walk_stack instead;
a CPU test proves the selected frame and locations are unchanged.

Document the protected state in each changed README. Extend the
boundary tests to every FlyDSL runtime by family, add per-route and
shipped-source checks, pin a full sha-bound inverse for the ported
runtimes, and keep out-of-root candidate tests reaching their
intended checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The validator review rejected the harness boundary of two ported tasks:
the guard accepts new non-protected files and the import closure runs
them, yet the README only described the closure. Say explicitly that
declared editable sources may import new task-local helper modules or
packages, which are part of the candidate and checked with it, as the
PR 127 closure tasks already document.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the existing import and introspection restrictions while extending the candidate dependency closure. Use frame-free line-level locations in bundled compatibility helpers and preserve static buffer extent calculation. Update fixture roots and exact inverse checks for the current main sources.
@irvineoy
irvineoy force-pushed the codex/harden-remaining-task-runtimes branch from 4782181 to e966fa7 Compare October 28, 2026 06:10
@irvineoy irvineoy changed the title Protect evaluator state in the remaining FlyDSL task runtimes Harden task validation and repair across the remaining FlyDSL tasks Oct 28, 2026
@irvineoy
irvineoy changed the base branch from codex/rl-task-gpu-quality to main October 28, 2026 06:11
@irvineoy
irvineoy marked this pull request as ready for review October 28, 2026 20:08
@irvineoy
irvineoy merged commit 0c6ab67 into main Oct 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant