From ffbf26e2e94ac82d702bcf2995671073d9caa119 Mon Sep 17 00:00:00 2001 From: Tequila Sunset Date: Tue, 11 Aug 2026 22:42:02 +0800 Subject: [PATCH] Make fuzz budgets deterministic --- Makefile | 9 +- ROADMAP.md | 18 ++-- design/release-0.1.0.md | 30 +++++-- design/release.md | 5 +- docs/release-0.1.0.md | 7 +- internal/testcheck/main_test.go | 16 ++++ internal/testcheck/makefile_release_test.go | 8 +- .../testcheck/testdata/fuzzfail/fuzz_test.go | 10 +++ .../release-0.1.0-issue-118-fuzz-budget.md | 85 +++++++++++++++++++ 9 files changed, 158 insertions(+), 30 deletions(-) create mode 100644 internal/testcheck/testdata/fuzzfail/fuzz_test.go create mode 100644 research/release-0.1.0-issue-118-fuzz-budget.md diff --git a/Makefile b/Makefile index a3503d4..c842104 100644 --- a/Makefile +++ b/Makefile @@ -3,9 +3,6 @@ # Keep release evidence in the specified order, including with make -j. .NOTPARALLEL: -# Caller-provided Make variables must not replace release test commands. -FUZZ_TIME ?= 10s - test: go run ./internal/testcheck ./... @@ -33,9 +30,9 @@ net: go run ./internal/testcheck -tags net ./examples/shadow/... fuzz: - go run ./internal/testcheck ./transport -run '^$$' -fuzz '^FuzzReadFrame$$' -fuzztime=$(FUZZ_TIME) - go run ./internal/testcheck . -run '^$$' -fuzz '^FuzzDecodeRequestContext$$' -fuzztime=$(FUZZ_TIME) - go run ./internal/testcheck ./internal/codegen -run '^$$' -fuzz '^FuzzParseGrainMarker$$' -fuzztime=$(FUZZ_TIME) + go run ./internal/testcheck ./transport -run '^$$' -fuzz '^FuzzReadFrame$$' -fuzztime=1600000x + go run ./internal/testcheck . -run '^$$' -fuzz '^FuzzDecodeRequestContext$$' -fuzztime=225000x + go run ./internal/testcheck ./internal/codegen -run '^$$' -fuzz '^FuzzParseGrainMarker$$' -fuzztime=2250000x resource: go run ./internal/testcheck ./internal/runtime -run '^TestRuntime_ActivationChurnReleasesOwnedResources$$' -count=1 diff --git a/ROADMAP.md b/ROADMAP.md index 33b874f..fd45d51 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -9,9 +9,10 @@ Grain Timers, State failure recovery, bounded Reminder delivery, the Silo Activation admission limit, and bounded resource tests are complete for 0.1.0. Portable benchmark probes are also complete. The fail-closed local release gate and fuzz checks are also complete. The external restart proof and public -document cleanup are complete. The exact tagged-module command is ready. The -hosted supported-system run and the final same-commit candidate proof are -complete. The successful tagged-module proof remains open. Action recovery +document cleanup are complete. The exact tagged-module command is ready. A +release-gate follow-up invalidated the earlier candidate proof. The hosted +supported-system run, final same-commit candidate proof, and successful +tagged-module proof remain open. Action recovery confirms Grain State before it commits an Application receipt. Pending actions recover in save order. An Invalid Reminder now gets one Terminal Result. Direct lifecycle proof for a mutable State alias and a @@ -188,14 +189,15 @@ The implementation batches are: before it commits an Application receipt. A fourth follow-up gives an Invalid Reminder one Terminal Result. A fifth follow-up proves the direct State failure lifecycle. A sixth follow-up migrates the exact `v0.0.5` - schema and proves the external upgrade path. The hosted platform run and - final same-commit candidate proof are complete. The successful - tagged-module run is still open. + schema and proves the external upgrade path. A seventh follow-up gives + each fuzz target a fixed iteration budget. The hosted platform run, final + same-commit candidate proof, and successful tagged-module run are open for + the new candidate. The exact scope and proof for each batch are in [design/release-0.1.0.md](design/release-0.1.0.md). Batches 1 through 7 and -Batch 8a through 8d are complete. Batch 8e candidate proof is complete; its -exact tagged-module proof remains open. +Batch 8a through 8d are complete. Batch 8e candidate and exact tagged-module +proofs remain open. ## Cluster preview diff --git a/design/release-0.1.0.md b/design/release-0.1.0.md index 5adab68..86d6d8a 100644 --- a/design/release-0.1.0.md +++ b/design/release-0.1.0.md @@ -391,8 +391,18 @@ Add these fuzz targets: Each fuzz target must contain valid and invalid seed inputs. A successful decode must satisfy the same size and round-trip rules as normal tests. `make fuzz` -must run each target separately for 10 seconds by default. Go must save a -failure in the normal fuzz corpus. +must run each target separately with a fixed iteration budget: + +- `FuzzReadFrame`: 1,600,000 iterations; +- `FuzzDecodeRequestContext`: 225,000 iterations; +- `FuzzParseGrainMarker`: 2,250,000 iterations. + +The commands must use Go's `-fuzztime=Nx` form and must hard-code these budgets. +A caller-provided Make variable must not reduce them. A fixed count gives every +runner the same minimum work and avoids the Go fuzz coordinator's racy +wall-clock deadline. A real target failure must still fail the command and Go +must save it in the normal fuzz corpus. The measured basis is in +[the issue 118 research note](../research/release-0.1.0-issue-118-fuzz-budget.md). Do not wrap a fuzz package in a process-wide goroutine count. The Go fuzz runner owns background goroutines, and their lifetime is not a package @@ -423,6 +433,7 @@ Proof: `ci`, `external`, or `external-tagged`; - a mutation that removes the zero-test rejection fails its behavior test; - each fuzz target runs its seed corpus and one focused fuzz session; +- the internal test command preserves a real fuzz callback failure; - `make ci` passes on Linux amd64; - all hosted platform jobs pass on the same commit. @@ -473,8 +484,9 @@ Proof: #### Batch 8e: Final candidate proof -**Status: In progress.** The final `master` candidate commands and hosted -platform jobs are complete. The exact tagged-module proof remains pending. +**Status: In progress.** A release-gate change invalidated the earlier final +candidate proof. The new final `master` candidate commands, hosted platform +jobs, and exact tagged-module proof remain pending. Before the final proof, complete the release audit for the upgrade from `v0.0.5`. That version stores Reminder identity in the `entity_type` and @@ -567,13 +579,15 @@ Batch 7 hardens generation and moves implementation code under `internal/`. Batch 8a adds Activation admission. Batch 8b adds bounded resource proof and portable benchmark probes. Batch 8c implements fail-closed local release checks, fixed tools, fuzz checks, and the hosted platform workflow. The hosted -workflow has passed on the final `master` candidate. Batch 8d implements the +workflow passed before the fixed fuzz-budget follow-up. Batch 8d implements the external restart proof and public document cleanup. A release audit follow-up passes dynamic Reminder names to typed methods. A second follow-up prevents a caller-provided Make variable from replacing the internal test command. A third follow-up restores Grain State before it commits an Application receipt. An Invalid Reminder now gets one Terminal Result. A fifth follow-up proves the mutable State alias and Reminder-turn State failure lifecycle. Batch 8e's -candidate commands, hosted proof, and final same-commit proof are complete. It -still owns the exact tagged-module proof. A sixth release audit follow-up -migrates the exact `v0.0.5` Reminder schema and adds the external upgrade proof. +candidate commands, hosted proof, final same-commit proof, and exact +tagged-module proof are pending for the new candidate. A sixth release audit +follow-up migrates the exact `v0.0.5` Reminder schema and adds the external +upgrade proof. A seventh follow-up gives every fuzz target a fixed iteration +budget above the earlier successful work count. diff --git a/design/release.md b/design/release.md index 522cf80..91c7781 100644 --- a/design/release.md +++ b/design/release.md @@ -90,8 +90,9 @@ The 0.0.x release-note question has a zero-maintenance answer: none is written. The earlier functional readiness work is complete. It includes public API comments, Runtime shutdown, errors, lifecycle, the example, observability, and benchmarks. The 0.1.0 contract adds composition and release evidence gates. -Public document cleanup, hosted jobs, and the final same-commit candidate proof -are complete. The final tagged-module proof remains open. See +Public document cleanup is complete. A fixed fuzz-budget follow-up invalidated +the earlier candidate proof. Hosted jobs, the final same-commit candidate +proof, and the final tagged-module proof remain open. See [ROADMAP.md](../ROADMAP.md). Cluster support remains a preview. A partition can misjudge healthy Silos. diff --git a/docs/release-0.1.0.md b/docs/release-0.1.0.md index 19a2106..694db0c 100644 --- a/docs/release-0.1.0.md +++ b/docs/release-0.1.0.md @@ -308,6 +308,7 @@ internal test command. Action recovery confirms requested State before it commits the Application receipt. The clean external-module restart proof passes locally on Linux. The exact `v0.0.5` migration and external upgrade proof also pass locally on Linux. An Invalid Reminder now gets one Terminal -Result. Public document cleanup is complete. Hosted -supported-system proof and the final same-commit candidate proof are complete. -The exact tagged-module run remains open. ROADMAP.md tracks the batches. +Result. Public document cleanup is complete. A fixed fuzz-budget follow-up +invalidated the earlier candidate proof. Hosted supported-system proof, the +final same-commit candidate proof, and the exact tagged-module run remain open. +ROADMAP.md tracks the batches. diff --git a/internal/testcheck/main_test.go b/internal/testcheck/main_test.go index 4e0662e..894e58c 100644 --- a/internal/testcheck/main_test.go +++ b/internal/testcheck/main_test.go @@ -38,3 +38,19 @@ func TestRunKeepsGoTestFailure(t *testing.T) { t.Fatalf("stdout = %q, want child test failure", stdout.String()) } } + +func TestRunKeepsFuzzFailure(t *testing.T) { + var stdout, stderr bytes.Buffer + code := run([]string{ + "-run", "^$", + "-fuzz", "^FuzzFail$", + "-fuzztime=1x", + "./testdata/fuzzfail", + }, &stdout, &stderr) + if code != 1 { + t.Fatalf("run exit = %d, want 1\nstdout:\n%s\nstderr:\n%s", code, stdout.String(), stderr.String()) + } + if !strings.Contains(stdout.String(), "intentional fuzz failure") { + t.Fatalf("stdout = %q, want child fuzz failure", stdout.String()) + } +} diff --git a/internal/testcheck/makefile_release_test.go b/internal/testcheck/makefile_release_test.go index 1a9ce51..5d2101d 100644 --- a/internal/testcheck/makefile_release_test.go +++ b/internal/testcheck/makefile_release_test.go @@ -38,9 +38,9 @@ var releaseCommandContracts = []struct { "go run ./internal/testcheck -tags net ./examples/shadow/...", "go run ./internal/testcheck ./internal/runtime -run '^TestRuntime_ActivationChurnReleasesOwnedResources$' -count=1", "go run ./internal/testcheck ./internal/timer -run '^TestPoller_LargeBacklogKeepsPagesAndWorkersBounded$' -count=1", - "go run ./internal/testcheck ./transport -run '^$' -fuzz '^FuzzReadFrame$' -fuzztime=10s", - "go run ./internal/testcheck . -run '^$' -fuzz '^FuzzDecodeRequestContext$' -fuzztime=10s", - "go run ./internal/testcheck ./internal/codegen -run '^$' -fuzz '^FuzzParseGrainMarker$' -fuzztime=10s", + "go run ./internal/testcheck ./transport -run '^$' -fuzz '^FuzzReadFrame$' -fuzztime=1600000x", + "go run ./internal/testcheck . -run '^$' -fuzz '^FuzzDecodeRequestContext$' -fuzztime=225000x", + "go run ./internal/testcheck ./internal/codegen -run '^$' -fuzz '^FuzzParseGrainMarker$' -fuzztime=2250000x", }, }, { @@ -70,7 +70,9 @@ func TestMakefile_ReleaseCommandsKeepTestcheck(t *testing.T) { {"GO_TEST=true"}, {"MAKE=true"}, {"GO_TEST=true", "MAKE=true"}, + {"FUZZ_TIME=1x"}, {"-j8"}, + {"-j8", "FUZZ_TIME=1x"}, {"-j8", "GO_TEST=true", "MAKE=true"}, } { overridden := makeDryRun(t, repository, contract.target, settings...) diff --git a/internal/testcheck/testdata/fuzzfail/fuzz_test.go b/internal/testcheck/testdata/fuzzfail/fuzz_test.go new file mode 100644 index 0000000..6c2cae0 --- /dev/null +++ b/internal/testcheck/testdata/fuzzfail/fuzz_test.go @@ -0,0 +1,10 @@ +package fuzzfail + +import "testing" + +func FuzzFail(f *testing.F) { + f.Add("seed") + f.Fuzz(func(t *testing.T, value string) { + t.Fatal("intentional fuzz failure") + }) +} diff --git a/research/release-0.1.0-issue-118-fuzz-budget.md b/research/release-0.1.0-issue-118-fuzz-budget.md new file mode 100644 index 0000000..b0e349a --- /dev/null +++ b/research/release-0.1.0-issue-118-fuzz-budget.md @@ -0,0 +1,85 @@ +# Issue 118 fuzz budget evidence + +## Observation + +The final `master` proof for +`884351d1b130ece20ede99c300995677ff202f9b` ran in GitHub Actions workflow +[31499798759](https://github.com/suraciii/gor/actions/runs/31499798759). +Linux stable used Go 1.26.5. `make ci` reached the last fuzz target and then +failed: + +```text +fuzz: elapsed: 10s, execs: 421658 (38686/sec), new interesting: 22 (total: 75) +--- FAIL: FuzzParseGrainMarker (10.08s) + context deadline exceeded +``` + +No product assertion failed. Go wrote no failing corpus input. The earlier two +targets completed 1,356,179 and 32,992 executions. The second target stopped +making progress before its wall-clock deadline but still passed. + +PR workflow +[31499526676](https://github.com/suraciii/gor/actions/runs/31499526676) +passed on `be9a3e02aae7b26a86d375f817bd5cca75a3261d`. The PR head and squash commit +have the same tree, `fe4ff904865aaf5fe259b9b7a5ea8d0103012d2b`. The result +therefore changed without a source-tree change. + +## Cause + +The release gate used `-fuzztime=10s`. Go implements a duration budget with a +context deadline. The coordinator can race with propagation of that normal +deadline and return `context deadline exceeded` as a test failure. The Go +project tracks this defect in +[golang/go#75804](https://github.com/golang/go/issues/75804). The upstream fix +in [golang/go#79199](https://github.com/golang/go/pull/79199) suppresses the +parent deadline as well as the child deadline. It was merged to Go master and +backported to the Go 1.27 branch, but the required Go 1.25 and Go 1.26.5 lanes +do not contain it. + +Retrying the workflow would not remove this race. Ignoring the error would also +hide real target timeouts. + +## Decision + +Use Go's fixed-iteration `-fuzztime=Nx` mode. Set each budget above both the +failed hosted run and the earlier successful ten-second run: + +| Target | Failed hosted run | Earlier successful run | Fixed budget | +| --- | ---: | ---: | ---: | +| `FuzzReadFrame` | 1,356,179 | 1,508,765 | 1,600,000 | +| `FuzzDecodeRequestContext` | 32,992 | 200,792 | 225,000 | +| `FuzzParseGrainMarker` | 421,658 | 2,185,654 | 2,250,000 | + +The fixed counts preserve a real fuzz session, require more work than either +recorded run, and remove only the wall-clock stop race. The commands remain +fail-closed for a panic, assertion, worker timeout, or invalid corpus entry. + +## Proof + +Working-tree validation passed before review: + +- `make fuzz` passed with exit 0 in 23.82 seconds at the first proposed fixed + counts; +- changing the first expected count by one made the release contract test fail + with exit 1, and restoring it made the test pass; +- `make ci` passed with exit 0 in 48.98 seconds with a cold test cache; +- `make external` passed with exit 0 in 31.82 seconds with a cold test cache, + including 20 external tests and the exact `v0.0.5` upgrade. + +Review found that the first proposed counts were lower than an earlier +successful run. The final counts above correct that defect. After the +correction: + +- `make fuzz` passed with exit 0 in 22.18 seconds and completed 1,600,000, + 225,000, and 2,250,000 executions; +- changing the first expected count from 1,600,000 to 1,599,999 made the + release contract test fail with exit 1; +- the internal test command preserved exit 1 and output from an intentional + fuzz callback failure; +- `make ci` passed with exit 0 in 107.83 seconds with a cold test cache, + including 639 default and race tests and all three fixed fuzz budgets; +- `make external` passed with exit 0 in 34.81 seconds with a cold test cache, + including 20 external tests and the exact `v0.0.5` upgrade. + +The mutation was restored. Exact-commit local proof and hosted proof remain +pending.