Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion PERF.md
Original file line number Diff line number Diff line change
Expand Up @@ -1663,7 +1663,7 @@ Pinned by `internal/session/window_policy_test.go`,
| Law | Where it is pinned |
| --- | --- |
| A warm tool-schema encode allocates **nothing**. The belt is append-only, so the memo hands back the slice it holds. | `internal/provider/alloclaws_test.go` |
| A warm transcript encode costs the **same** at 81 turns as at 8 — 1 allocation automatic, 8 with breakpoints. Encode runs once per call, so a per-call cost linear in the transcript is a per-run cost quadratic in the run. | `internal/provider/alloclaws_test.go` |
| A warm transcript encode costs the **same** at 81 turns as at 8 — 1 allocation automatic; with breakpoints, the result slice, the parts slice the marked system message is expanded into, and encoding/json's own price for the two marked marshals, measured in-process (8 in all on Go 1.26, 11 on Go 1.27 — both measured). This package's own allocations are named, and an extra allocation of its own anywhere on that path, inside `marshalMarked` included, fails. Encode runs once per call, so a per-call cost linear in the transcript is a per-run cost quadratic in the run. | `internal/provider/alloclaws_test.go` |
| The babble guard builds **one** zlib writer per stream and Resets it per window; a window costs at most 4 allocations. A writer per window is a hundred kilobytes of deflate state per five hundred bytes of reply. | `internal/provider/alloclaws_test.go` |
| The hub's backlog fold is **amortized constant per delta**: ten times the deltas for less than twice the allocations. `Text += delta` is quadratic — 1.6 GB of copying over one long reply, under the hub's lock. | `internal/session/alloclaws_test.go` |
| Splitting or shape-checking a **4,000-line paste** allocates at most **10 times** the bytes of 1,000 lines. A linear parser measured 5.1 (slices grow by doubling, so up to twice either side of fourfold); the parser that copied the rest of the paste at every word measured 15.3 and froze a real 4,000-line paste for twenty seconds. The gate counts bytes because that copy is one allocation per word, so an allocation count grows fourfold either way. | `internal/tui3/imagepaste_test.go` |
Expand Down
9 changes: 7 additions & 2 deletions cmd/codeaf/do_engine_contract_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,12 @@ func TestDoOnTheRunEngineSignsWorkerCommitsUnlessContributingForbids(t *testing.
func TestDoOnTheRunEngineSignsFirstCommitOnUnbornBranch(t *testing.T) {
beltRunEnv(t)
t.Setenv("CODEAF_PLANDB_BIN", beltPlandbDoor(t))
workspace := t.TempDir()
// Canonical for the same reason beltRepoWorkspace resolves its folder: the
// envelope names files under the engine's resolved root.
workspace, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
beltGit(t, workspace, "init")
beltGit(t, workspace, "checkout", "-b", "work")
seat := &beltSeat{
Expand All @@ -240,7 +245,7 @@ func TestDoOnTheRunEngineSignsFirstCommitOnUnbornBranch(t *testing.T) {
},
}
var stdout, stderr strings.Builder
err := doErrand(doRequest{
err = doErrand(doRequest{
task: "add hello.txt", workspace: workspace, asJSON: true,
timeout: 60 * time.Second, slots: bound(1), stdout: &stdout, stderr: &stderr,
newBeltCompleter: func(string) session.Completer { return seat },
Expand Down
10 changes: 9 additions & 1 deletion cmd/codeaf/do_engine_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,15 @@ func beltRepoWorkspace(t *testing.T) string {
if _, err := exec.LookPath("git"); err != nil {
t.Skip("git is not on PATH")
}
dir := t.TempDir()
// The engine records every path under the repository's canonical root
// (internal/session repositoryRoot resolves symlinks so one spelling serves
// every later path and lock), and on macOS t.TempDir() is spelled through
// the /var -> /private/var link. Resolving it here lets a test compare the
// files the envelope names against the workspace it built, on any host.
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
beltGit(t, dir, "init")
beltGit(t, dir, "checkout", "-b", "work")
if err := os.WriteFile(filepath.Join(dir, "README.md"), []byte("the project\n"), 0o644); err != nil {
Expand Down
20 changes: 20 additions & 0 deletions docs/changes/unreleased/1722-test-suite-green-on-macos.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
kind: fixed
title: the test suite is green on macOS, and the allocation law prices its marshals itself
pr: 1722
surface: [engine]
invalidates:
- "the ten tests that failed `make pr-ready` on a clean dev on a Mac were a host or toolchain assumption in each test, never a product fault or a flake — CI on ubuntu was green for the same commit."
- "`internal/provider/alloclaws_test.go` no longer names 8 for a warm breakpoints encode; encoding/json's own price is measured in-process, this package's own allocations are named, and nothing more is allowed (8 in all on Go 1.26, 11 on Go 1.27 — both measured)."
---

Four `internal/session` tests that hold a task under a memory floor no machine
meets (`1 << 40` MiB) state their machine through `littleMemoryHost`, a `readHost`
seam the governors are built over, because the real reading is `/proc/meminfo` and a
host without it admits everything; they now run and pass on every host instead
of holding only on Linux. Four `cmd/codeaf` engine tests, the standing
isolation test and the held-run reopen test compare against the canonical temp
folder, since the engine records the resolved root and macOS spells `/var`
through `/private/var`. The warm breakpoints encode law measures encoding/json's
own price in-process, names this package's own result and parts slice allocations,
and allows nothing more anywhere on the warm path, inside `marshalMarked` included.
161 changes: 148 additions & 13 deletions internal/provider/alloclaws_test.go
Original file line number Diff line number Diff line change
@@ -1,8 +1,13 @@
package provider

import (
"encoding"
"encoding/json"
"fmt"
"reflect"
"testing"

"github.com/Agent-Field/agentfield/sdk/go/ai"
)

// THE LAWS THIS FILE DEFENDS ARE ABOUT WORK, NOT ABOUT TIME.
Expand Down Expand Up @@ -52,6 +57,78 @@ func TestTheWarmToolEncodeAllocatesNothing(t *testing.T) {
}
}

// markedJSONPrice prices only the encoding/json calls the benchmark's two
// marked shapes are designed to make. Their inputs are built before measuring,
// so this baseline cannot absorb an allocation added inside marshalMarked or
// the memo. A new fixture shape needs its own price rather than a guess.
func markedJSONPrice(t *testing.T, message ai.Message) float64 {
t.Helper()
// A wire type's own marshaler makes encoding/json's measured price include this package's work.
// A deliberate marshaler needs named overhead here and in PERF.md in the same change.
for _, wireType := range []reflect.Type{
reflect.TypeFor[markedTextPart](),
reflect.TypeFor[markedMessage](),
reflect.TypeFor[markedToolMessage](),
} {
for _, candidate := range []reflect.Type{wireType, reflect.PointerTo(wireType)} {
for _, marshaler := range []reflect.Type{
reflect.TypeFor[json.Marshaler](),
reflect.TypeFor[encoding.TextMarshaler](),
} {
if candidate.Implements(marshaler) {
t.Fatalf("%v implements %v: encoding/json's price is measured only while the wire types "+
"have no marshaler of their own. A marshaler added on purpose is this package's work "+
"and must be priced as a named overhead here and in PERF.md together.", candidate, marshaler)
}
}
}
}
if len(message.Content) != 1 || message.Content[0].Type != "text" || len(message.ToolCalls) != 0 {
t.Fatalf("marked JSON price has no law for role %q with %d content parts and %d tool calls",
message.Role, len(message.Content), len(message.ToolCalls))
}
price := func(marshal func() ([]byte, error)) float64 {
return testing.AllocsPerRun(100, func() {
if _, err := marshal(); err != nil {
t.Fatalf("pricing encoding/json: %v", err)
}
})
}
part := message.Content[0]
switch message.Role {
case "system":
if message.ToolCallID != "" {
t.Fatalf("marked JSON price has no law for a system message with tool id %q", message.ToolCallID)
}
marshalPart := func() ([]byte, error) {
return json.Marshal(markedTextPart{
Type: part.Type, Text: part.Text, CacheControl: ephemeralBreakpoint,
})
}
raw, err := marshalPart()
if err != nil {
t.Fatalf("building marked JSON price inputs: %v", err)
}
parts := []json.RawMessage{raw}
return price(marshalPart) + price(func() ([]byte, error) {
return json.Marshal(markedMessage{Role: message.Role, Content: parts})
})
case "tool":
if message.ToolCallID == "" {
t.Fatal("marked JSON price has no law for a tool result without an id")
}
return price(func() ([]byte, error) {
return json.Marshal(markedToolMessage{
Role: message.Role, Content: part.Text,
ToolCallID: message.ToolCallID, CacheControl: ephemeralBreakpoint,
})
})
default:
t.Fatalf("marked JSON price has no law for role %q", message.Role)
return 0
}
}

// THE WARM TRANSCRIPT ENCODE COSTS THE SAME AT EIGHTY TURNS AS AT EIGHT.
//
// This is the law the memo exists for, stated as the only thing that can prove
Expand All @@ -60,20 +137,38 @@ func TestTheWarmToolEncodeAllocatesNothing(t *testing.T) {
// linear in transcript length is a per-run cost quadratic in the length of the
// run — which is exactly what it was before memo.go.
//
// The two constants are named rather than merely bounded, because each of them
// is a specific thing and a change to either is a change worth reading:
// This package's own allocations are named rather than merely bounded, because
// each is a specific thing and a change to one is a change worth reading:
//
// - automatic: 1 — the []json.RawMessage the caller is handed. Every element
// of it is a slice the memo already holds. There is nothing else to pay.
// - breakpoints: 8 — that same slice, plus the two marked positions, which are
// re-derived per call BY DESIGN. The tail marker rolls forward every turn, so
// a memo of the marked form would be a cache of the one thing that changes
// (memo.go says this where it explains what is deliberately not memoized).
// Two marshals, whatever they cost, and never 244.
// - breakpoints: that same slice, the parts slice the marked system message is
// expanded into, and encoding/json's price for the two marked positions.
// The tool result has no overhead of its own. The marked forms are re-derived
// per call BY DESIGN: the tail marker rolls forward every turn, so a memo of
// it would cache the one thing that changes (memo.go explains this choice).
//
// ONLY ENCODING/JSON'S PRICE IS MEASURED IN THIS PROCESS. The complete
// breakpoints path cost 8 on Go 1.26 and 11 on Go 1.27 with the memo unchanged,
// so a total copied from one toolchain turned a Go upgrade into a red that CI
// could not see. The baseline now calls encoding/json directly with this
// package's wire types and prebuilt inputs; it never calls marshalMarked or
// encodeMessages. This package's own costs stay named, and each marked position
// is checked before the whole warm path. An extra allocation of this package's
// own ANYWHERE ON THAT PATH, inside marshalMarked included, therefore fails
// instead of raising its own allowance.
//
// The equality across sizes is the load-bearing assertion; the constants are the
// teaching. One-per-message at 81 turns would be 244.
func TestTheWarmTranscriptEncodeCostsTheSameAtEightyTurnsAsAtEight(t *testing.T) {
const (
// resultSliceCost is the slice the caller needs to hold this call's encoded transcript.
resultSliceCost = 1
// partsSliceCost is the storage needed to expand the marked system message into array form.
partsSliceCost = 1
// toolOverhead is zero because the marked tool result needs no intermediate storage.
toolOverhead = 0
)
warmEncode := func(t *testing.T, turns int, dialect cacheDialect) float64 {
t.Helper()
messages := benchTranscript(turns)
Expand All @@ -88,29 +183,69 @@ func TestTheWarmTranscriptEncodeCostsTheSameAtEightyTurnsAsAtEight(t *testing.T)
})
}

messages := benchTranscript(81)
placed := breakpointsFor(messages)
if placed.system < 0 || placed.tail < 0 {
t.Fatal("the benchmark transcript must place both a system and a tail breakpoint")
}
systemJSON := markedJSONPrice(t, messages[placed.system])
tailJSON := markedJSONPrice(t, messages[placed.tail])
for _, position := range []struct {
name string
index int
jsonCost float64
ownCost float64
}{
{"system", placed.system, systemJSON, partsSliceCost},
{"tail", placed.tail, tailJSON, toolOverhead},
} {
allocations := testing.AllocsPerRun(100, func() {
if _, err := marshalMarked(messages[position.index]); err != nil {
t.Fatalf("marked %s position: %v", position.name, err)
}
})
t.Logf("marked %s position: encoding/json=%.0f package=%.0f total=%.0f",
position.name, position.jsonCost, position.ownCost, allocations)
if want := position.jsonCost + position.ownCost; allocations != want {
change := "Fewer allocations mean the marked shape or its named overhead has changed."
if allocations > want {
change = "An extra allocation inside marshalMarked must fail rather than raise its own allowance."
}
t.Fatalf("marked %s position (%d) costs %.0f allocations, want %.0f: "+
"encoding/json's price is measured in-process (%.0f), and this package's "+
"own overhead is named (%.0f). %s If this is deliberate, update markedJSONPrice's "+
"shapes, the named overhead constants, and the PERF.md row together.",
position.name, position.index, allocations, want, position.jsonCost, position.ownCost, change)
}
}

for _, dialect := range []struct {
name string
d cacheDialect
want float64
}{
{"automatic", cacheDialectAutomatic, 1},
{"breakpoints", cacheDialectBreakpoints, 8},
{"automatic", cacheDialectAutomatic, resultSliceCost},
{"breakpoints", cacheDialectBreakpoints, resultSliceCost + systemJSON + partsSliceCost + tailJSON + toolOverhead},
} {
t.Run(dialect.name, func(t *testing.T) {
// 8 and 81 turns are the ends of the range BENCHMARKS.md records for
// real runs, and they are 25 and 244 messages long.
small, large := warmEncode(t, 8, dialect.d), warmEncode(t, 81, dialect.d)
t.Logf("warm %s encode: 8 turns=%.0f 81 turns=%.0f law=%.0f",
dialect.name, small, large, dialect.want)
if small != large {
t.Fatalf("a warm encode costs %.0f allocations at 8 turns and %.0f at 81 — "+
"the memo's whole promise is that this number does not move with the "+
"transcript, and a cost that grows here is a per-run cost that grows "+
"quadratically (memo.go)", small, large)
}
if large != dialect.want {
t.Fatalf("a warm %s encode costs %.0f allocations, and the law is %.0f. "+
"If this is a deliberate change — a different marked-message shape, a "+
"different result slice — move the number here and in PERF.md together; "+
"if it is not, something on the warm path stopped being memoized.",
t.Fatalf("a warm %s encode costs %.0f allocations, and the law is %.0f: "+
"encoding/json's price is measured in-process, and this package's own "+
"allocations are named. An extra allocation of this package's own "+
"anywhere on the warm path, inside marshalMarked included, fails. If "+
"this is deliberate, change the law here and in PERF.md together; if "+
"it is not, the warm path is doing work it should not.",
dialect.name, large, dialect.want)
}
})
Expand Down
14 changes: 13 additions & 1 deletion internal/session/run_admission_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,22 @@ import (
"github.com/Agent-Field/codeaf/internal/config"
)

func TestRunAdmissionUsesRealHostMemoryAndSharedLanes(t *testing.T) {
// requireHostMemoryReading skips a test that is about the REAL reading. It is
// /proc/meminfo (hostMemoryMB), and a host without it — macOS — leaves the
// governor unknown, and an unknown governor admits everything
// (admissionGovernor.admits: silence is not pressure). A test of held-run
// behaviour that merely needs a hold states its machine with littleMemoryHost
// instead and runs everywhere; only a test whose claim is about this host's
// own numbers belongs behind this skip.
func requireHostMemoryReading(t *testing.T) {
t.Helper()
if _, err := os.Stat("/proc/meminfo"); err != nil {
t.Skip("host has no proc memory reading")
}
}

func TestRunAdmissionUsesRealHostMemoryAndSharedLanes(t *testing.T) {
requireHostMemoryReading(t)
lanes := NewTaskLanes()
gate := NewRunAdmission(0, 1<<40, lanes)
if gate == nil || gate.MayStart() {
Expand Down
6 changes: 5 additions & 1 deletion internal/session/standing_isolation_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,12 @@ func TestStandingIsolationRecordsItsCopyBeforeWorkerInitialization(t *testing.T)
if _, err := runner.Run(context.Background(), item, runDir, ""); !errors.Is(err, want) {
t.Fatalf("initialization error changed: %v", err)
}
// The record keeps the repository's canonical root ([repositoryRoot]), so
// the comparison resolves symlinks too: on macOS t.TempDir() answers under
// /var/folders while the same directory's canonical spelling is under
// /private/var/folders, and a raw comparison fails a correct record.
trees := loadStandingTrees(runDir)
if len(trees) != 1 || trees[0].Root != repo || trees[0].Home == "" || trees[0].HomeSha == "" {
if len(trees) != 1 || trees[0].Root != canonicalPath(repo) || trees[0].Home == "" || trees[0].HomeSha == "" {
t.Fatalf("lost recovery identity: %+v", trees)
}
if got := currentBranch(trees[0].Dir); got != trees[0].Branch {
Expand Down
13 changes: 11 additions & 2 deletions internal/session/task_pressure.go
Original file line number Diff line number Diff line change
Expand Up @@ -365,7 +365,7 @@ func newAdmissionGovernor(maxLoad float64, minFreeMB int) *admissionGovernor {
return &admissionGovernor{
maxLoad: maxLoad,
minFreeMB: minFreeMB,
read: hostReading,
read: readHost,
now: time.Now,
}
}
Expand All @@ -378,7 +378,7 @@ func newAdmissionGovernorForProfile(maxLoad float64, minFreeMB int, profileDir s
governor := newAdmissionGovernor(maxLoad, minFreeMB)
// An empty profile selects the ordinary CODEAF_HOME profile too.
if governor == nil {
governor = &admissionGovernor{read: hostReading, now: time.Now}
governor = &admissionGovernor{read: readHost, now: time.Now}
}
governor.settings = func() (float64, int) {
return config.TaskAdmissionLimitsAt(profileDir, maxLoad, minFreeMB)
Expand Down Expand Up @@ -563,6 +563,15 @@ func (g *admissionGovernor) footprintLocked() int {
return max(share, g.peakShareMB)
}

// readHost is the reading every governor is built over, and it is a variable
// for one reason: a test of HELD work needs the hold to be there at the first
// StartTask, before any governor exists for it to reach into through the
// `read` field, and on a host with no /proc — macOS — the real reading is
// unknown and an unknown reading holds nothing. A test states its machine
// here (littleMemoryHost in task_pressure_test.go) and restores this when it
// ends. Nothing outside a test assigns it.
var readHost = hostReading

// hostReading asks this machine what it is carrying. Every part is read
// independently, and a part that cannot be read is reported as zero rather
// than as a failure of the whole: a kernel with a loadavg and no meminfo still
Expand Down
1 change: 1 addition & 0 deletions internal/session/task_pressure_external_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ func TestMachineAdmissionNoticesSettingsFromAnotherProcess(t *testing.T) {
}

func TestNewAdmissionReadsSettingsChangedBeforeItsCreation(t *testing.T) {
littleMemoryHost(t)
for _, profileKind := range []string{"named", "default"} {
t.Run(profileKind, func(t *testing.T) {
profile := t.TempDir()
Expand Down
15 changes: 15 additions & 0 deletions internal/session/task_pressure_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,21 @@ func loaded() machineReading {
return reading
}

// littleMemoryHost makes every governor built for the rest of the test see a
// quiet machine with one gibibyte available, so a test that holds work under
// a 1 EiB floor (`TaskMinFreeMB = 1 << 40` MiB) is held on every host. The real
// reading is /proc/meminfo, and a host without it — macOS — reports unknown,
// which admits everything; stating the machine here instead of skipping
// keeps the held-run tests running wherever the suite is run.
func littleMemoryHost(t *testing.T) {
t.Helper()
reading := roomFor(0)
reading.availableMB = 1024
previous := readHost
readHost = func() (machineReading, bool) { return reading, true }
t.Cleanup(func() { readHost = previous })
}

// fanRunner is a scripted runner that only records a start: every node it is
// handed stays running until the test settles it, ON THE TEST'S OWN GOROUTINE,
// so the frontier pass a settle causes has finished before the next line reads
Expand Down
Loading
Loading