Skip to content

Commit 18f72fa

Browse files
donislawdevclaude
donislawdev
and
claude
authored
preset: a set asks each format for its smallest size once, and plans one file once (#132)
* guard: every preset's expansion is pinned, not only size-boundaries TestEjectingAPresetGivesTheBytesItAlwaysGave pinned one case of one preset. The next change is to how upload-validation, tabular-import, text-encoding and empty-and-minimal work their sets out, and none of their bytes had a guard. Nineteen cases now, measured on the tree before that change: every preset, other formats in allow, and three refusals whose words carry a format's smallest size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * preset: a set asks each format for its smallest size once, and plans one file once Expanding upload-validation took 137-142 ms and 57 MB, tabular-import 16-31 ms and 39 MB, every time a value changed - in the window, on every key typed into a preset's setting or into the batch screen with a base. The profile named the cause. A format's smallest size is found by planning it at growing sizes, which for a picture is encoding one, and the presets asked it afresh: sampleFor once for every file about a name or an insides - the same question format.SmallestWithLabel already remembers - and setFile.bytes, the boundary set and the encoding set with settings or a seed, which nothing remembered. And upload-validation plans the same small picture under a dozen names, one encoding per name. - sampleFor asks format.SmallestWithLabel. - format.SmallestRemembered keeps SmallestAccepted by format and request, at most 4096 answers. SmallestWithLabel is that with the label on. - format.RequestKey writes a request as text no other request shares, with the values quoted so that a delimiter of "|" cannot join two settings. - uploadSet.reachable skips a file whose question it has already asked. The same request gets the same refusal and need, and the deepest shortfall keeps the first file that reached it, so no answer moves. The bytes of all nineteen pinned expansions and the words of three refusals are the same as before. A changed value now allocates 9.6 MB and takes 20-36 ms for upload-validation, 19.8 MB for tabular-import. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * guard: a new value asks each question of a preset's set once, and a request's key holds every field TestANewUploadLimitDoesNotAskTheSameQuestionsAgain and TestANewRowCountWorksTheSheetsSmallestSizeOutOnce hold the allocation of an expansion at a value the process has not seen, least of five, with the line between what was measured with the answers remembered and without. Each first asks that the set still holds what it is about - files asking one question under several names, a sheet at its floor. TestEveryFieldOfARequestIsInItsKey sets every field of format.Request in turn by reflection and wants a key of its own, and wants a value holding a space and an equals sign kept apart from two settings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * preset: the question already asked is told apart in a function of its own, so reachable stays two deep Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * changelog: changing a setting of a preset is about four times faster Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 364cda2 commit 18f72fa

9 files changed

Lines changed: 398 additions & 32 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -494,6 +494,13 @@ because it turns other people's test suites red.
494494
again for each one. Both are now done once, and the window uses less memory
495495
while you type.
496496

497+
- **Changing a setting of a preset is about four times faster.** With
498+
`upload-validation`, changing one of its settings held the window for
499+
about 0.15 seconds, and so did every key typed on `Several batches` built
500+
on it. Both now take about 0.03 seconds. Switching the base preset on
501+
`Several batches` to `upload-validation` went from about 0.3 to 0.07
502+
seconds. The sets the presets build are byte for byte the same as before.
503+
497504
- **Typing on the `Presets` screen no longer lags.** With `upload-validation`
498505
chosen, every key typed into a box held the window for about 0.3 seconds,
499506
and about 0.07 seconds with `tabular-import`, because the preset was worked

‎internal/format/registry.go‎

Lines changed: 73 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import (
44
"fmt"
55
"sort"
66
"strconv"
7+
"strings"
78
"sync"
89
)
910

@@ -49,31 +50,93 @@ func Register(d Descriptor) {
4950
// expansion afresh against 0.51 MB remembered, and a keystroke that cost
5051
// 380 ms in the real window (docs/GUI-MEMORY-2026-09-23.md section 2.3).
5152
//
52-
// Keyed by id, which is safe because Register refuses a second descriptor
53-
// under one. Here rather than beside its caller because this file is already
54-
// where the registry's reads meet its writes: the window settles from its
55-
// worker as well as from its own goroutine. The size is worked out without
56-
// the lock held, because planning an archive reads the registry itself.
53+
// Keyed by id and request - see SmallestRemembered - which is safe because
54+
// Register refuses a second descriptor under one id. Here rather than beside
55+
// its caller because this file is already where the registry's reads meet its
56+
// writes: the window settles from its worker as well as from its own
57+
// goroutine. The size is worked out without the lock held, because planning an
58+
// archive reads the registry itself.
5759
func SmallestWithLabel(d Descriptor) int64 {
60+
return SmallestRemembered(d, Request{Label: true})
61+
}
62+
63+
// SmallestRemembered is d.SmallestAccepted(r), worked out once per format and
64+
// request and remembered.
65+
//
66+
// SmallestWithLabel's reasoning, for the questions that carry settings or a
67+
// seed. The presets ask the floor of a file with its dialect, of a sheet with
68+
// its rows and columns, of the boundary set with seed 1 - and asked it afresh
69+
// at every expansion, which on a picture is encoding one. Measured 2026-09-23:
70+
// 29% of expanding tabular-import (docs/GUI-MEMORY-2026-09-23.md section 4j).
71+
//
72+
// The size a request asks for and SizeFromContents are left out, because
73+
// SmallestAccepted sets both itself. A request with contents is worked out
74+
// every time - see RequestKey.
75+
//
76+
// At most smallestCeiling answers are kept. The key grows with values somebody
77+
// types - the rows of a sheet - so a long session would otherwise keep one for
78+
// every number ever typed. Past the ceiling the memory starts again, which
79+
// costs one working out per question and nothing else.
80+
func SmallestRemembered(d Descriptor, r Request) int64 {
81+
r.Bytes, r.SizeFromContents = 0, false
82+
key, ok := RequestKey(d.ID, r)
83+
if !ok {
84+
return d.SmallestAccepted(r)
85+
}
5886
smallestMu.Lock()
59-
known, ok := smallestKnown[d.ID]
87+
known, found := smallestKnown[key]
6088
smallestMu.Unlock()
61-
if ok {
89+
if found {
6290
return known
6391
}
64-
size := d.SmallestAccepted(Request{Label: true})
92+
size := d.SmallestAccepted(r)
6593
smallestMu.Lock()
66-
smallestKnown[d.ID] = size
94+
if len(smallestKnown) >= smallestCeiling {
95+
smallestKnown = map[string]int64{}
96+
}
97+
smallestKnown[key] = size
6798
smallestMu.Unlock()
6899
return size
69100
}
70101

71-
// smallestKnown is what SmallestWithLabel has worked out, by format id.
102+
// smallestCeiling is how many answers SmallestRemembered keeps - a few hundred
103+
// bytes each, so the most it holds is about a megabyte.
104+
const smallestCeiling = 4096
105+
106+
// smallestKnown is what SmallestRemembered has worked out, by RequestKey.
72107
var (
73108
smallestMu sync.Mutex
74109
smallestKnown = map[string]int64{}
75110
)
76111

112+
// RequestKey is one request to one format written as text that no other
113+
// request shares, so that something worked out for it can be kept under it.
114+
//
115+
// Every field of Request that can change a plan is in it, and a guard sets
116+
// each field in turn to hold that true when Request grows. The values are
117+
// quoted, because a setting's value is text somebody typed and may hold any
118+
// separator - a CSV delimiter of "|" would otherwise read as the start of a
119+
// second setting, and two requests sharing a key share an answer.
120+
//
121+
// A request with contents has no key. What an archive holds is a list of
122+
// formats with sizes of their own, and nothing asks such a request twice.
123+
func RequestKey(id string, r Request) (string, bool) {
124+
if len(r.Contains) > 0 {
125+
return "", false
126+
}
127+
names := make([]string, 0, len(r.Properties))
128+
for name := range r.Properties {
129+
names = append(names, name)
130+
}
131+
sort.Strings(names)
132+
var b strings.Builder
133+
fmt.Fprintf(&b, "%q %d %t %t %d", id, r.Bytes, r.SizeFromContents, r.Label, r.Seed)
134+
for _, name := range names {
135+
fmt.Fprintf(&b, " %q=%q", name, r.Properties[name])
136+
}
137+
return b.String(), true
138+
}
139+
77140
// SortChoices puts a closed set in the order somebody looks for a value in.
78141
//
79142
// Here rather than in the menu that draws them, and that is the whole point:

‎internal/guard/presetbytes_test.go‎

Lines changed: 44 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -23,29 +23,60 @@ import (
2323
// was taken by hand on 2026-09-08 and again either side of that move on
2424
// 2026-09-22, both times 1298 B and this sum. This is that measurement kept.
2525
//
26+
// Every preset since 2026-09-24, and a refusal's words as well as a source's
27+
// bytes. Until then only size-boundaries was pinned, and the change that day
28+
// was to how the other four work their sets out: asking the format for its
29+
// smallest size once rather than for every file, and planning a file the set
30+
// holds twelve times once (docs/GUI-MEMORY-2026-09-23.md section 4j). Neither
31+
// may move a byte, so the gate came first and was measured on the tree before
32+
// the change. The cases are the ones that reach what changed - other formats
33+
// in allow, which ask for other floors, a limit small enough that a refusal
34+
// names the floor, and a spread narrow enough to reach it.
35+
//
2636
// What to do when it goes red: decide, rather than update. The sum moving is a
2737
// breaking change under D11 - a major, a Breaking entry in the changelog, and
2838
// the owner's decision, because untouchable rule 12 says the assistant does not
2939
// raise the version. A refactor that moved it is a refactor to undo.
3040
func TestEjectingAPresetGivesTheBytesItAlwaysGave(t *testing.T) {
41+
// bytes and sum are the whole document, refused the whole refusal. The
42+
// first row was measured 2026-09-08, the rest on 2026-09-24 before the
43+
// change they guard.
3144
pinned := []struct {
32-
id string
33-
args preset.Args
34-
// bytes and sum are the whole document, measured 2026-09-08 and
35-
// unchanged since.
36-
bytes int
37-
sum string
45+
id string
46+
args preset.Args
47+
bytes int
48+
sum string
49+
refused string
3850
}{
39-
{
40-
id: "size-boundaries",
41-
args: preset.Args{"limit": "10mb", "format": "pdf"},
42-
bytes: 1298,
43-
sum: "2733cf63db40465fb97e26790d668d65ea01f5e94927a44ddf0869399beee2bb",
44-
},
51+
{id: "size-boundaries", args: preset.Args{"format": "pdf", "limit": "10mb"}, bytes: 1298, sum: "2733cf63db40465fb97e26790d668d65ea01f5e94927a44ddf0869399beee2bb"},
52+
{id: "size-boundaries", args: preset.Args{"format": "png", "limit": "1mb"}, refused: "the preset size-boundaries cannot build this set - under_1mb would be 0 B, and a file cannot be smaller than nothing. Raise the limit above 1048650 B, narrow the spread, or choose a format with a smaller minimum. The limit asked for was 1048576 B."},
53+
{id: "size-boundaries", args: preset.Args{"format": "jpg", "limit": "10mb"}, bytes: 1298, sum: "29c7e0a133fb97fdf9d19fb40d0d96ad97c4a1fef67556a9dc734b520f7d9b09"},
54+
{id: "size-boundaries", args: preset.Args{"format": "jpg", "limit": "2kb", "spread": "1kb"}, bytes: 640, sum: "1346f4a7ae514fe2d15de426b910e308442f66fa97b485932d8fa1f5df56b910"},
55+
{id: "size-boundaries", args: preset.Args{"format": "jpg", "limit": "300", "spread": "100"}, refused: "the preset size-boundaries cannot build this set - under_100 would be 200 B and the smallest JPG this build makes is 602 B. Raise the limit above 702 B, narrow the spread, or choose a format with a smaller minimum. The limit asked for was 300 B."},
56+
{id: "upload-validation", args: preset.Args{}, bytes: 4116, sum: "a75039d859ee25d5ea5fd463ac2a774aacb1c45d013cf7ef3b80d46e0688fb78"},
57+
{id: "upload-validation", args: preset.Args{"limit": "5mb"}, bytes: 4108, sum: "4b7e716e2e199837b3c2bef228921e0087e894c49a8cf868a246c0ac636b9a35"},
58+
{id: "upload-validation", args: preset.Args{"limit": "3kb"}, refused: "the preset upload-validation cannot build this set - allowed_pdf would be 1536 B and the smallest PDF this build makes is 3415 B. Raise the limit to 6830 B or more, or take pdf out of the allowed types. The limit asked for was 3072 B."},
59+
{id: "upload-validation", args: preset.Args{"allow": "docx,gif"}, bytes: 3816, sum: "33975195adf794dd9f6e96dbec8a9775bfbda3c4b7dfd3524475f70c39496059"},
60+
{id: "upload-validation", args: preset.Args{"allow": "xlsx,ico,wav"}, bytes: 4125, sum: "c46aba3ad1aa6bf3ac2442bde058b9802b4d4f72375fd2f7cbffc6fab1620d6f"},
61+
{id: "upload-validation", args: preset.Args{"bulk": "3", "far-over": "off"}, bytes: 3933, sum: "98bc51d5edb94a80fb764a03915eccc6997b70fa8eac153578320b3128cb6d95"},
62+
{id: "upload-validation", args: preset.Args{"deny": "exe,js"}, bytes: 3767, sum: "55d9946e7233761716849f52517cd58f7a38c8748f4cdae8620afade0681621c"},
63+
{id: "tabular-import", args: preset.Args{}, bytes: 3369, sum: "fad20b41756327a6e85da93715fb09b20db394fb23e4b3ce22bdaf6e11b20726"},
64+
{id: "tabular-import", args: preset.Args{"rows": "100"}, bytes: 3367, sum: "ee28ae5421d4717fb24ee6dfbef53f7a54e3a00105e5015f884d59768181d552"},
65+
{id: "tabular-import", args: preset.Args{"columns": "5"}, bytes: 3367, sum: "580d526546720b851ac7d834b97c163f5ff50b29caeff5b27fcc1ba00e5bb952"},
66+
{id: "text-encoding", args: preset.Args{}, bytes: 4570, sum: "de27b9dc6c646baebaa0b16019ba3ce15d0f1d145941d376b47263de242998e6"},
67+
{id: "text-encoding", args: preset.Args{"sample": "8kb"}, bytes: 4570, sum: "f87c73864e5f517abb08b50393cd9a1681a90a30560c1d14dfddcf31e8037479"},
68+
{id: "empty-and-minimal", args: preset.Args{}, bytes: 3811, sum: "80641962ac9dfb303f812fd78e0a0d1080f714094d159d7b10448291debb9279"},
69+
{id: "empty-and-minimal", args: preset.Args{"formats": "jpg,png,txt"}, bytes: 743, sum: "4fd23e4b06a2e27ede987ab48a2cc302cc2accb9f948c7d5d25f675681f31f69"},
4570
}
4671

4772
for _, want := range pinned {
4873
expanded, err := preset.Expand(want.id, want.args)
74+
if want.refused != "" {
75+
if err == nil || err.Error() != want.refused {
76+
t.Errorf("%s at %v was refused with\n %q\nuntil now, and now gives\n %v", want.id, want.args, want.refused, err)
77+
}
78+
continue
79+
}
4980
if err != nil {
5081
t.Errorf("%s refused %v: %v", want.id, want.args, err)
5182
continue
@@ -55,8 +86,7 @@ func TestEjectingAPresetGivesTheBytesItAlwaysGave(t *testing.T) {
5586
if len(expanded.Source) == want.bytes && got == want.sum {
5687
continue
5788
}
58-
t.Errorf("ejecting %s at %v gives %d B and %s, and it has given %d B and %s since "+
59-
"2026-09-08.\n"+
89+
t.Errorf("ejecting %s at %v gives %d B and %s, and it has given %d B and %s until now.\n"+
6090
"Every manifest written from this preset carries a hash of these bytes, so this is a "+
6191
"breaking change under D11 rather than a number to update here.\n%s",
6292
want.id, want.args, len(expanded.Source), got, want.bytes, want.sum, expanded.Source)

‎internal/guard/presetcost_test.go‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
package guard
2+
3+
import (
4+
"fmt"
5+
"runtime"
6+
"testing"
7+
8+
"github.com/donislawdev/TestingFilesGenerator/internal/format"
9+
_ "github.com/donislawdev/TestingFilesGenerator/internal/format/all"
10+
"github.com/donislawdev/TestingFilesGenerator/internal/preset"
11+
"github.com/donislawdev/TestingFilesGenerator/internal/recipe"
12+
)
13+
14+
// A new upload limit does not ask the same questions again.
15+
//
16+
// Expanding upload-validation took 137-142 ms and 57 MB whenever a value
17+
// changed, which in the window is every key typed into one of its settings or
18+
// into the batch screen built on it. Two causes, both a question asked again
19+
// with the answer already known: the smallest size of a format, found for
20+
// every file about a name by encoding pictures at growing sizes, and one small
21+
// picture planned under a dozen names (docs/GUI-MEMORY-2026-09-23.md section
22+
// 4j). Measured 2026-09-24, least of five: 9.58 MB with both answered once,
23+
// 18.95 MB with the picture planned under every name, 44.29 MB with the floor
24+
// worked out for every file, 9.67 MB under -race. The line sits between the
25+
// first two.
26+
//
27+
// Asked with a limit this process has not expanded before, because the same
28+
// limit twice is the window's memory's business and says nothing about this.
29+
// The least of several readings, because the counter is the whole process's
30+
// and a reading can only be too high - see
31+
// TestTheMinimalSetIsWorkedOutOnceAndNotAtEveryExpansion.
32+
//
33+
// If it goes red after a change that is not about this, measure the four
34+
// numbers above again before moving the line: a line moved to the new reading
35+
// no longer stands between anything.
36+
func TestANewUploadLimitDoesNotAskTheSameQuestionsAgain(t *testing.T) {
37+
const ceiling = 13<<20 + 1<<19 // 13.5 MB
38+
// The state this is about: files that ask one question under several
39+
// names. A set without them would expand cheaply whatever this code did.
40+
if most := mostFilesAskingOneQuestion(t, "upload-validation", preset.Args{}); most < 5 {
41+
t.Fatalf("the most files of upload-validation asking one question is %d, so the set no longer holds what this guard is about", most)
42+
}
43+
least := leastAllocatedByAnExpansion(t, "upload-validation", func(i int) preset.Args {
44+
return preset.Args{"limit": fmt.Sprintf("%dmb", 21+i)}
45+
})
46+
if least > ceiling {
47+
t.Errorf("expanding upload-validation at a new limit allocated %d bytes, over %d - "+
48+
"a format's smallest size or a file already planned is being worked out again", least, ceiling)
49+
}
50+
}
51+
52+
// A new row count works the sheet's smallest size out once, not twice.
53+
//
54+
// tabular-import asks for its sheet at the smallest size the rows and columns
55+
// allow, and finding that size means building the sheet at growing sizes. The
56+
// size was asked twice per expansion - once to check the file, once to write
57+
// it into the set - and nothing remembered it. A new row count does change the
58+
// sheet, so one working out is the real work and the second was the waste.
59+
// The wide CSV, the other file asked at its floor, is the same at every
60+
// expansion and now comes from memory.
61+
//
62+
// Measured 2026-09-24 at 300 rows, least of five: 19.84 MB remembered, 38.59 MB
63+
// worked out twice, 20.63 MB under -race. The line sits between. The numbers
64+
// grow with the rows - 700 rows allocate 39 MB remembered - so the rows asked
65+
// here stay where they were measured.
66+
func TestANewRowCountWorksTheSheetsSmallestSizeOutOnce(t *testing.T) {
67+
const ceiling = 27 << 20
68+
if sheets := filesOfFormat(t, "tabular-import", preset.Args{}, "xlsx"); sheets != 1 {
69+
t.Fatalf("tabular-import holds %d sheets, so the set no longer holds the file this guard is about", sheets)
70+
}
71+
least := leastAllocatedByAnExpansion(t, "tabular-import", func(i int) preset.Args {
72+
return preset.Args{"rows": fmt.Sprintf("%d", 300+i)}
73+
})
74+
if least > ceiling {
75+
t.Errorf("expanding tabular-import at a new row count allocated %d bytes, over %d - "+
76+
"the sheet's smallest size is being worked out more than once", least, ceiling)
77+
}
78+
}
79+
80+
// leastAllocatedByAnExpansion expands a preset once, then five times with
81+
// values it has not been given, and returns the least any of the five
82+
// allocated.
83+
func leastAllocatedByAnExpansion(t *testing.T, id string, fresh func(int) preset.Args) uint64 {
84+
t.Helper()
85+
if _, err := preset.Expand(id, preset.Args{}); err != nil {
86+
t.Fatalf("%s did not expand, so nothing was asked: %v", id, err)
87+
}
88+
least := ^uint64(0)
89+
for i := 0; i < 5; i++ {
90+
var before, after runtime.MemStats
91+
runtime.ReadMemStats(&before)
92+
if _, err := preset.Expand(id, fresh(i)); err != nil {
93+
t.Fatalf("%s at %v: %v", id, fresh(i), err)
94+
}
95+
runtime.ReadMemStats(&after)
96+
if spent := after.TotalAlloc - before.TotalAlloc; spent < least {
97+
least = spent
98+
}
99+
}
100+
return least
101+
}
102+
103+
// mostFilesAskingOneQuestion is how many targets of an expanded set ask their
104+
// format the same thing, at most - one format, one size, one set of settings.
105+
func mostFilesAskingOneQuestion(t *testing.T, id string, args preset.Args) int {
106+
t.Helper()
107+
same := map[string]int{}
108+
most := 0
109+
for _, target := range expandedTargets(t, id, args) {
110+
if len(target.Sizes) == 0 {
111+
continue
112+
}
113+
key, ok := format.RequestKey(target.Format, format.Request{
114+
Label: true, Properties: target.Properties, Bytes: target.Sizes[0],
115+
})
116+
if !ok {
117+
continue
118+
}
119+
same[key]++
120+
if same[key] > most {
121+
most = same[key]
122+
}
123+
}
124+
return most
125+
}
126+
127+
func filesOfFormat(t *testing.T, id string, args preset.Args, formatID string) int {
128+
t.Helper()
129+
n := 0
130+
for _, target := range expandedTargets(t, id, args) {
131+
if target.Format == formatID {
132+
n++
133+
}
134+
}
135+
return n
136+
}
137+
138+
func expandedTargets(t *testing.T, id string, args preset.Args) []recipe.Target {
139+
t.Helper()
140+
expanded, err := preset.Expand(id, args)
141+
if err != nil {
142+
t.Fatalf("%s did not expand: %v", id, err)
143+
}
144+
rec, err := recipe.Parse(expanded.Source, id)
145+
if err != nil {
146+
t.Fatalf("%s expanded into a recipe that does not read: %v", id, err)
147+
}
148+
return rec.Targets
149+
}

0 commit comments

Comments
 (0)