diff --git a/CHANGELOG.md b/CHANGELOG.md index 0070cdba..dea16f1a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -195,6 +195,54 @@ because it turns other people's test suites red. ### Added +- **Three more presets: `upload-validation`, `text-encoding` and + `tabular-import`.** Run `tfg preset list` for all five, or + `tfg preset show ` for what one takes and what it would produce before + it writes anything. + + **`upload-validation`** answers "does my upload form take what it should and + turn the rest away?" - 71 files in eight groups at its defaults, 115 MB. A + file a byte under your limit, one at it, one a byte over and one twice it. A + real file of every type you allow, which is the positive control. A file for + every extension you deny, an SVG and an HTML among them, because both are + routinely taken for a picture and for plain text. A PDF named `.jpg`. A file + with no extension, one named `PHOTO.JPG` and one named `invoice.jpg.exe`. A + name 204 characters long, a name outside ASCII, and a name with spaces and + brackets that is perfectly legal. And fifty files at once. + + `--limit` is the size your form declares, and the run says out loud when you + did not give one, because a set built around our placeholder says nothing + about your form. `--allow` and `--deny` are lists with commas. An extension + in `--deny` that this build has no format for - `exe`, `sh` - still gets a + file under that name, holding plain text, and the run says so: it tests a + form reading the end of a name, not one reading what is inside. + `--far-over 10x` asks for a file ten times the limit rather than twice it, + `--far-over off` leaves it out, and `--bulk 0` leaves the mass upload out. + Anything a setting empties is named in the output rather than quietly + missing. What this set does **not** do is put a path in a file name: + `../../etc/passwd` is not a name this tool will write, deliberately. + + **`text-encoding`** answers "does my reader know which encoding a file is + in, or is it guessing?" - 20 files, 80 kB. TXT, MD and XML in UTF-8, + UTF-16LE and UTF-16BE, each with and without a byte order mark, plus CSV and + LOG with LF and with CRLF endings. UTF-8 expects `accept` with or without a + mark. UTF-16 expects `unspecified`: whether your system handles it at all is + your policy, and the manifest does not invent it. Two combinations are left + out and said out loud, because XML in UTF-16 has to open with a mark. No + format in this build carries an encoding and a line ending at once, so the + two halves are separate files rather than one grid - the run says that too. + `--sample` sets how big each file is, and refuses an odd number, because a + file in UTF-16 always has an even number of bytes. + + **`tabular-import`** answers "does my table import survive what real tools + export?" - 13 files, 2.9 MB. One CSV per dialect: comma, semicolon, pipe and + tab, LF and CRLF, with and without a header row, and three quoting styles, + one setting at a time against a base so a failure names its cause. A CSV + with more columns than a spreadsheet will show, which expects `unspecified` + with `count_limit`. A spreadsheet of `--rows` by `--columns`, written at + exactly the size that many cells package to. And the same JSON records + written indented, minified and one to a line. + - **A second preset: `empty-and-minimal`.** It answers "does a file that is valid and as small as the format allows get through?" and builds the smallest legal file of every format this build has, plus a file of nought diff --git a/internal/cli/preset.go b/internal/cli/preset.go index f2cdf5a8..99b7b8d8 100644 --- a/internal/cli/preset.go +++ b/internal/cli/preset.go @@ -9,6 +9,7 @@ import ( "sort" "strings" + "github.com/donislawdev/TestingFilesGenerator/internal/core" "github.com/donislawdev/TestingFilesGenerator/internal/engine" "github.com/donislawdev/TestingFilesGenerator/internal/format" "github.com/donislawdev/TestingFilesGenerator/internal/manifest" @@ -233,23 +234,30 @@ func presetNamed(args []string) string { // for a flag - a parse that succeeded means every name was defined. Names the // command does define are skipped for the same reason: with --preset given, // --limit is defined, and the failure was about something else entirely. -func parameterWithoutItsPreset(fs *flag.FlagSet, args []string) (name, owner string) { +func parameterWithoutItsPreset(fs *flag.FlagSet, args []string) (name string, owners []string) { for _, a := range args { if a == "--" { - return "", "" + return "", nil } - if !strings.HasPrefix(a, "-") { - continue - } - candidate, _, _ := strings.Cut(strings.TrimLeft(a, "-"), "=") - if fs.Lookup(candidate) != nil { - continue - } - if owner := preset.Declaring(candidate); owner != "" { - return candidate, owner + candidate, declared := presetParameterIn(fs, a) + if len(declared) > 0 { + return candidate, declared } } - return "", "" + return "", nil +} + +// presetParameterIn is the preset parameter one argument names, and every +// preset that declares it. +func presetParameterIn(fs *flag.FlagSet, arg string) (string, []string) { + if !strings.HasPrefix(arg, "-") { + return "", nil + } + candidate, _, _ := strings.Cut(strings.TrimLeft(arg, "-"), "=") + if fs.Lookup(candidate) != nil { + return "", nil + } + return candidate, preset.Declaring(candidate) } // addPresetFlags puts the parameters of the named preset on the generate flag @@ -281,19 +289,56 @@ func addPresetFlags(fs *flag.FlagSet, args []string, errOut io.Writer) int { // It reports whether it answered, so the caller knows whether the complaint it // held back still has to be let through. func explainUndefinedFlag(fs *flag.FlagSet, args []string, errOut io.Writer) bool { - name, owner := parameterWithoutItsPreset(fs, args) + name, owners := parameterWithoutItsPreset(fs, args) if name == "" { return false } // The second sentence names both roads, because since 2026-09-22 a // recipe file can build on the preset too - and beside a file the flag // does not exist either, the file's with section is where the value goes. + // + // Every owner is named, because since 2026-09-22 two presets ask for the + // limit a system declares and both call it that. Naming one of them sent + // the reader who meant the other to add the wrong preset, in a sentence + // that read as certain. fmt.Fprintf(errOut, - "tfg: --%s is a parameter of the preset %s, so it only exists beside it. Add --preset %s, put %s under with: in a recipe that extends it, or drop --%s.\n", - name, owner, owner, name, name) + "tfg: --%s is a parameter of %s, so it only exists beside %s. Add %s, put %s under with: in a recipe that extends %s, or drop --%s.\n", + name, presetsNamed(owners), oneOfThem(owners), presetFlagsFor(owners), + name, oneOfThem(owners), name) return true } +// presetsNamed, oneOfThem and presetFlagsFor word one sentence for one owner +// and for several, so it reads as English either way rather than as a list with +// a noun beside it that does not agree. +func presetsNamed(owners []string) string { + return core.Noun(len(owners), "the preset ", "the presets ") + joinWithOr(owners) +} + +func oneOfThem(owners []string) string { + return core.Noun(len(owners), "it", "one of them") +} + +func presetFlagsFor(owners []string) string { + flags := make([]string, 0, len(owners)) + for _, id := range owners { + flags = append(flags, "--preset "+id) + } + return joinWithOr(flags) +} + +// joinWithOr writes a list the way a sentence takes one. A comma between every +// pair reads as an enumeration and this sentence is prose. +func joinWithOr(items []string) string { + switch len(items) { + case 0: + return "" + case 1: + return items[0] + } + return strings.Join(items[:len(items)-1], ", ") + " or " + items[len(items)-1] +} + // describingFlagsBeside is describingFlagsGiven minus what this preset reads. // // A preset lays out a whole set, so a flag describing one target has no target diff --git a/internal/guard/branching_test.go b/internal/guard/branching_test.go index 9f486fd5..9a4588a5 100644 --- a/internal/guard/branching_test.go +++ b/internal/guard/branching_test.go @@ -52,7 +52,12 @@ const ( // more than the depth does. // Lowered from 52 on 2026-09-05: splitting preflight out took one function // out of the band. The ratchet only tightens. - crowdedDepthFunctions = 51 + // Lowered from 51 on 2026-09-22: the scan for a preset parameter typed + // without its preset became two functions when it learnt to answer with + // every owner, and the loop left behind is two deep rather than three. The + // four presets that arrived the same day were flattened to hold the number + // where it was - this is the one that went below it. + crowdedDepthFunctions = 50 // An axis this set does not watch. crowding() asks n >= band, so nothing // reaches it. diff --git a/internal/guard/generatewindow_test.go b/internal/guard/generatewindow_test.go index 05064a68..f837c524 100644 --- a/internal/guard/generatewindow_test.go +++ b/internal/guard/generatewindow_test.go @@ -149,24 +149,7 @@ func TestTheWindowDrawsAFieldForEveryDeclaredProperty(t *testing.T) { if bad := wrongKindOfControl(p, control); bad != "" { t.Errorf("%s.%s is %s", d.ID, p.Name, bad) } - // A closed set says what it takes with its menu rather than in - // prose, since 2026-08-19 (O105). What it still has to say is what - // it is FOR - the sentence spelling twenty format names out under - // a menu offering the same twenty was two lines of duplication on - // a screen that does not fit as it is. - want := p.Allowed() - if p.Kind == format.PropertyChoice { - want = p.Detail - } - if shown := everythingSaid(content); want != "" && !strings.Contains(shown, want) { - t.Errorf("the field for %s.%s does not say %q", d.ID, p.Name, want) - } - if p.Kind == format.PropertyChoice { - if shown := everythingSaid(content); strings.Contains(shown, p.Allowed()) { - t.Errorf("the field for %s.%s lists its values in prose (%q) as well as in the "+ - "menu above them", d.ID, p.Name, p.Allowed()) - } - } + saysWhatItTakes(t, everythingSaid(content), d.ID, p) checked++ } @@ -203,6 +186,35 @@ func declares(d format.Descriptor, name string) bool { return false } +// saysWhatItTakes checks the sentence under a field against the declaration it +// was drawn from. +// +// A closed set says what it takes with its menu rather than in prose, since +// 2026-08-19 (O105). What it still has to say is what it is FOR - the sentence +// spelling twenty format names out under a menu offering the same twenty was +// two lines of duplication on a screen that does not fit as it is. +// +// Shared with the preset screen since 2026-09-22, when the first preset +// parameter with a closed set arrived and the screen it is on asked for the +// prose the other screen had been told not to write. One rule, one place: two +// screens drawing a field from one declaration cannot be judged by two rules +// without one of them being wrong. +func saysWhatItTakes(t *testing.T, shown, owner string, p format.Property) { + t.Helper() + + want := p.Allowed() + if p.Kind == format.PropertyChoice { + want = p.Detail + } + if want != "" && !strings.Contains(shown, want) { + t.Errorf("the field for %s.%s does not say %q", owner, p.Name, want) + } + if p.Kind == format.PropertyChoice && strings.Contains(shown, p.Allowed()) { + t.Errorf("the field for %s.%s lists its values in prose (%q) as well as in the "+ + "menu above them", owner, p.Name, p.Allowed()) + } +} + // wrongKindOfControl says when a declaration got a control that cannot express // it. A closed set drawn as a box to type in is how a value gets misspelled. func wrongKindOfControl(p format.Property, control fyne.CanvasObject) string { diff --git a/internal/guard/parity_test.go b/internal/guard/parity_test.go index 3a5795f2..9d7c79df 100644 --- a/internal/guard/parity_test.go +++ b/internal/guard/parity_test.go @@ -71,6 +71,22 @@ var reachableFromTheWindow = []string{ // menu. "preset:size-boundaries.format", + // The three presets of 2026-09-22, and every parameter of each. Drawn by + // the same code as the two above, because a preset parameter IS a + // format.Property - which is what made far-over, the first closed set any + // preset has declared, arrive as a menu with no window code at all. + "preset:text-encoding", + "preset:text-encoding.sample", + "preset:tabular-import", + "preset:tabular-import.rows", + "preset:tabular-import.columns", + "preset:upload-validation", + "preset:upload-validation.limit", + "preset:upload-validation.allow", + "preset:upload-validation.deny", + "preset:upload-validation.far-over", + "preset:upload-validation.bulk", + // A recipe that builds on a preset, since 2026-09-22: the switch and the // menu on the batch screen are the extends key, and the chosen preset's // parameters under it are the with section, drawn from the declaration diff --git a/internal/guard/preset_test.go b/internal/guard/preset_test.go index 4d501f7c..76c3b78a 100644 --- a/internal/guard/preset_test.go +++ b/internal/guard/preset_test.go @@ -5,6 +5,7 @@ import ( "strings" "testing" + "github.com/donislawdev/TestingFilesGenerator/internal/format" _ "github.com/donislawdev/TestingFilesGenerator/internal/format/all" "github.com/donislawdev/TestingFilesGenerator/internal/preset" "github.com/donislawdev/TestingFilesGenerator/internal/recipe" @@ -46,6 +47,22 @@ func TestEveryPresetExpandsIntoARecipeThisBuildAccepts(t *testing.T) { if target.Group == "" { t.Errorf("target %q carries no group, so nothing can assert about the class it belongs to", target.ID) } + // The parser does not resolve formats - the engine does, and + // "tfg validate" answers with exit 4 and the list of known + // ones. So a preset naming a format nobody registered produces + // a recipe that PARSES and a run that cannot start, and this + // guard read "the recipe it produced does not parse" and said + // nothing about it. + // + // Measured on 2026-09-22 by mutation: putting an x on the end + // of the format id in the boundary set left this guard green, + // and it had been green against that mutation on the commit + // before as well. The entry was proving nothing rather than + // something breaking. + if _, err := format.Get(target.Format); err != nil { + t.Errorf("target %q asks for the format %q and this build has none: %v", + target.ID, target.Format, err) + } } }) } diff --git a/internal/guard/presetbytes_test.go b/internal/guard/presetbytes_test.go new file mode 100644 index 00000000..0a26aa1c --- /dev/null +++ b/internal/guard/presetbytes_test.go @@ -0,0 +1,64 @@ +package guard + +import ( + "crypto/sha256" + "encoding/hex" + "testing" + + _ "github.com/donislawdev/TestingFilesGenerator/internal/format/all" + "github.com/donislawdev/TestingFilesGenerator/internal/preset" +) + +// What a preset ejects is bytes somebody else's manifest remembers. +// +// The manifest carries a recipe_hash, computed from the source a run consumed, +// and a preset's source is what eject prints - the same bytes, which is the +// whole of PR5. So a tidy-up that moved a space would tell everyone holding a +// record from an earlier run that their recipe had changed, and nothing in this +// tree would have said a word. +// +// PRESET-FEASIBILITY-2026-09-08.md section 5 asked for a measured gate on +// exactly one refactor: pulling the set a declared limit produces out of +// size-boundaries so that upload-validation could use it too. The measurement +// was taken by hand on 2026-09-08 and again either side of that move on +// 2026-09-22, both times 1298 B and this sum. This is that measurement kept. +// +// What to do when it goes red: decide, rather than update. The sum moving is a +// breaking change under D11 - a major, a Breaking entry in the changelog, and +// the owner's decision, because untouchable rule 12 says the assistant does not +// raise the version. A refactor that moved it is a refactor to undo. +func TestEjectingAPresetGivesTheBytesItAlwaysGave(t *testing.T) { + pinned := []struct { + id string + args preset.Args + // bytes and sum are the whole document, measured 2026-09-08 and + // unchanged since. + bytes int + sum string + }{ + { + id: "size-boundaries", + args: preset.Args{"limit": "10mb", "format": "pdf"}, + bytes: 1298, + sum: "2733cf63db40465fb97e26790d668d65ea01f5e94927a44ddf0869399beee2bb", + }, + } + + for _, want := range pinned { + expanded, err := preset.Expand(want.id, want.args) + if err != nil { + t.Errorf("%s refused %v: %v", want.id, want.args, err) + continue + } + sum := sha256.Sum256(expanded.Source) + got := hex.EncodeToString(sum[:]) + if len(expanded.Source) == want.bytes && got == want.sum { + continue + } + t.Errorf("ejecting %s at %v gives %d B and %s, and it has given %d B and %s since "+ + "2026-09-08.\n"+ + "Every manifest written from this preset carries a hash of these bytes, so this is a "+ + "breaking change under D11 rather than a number to update here.\n%s", + want.id, want.args, len(expanded.Source), got, want.bytes, want.sum, expanded.Source) + } +} diff --git a/internal/guard/presetsilence_test.go b/internal/guard/presetsilence_test.go new file mode 100644 index 00000000..82fbe6cb --- /dev/null +++ b/internal/guard/presetsilence_test.go @@ -0,0 +1,260 @@ +package guard + +import ( + "strings" + "testing" + + "github.com/donislawdev/TestingFilesGenerator/internal/format" + _ "github.com/donislawdev/TestingFilesGenerator/internal/format/all" + "github.com/donislawdev/TestingFilesGenerator/internal/preset" + "github.com/donislawdev/TestingFilesGenerator/internal/recipe" +) + +// A preset asked for something that empties one of its groups says so. +// +// Untouchable rule 6, on the one path where it is easiest to break: nothing +// FAILS. The set expands, the run succeeds, the manifest is honest about every +// file it holds - and it holds five groups where the card describes six, +// because a parameter the caller set emptied one. A set missing a group looks +// exactly like a set that never had it, and somebody goes looking for files +// that were never going to be there. +// +// Written on 2026-09-22 with the three presets that made it reachable. Until +// then no parameter of any preset could empty anything. +func TestAGroupAPresetLeavesOutIsSaidOutLoud(t *testing.T) { + cases := []struct { + id string + args preset.Args + gone string + about string + }{ + { + id: "empty-and-minimal", args: preset.Args{"formats": "zip"}, gone: "empty", + about: "no archive has a legal empty form, so a set of archives has no empty half", + }, + { + id: "upload-validation", args: preset.Args{"bulk": "0"}, gone: "bulk-upload", + about: "nought files in the mass upload is a legal thing to ask for", + }, + { + id: "upload-validation", args: preset.Args{"allow": "jpg"}, gone: "extension-content-mismatch", + about: "one allowed type leaves nothing to name a file wrongly after", + }, + { + id: "upload-validation", args: preset.Args{"far-over": "off"}, gone: "", + about: "the file well past the limit was turned off", + }, + } + + for _, c := range cases { + t.Run(c.id+" "+c.about, func(t *testing.T) { + full := groupsOf(t, c.id, preset.Args{}) + narrowed := groupsOf(t, c.id, c.args) + + // The state this guard is about, asserted rather than assumed. + // A parameter that stopped emptying the group would leave this + // guard green while saying nothing - O118, a dozen times over. + if c.gone != "" { + if !full[c.gone] { + t.Fatalf("%s at its defaults has no group called %q, so this case is not about anything", + c.id, c.gone) + } + if narrowed[c.gone] { + t.Fatalf("%v was supposed to empty the group %q and did not", c.args, c.gone) + } + } + + // And the set says more than it does when nothing is missing. The + // sentence itself is not matched: a guard that looks for words + // pins the wording rather than the behaviour, and the wording is + // meant to improve. + said := notesOf(t, c.id, c.args) + for _, always := range notesOf(t, c.id, preset.Args{}) { + said = without(said, always) + } + if len(said) == 0 { + t.Errorf("%s asked for %v leaves something out and says nothing it does not say at its defaults", + c.id, c.args) + } + }) + } +} + +// groupsOf is every group the set holds at these values. +func groupsOf(t *testing.T, id string, args preset.Args) map[string]bool { + t.Helper() + + expanded, err := preset.Expand(id, args) + if err != nil { + t.Fatalf("%s refused %v: %v", id, args, err) + } + doc, err := recipe.Parse(expanded.Source, id+".yaml") + if err != nil { + t.Fatalf("%s wrote a recipe this build cannot read: %v", id, err) + } + out := map[string]bool{} + for _, target := range doc.Targets { + out[target.Group] = true + } + return out +} + +func notesOf(t *testing.T, id string, args preset.Args) []string { + t.Helper() + + expanded, err := preset.Expand(id, args) + if err != nil { + t.Fatalf("%s refused %v: %v", id, args, err) + } + return expanded.Notes() +} + +func without(all []string, one string) []string { + out := make([]string, 0, len(all)) + for _, said := range all { + if said != one { + out = append(out, said) + } + } + return out +} + +// The encoding set holds every combination this build can write and none of the +// ones it cannot. +// +// Both halves, because either one alone is the guard that proves nothing. A set +// that simply expanded would pass a check for "no refused cell" by holding +// three files, and a check for "every cell" would demand two that XML refuses - +// the specification says a UTF-16 entity opens with a byte order mark, and the +// format enforces it. +// +// The registry is asked which is which rather than this file knowing. The rule +// belongs to XML today, and the preset was written so that the next text format +// bringing its own rule needs no line here and none there. +func TestTheEncodingSetHoldsEveryCombinationThisBuildCanWrite(t *testing.T) { + expanded, err := preset.Expand("text-encoding", preset.Args{}) + if err != nil { + t.Fatalf("the preset refused its own defaults: %v", err) + } + doc, err := recipe.Parse(expanded.Source, "text-encoding.yaml") + if err != nil { + t.Fatalf("the preset wrote a recipe this build cannot read: %v\n%s", err, expanded.Source) + } + + held := map[string]bool{} + for _, target := range doc.Targets { + held[cellKey(target.Format, target.Properties)] = true + } + + var refused, written int + for _, desc := range format.All() { + encoding, ok := settingNamed(desc, "encoding") + if !ok { + continue + } + for _, cell := range cellsOf(desc, encoding) { + key := cellKey(desc.ID, cell) + if formatRefuses(desc, cell) { + refused++ + if held[key] { + t.Errorf("the set holds %s, which this build refuses to write", key) + } + continue + } + written++ + if !held[key] { + t.Errorf("this build writes %s and the set leaves it out", key) + } + } + } + + if refused == 0 { + t.Error("no combination of encoding and byte order mark is refused by any format in this " + + "build, so the half of this guard about what the set leaves OUT is asserting nothing") + } + if written == 0 { + t.Fatal("no combination is writable, so this guard is asserting nothing at all") + } + t.Logf("%d combination(s) written and in the set, %d refused and left out", written, refused) +} + +// cellsOf is every combination of encoding and mark one format declares. +func cellsOf(desc format.Descriptor, encoding format.Property) []map[string]string { + marks := []string{""} + if _, ok := settingNamed(desc, "bom"); ok { + marks = []string{"false", "true"} + } + var out []map[string]string + for _, name := range encoding.Choices { + for _, mark := range marks { + cell := map[string]string{"encoding": name} + if mark != "" { + cell["bom"] = mark + } + out = append(out, cell) + } + } + return out +} + +// formatRefuses asks the format itself whether it will write this combination, +// at the size the format names as its smallest for it - so an answer that comes +// back is about the settings and not about the room they need. +func formatRefuses(desc format.Descriptor, cell map[string]string) bool { + r := format.Request{Label: true, Properties: cell} + r.Bytes = desc.SmallestAccepted(r) + _, err := desc.Generator.Plan(r) + return err != nil +} + +func cellKey(id string, props map[string]string) string { + mark := props["bom"] + if mark == "" { + mark = "false" + } + return id + " " + props["encoding"] + " bom=" + mark +} + +// declaredSetting is one setting of a format as the format declares it. +func settingNamed(desc format.Descriptor, name string) (format.Property, bool) { + for _, p := range desc.Properties { + if p.Name == name { + return p, true + } + } + return format.Property{}, false +} + +// A list of formats refused for a name this build does not have says what it +// does have. +// +// The person who typed heic has no other way to find out. The refusal for a +// single format has named the list since the registry was written, and a list +// parameter is where it would be dropped without anybody noticing: the value is +// split first, so the refusal is raised by the preset rather than by +// format.Get, and a preset that worded its own sentence would say "unknown +// format" and stop. +func TestAListOfFormatsRefusedNamesWhatThisBuildHas(t *testing.T) { + lists := []struct{ id, param string }{ + {"empty-and-minimal", "formats"}, + {"upload-validation", "allow"}, + } + if len(format.IDs()) == 0 { + t.Fatal("this build registers no format, so there is no list for a refusal to name") + } + + for _, list := range lists { + _, err := preset.Expand(list.id, preset.Args{list.param: "heic"}) + if err == nil { + t.Errorf("%s took %s=heic and this build has no such format", list.id, list.param) + continue + } + for _, id := range format.IDs() { + if !strings.Contains(err.Error(), id) { + t.Errorf("%s refusing %s=heic does not name %s, which this build does have: %s", + list.id, list.param, id, err) + break + } + } + } +} diff --git a/internal/guard/presetwindow_test.go b/internal/guard/presetwindow_test.go index 8bf2415a..8fa955e6 100644 --- a/internal/guard/presetwindow_test.go +++ b/internal/guard/presetwindow_test.go @@ -198,10 +198,7 @@ func TestTheWindowDrawsAFieldForEveryPresetParameter(t *testing.T) { if bad := wrongKindOfControl(param, control); bad != "" { t.Errorf("%s.%s is %s", p.ID, param.Name, bad) } - if shown := everythingSaid(content); !strings.Contains(shown, param.Allowed()) { - t.Errorf("the field for %s.%s does not say what it takes (%q)", - p.ID, param.Name, param.Allowed()) - } + saysWhatItTakes(t, everythingSaid(content), p.ID, param) checked++ } } diff --git a/internal/guard/screenpixels_test.go b/internal/guard/screenpixels_test.go index eb39e780..4f7bf2fb 100644 --- a/internal/guard/screenpixels_test.go +++ b/internal/guard/screenpixels_test.go @@ -442,13 +442,29 @@ func screenScenes() []screenScene { // else on the form. {name: "preset-menu-setting", tab: text.TabPresets(), set: func(t *testing.T, s scene) { - // The only preset that reads a global flag, so it is the only - // one with a menu among its settings. + // The only preset that reads a GLOBAL flag, so the menu this + // opens is one drawn from preset.Global rather than from the + // preset's own declaration. upload-validation has a menu among + // its settings too since 2026-09-22 - see the screen below, + // which is where a menu the preset declares itself is looked at. menuUnder(t, s.tab, text.FieldPreset()).SetSelected("size-boundaries") }, after: func(t *testing.T, s scene) { menuUnder(t, s.tab, text.SettingLabel("format")).Tapped(&fyne.PointEvent{}) }}, + // The widest form this screen can be asked to draw: five settings, of + // four different kinds, one of them a closed set the preset declares + // itself. + // + // It is here because nobody had looked. Every preset until 2026-09-22 + // declared one setting or two, so "what does this form do with five" + // was answered by reading preset.go and finding a VScroll, which is a + // reading rather than a measurement - and UX.md section 7.0 gate 1 + // counts a state with no picture as a state nobody has seen. + {name: "preset-many-settings", tab: text.TabPresets(), + set: func(t *testing.T, s scene) { + menuUnder(t, s.tab, text.FieldPreset()).SetSelected("upload-validation") + }}, // The recipe screen, which arrived on 2026-08-18. It has states neither // of the others can be put into, and every one of them is here because a diff --git a/internal/guard/tabularset_test.go b/internal/guard/tabularset_test.go new file mode 100644 index 00000000..edbda1b3 --- /dev/null +++ b/internal/guard/tabularset_test.go @@ -0,0 +1,146 @@ +package guard + +import ( + "testing" + + "github.com/donislawdev/TestingFilesGenerator/internal/format" + _ "github.com/donislawdev/TestingFilesGenerator/internal/format/all" + "github.com/donislawdev/TestingFilesGenerator/internal/preset" + "github.com/donislawdev/TestingFilesGenerator/internal/recipe" +) + +// The table preset takes the counts the spreadsheet itself takes. +// +// It has to declare them rather than read them: a preset is registered at init +// and the format registry has not necessarily finished filling by then, which +// is why preset.Global reads the formats when it is called instead. So the +// range is written down, and a written down number goes stale green. This is +// the thing that reddens instead. +// +// What it protects: a field on a screen and a refusal on the command line, both +// built from the declaration. A preset offering "1 to 200000 rows" beside a +// build whose sheet stops at 50000 would take the value, draw the field, refuse +// the run - and blame the sheet for a range this preset had promised. +func TestTheTabularPresetTakesTheRangesTheSheetDeclares(t *testing.T) { + sheet, err := format.Get("xlsx") + if err != nil { + t.Fatalf("this build has no spreadsheet, so the preset built on one cannot be checked: %v", err) + } + p, err := preset.Get("tabular-import") + if err != nil { + t.Fatalf("this build has no tabular-import preset: %v", err) + } + + checked := 0 + for _, param := range p.Parameters { + declared, ok := settingNamed(sheet, param.Name) + if !ok { + t.Errorf("the preset declares a parameter called %q and the spreadsheet has no such "+ + "setting, so nothing says what its range should be", param.Name) + continue + } + if param.Min != declared.Min || param.Max != declared.Max { + t.Errorf("the preset takes %s from %d to %d and the spreadsheet takes %d to %d - "+ + "the field and the refusal are built from the first and the run is judged by the second", + param.Name, param.Min, param.Max, declared.Min, declared.Max) + } + if param.Unit != declared.Unit { + t.Errorf("the preset counts %s in %q and the spreadsheet counts it in %q", + param.Name, param.Unit, declared.Unit) + } + checked++ + } + if checked == 0 { + t.Fatal("no parameter was compared - this guard would pass against any range ever written") + } + t.Logf("%d parameter(s) with the range the spreadsheet declares", checked) +} + +// Every dialect the table format declares is in the set. +// +// The group is built by walking what CSV declares rather than by a list written +// in the preset, and this is what holds that true from the other side: a fifth +// dialect setting has to arrive in the set without anybody adding it, and one +// that stopped arriving has to redden. +// +// One file per value, minus the one the base already stands for. The base is a +// file of its own holding every setting at its default, so the count is exact +// rather than a lower bound: a set where one axis quietly produced nothing +// would still hold files for the others and still look like a set. +// +// What it does NOT check is which of those files expect accept and which say +// the answer is the reader's policy. That line is drawn in the preset, by a +// standard rather than by the registry - RFC 4180 asks for CRLF and for +// quoting, and no registry knows that. +func TestEveryDialectTheTableDeclaresIsInTheSet(t *testing.T) { + csv, err := format.Get("csv") + if err != nil { + t.Fatalf("this build has no csv, so the dialect half of this set cannot be checked: %v", err) + } + expanded, err := preset.Expand("tabular-import", preset.Args{}) + if err != nil { + t.Fatalf("the preset refused its own defaults: %v", err) + } + doc, err := recipe.Parse(expanded.Source, "tabular-import.yaml") + if err != nil { + t.Fatalf("the preset wrote a recipe this build cannot read: %v\n%s", err, expanded.Source) + } + + dialects := targetsInGroup(doc, "csv-dialects") + if len(dialects) == 0 { + t.Fatal("the set holds no dialect group at all, so this guard is asserting nothing") + } + + axes := 0 + wanted := 1 // the base, which stands for every setting at its default + for _, p := range csv.Properties { + if p.Kind != format.PropertyChoice && p.Kind != format.PropertyBool { + continue + } + axes++ + wanted += len(valuesOfSetting(p)) - 1 + assertAxisIsVaried(t, dialects, p) + } + if axes == 0 { + t.Fatal("csv declares no setting that names a shape, so there is no dialect to vary") + } + if len(dialects) != wanted { + t.Errorf("the dialect group holds %d files and %d setting(s) with their values come to %d", + len(dialects), axes, wanted) + } + t.Logf("%d dialect file(s) across %d setting(s)", len(dialects), axes) +} + +// assertAxisIsVaried checks that every value of one setting appears somewhere in +// the group, the default included - it is the base that carries that one. +func assertAxisIsVaried(t *testing.T, dialects []recipe.Target, p format.Property) { + t.Helper() + + seen := map[string]bool{} + for _, target := range dialects { + seen[target.Properties[p.Name]] = true + } + for _, value := range valuesOfSetting(p) { + if !seen[value] { + t.Errorf("csv takes %s=%s and no file of the dialect group is written that way", + p.Name, value) + } + } +} + +func valuesOfSetting(p format.Property) []string { + if p.Kind == format.PropertyBool { + return []string{"false", "true"} + } + return p.Choices +} + +func targetsInGroup(doc *recipe.Recipe, group string) []recipe.Target { + var out []recipe.Target + for _, target := range doc.Targets { + if target.Group == group { + out = append(out, target) + } + } + return out +} diff --git a/internal/guard/testdata/screens/preset-many-settings.png b/internal/guard/testdata/screens/preset-many-settings.png new file mode 100644 index 00000000..070d9bc3 Binary files /dev/null and b/internal/guard/testdata/screens/preset-many-settings.png differ diff --git a/internal/guard/testdata/screens/preset-many-settings.xml b/internal/guard/testdata/screens/preset-many-settings.xml new file mode 100644 index 00000000..a529c7df --- /dev/null +++ b/internal/guard/testdata/screens/preset-many-settings.xml @@ -0,0 +1,493 @@ + + + + + + + + + + + Single batch + + + + + Presets + + + + + Several batches + + + + + About + + + + + + + + + + + + + + + + + Presets + + + + + + + + + Ready-made sets of files, each built to answer one question about the system under test. + + + + + + + + + + + The question + + + + Preset + + + + + + + + + + + + + upload-validation + + + + + + + + + + + + + + + + Does my upload form take what it should and turn the rest away? + + + + + + + + + Typically finds: + + + + + + + + + + + a limit enforced in the browser and not on the server + + + + + • + + + + + + + an SVG or an HTML file taken for a picture or for plain text, which is a way to get a script past a form + + + + + • + + + + + + + a file checked by its extension and never opened, so a PDF named .jpg goes through + + + + + • + + + + + + + a form that reads the whole body into memory before it looks at how big it is + + + + + • + + + + + + + an upload named PHOTO.JPG turned away where photo.jpg is taken, or the other way round + + + + + • + + + + + + + a name with spaces, brackets or characters outside ASCII written to disk unchanged + + + + + • + + + + + + + + + + + Settings + + + + + Limit + + + + + + + + + + + + + + + 10mb + + + + + + + + + + + + + + + + + + + + + + + + + Allow + + + + + + + + + + + + + + + jpg,png,pdf + + + + + + + + + + + + + + + + Deny + + + + + + + + + + + + + + + svg,html,exe,sh + + + + + + + + + + + + + + + + Far-over + + + + + + + + + + + + + 2x + + + + + + + + + + + + + + Bulk + + + + + + + + + + + + + + + 50 + + + + + + + + + + + + + + + + + + + + + Output + + + + Output directory + + * + + + + + + + + + + + + + + + + /tfg/out + + + + + + + + Choose... + + + + + + + + + + Seed + + * + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + + + + + + + + + + + + + + + + Preview + + + + + Generate + + + + + + + + + + + 71 files · 115.1 MB (120 639 488 B) · html, jpg, pdf, png, svg, txt · will go to /tfg/out + + + + + + + + + + + + + + + + Donate + + + + + + + + + + + + + diff --git a/internal/guard/testdata/screens/preset-menu.png b/internal/guard/testdata/screens/preset-menu.png index 1c2fa78f..862eaa7e 100644 Binary files a/internal/guard/testdata/screens/preset-menu.png and b/internal/guard/testdata/screens/preset-menu.png differ diff --git a/internal/guard/testdata/screens/preset-menu.xml b/internal/guard/testdata/screens/preset-menu.xml index b3c6c87f..47dcbd1c 100644 --- a/internal/guard/testdata/screens/preset-menu.xml +++ b/internal/guard/testdata/screens/preset-menu.xml @@ -340,16 +340,16 @@ - - - - - - - - - - + + + + + + + + + + @@ -363,12 +363,39 @@ size-boundaries + + + + tabular-import + + + + + + text-encoding + + + + + + upload-validation + + + + + + + + + + + diff --git a/internal/guard/uploadset_test.go b/internal/guard/uploadset_test.go new file mode 100644 index 00000000..5aaa511d --- /dev/null +++ b/internal/guard/uploadset_test.go @@ -0,0 +1,94 @@ +package guard + +import ( + "strings" + "testing" + + "github.com/donislawdev/TestingFilesGenerator/internal/format" + _ "github.com/donislawdev/TestingFilesGenerator/internal/format/all" + "github.com/donislawdev/TestingFilesGenerator/internal/preset" + "github.com/donislawdev/TestingFilesGenerator/internal/recipe" +) + +// A type cannot be allowed and denied at once. +// +// The two lists never saw each other, so "--allow pdf --deny pdf" laid out a +// set holding a PDF expecting accept and a PDF expecting reject for +// extension_rule. Both reach the manifest, so any suite running that set +// contradicts itself whatever the system under test does - and nothing said a +// word, which is untouchable rule 6 on the silence that is hardest to notice: +// the one where nothing fails. +// +// Found by review on 2026-09-22, and the whole set is refused rather than one +// half dropped, because dropping a half chooses for somebody which of the two +// they meant. +func TestAnExtensionCannotBeAllowedAndDeniedAtOnce(t *testing.T) { + // The state this guard is about, asserted rather than assumed: the + // declared defaults have to be buildable, or the refusal below would be + // the preset refusing everything. + if _, err := preset.Expand("upload-validation", preset.Args{}); err != nil { + t.Fatalf("the preset refuses its own defaults, so nothing here is about an overlap: %v", err) + } + + for _, id := range []string{"pdf", "targz"} { + _, err := preset.Expand("upload-validation", preset.Args{"allow": id, "deny": id}) + if err == nil { + t.Errorf("%s is allowed and denied at once and the set was built anyway - one of its "+ + "files expects accept and another expects reject, and both are %s", id, id) + continue + } + if !strings.Contains(err.Error(), id) { + t.Errorf("the refusal for %s allowed and denied does not name it: %s", id, err) + } + } +} + +// Every file of the denied group is named with the extension the registry +// declares for it. +// +// A format's extension is not always a dot and its id. targz is written +// .tar.gz, so a denied file built from the id alone was called denied.targz - +// a name no upload form has a rule about, which is the one thing that group +// exists to test. Found on 2026-09-22 while checking the overlap above. +// +// An entry this build has no format for keeps the extension as typed, which is +// the point of taking those at all. +func TestADeniedFileIsNamedWithTheExtensionTheRegistryDeclares(t *testing.T) { + const denied = "targz,zip,exe" + + expanded, err := preset.Expand("upload-validation", preset.Args{"deny": denied}) + if err != nil { + t.Fatalf("the preset refused %q: %v", denied, err) + } + doc, err := recipe.Parse(expanded.Source, "upload-validation.yaml") + if err != nil { + t.Fatalf("the preset wrote a recipe this build cannot read: %v", err) + } + + checked := 0 + for _, target := range doc.Targets { + if target.Group != "denied-types" { + continue + } + checked++ + want := "." + strings.TrimPrefix(target.ID, "denied_") + if desc, err := format.Get(strings.TrimPrefix(target.ID, "denied_")); err == nil { + want = desc.Extension + } + if !strings.HasSuffix(target.Name, want) { + t.Errorf("the denied file %s is called %q and this build writes that format as %q", + target.ID, target.Name, want) + } + } + if checked != 3 { + t.Fatalf("%d denied file(s) for a list of three, so this guard is not looking at what it thinks", + checked) + } + // One of the three has a multi-part extension and one is outside the + // registry, or the two halves of this guard are the same half twice. + if desc, err := format.Get("targz"); err != nil || desc.Extension == ".targz" { + t.Errorf("targz is written %q in this build, so it no longer stands for the case this "+ + "guard was written about - find another format whose extension is not a dot and its id", + desc.Extension) + } +} diff --git a/internal/preset/build.go b/internal/preset/build.go index 420ed9b3..46eaafbb 100644 --- a/internal/preset/build.go +++ b/internal/preset/build.go @@ -2,6 +2,7 @@ package preset import ( "fmt" + "strconv" "strings" "github.com/donislawdev/TestingFilesGenerator/internal/format" @@ -131,6 +132,76 @@ func knownFormat(item string) string { // lower is the keep for a list of names the registry spells in lower case. func lower(item string) string { return strings.ToLower(item) } +// setFile is one file of a preset's set: which format writes it, what it is +// called, what settings make it what it is, and what a reasonably built system +// should do with it. +// +// Shared because two presets lay their sets out this way and a third is the +// point at which a shape becomes a type. What it is NOT is every file of every +// preset: the minimal set and the encoding set each name their files from one +// varying part and compute the rest, which is a rule of their own rather than a +// field here. +type setFile struct { + id, name, group string + desc format.Descriptor + props map[string]string + // count is how many files this one entry stands for. Nought and one both + // mean a single file, because a set with one of something says "1" and a + // set with none of it leaves the entry out entirely. + count int + size int64 + // atFloor asks for the smallest size this format takes for these settings, + // whatever that turns out to be. A separate field rather than a nought in + // size, because nought is a real size and a file of no bytes is a case two + // of these presets are about - the sentinel that collides with a legal + // value is how a guard ends up testing the wrong thing. + atFloor bool + expected, reason string +} + +// bytes is the size this file is asked for. +func (f setFile) bytes() int64 { + if f.atFloor { + return f.desc.SmallestAccepted(format.Request{Label: true, Properties: f.props}) + } + return f.size +} + +// refused is what the format says about this file before anything is written, +// or nil when it will produce it. +// +// Every set built from setFile asks this of every file it holds, which is PR7 +// one level down: a set missing the three files the run was about still looks +// like a set. +func (f setFile) refused() error { + r := format.Request{Label: true, Properties: f.props} + r.Bytes = f.bytes() + _, err := f.desc.Generator.Plan(r) + return err +} + +func (f setFile) draft() recipe.TargetDraft { + count := "1" + if f.count > 1 { + count = strconv.Itoa(f.count) + } + return recipe.TargetDraft{ + ID: f.id, Format: f.desc.ID, Count: count, + Size: strconv.FormatInt(f.bytes(), 10), Name: f.name, Group: f.group, + Expected: f.expected, ExpectedReason: f.reason, + Properties: f.props, + } +} + +// draftsOf is a whole set as targets, ready for the composer. +func draftsOf(files []setFile) []recipe.TargetDraft { + out := make([]recipe.TargetDraft, 0, len(files)) + for _, f := range files { + out = append(out, f.draft()) + } + return out +} + // plan is the set a preset lays out, ready to be written. // // A preset describes targets and this turns them into source. PR5 asks for diff --git a/internal/preset/limitset.go b/internal/preset/limitset.go new file mode 100644 index 00000000..2c3d229b --- /dev/null +++ b/internal/preset/limitset.go @@ -0,0 +1,165 @@ +package preset + +import ( + "fmt" + "strconv" + "strings" + + "github.com/donislawdev/TestingFilesGenerator/internal/format" + "github.com/donislawdev/TestingFilesGenerator/internal/recipe" +) + +// The set a declared limit produces: the limit itself, and one file either +// side of it for every distance asked for. +// +// It lives on its own because two presets lay it out. size-boundaries is a +// microscope on this one axis and reaches as far either side as it is told. +// upload-validation is a survey across many axes, of which size is one, and +// takes a single step either side plus one file well past the limit - the +// verdict of PRESET-FEASIBILITY-2026-09-08.md section 5, where the alternative +// was two presets doing literally the same thing. +// +// The sentence "under the limit is accepted, over it is refused for size_limit" +// then exists once. Writing it twice is what the same document calls an +// invitation for the two to drift, and the drift would be invisible: both sets +// would still run, still verify, and disagree about what a limit means. +// +// D11 gate on the extraction, measured 2026-09-22: eject size-boundaries at its +// defaults gives 1298 B and the same sha256 recorded on 2026-09-08, because the +// bytes of an ejected recipe reach other people's manifests as a recipe_hash. + +// offset is one step either side of the limit, with the text it was written as +// so the files can name themselves after it. +type offset struct { + text string + bytes int64 +} + +// step is one file of the set: how big, what it is called, and what a +// reasonably built system should do with it. +type step struct { + id string + size int64 + accept bool +} + +// limitSet is a declared limit and everything needed to lay files out around it. +type limitSet struct { + // preset and setting name what a refusal is about: which preset could not + // build the set, and which of its parameters a window should mark. + preset, setting string + // group is what the files call themselves collectively in the manifest, so + // a test can assert on the whole class at once. + group string + desc format.Descriptor + limit int64 + // limitText is the limit as it was typed. It leads every file name, so two + // sets built around different limits cannot be told apart only by opening + // the files - reported from use on 2026-08-11, where a directory holding + // two runs was a directory of guesses. + limitText string + spread []offset +} + +// steps lays the set out, largest distance below the limit first, then the +// limit, then upward. The order is the order somebody reads a table in. +func (s limitSet) steps() []step { + out := make([]step, 0, 2*len(s.spread)+1) + for i := len(s.spread) - 1; i >= 0; i-- { + out = append(out, step{ + id: "under_" + s.spread[i].text, size: s.limit - s.spread[i].bytes, accept: true, + }) + } + out = append(out, step{id: "at_limit", size: s.limit, accept: true}) + for _, o := range s.spread { + out = append(out, step{id: "over_" + o.text, size: s.limit + o.bytes, accept: false}) + } + return out +} + +// reachable refuses the whole set when any one file of it is out of reach. +// +// PR7, and the untouchable rule about silence. A set missing three of its seven +// files still looks like a set, and the three that are missing are the ones the +// run was about - the ones nearest the limit. +func (s limitSet) reachable(set []step) error { + floor := s.desc.SmallestAccepted(format.Request{Seed: 1, Label: true}) + for _, one := range set { + if one.size >= floor { + continue + } + what := fmt.Sprintf("%s would be %d B and the smallest %s this build makes is %d B", + one.id, one.size, strings.ToUpper(s.desc.ID), floor) + if one.size <= 0 { + what = fmt.Sprintf("%s would be %d B, and a file cannot be smaller than nothing", one.id, one.size) + } + return &ImpossibleError{ + Preset: s.preset, + // The limit rather than the spread, although the sentence offers + // both ways out. The limit is the one number the set is measured + // from, so it is where somebody types first - and a message can + // only stand beside one box. + Setting: s.setting, + Detail: what, + Hint: fmt.Sprintf( + // The settings are named without a leading dash on purpose. This + // sentence is built in the engine and both surfaces show it word + // for word, so a spelling only one of them has sends the other's + // reader translating: the window labels these fields "limit" and + // "spread", and there is no "--limit" anywhere on it. Seen on + // screen 2026-08-11, O79. + "Raise the {setting} above %d B, narrow the spread, or choose a format with a smaller minimum. The {setting} asked for was %d B.", + floor+deepest(set, s.limit), s.limit), + } + } + return nil +} + +// deepest is how far below the limit the set reaches, so the hint can name a +// limit that would work rather than only the one that did not. +func deepest(set []step, limit int64) int64 { + var found int64 + for _, one := range set { + if d := limit - one.size; d > found { + found = d + } + } + return found +} + +// drafts is the set as targets, ready for the composer. +// +// This used to print the document itself, line by line, with a comment saying +// that was safe because every value was one the package built itself. The +// comment was wrong until 2026-08-05: the id carries the caller's own text, so +// "1\rB" reached the document raw and broke it, because the size parser trims +// the ends and that carriage return sat in the middle. Found by fuzzing rather +// than by reading. +// +// parseSpread refuses that character now, and this no longer writes YAML at +// all - plan.source hands the values to the marshaller, which does the quoting +// and owns the shape of the document. Two defences rather than one, and the +// second one cannot be forgotten by the next preset. +func (s limitSet) drafts(set []step) []recipe.TargetDraft { + out := make([]recipe.TargetDraft, 0, len(set)) + for _, one := range set { + draft := recipe.TargetDraft{ + ID: one.id, + Format: s.desc.ID, + Count: "1", + Size: strconv.FormatInt(one.size, 10), + // The id stays as it was. It derives the seed, so putting the limit + // in it would move the bytes of every file in this set for a change + // that is about telling two directories apart. + Name: s.limitText + "_" + one.id + s.desc.Extension, + Group: s.group, + Expected: "accept", + } + if !one.accept { + draft.Expected = "reject" + draft.ExpectedReason = "size_limit" + } + out = append(out, draft) + } + return out +} diff --git a/internal/preset/preset.go b/internal/preset/preset.go index d63ef1da..bffa17b4 100644 --- a/internal/preset/preset.go +++ b/internal/preset/preset.go @@ -313,9 +313,16 @@ func (e *ImpossibleError) InTheWordsOf(name string) string { if name == "" { name = e.Setting } - return core.InTheWordsOf( - fmt.Sprintf("the preset %s cannot build this set - %s. %s", e.Preset, e.Detail, e.Hint), - name) + said := fmt.Sprintf("the preset %s cannot build this set - %s", e.Preset, e.Detail) + // A refusal that came up from the registry already ends with what to do + // about it, and a second sentence after that one is worse than no second + // sentence: "Ask for fewer rows or fewer columns. Ask for a set the rows + // can carry" is the tool saying the same thing twice and vaguer the second + // time. Seen on 2026-09-22 while the tabular set was being wired up. + if e.Hint == "" { + return core.InTheWordsOf(said+".", name) + } + return core.InTheWordsOf(said+". "+e.Hint, name) } // AboutSetting lets a window put this message beside the box that caused it, @@ -410,22 +417,36 @@ func IDs() []string { return ids() } -// Declaring is the preset that declares a parameter of this name, or empty. +// Declaring is every preset that declares a parameter of this name, by id. // // Parameter names and global flag names share one namespace, which is what // lets "--preset size-boundaries --limit 10mb" read as one sentence. The cost // of that is a flag which exists in one invocation and not in the next, so // typing --limit without its preset has to be answered with something better // than "not defined". This is how the command line finds out whose it is. -func Declaring(name string) string { +// +// Every owner rather than the first, since 2026-09-22. Two presets ask for the +// number a system declares as its limit and both call it limit - deliberately, +// because a reader who learns one word and types it at the other has learnt the +// right word. Answering with only the first left that reader pointed at the +// preset they were not using, with a sentence that read as certain. +func Declaring(name string) []string { + var out []string for _, id := range ids() { - for _, param := range registry[id].Parameters { - if param.Name == name { - return id - } + if declares(registry[id], name) { + out = append(out, id) } } - return "" + return out +} + +func declares(p Preset, name string) bool { + for _, param := range p.Parameters { + if param.Name == name { + return true + } + } + return false } func ids() []string { diff --git a/internal/preset/sizeboundaries.go b/internal/preset/sizeboundaries.go index 0b2ca88f..061d9348 100644 --- a/internal/preset/sizeboundaries.go +++ b/internal/preset/sizeboundaries.go @@ -2,12 +2,10 @@ package preset import ( "fmt" - "strconv" "strings" "github.com/donislawdev/TestingFilesGenerator/internal/core" "github.com/donislawdev/TestingFilesGenerator/internal/format" - "github.com/donislawdev/TestingFilesGenerator/internal/recipe" ) const ( @@ -59,13 +57,6 @@ func init() { }) } -// offset is one step either side of the limit, with the text it was written as -// so the files can name themselves after it. -type offset struct { - text string - bytes int64 -} - // spreadList is how the distances either side of the limit are written. // // The shared parser does the splitting, the duplicate and the refusal, and this @@ -160,30 +151,6 @@ func parseSpread(raw string) ([]offset, error) { return out, nil } -// step is one file of the set: how big, what it is called, and what a -// reasonably built system should do with it. -type step struct { - id string - size int64 - accept bool -} - -// steps lays the set out, largest distance below the limit first, then the -// limit, then upward. The order is the order somebody reads a table in. -func steps(limit int64, spread []offset) []step { - out := make([]step, 0, 2*len(spread)+1) - for i := len(spread) - 1; i >= 0; i-- { - out = append(out, step{ - id: "under_" + spread[i].text, size: limit - spread[i].bytes, accept: true, - }) - } - out = append(out, step{id: "at_limit", size: limit, accept: true}) - for _, o := range spread { - out = append(out, step{id: "over_" + o.text, size: limit + o.bytes, accept: false}) - } - return out -} - func expandSizeBoundaries(args Args) ([]byte, error) { limit, err := core.ParseSize(args["limit"]) if err != nil { @@ -233,100 +200,17 @@ func expandSizeBoundaries(args Args) ([]byte, error) { } } - set := steps(limit, spread) - if err := reachable(set, desc, limit); err != nil { + around := limitSet{ + preset: boundariesID, setting: "limit", group: boundariesID, + desc: desc, limit: limit, limitText: limitText, spread: spread, + } + set := around.steps() + if err := around.reachable(set); err != nil { return nil, err } return plan{ preset: boundariesID, question: boundariesQuestion, - targets: draftsOfSteps(set, desc, limitText), + targets: around.drafts(set), }.source() } - -// reachable refuses the whole set when any one file of it is out of reach. -// -// PR7, and the untouchable rule about silence. A set missing three of its seven -// files still looks like a set, and the three that are missing are the ones the -// run was about - the ones nearest the limit. -func reachable(plan []step, desc format.Descriptor, limit int64) error { - floor := desc.SmallestAccepted(format.Request{Seed: 1, Label: true}) - for _, s := range plan { - if s.size >= floor { - continue - } - what := fmt.Sprintf("%s would be %d B and the smallest %s this build makes is %d B", - s.id, s.size, strings.ToUpper(desc.ID), floor) - if s.size <= 0 { - what = fmt.Sprintf("%s would be %d B, and a file cannot be smaller than nothing", s.id, s.size) - } - return &ImpossibleError{ - Preset: boundariesID, - // The limit rather than the spread, although the sentence offers - // both ways out. The limit is the one number the set is measured - // from, so it is where somebody types first - and a message can - // only stand beside one box. - Setting: "limit", - Detail: what, - Hint: fmt.Sprintf( - // The settings are named without a leading dash on purpose. This - // sentence is built in the engine and both surfaces show it word - // for word, so a spelling only one of them has sends the other's - // reader translating: the window labels these fields "limit" and - // "spread", and there is no "--limit" anywhere on it. Seen on - // screen 2026-08-11, O79. - "Raise the {setting} above %d B, narrow the spread, or choose a format with a smaller minimum. The {setting} asked for was %d B.", - floor+largest(plan, limit), limit), - } - } - return nil -} - -// largest is how far below the limit the set reaches, so the hint can name a -// limit that would work rather than only the one that did not. -func largest(plan []step, limit int64) int64 { - var deepest int64 - for _, s := range plan { - if d := limit - s.size; d > deepest { - deepest = d - } - } - return deepest -} - -// draftsOfSteps is the set as targets, ready for the composer. -// -// This used to print the document itself, line by line, with a comment saying -// that was safe because every value was one the package built itself. The -// comment was wrong until 2026-08-05: the id carries the caller's own text, so -// "1\rB" reached the document raw and broke it, because the size parser trims -// the ends and that carriage return sat in the middle. Found by fuzzing rather -// than by reading. -// -// parseSpread refuses that character now, and this no longer writes YAML at -// all - plan.source hands the values to the marshaller, which does the quoting -// and owns the shape of the document. Two defences rather than one, and the -// second one cannot be forgotten by the next preset. -func draftsOfSteps(set []step, desc format.Descriptor, limitText string) []recipe.TargetDraft { - out := make([]recipe.TargetDraft, 0, len(set)) - for _, s := range set { - draft := recipe.TargetDraft{ - ID: s.id, - Format: desc.ID, - Count: "1", - Size: strconv.FormatInt(s.size, 10), - // The id stays as it was. It derives the seed, so putting the limit - // in it would move the bytes of every file in this set for a change - // that is about telling two directories apart. - Name: limitText + "_" + s.id + desc.Extension, - Group: boundariesID, - Expected: "accept", - } - if !s.accept { - draft.Expected = "reject" - draft.ExpectedReason = "size_limit" - } - out = append(out, draft) - } - return out -} diff --git a/internal/preset/tabularimport.go b/internal/preset/tabularimport.go new file mode 100644 index 00000000..9e943e18 --- /dev/null +++ b/internal/preset/tabularimport.go @@ -0,0 +1,310 @@ +package preset + +import ( + "strconv" + "strings" + + "github.com/donislawdev/TestingFilesGenerator/internal/format" +) + +const ( + tabularID = "tabular-import" + + rowsParam = "rows" + columnsParam = "columns" + defaultRows = "1000" + defaultColumns = "10" + + // The ranges the spreadsheet declares, written here because a preset is + // registered at init and the format registry has not necessarily finished + // filling by then - the same reason Global() reads the formats when it is + // called rather than when it is declared. + // + // A number copied by hand goes stale green, so this pair is not left to + // care: TestTheTabularPresetTakesTheRangesTheSheetDeclares asks the + // registry and reddens when they drift. + sheetRowsMax = 200000 + sheetColumnsMax = 32768 + + dialectGroup = "csv-dialects" + wideGroup = "csv-wide" + sheetGroup = "xlsx-rows" + layoutGroup = "json-layouts" + + // tableSample is how big the files that are about SHAPE rather than size + // are - the dialects and the layouts. Big enough to hold hundreds of rows, + // so a reader that copes with the first row and not the hundredth is + // caught, and small enough that eleven of them are noise beside the + // spreadsheet. + tableSample = 64 << 10 + + // The settings this set varies, spelled here because no package owns these + // names the way textenc owns the encoding. What keeps them honest is the + // registry: the axes are read from what CSV declares, and these are only + // used to say which of them is a matter of policy. + csvDelimiter = "delimiter" + csvHeader = "header" + csvQuoteStyle = "quote_style" + jsonFormat = "formatting" + + // tabularQuestion is announced by the preset AND written into the header of + // an ejected recipe. Named once rather than typed twice. + tabularQuestion = "Does my table import survive what real tools export?" +) + +func init() { + Register(Preset{ + ID: tabularID, + Title: "Tabular import", + Question: tabularQuestion, + + Parameters: []format.Property{ + { + Name: rowsParam, Kind: format.PropertyInt, + Min: 1, Max: sheetRowsMax, Unit: "rows", + Default: defaultRows, + Detail: "How many rows the spreadsheet holds. It is written at exactly the size " + + "that many rows package to, so the budget above moves with this.", + }, + { + Name: columnsParam, Kind: format.PropertyInt, + Min: 1, Max: sheetColumnsMax, Unit: "columns", + Default: defaultColumns, + Detail: "How many columns each row of the spreadsheet has. Rows times columns " + + "has a ceiling, and asking past it is refused before anything is written.", + }, + }, + + Requires: []string{"MVP"}, + Catches: []string{ + "a semicolon file read as one column, because the delimiter was assumed rather than looked for", + "a CRLF file split into rows with an empty row after each one", + "a headerless table whose first row of data is eaten as column names", + "an import that keeps the columns it can show and drops the rest without a word", + "a reader that takes JSON records one line at a time and stops at the first indented document", + }, + + Expand: expandTabularImport, + }) +} + +// dialectPolicy says whether changing one CSV setting produces a file any +// reader that takes CSV has to cope with, or a file whose acceptance is that +// reader's own policy. +// +// The line is drawn where a standard draws it. RFC 4180 asks for CRLF and for +// quoting, so a reader that breaks on either has a defect rather than a policy. +// A semicolon file is what a European spreadsheet exports and it is a different +// dialect - a reader that takes commas only and says so is a correct reader, +// and MF5 forbids inventing the answer for it. A table with no header row is +// the same kind of question. +// +// Every axis CSV declares reaches the set whether or not it is named here - +// TestEveryDialectTheTableDeclaresIsInTheSet asks the registry for that. What +// an axis missing from this map loses is the stronger half of its verdict: it +// lands on the policy side and says "your call" where it might have said "this +// has to work". A weaker claim rather than a wrong one, which is the right way +// round for a default nobody chose. +var dialectPolicy = map[string]bool{ + lineEndingSetting: false, + csvQuoteStyle: false, + csvDelimiter: true, + csvHeader: true, +} + +// dialectFiles is the base table and one file for every other value of every +// setting CSV varies by. +// +// One axis at a time rather than the product, and that is the whole design of +// this group. Four delimiters times two line endings times two headers times +// three quote styles is forty eight files that say LESS than eight, because a +// failure in one of them names no cause. Eight files name one setting each. +// +// The base is a file of its own rather than four files restating it. Every +// setting at its declared default appears once, under the name default, and +// each of the other seven differs from it in exactly one place. +func dialectFiles(csv format.Descriptor) []setFile { + base := map[string]string{} + for _, axis := range dialectAxes(csv) { + base[axis.Name] = axis.Default + } + out := []setFile{{ + id: "dialect_default", name: "default.csv", group: dialectGroup, + desc: csv, props: base, size: tableSample, expected: "accept", + }} + + for _, axis := range dialectAxes(csv) { + out = append(out, dialectVariants(csv, base, axis)...) + } + return out +} + +// dialectVariants is one file for every value of one setting except the one the +// base already stands for. +func dialectVariants(csv format.Descriptor, base map[string]string, axis format.Property) []setFile { + var out []setFile + for _, value := range valuesOf(axis) { + if value == axis.Default { + continue + } + file := setFile{ + id: "dialect_" + axis.Name + "_" + value, + name: axis.Name + "_" + value + ".csv", + group: dialectGroup, desc: csv, props: besides(base, axis.Name, value), + size: tableSample, expected: "accept", + } + if dialectPolicy[axis.Name] { + file.expected, file.reason = "unspecified", "none" + } + out = append(out, file) + } + return out +} + +// besides is the base settings with one of them changed, leaving the base as +// it was. Every file of the group differs from it in exactly one place, so the +// copy is what keeps that true. +func besides(base map[string]string, name, value string) map[string]string { + out := make(map[string]string, len(base)) + for k, v := range base { + out[k] = v + } + out[name] = value + return out +} + +// dialectAxes is every setting of a format that names a shape rather than a +// size, in the order the format declares them. +// +// Read from the registry, so a fifth dialect setting joins the set without a +// line changing here. The whole numbers are left out because they are counts +// rather than dialects, and the one that matters has a group of its own. +func dialectAxes(desc format.Descriptor) []format.Property { + var out []format.Property + for _, p := range desc.Properties { + if p.Kind == format.PropertyChoice || p.Kind == format.PropertyBool { + out = append(out, p) + } + } + return out +} + +// valuesOf is what one setting can be, for a choice and for a switch. +func valuesOf(p format.Property) []string { + if p.Kind == format.PropertyBool { + return []string{"false", "true"} + } + return p.Choices +} + +// wideFile is the table with more columns than a spreadsheet will show. +// +// The count is the most this build writes rather than a number chosen here, +// because the registry is the only place that knows it and the file is about +// the ceiling. Its size is whatever that many columns need, so this one file +// sets the floor of the whole set and cannot be made smaller. +func wideFile(csv format.Descriptor) setFile { + columns, _ := declared(csv, columnsParam) + return setFile{ + id: "wide_table", name: "wide.csv", group: wideGroup, desc: csv, + props: map[string]string{columnsParam: strconv.FormatInt(columns.Max, 10)}, + atFloor: true, + // A spreadsheet shows what it can show and drops the rest without + // saying so. Whether an import should refuse such a table, truncate it + // or take it whole is its owner's decision, so this is a position + // rather than a promise - MF5. + expected: "unspecified", reason: "count_limit", + } +} + +// sheetFile is the spreadsheet at the row and column counts asked for. +func sheetFile(xlsx format.Descriptor, rows, columns string) setFile { + return setFile{ + id: "sheet", name: "sheet.xlsx", group: sheetGroup, desc: xlsx, + props: map[string]string{rowsParam: rows, columnsParam: columns}, + atFloor: true, + expected: "accept", + } +} + +// layoutFiles is the same records written the three ways a document can be +// laid out. +// +// All three are accepted by every JSON reader - the format says so in its own +// declaration - so this group is stated rather than left open. What it finds is +// a consumer that is not a JSON reader at all but a loop over lines. +func layoutFiles(js format.Descriptor) []setFile { + layout, ok := declared(js, jsonFormat) + if !ok { + return nil + } + out := make([]setFile, 0, len(layout.Choices)) + for _, value := range layout.Choices { + out = append(out, setFile{ + id: "layout_" + strings.ReplaceAll(value, "-", "_"), + name: value + ".json", group: layoutGroup, desc: js, + props: map[string]string{jsonFormat: value}, + size: tableSample, expected: "accept", + }) + } + return out +} + +// refusedTable turns a refusal from the registry into one about the setting +// somebody typed. +// +// The sheet refuses rows times columns above its ceiling, and the sentence it +// writes is the right one - it names both counts, the limit and why there is +// one. What it cannot know is that those two numbers came from parameters of a +// preset rather than from a recipe, so this is where the refusal learns which +// box to stand beside. +func refusedTable(f setFile) error { + err := f.refused() + if err == nil { + return nil + } + return &ImpossibleError{ + Preset: tabularID, + Setting: settingBehind(f), + // No hint of our own. The sheet's refusal ends with what to do about it + // and says it better than a general sentence could, because it knows + // which of the two counts it was and by how much. + Detail: strings.TrimPrefix(err.Error(), f.desc.ID+": "), + } +} + +// settingBehind is the parameter a file of this set was built from, so a +// refusal can stand beside the box that caused it. Empty where the file is +// built from no parameter at all. +func settingBehind(f setFile) string { + if f.group == sheetGroup { + return rowsParam + } + return "" +} + +func expandTabularImport(args Args) ([]byte, error) { + csv, err := format.Get("csv") + if err != nil { + return nil, err + } + xlsx, err := format.Get("xlsx") + if err != nil { + return nil, err + } + js, err := format.Get("json") + if err != nil { + return nil, err + } + + files := dialectFiles(csv) + files = append(files, wideFile(csv), sheetFile(xlsx, args[rowsParam], args[columnsParam])) + files = append(files, layoutFiles(js)...) + + for _, f := range files { + if err := refusedTable(f); err != nil { + return nil, err + } + } + return plan{preset: tabularID, question: tabularQuestion, targets: draftsOf(files)}.source() +} diff --git a/internal/preset/textencoding.go b/internal/preset/textencoding.go new file mode 100644 index 00000000..983f7a4b --- /dev/null +++ b/internal/preset/textencoding.go @@ -0,0 +1,413 @@ +package preset + +import ( + "errors" + "fmt" + "strconv" + "strings" + + "github.com/donislawdev/TestingFilesGenerator/internal/core" + "github.com/donislawdev/TestingFilesGenerator/internal/format" + "github.com/donislawdev/TestingFilesGenerator/internal/format/textenc" + "github.com/donislawdev/TestingFilesGenerator/internal/recipe" +) + +const ( + encodingID = "text-encoding" + // sampleParam is not called size, and that is forced rather than chosen: a + // preset parameter is a flag, --size is already one, and the build refuses + // a preset that shadows it. See clashingParameter in internal/cli. + sampleParam = "sample" + defaultSample = "4kb" + + encodingGroup = "encoding" + lineEndingGroup = "line-endings" + + // lineEndingSetting is the second axis. Written here rather than imported, + // because no package owns the name the way textenc owns the first one - csv + // declares a constant for it and log writes the string. What keeps this + // honest is not the spelling but the guard: the set is built from whatever + // the registry says carries this setting, so a format that gains it joins + // the set and a renamed setting empties the group and reddens. + lineEndingSetting = "line_ending" + + // encodingQuestion is announced by the preset AND written into the header + // of an ejected recipe. Named once rather than typed twice. + encodingQuestion = "Does my reader know which encoding a file is in, or is it guessing?" +) + +func init() { + Register(Preset{ + ID: encodingID, + Title: "Text encoding", + Question: encodingQuestion, + + Parameters: []format.Property{ + { + Name: sampleParam, Kind: format.PropertySize, + Default: defaultSample, + Detail: "How big each file of the set is. UTF-16 stores two bytes for every " + + "character, so an odd number is refused.", + }, + }, + + Requires: []string{"MVP"}, + Catches: []string{ + "a reader that assumes UTF-8 and shows a UTF-16 file as one character in three, or as rows of boxes", + "a byte order mark read as content, so the first field of an import starts with three stray characters", + "an importer that guesses the encoding from the opening bytes and guesses differently for a longer file", + "a CRLF file split into rows with an empty row after each one, or a carriage return kept inside the last field", + }, + + Says: saidAboutTheEncodingSet, + Expand: expandTextEncoding, + }) +} + +// textFile is one file of the set: a format, the settings that make it what it +// is, and what a reasonably built reader should do with it. +type textFile struct { + desc format.Descriptor + group string + props map[string]string + // label is what the file is named after - the setting that distinguishes it + // from its neighbours, such as "utf-16le_bom" or "crlf". + label string + expected, reason string +} + +func (f textFile) id() string { + return strings.ReplaceAll(f.group, "-", "_") + "_" + f.desc.ID + "_" + f.label +} + +func (f textFile) name() string { return f.label + f.desc.Extension } + +func (f textFile) draft(size int64) recipe.TargetDraft { + return recipe.TargetDraft{ + ID: f.id(), Format: f.desc.ID, Count: "1", + Size: strconv.FormatInt(size, 10), Name: f.name(), Group: f.group, + Expected: f.expected, ExpectedReason: f.reason, + Properties: f.props, + } +} + +// carrying is every format in this build that declares all of these settings. +// +// Asked of the registry rather than written down, which is the whole shape of +// this preset. The card for it, written on 2026-09-08, said the set was +// "utf-8 / utf-16 times a byte order mark times CRLF and LF on txt, md, csv and +// log" - and that set does not exist: txt has no line ending, csv has no +// encoding, and the two axes have no format in common. A list copied by hand +// goes stale green, so this one is not copied. +func carrying(settings ...string) []format.Descriptor { + var out []format.Descriptor + for _, id := range format.IDs() { + desc, err := format.Get(id) + if err == nil && declaresAll(desc, settings) { + out = append(out, desc) + } + } + return out +} + +// declaresAll says whether one format has every one of these settings. +func declaresAll(desc format.Descriptor, settings []string) bool { + for _, want := range settings { + if _, ok := declared(desc, want); !ok { + return false + } + } + return true +} + +// declared is one setting of a format as the format declares it. +func declared(desc format.Descriptor, name string) (format.Property, bool) { + for _, p := range desc.Properties { + if p.Name == name { + return p, true + } + } + return format.Property{}, false +} + +// encodingCells is the encoding half of the set, with the combinations this +// build refuses left out and named. +// +// The product is not full and cannot be generated blind. XML in UTF-16 has to +// open with a byte order mark - the specification says so and the format +// refuses the combination - so two cells of eighteen do not exist. Generating +// them would refuse the whole set over a cell nobody asked for and nobody could +// remove, because we laid it out rather than them. +// +// Which cells those are is asked of the format rather than written here. The +// rule belongs to XML today and the next text format may have its own. +func encodingCells() (files []textFile, left []string) { + for _, desc := range carrying(textenc.Setting) { + kept, dropped := cellsOfFormat(desc) + files = append(files, kept...) + left = append(left, dropped...) + } + return files, left +} + +// cellsOfFormat is the encoding half for one format. +func cellsOfFormat(desc format.Descriptor) (files []textFile, left []string) { + enc, _ := declared(desc, textenc.Setting) + for _, name := range enc.Choices { + kept, dropped := cellsOfEncoding(desc, name) + files = append(files, kept...) + left = append(left, dropped...) + } + return files, left +} + +// cellsOfEncoding is one format in one encoding, with and without a mark. +func cellsOfEncoding(desc format.Descriptor, encoding string) (files []textFile, left []string) { + for _, mark := range marks(desc) { + props := map[string]string{textenc.Setting: encoding} + label := encoding + if mark != "" { + props[textenc.SettingBOM] = mark + } + if mark == "true" { + label += "_bom" + } + if why := refusedOutright(desc, props); why != "" { + left = append(left, fmt.Sprintf("%s as %s (%s)", desc.ID, label, why)) + continue + } + files = append(files, textFile{ + desc: desc, group: encodingGroup, props: props, label: label, + expected: outcomeFor(encoding), reason: reasonFor(encoding), + }) + } + return files, left +} + +// marks is the byte order mark axis for one format, or one empty entry for a +// format that has an encoding and no mark to go with it. +// +// Both spellings rather than the declared choices, because a bool declares no +// closed set and a window draws it as a switch with two positions. +func marks(desc format.Descriptor) []string { + if _, ok := declared(desc, textenc.SettingBOM); !ok { + return []string{""} + } + return []string{"false", "true"} +} + +// outcomeFor and reasonFor are the owner's decision of 2026-09-22, and the +// line they draw is between what every reader has to handle and what is +// somebody's declared policy. +// +// UTF-8 is the one encoding a modern reader cannot decline, with or without a +// mark: the mark is legal there, and a reader showing it as stray characters +// has a defect we can name. Whether a system handles UTF-16 at all is its own +// policy - "text uploads, UTF-8 only" is a correct system, not a broken one - +// and MF5 says we do not invent that answer. +func outcomeFor(encoding string) string { + if encoding == textenc.UTF8 { + return "accept" + } + return "unspecified" +} + +func reasonFor(encoding string) string { + if encoding == textenc.UTF8 { + return "" + } + return "encoding_invalid" +} + +// lineEndingCells is the other half, and it is a separate half rather than a +// second axis of the first. +// +// No format in this build carries both settings, measured 2026-09-22 against +// the registry: an encoding belongs to md, txt and xml, a line ending to csv +// and log. The set says so out loud, because "where is the UTF-16 CSV" is the +// first question somebody reading it asks. +func lineEndingCells() []textFile { + var out []textFile + for _, desc := range carrying(lineEndingSetting) { + ending, _ := declared(desc, lineEndingSetting) + for _, name := range ending.Choices { + out = append(out, textFile{ + desc: desc, + group: lineEndingGroup, + props: map[string]string{lineEndingSetting: name}, + label: name, + // Both endings are legal in both formats and a reader that + // handles one has no excuse for the other, so this half of the + // set is stated rather than left open. + expected: "accept", + }) + } + } + return out +} + +// refusedOutright is why this format will not take these settings at any size, +// or empty when it will. +// +// The format answers rather than this file. Asked at the size the format itself +// names as its smallest for these settings, so a refusal that comes back is +// about the settings and not about the room they need. +func refusedOutright(desc format.Descriptor, props map[string]string) string { + r := format.Request{Label: true, Properties: props} + r.Bytes = desc.SmallestAccepted(r) + _, err := desc.Generator.Plan(r) + var bad *format.PropertyValueError + if errors.As(err, &bad) { + return bad.Reason + } + return "" +} + +// evenEnough refuses a size no file of this set could have. +// +// A whole file in UTF-16 has an even number of bytes, so an odd sample is not a +// file too small - it is a value that cannot be written, and the difference +// matters to the person reading the refusal. Left to the run it would arrive as +// a complaint about one file of twenty, with a floor a byte above what was +// asked for and advice to raise it. +// +// The codec answers, and it is asked without a mark so that the only thing left +// that can refuse is the width. The numbers in the message are its own. +func evenEnough(size int64) error { + for _, name := range encodingsInTheSet() { + codec, err := textenc.Parse(encodingID, map[string]string{textenc.Setting: name}) + if err != nil { + continue + } + var below *format.BelowMinimumError + if errors.As(codec.Check(encodingID, size), &below) { + return &ImpossibleError{ + Preset: encodingID, + Setting: sampleParam, + Detail: fmt.Sprintf("the set holds files in %s, and %s", name, below.Reason), + Hint: fmt.Sprintf("Set the {setting} to %d B or %d B.", + size-1, below.Minimum), + } + } + } + return nil +} + +// encodingsInTheSet is every encoding any format of this set is written in, +// once each and in the order the formats declare them. +func encodingsInTheSet() []string { + var out []string + seen := map[string]bool{} + for _, desc := range carrying(textenc.Setting) { + enc, _ := declared(desc, textenc.Setting) + out = appendUnseen(out, seen, enc.Choices) + } + return out +} + +func appendUnseen(out []string, seen map[string]bool, more []string) []string { + for _, name := range more { + if !seen[name] { + seen[name] = true + out = append(out, name) + } + } + return out +} + +// roomEnough refuses the whole set when any one file of it is out of reach. +// +// The same rule as the boundary set, and the same reason: a set missing four of +// its twenty files still looks like a set, and the four that are missing are +// whichever ones the reader was weakest at. The floor named is the highest of +// them, because raising the sample to the first one would only produce the next +// refusal. +func roomEnough(files []textFile, size int64) error { + var floor int64 + var tallest textFile + for _, f := range files { + r := format.Request{Label: true, Properties: f.props} + r.Bytes = size + if _, err := f.desc.Generator.Plan(r); err == nil { + continue + } + if need := f.desc.SmallestAccepted(r); need > floor { + floor, tallest = need, f + } + } + if floor == 0 { + return nil + } + return &ImpossibleError{ + Preset: encodingID, + Setting: sampleParam, + Detail: fmt.Sprintf("%s written as %s cannot be smaller than %d B, and every file of this set is the same size", + strings.ToUpper(tallest.desc.ID), tallest.label, floor), + Hint: fmt.Sprintf("Set the {setting} to %d B or more.", floor), + } +} + +// saidAboutTheEncodingSet names what the set does not hold. +// +// Two silences, and untouchable rule 6 is about both. The combinations this +// build refuses are left out, so a set of sixteen arrives where eighteen were +// described. And the two halves look like one grid and are not - somebody +// reading the file list will look for the UTF-16 CSV that no format in this +// build can produce. +func saidAboutTheEncodingSet(Args) []string { + var out []string + cells, left := encodingCells() + if len(left) > 0 { + out = append(out, fmt.Sprintf( + "this build refuses %s of the encoding set, so it holds %s rather than %d. Left out: %s.", + core.Count(len(left), "combination", "combinations"), + core.Count(len(cells), "file", "files"), + len(cells)+len(left), strings.Join(left, ", "))) + } + if both := carrying(textenc.Setting, lineEndingSetting); len(both) == 0 { + out = append(out, fmt.Sprintf( + "no format in this build carries an encoding and a line ending at once, so the two halves of this set are separate files rather than one grid. Encodings: %s. Line endings: %s.", + strings.Join(idsOf(carrying(textenc.Setting)), ", "), + strings.Join(idsOf(carrying(lineEndingSetting)), ", "))) + } + return out +} + +func idsOf(descs []format.Descriptor) []string { + out := make([]string, 0, len(descs)) + for _, d := range descs { + out = append(out, d.ID) + } + return out +} + +func expandTextEncoding(args Args) ([]byte, error) { + size, err := core.ParseSize(args[sampleParam]) + if err != nil { + return nil, fmt.Errorf("%s: %w", sampleParam, err) + } + // Before the files are laid out, because this refusal is about the value + // somebody typed and the one below it is about the set that value asks for. + if err := evenEnough(size); err != nil { + return nil, err + } + + cells, _ := encodingCells() + files := append(cells, lineEndingCells()...) + if len(files) == 0 { + return nil, &ImpossibleError{ + Preset: encodingID, + Detail: "no format in this build carries an encoding or a line ending, so there is no set to build", + Hint: "Run \"tfg formats\" to see what this build has.", + } + } + if err := roomEnough(files, size); err != nil { + return nil, err + } + + targets := make([]recipe.TargetDraft, 0, len(files)) + for _, f := range files { + targets = append(targets, f.draft(size)) + } + return plan{preset: encodingID, question: encodingQuestion, targets: targets}.source() +} diff --git a/internal/preset/uploadset.go b/internal/preset/uploadset.go new file mode 100644 index 00000000..4dc37e7b --- /dev/null +++ b/internal/preset/uploadset.go @@ -0,0 +1,561 @@ +package preset + +import ( + "errors" + "fmt" + "sort" + "strconv" + "strings" + + "github.com/donislawdev/TestingFilesGenerator/internal/core" + "github.com/donislawdev/TestingFilesGenerator/internal/format" +) + +// The set upload-validation lays out, and the values it is settled on. +// +// Apart from the file beside it, which is what the preset ANNOUNCES - its +// question, its parameters and the sentences it says out loud about what the +// set came out as. This is how the eight groups of files are built and which of +// them a parameter can empty. The two were one file of 498 lines of code on +// 2026-09-22 and the ceiling is 408, so the split follows what the parts do +// rather than where the count happened to fall. + +// allowList and denyList are the two lists this preset takes. They are not the +// same kind of value, which is the decision of 2026-09-22 written down. +// +// An allowed type has to become a real file of that type, so its values are +// formats the registry has. A denied one is about the EXTENSION: the measured +// set turns away .exe and .sh, and this build has no format called either - +// installer.exe would be a ZIP in any real system and deploy.sh is text. So a +// denied entry is a format id where the registry knows one and a bare extension +// otherwise, and the file it produces says out loud what is inside it. +var allowList = commaList{ + preset: uploadID, + param: allowParam, + empty: "no types were allowed, so the set has no positive control and no file to name wrongly", + check: knownFormat, + keep: lower, + duplicate: repeatedAllowed, +} + +var denyList = commaList{ + preset: uploadID, + param: denyParam, + empty: "no extensions were denied, so there is nothing for the form to turn away", + check: checkExtension, + keep: lower, + duplicate: repeatedDenied, +} + +func repeatedAllowed(first string) string { + return fmt.Sprintf( + "it is the same type as %q and the set would hold that file twice. Every allowed type appears once, because each one stands for one path through your form", + first) +} + +func repeatedDenied(first string) string { + return fmt.Sprintf( + "it is the same extension as %q and the set would hold that file twice. Every denied extension appears once, because each one stands for one rule your form has", + first) +} + +// checkExtension answers why a piece of the deny list is not an extension. +// +// An extension rather than a format, because this list is about the name. The +// value reaches a file name, so it is made of what a name is made of and +// nothing else - the same defence the boundary set's distances get, and for the +// same reason rather than by analogy. +func checkExtension(item string) string { + if strings.HasPrefix(item, ".") { + return fmt.Sprintf("an extension is written without its dot, so %q rather than %q", + strings.TrimPrefix(item, "."), item) + } + if len(item) > longestExtension { + return fmt.Sprintf("it is %d characters long and an extension here is at most %d", + len(item), longestExtension) + } + if bad := firstNotAlphanumeric(item); bad != "" { + return fmt.Sprintf( + "it holds %s, and an extension is written with letters and digits - such as exe, sh or svg. Its text becomes the end of a file name", bad) + } + return "" +} + +// firstNotAlphanumeric names the first character that cannot appear in an +// extension, quoted so a space or a control character is visible in the message. +func firstNotAlphanumeric(item string) string { + for _, r := range item { + switch { + case r >= '0' && r <= '9', r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z': + default: + return fmt.Sprintf("%q", r) + } + } + return "" +} + +// deniedEntry is one extension the form is meant to turn away, and what this +// build can put inside a file of that name. +type deniedEntry struct { + ext string + // desc writes the bytes. It is the format of that name where the registry + // has one, and the filler otherwise. + desc format.Descriptor + // known says the file really is what its name claims. Where it is false the + // run says so out loud, because a test of a content sniffer would otherwise + // pass against a file that never held what it claimed to. + known bool +} + +// extension is what a file of this entry is named with, dot included. +// +// A format the registry has answers for itself, because an extension is not +// always a dot and an id: targz is written .tar.gz, and denied.targz would be +// a file no upload form has a rule about - which is the one thing this group +// exists to test. +func (e deniedEntry) extension() string { + if e.known { + return e.desc.Extension + } + return "." + e.ext +} + +// uploadSet is one survey settled on its parameters. +type uploadSet struct { + limit int64 + limitText string + allowed []format.Descriptor + denied []deniedEntry + // farOver is how many times the limit the one big file is, or nought when + // that file was turned off. + farOver int64 + bulk int + filler format.Descriptor +} + +func settleUpload(args Args) (uploadSet, error) { + var s uploadSet + limit, err := core.ParseSize(args[uploadLimitParam]) + if err != nil { + return s, fmt.Errorf("%s: %w", uploadLimitParam, err) + } + // The limit leads the names of the files built around it, so its text + // reaches a file name and gets the check a distance gets. + s.limitText = strings.TrimSpace(args[uploadLimitParam]) + if bad := firstUnusable(s.limitText); bad != "" { + return s, fmt.Errorf( + "%s: it holds %s, and a limit is written with digits, letters and a dot - "+ + "such as 10mb, 512 or 1.5gb. Its text becomes part of every file name", + uploadLimitParam, bad) + } + s.limit = limit + + if s.allowed, err = allowedFormats(args[allowParam]); err != nil { + return s, err + } + if s.denied, err = deniedExtensions(args[denyParam]); err != nil { + return s, err + } + if err := listsAgree(s.allowed, s.denied); err != nil { + return s, err + } + if s.farOver, err = farOverTimes(args[farOverParam]); err != nil { + return s, err + } + if s.bulk, err = strconv.Atoi(args[bulkParam]); err != nil { + return s, fmt.Errorf("%s: %q is not a whole number of files", bulkParam, args[bulkParam]) + } + s.filler, err = format.Get(fillerFormat) + return s, err +} + +// listsAgree refuses an extension that is allowed and denied at once. +// +// Neither list can see the other, so "--allow pdf --deny pdf" laid a set out +// holding allowed_pdf.pdf expecting accept and denied.pdf expecting reject for +// extension_rule. Both expectations reach the manifest, so any suite running +// that set contradicts itself whatever the system under test does - and the run +// said nothing, which is untouchable rule 6 on the worst kind of silence: the +// one where nothing fails. +// +// The whole set is refused rather than one half dropped, because dropping a +// half is choosing for somebody which of the two they meant. Found by review on +// 2026-09-22, not by a guard, and there is one now. +func listsAgree(allowed []format.Descriptor, denied []deniedEntry) error { + turned := make(map[string]string, len(denied)) + for _, entry := range denied { + turned[entry.extension()] = entry.ext + } + for _, desc := range allowed { + written, both := turned[desc.Extension] + if !both { + continue + } + return &ImpossibleError{ + Preset: uploadID, Setting: denyParam, + Detail: fmt.Sprintf( + "%s is allowed and %s is denied, and both name a file ending %s - so the set would hold one of them to be taken and one to be turned away", + desc.ID, written, desc.Extension), + Hint: fmt.Sprintf("Take %s out of the %s list, or %s out of the %s list.", + desc.ID, allowParam, written, denyParam), + } + } + return nil +} + +// allowedFormats is the allow list in registry order. +// +// Registry order rather than the order somebody typed, for the reason the +// minimal set found the hard way on 2026-09-22: walking the typing makes +// "--allow jpg,png" and "--allow png,jpg" two different recipes, two different +// recipe hashes and two file lists nobody can compare. +func allowedFormats(raw string) ([]format.Descriptor, error) { + ids, err := allowList.parse(raw) + if err != nil { + return nil, err + } + wanted := map[string]bool{} + for _, id := range ids { + wanted[id] = true + } + var out []format.Descriptor + for _, id := range format.IDs() { + if desc, err := format.Get(id); err == nil && wanted[id] { + out = append(out, desc) + } + } + return out, nil +} + +// deniedExtensions is the deny list in alphabetical order. +// +// Alphabetical rather than the registry's, because half these entries are not +// in the registry and an order that covers only half of a list is not an order. +func deniedExtensions(raw string) ([]deniedEntry, error) { + items, err := denyList.parse(raw) + if err != nil { + return nil, err + } + sort.Strings(items) + filler, err := format.Get(fillerFormat) + if err != nil { + return nil, err + } + out := make([]deniedEntry, 0, len(items)) + for _, item := range items { + entry := deniedEntry{ext: item, desc: filler} + if desc, err := format.Get(item); err == nil { + entry.desc, entry.known = desc, true + } + out = append(out, entry) + } + return out, nil +} + +// farOverTimes reads the multiplier, or nought for the file turned off. +func farOverTimes(raw string) (int64, error) { + if raw == farOverOff { + return 0, nil + } + times, err := strconv.ParseInt(strings.TrimSuffix(raw, "x"), 10, 64) + if err != nil || times < 2 { + return 0, &format.PropertyValueError{ + Format: uploadID, Key: farOverParam, Value: raw, + Reason: "it has to be a number of times the limit, written with an x - 2x or 10x - or off", + } + } + return times, nil +} + +func expandUploadValidation(args Args) ([]byte, error) { + s, err := settleUpload(args) + if err != nil { + return nil, err + } + + // The three files around the limit come from the same code the boundary set + // uses, so "under the limit is accepted, over it is refused for size_limit" + // exists once rather than in two presets that could drift. + around := limitSet{ + preset: uploadID, setting: uploadLimitParam, group: sizeGroup, + desc: s.allowed[0], limit: s.limit, limitText: s.limitText, + spread: []offset{{text: "1b", bytes: 1}}, + } + steps := around.steps() + if err := around.reachable(steps); err != nil { + return nil, err + } + + files := s.files() + if err := s.reachable(files); err != nil { + return nil, err + } + targets := append(around.drafts(steps), draftsOf(files)...) + return plan{preset: uploadID, question: uploadQuestion, targets: targets}.source() +} + +// files is every file of the set except the three around the limit. +func (s uploadSet) files() []setFile { + var out []setFile + out = append(out, s.farOverFiles()...) + out = append(out, s.degenerateFiles()...) + out = append(out, s.allowedFiles()...) + out = append(out, s.deniedFiles()...) + out = append(out, s.mismatchFiles()...) + out = append(out, s.anomalyFiles()...) + out = append(out, s.nameFiles()...) + out = append(out, s.bulkFiles()...) + return out +} + +// reachable refuses the whole set when any one file of it is out of reach. +// +// Only the files measured from the limit can be, because everything else is +// asked for at the sample size or at the format's own floor, whichever is +// larger. So the refusal always has the same way out and can name the limit +// that would work: the floor, scaled back up by the share of the limit this +// file is. +func (s uploadSet) reachable(files []setFile) error { + var worst shortfall + for _, f := range files { + short, err := s.shortfallOf(f) + if err != nil { + return err + } + // The deepest shortfall rather than the first, so the limit the + // refusal names fixes every file at once. Naming the first would send + // somebody back for the next refusal, and RC7 says a file fixed one + // error per run is the cheapest way to make them stop using the tool. + if short.need > worst.need { + worst = short + } + } + if worst.need == 0 { + return nil + } + return s.cannotReach(worst) +} + +// shortfall is one file that is smaller than its format will write, and the +// limit at which it would stop being. +type shortfall struct { + file setFile + floor int64 + need int64 +} + +// shortfallOf measures one file. The error beside it is a refusal raising the +// limit would not fix, which goes straight back rather than being weighed. +func (s uploadSet) shortfallOf(f setFile) (shortfall, error) { + err := f.refused() + if err == nil { + return shortfall{}, nil + } + var below *format.BelowMinimumError + if !errors.As(err, &below) { + return shortfall{}, &ImpossibleError{ + Preset: uploadID, Setting: uploadLimitParam, + Detail: strings.TrimPrefix(err.Error(), f.desc.ID+": "), + } + } + return shortfall{ + file: f, floor: below.Minimum, + need: wouldReach(below.Minimum, f.bytes(), s.limit), + }, nil +} + +func (s uploadSet) cannotReach(short shortfall) error { + return &ImpossibleError{ + Preset: uploadID, Setting: uploadLimitParam, + // The same sentence the boundary set writes for the same situation. + // Splicing the format's own reason in instead read as a contradiction: + // it ends "already needs that much", and after "would be 1024 B" the + // words pointed at the wrong number. + Detail: fmt.Sprintf("%s would be %d B and the smallest %s this build makes is %d B", + short.file.id, short.file.bytes(), strings.ToUpper(short.file.desc.ID), short.floor), + Hint: fmt.Sprintf( + "Raise the {setting} to %d B or more, or take %s out of the allowed types. The {setting} asked for was %d B.", + short.need, short.file.desc.ID, s.limit), + } +} + +// wouldReach is the limit at which a file this far below its floor would reach +// it, given that the file is a fixed share of the limit. Rounded up, because a +// limit that lands a byte short is advice that fails when it is followed. +func wouldReach(floor, size, limit int64) int64 { + if size <= 0 { + return limit + } + return (floor*limit + size - 1) / size +} + +// sampleFor is how big a file that is about its name or its insides should be: +// the sample, or the format's own floor where that is larger. +// +// The larger of the two rather than the sample, so that a format with a floor +// above it cannot turn a file about a NAME into a refusal about a size. +func sampleFor(desc format.Descriptor) int64 { + if floor := desc.SmallestAccepted(format.Request{Label: true}); floor > uploadSample { + return floor + } + return uploadSample +} + +// farOverFiles is the one file well past the limit, or none when it was turned +// off. +func (s uploadSet) farOverFiles() []setFile { + if s.farOver == 0 { + return nil + } + times := strconv.FormatInt(s.farOver, 10) + "x" + desc := s.allowed[0] + return []setFile{{ + id: "far_over_" + times, group: sizeGroup, desc: desc, + name: s.limitText + "_far_over_" + times + desc.Extension, + size: s.limit * s.farOver, + expected: "reject", reason: "size_limit", + }} +} + +// degenerateFiles is the upload of nothing at all. +// +// Written by the filler rather than by an allowed format, because a file of +// nought bytes is bytes of no kind and most formats have no such thing. The +// name still carries an allowed extension, which is what makes it the case a +// form really meets: somebody pressed upload on an empty file. +func (s uploadSet) degenerateFiles() []setFile { + return []setFile{{ + id: "empty", group: degenerateGroup, desc: s.filler, + name: "empty" + s.allowed[0].Extension, size: 0, + // Legal, and what a form should do with it is its own decision - + // storage keeps it, an upload form usually turns it away, and both are + // defensible. MF5. + expected: "unspecified", reason: "size_zero", + }} +} + +// allowedFiles is one real file of every allowed type, comfortably inside the +// limit. If these fail, every refusal the rest of the set reports means nothing. +func (s uploadSet) allowedFiles() []setFile { + out := make([]setFile, 0, len(s.allowed)) + for _, desc := range s.allowed { + out = append(out, setFile{ + id: "allowed_" + desc.ID, group: allowedGroup, desc: desc, + name: "allowed_" + desc.ID + desc.Extension, + size: s.limit / halfShare, expected: "accept", + }) + } + return out +} + +// deniedFiles is one file per denied extension. +func (s uploadSet) deniedFiles() []setFile { + out := make([]setFile, 0, len(s.denied)) + for _, entry := range s.denied { + out = append(out, setFile{ + id: "denied_" + entry.ext, group: deniedGroup, desc: entry.desc, + name: "denied" + entry.extension(), size: sampleFor(entry.desc), + expected: "reject", reason: "extension_rule", + }) + } + return out +} + +// mismatchFiles is every allowed type under the name of the next one. +// +// Built from the allow list rather than from three names written here, so the +// group is about the types the reader said it takes. Each file is something the +// form accepts, wearing the extension of something else it accepts - which is +// the shape that gets past a check made on the name. +// +// It needs two allowed types and there is nothing to say when there is one. +func (s uploadSet) mismatchFiles() []setFile { + if len(s.allowed) < 2 { + return nil + } + out := make([]setFile, 0, len(s.allowed)) + for i, named := range s.allowed { + inside := s.allowed[(i+1)%len(s.allowed)] + out = append(out, setFile{ + id: "mismatch_" + named.ID, group: mismatchGroup, desc: inside, + name: inside.ID + "_as_" + named.ID + named.Extension, + size: sampleFor(inside), expected: "reject", reason: "mime_mismatch", + }) + } + return out +} + +// anomalyFiles is three names that are not quite an extension. +func (s uploadSet) anomalyFiles() []setFile { + desc := s.allowed[0] + size := sampleFor(desc) + out := []setFile{ + { + id: "no_extension", group: anomalyGroup, desc: desc, + name: "invoice", size: size, + // Whether a form insists on an extension is its own rule, and both + // answers are defensible. + expected: "unspecified", reason: "extension_rule", + }, + { + id: "uppercase_extension", group: anomalyGroup, desc: desc, + name: "PHOTO" + strings.ToUpper(desc.Extension), size: size, + expected: "unspecified", reason: "extension_rule", + }, + } + // The double extension needs something denied to end with. The deny list + // cannot be empty - the parser refuses one that names nothing - so this is + // a guard against a list that arrived some other way rather than a case. + if len(s.denied) > 0 { + out = append(out, setFile{ + id: "double_extension", group: anomalyGroup, desc: desc, + name: "invoice" + desc.Extension + s.denied[0].extension(), size: size, + expected: "reject", reason: "extension_rule", + }) + } + return out +} + +// nameFiles is three names a form has to write to disk, or refuse for a reason +// it can say out loud. +// +// The names are what they are on purpose: a long one for a column that is 255 +// characters wide, one outside ASCII for a path handled as bytes, and one with +// spaces and brackets that is perfectly legal and still breaks a script that +// did not quote it. +func (s uploadSet) nameFiles() []setFile { + desc := s.allowed[0] + size := sampleFor(desc) + return []setFile{ + { + id: "long_name", group: nameGroup, desc: desc, + name: strings.Repeat("a", 204) + desc.Extension, size: size, + expected: "unspecified", reason: "filename_too_long", + }, + { + id: "outside_ascii", group: nameGroup, desc: desc, + name: "wiadomość_日本語_🎉" + desc.Extension, size: size, + expected: "unspecified", reason: "filename_invalid", + }, + { + id: "spaces_and_brackets", group: nameGroup, desc: desc, + name: "my report (final) v2" + desc.Extension, size: size, + // A legal name on every filesystem this runs on, so this one is a + // promise rather than a question. + expected: "accept", + }, + } +} + +// bulkFiles is the mass upload, or none when it was turned off. +func (s uploadSet) bulkFiles() []setFile { + if s.bulk == 0 { + return nil + } + desc := s.allowed[0] + return []setFile{{ + id: "bulk", group: bulkGroup, desc: desc, count: s.bulk, + name: "bulk_{index:04}" + desc.Extension, + size: s.limit / bulkShare, expected: "accept", + }} +} diff --git a/internal/preset/uploadvalidation.go b/internal/preset/uploadvalidation.go new file mode 100644 index 00000000..154025c7 --- /dev/null +++ b/internal/preset/uploadvalidation.go @@ -0,0 +1,213 @@ +package preset + +import ( + "fmt" + "strings" + + "github.com/donislawdev/TestingFilesGenerator/internal/core" + "github.com/donislawdev/TestingFilesGenerator/internal/format" +) + +const ( + uploadID = "upload-validation" + + // limit is spelled the same as the boundary set's, and that is the point + // rather than a collision. Two presets asking for the number a system + // declares as its limit have to ask for it in the same word, or somebody + // learns one and types the other. The shared namespace is what makes + // "--preset upload-validation --limit 5mb" read as one sentence, and + // Declaring names every owner so the refusal beside a bare --limit can too. + uploadLimitParam = "limit" + allowParam = "allow" + denyParam = "deny" + farOverParam = "far-over" + bulkParam = "bulk" + + defaultUploadLimit = "10mb" + defaultAllow = "jpg,png,pdf" + defaultDeny = "svg,html,exe,sh" + // The owner's decision of 2026-09-22. Ten times the limit is the case worth + // having - a form that reads the whole body into memory before it looks at + // the size dies on it - but at the placeholder limit that one file is 100 MB + // of the 205 MB an untouched run would write. Twice the limit asks the same + // question of a form that checks after reading, and somebody who wants the + // heavier case types --far-over 10x. + defaultFarOver = "2x" + farOverOff = "off" + defaultBulk = "50" + // A mass upload of ten thousand files is already a thousand times the limit + // in bytes and far past what any form takes at once. The ceiling is here so + // that a slip of the keyboard is refused by the declaration rather than + // discovered as a full disk. + mostBulk = 10000 + + // The fractions of the limit the set is built from. Half the limit is + // comfortably inside it for the positive control, and a tenth of it is a + // file somebody would really upload fifty of at once. + halfShare = 2 + bulkShare = 10 + + // uploadSample is how big a file that is about its NAME or what is INSIDE + // it should be. Those files say nothing about size, so they are as small as + // they can be without being about smallness too. + uploadSample = 4 << 10 + + // longestExtension is what this preset will take as one. Nothing on any + // filesystem here is near it - it is a bound rather than a rule, so that a + // pasted sentence is refused as a sentence instead of becoming a file name. + longestExtension = 16 + + sizeGroup = "size-limit" + degenerateGroup = "degenerate" + allowedGroup = "allowed-types" + deniedGroup = "denied-types" + mismatchGroup = "extension-content-mismatch" + anomalyGroup = "extension-anomalies" + nameGroup = "hostile-names" + bulkGroup = "bulk-upload" + + // fillerFormat is what a file holds when the set needs bytes of no + // particular kind: an empty file, or a name whose extension the registry + // has never heard of. + fillerFormat = "txt" + + // uploadQuestion is announced by the preset AND written into the header of + // an ejected recipe. Named once rather than typed twice. + uploadQuestion = "Does my upload form take what it should and turn the rest away?" +) + +func init() { + Register(Preset{ + ID: uploadID, + Title: "Upload validation", + Question: uploadQuestion, + + Parameters: []format.Property{ + { + Name: uploadLimitParam, Kind: format.PropertySize, + Default: defaultUploadLimit, + Detail: "The size limit your upload form declares. This set takes one step either " + + "side of it - for a file at every distance, run the size-boundaries preset.", + }, + { + Name: allowParam, Kind: format.PropertyText, + Shape: "format ids separated by commas", + Default: defaultAllow, + Detail: "Which types your form is supposed to accept. Each one becomes a real file " + + "of that type, and they are the positive control of the whole set.", + }, + { + Name: denyParam, Kind: format.PropertyText, + Shape: "extensions separated by commas", + Default: defaultDeny, + Detail: "Which extensions your form is supposed to turn away. An extension this build " + + "has no format for still gets a file under that name, holding plain text.", + }, + { + Name: farOverParam, Kind: format.PropertyChoice, + Choices: []string{"10x", "2x", farOverOff}, + Default: defaultFarOver, + Detail: "How far past the limit the one big file goes. Turn it off where writing " + + "several times the limit is not worth the disk.", + }, + { + Name: bulkParam, Kind: format.PropertyInt, + Min: 0, Max: mostBulk, Unit: "files", + Default: defaultBulk, + Detail: "How many files the mass upload holds. Nought leaves that group out " + + "of the set altogether.", + }, + }, + + SaidWhenDefaulted: map[string]string{ + uploadLimitParam: "no limit was given, so this set is built around " + defaultUploadLimit + + " - that is our placeholder and not your form's limit. Pass the limit your form declares, or the files say nothing about it.", + }, + + Requires: []string{"MVP"}, + Catches: []string{ + "a limit enforced in the browser and not on the server", + "an SVG or an HTML file taken for a picture or for plain text, which is a way to get a script past a form", + "a file checked by its extension and never opened, so a PDF named .jpg goes through", + "a form that reads the whole body into memory before it looks at how big it is", + "an upload named PHOTO.JPG turned away where photo.jpg is taken, or the other way round", + "a name with spaces, brackets or characters outside ASCII written to disk unchanged", + }, + + Says: saidAboutTheUploadSet, + Expand: expandUploadValidation, + }) +} + +// saidAboutTheUploadSet names what this set is not, given what it was asked for. +// +// Three silences, and untouchable rule 6 is about all of them. A group that a +// parameter emptied looks exactly like a group that was forgotten. And a file +// whose inside is a stand-in reads as a real one: somebody testing a content +// sniffer against denied.exe would be testing it against plain text and would +// never find out from the file. +func saidAboutTheUploadSet(args Args) []string { + s, err := settleUpload(args) + if err != nil { + // Expand is about to refuse these same values with a message that names + // the one that is wrong. A sentence here would be a second opinion. + return nil + } + var out []string + if stood := standIns(s.denied); len(stood) > 0 { + out = append(out, fmt.Sprintf( + "this build has no format called %s, so %s %s plain text under %s. That tests a form reading the end of a name, not one reading what is inside.", + joinWithOr(stood), joinWithAnd(namesOf(stood)), + core.Noun(len(stood), "holds", "hold"), + core.Noun(len(stood), "that name", "those names"))) + } + if s.farOver == 0 { + out = append(out, "far-over is off, so nothing in this set is well past the limit. The largest file is one byte over it.") + } + if s.bulk == 0 { + out = append(out, "bulk is nought, so this set holds no mass upload.") + } + if len(s.allowed) < 2 { + out = append(out, "only one type is allowed, so the set holds no file named as one allowed type and filled with another - that needs two.") + } + return out +} + +// standIns is the denied extensions this build has no format for. +func standIns(denied []deniedEntry) []string { + var out []string + for _, entry := range denied { + if !entry.known { + out = append(out, entry.ext) + } + } + return out +} + +// joinWithAnd and joinWithOr write a list the way a sentence takes one. +// +// A comma between every pair is how a machine writes a list and it reads as an +// enumeration rather than as a sentence: "denied.exe, denied.sh holds plain +// text" has no number to agree with. The text rules in CLAUDE.md ask for a +// sentence, so the last pair gets its conjunction. +func joinWithAnd(items []string) string { return joinWith(items, "and") } + +func joinWithOr(items []string) string { return joinWith(items, "or") } + +func joinWith(items []string, conjunction string) string { + switch len(items) { + case 0: + return "" + case 1: + return items[0] + } + return strings.Join(items[:len(items)-1], ", ") + " " + conjunction + " " + items[len(items)-1] +} + +func namesOf(extensions []string) []string { + out := make([]string, 0, len(extensions)) + for _, ext := range extensions { + out = append(out, "denied."+ext) + } + return out +} diff --git a/web/content/en/site.json b/web/content/en/site.json index 432ca61c..af68da73 100644 --- a/web/content/en/site.json +++ b/web/content/en/site.json @@ -96,7 +96,10 @@ }, "presets": { "empty-and-minimal": "Does a file that is valid and as small as the format allows get through?", - "size-boundaries": "Is a size limit enforced exactly where it is declared?" + "size-boundaries": "Is a size limit enforced exactly where it is declared?", + "tabular-import": "Does my table import survive what real tools export?", + "text-encoding": "Does my reader know which encoding a file is in, or is it guessing?", + "upload-validation": "Does my upload form take what it should and turn the rest away?" }, "commands": { "generate": "produce files, from a recipe or from flags", diff --git a/web/content/pl/site.json b/web/content/pl/site.json index 9bd1fc00..bbd50cd4 100644 --- a/web/content/pl/site.json +++ b/web/content/pl/site.json @@ -96,7 +96,10 @@ }, "presets": { "empty-and-minimal": "Czy plik poprawny i najmniejszy, na jaki format pozwala, przechodzi?", - "size-boundaries": "Czy limit rozmiaru działa dokładnie tam, gdzie jest zadeklarowany?" + "size-boundaries": "Czy limit rozmiaru działa dokładnie tam, gdzie jest zadeklarowany?", + "tabular-import": "Czy import tabeli poradzi sobie z tym, co eksportują prawdziwe narzędzia?", + "text-encoding": "Czy mój czytnik wie, w jakim kodowaniu jest plik, czy zgaduje?", + "upload-validation": "Czy mój formularz przesyłania plików przyjmuje to, co powinien, i odrzuca resztę?" }, "commands": { "generate": "tworzy pliki, z przepisu albo z flag", diff --git a/web/public/docs/index.html b/web/public/docs/index.html index 238dbe56..b5716b89 100644 --- a/web/public/docs/index.html +++ b/web/public/docs/index.html @@ -315,6 +315,18 @@

What is a preset?

size-boundaries

Is a size limit enforced exactly where it is declared?

+
  • +

    tabular-import

    +

    Does my table import survive what real tools export?

    +
    • +
  • +

    text-encoding

    +

    Does my reader know which encoding a file is in, or is it guessing?

    +
    • +
  • +

    upload-validation

    +

    Does my upload form take what it should and turn the rest away?

    + 
    • tfg preset list
 tfg preset show size-boundaries
diff --git a/web/public/pl/dokumentacja/index.html b/web/public/pl/dokumentacja/index.html
index 5b59a2e6..61922942 100644
--- a/web/public/pl/dokumentacja/index.html
+++ b/web/public/pl/dokumentacja/index.html
@@ -316,6 +316,18 @@ 
    Czym jest preset?
    
     
    size-boundaries
    
     
    Czy limit rozmiaru działa dokładnie tam, gdzie jest zadeklarowany?
    
   
+  
  • 
+    
    tabular-import
    
+    
    Czy import tabeli poradzi sobie z tym, co eksportują prawdziwe narzędzia?
    
+  
    • 
+  
  • 
+    
    text-encoding
    
+    
    Czy mój czytnik wie, w jakim kodowaniu jest plik, czy zgaduje?
    
+  
    • 
+  
  • 
+    
    upload-validation
    
+    
    Czy mój formularz przesyłania plików przyjmuje to, co powinien, i odrzuca resztę?
    
+  
    • 
 
   
    tfg preset list
 tfg preset show size-boundaries