fix(requestflag): ignore a leading UTF-8 BOM in YAML/JSON input

[thegoodengineer]

Summary

A request file that starts with a UTF-8 byte order mark is not read as an object when it is piped to the CLI. Windows tools write this BOM routinely, for example Set-Content -Encoding UTF8 in Windows PowerShell 5.1. The file looks correct in an editor, and the CLI either rejects it with a misleading error or sends the wrong request.

Problem

Piped stdin and JSON flag values are decoded by requestflag.UnmarshalYAMLOrJSON, which passes the bytes to go-yaml. go-yaml treats a leading BOM as content (goccy/go-yaml#906, still open). A JSON document therefore decodes as a single string, and a YAML document keeps the BOM attached to its first key.

These inputs were all written by Windows PowerShell 5.1 with Set-Content -Encoding UTF8 (each file starts with ef bb bf) and piped with <. Requests were captured with a local echo server, comparing main with this branch:

Command	main	This branch
responses create < body.json	Sends the entire file as a JSON string body, "\ufeff{\"model\":\"gpt-test\",...}", and exits 0	Sends {"input":"hi","model":"gpt-test"}
responses create --instructions "be brief" < body.json	Fails with Cannot merge flags with a body that is not a map: {"model":"gpt-test","input":"hi"}	Sends the merged body
responses create < body.yaml	Sends "\ufeffmodel":"gpt-test", so the request has no model	Sends "model":"gpt-test"
responses retrieve < get.json containing {"response_id":"resp_123"}	Fails with Required flag "response-id" not set	Requests /responses/resp_123

Because the BOM is invisible, the last three are hard to diagnose: the error prints a map while saying it is not a map, and the "missing" flag is right there in the file.

Fix

Strip one leading UTF-8 BOM in UnmarshalYAMLOrJSON before decoding. YAML allows a BOM at the start of a stream, and RFC 8259 section 8.1 lets JSON parsers ignore one. Both decode sites have gone through this helper since #195, so stdin and JSON flag values are both covered.

Scope:

• Only a single leading BOM is removed. U+FEFF anywhere else, including inside string values, is still treated as content.
• Only YAML/JSON decoding is affected. @file contents, file uploads, and binary bodies read from stdin are untouched.
• Input without a BOM reaches go-yaml unchanged. The JSON exponent handling from fix(requestflag): preserve JSON numbers written in exponent form #195 still applies once the BOM is removed.
• UTF-16 input, which is what Windows PowerShell 5.1 writes with > or Out-File by default, is not handled here. That needs transcoding rather than trimming, and RFC 8259 requires JSON to be UTF-8, so I left it for a separate change.

Tests

• internal/requestflag/unmarshal_test.go covers a BOM before a JSON object, a JSON array, a YAML mapping (the first key stays intact), and JSON containing an exponent number, so the two workarounds are shown to compose. It also covers a map flag value, and includes a guard that U+FEFF inside a string value is preserved.
• cmd/openai/main_stdinbom_test.go runs the real entrypoint with BOM-prefixed stdin in three cases: a JSON body merged with --instructions, a YAML body, and a JSON path parameter for responses retrieve. It checks the request path and body the server receives, using the runMainDispatchWithStdin helper added in fix(requestflag): preserve JSON numbers written in exponent form #195.

With the one-line fix disabled, all three end-to-end cases fail with the symptoms above:

JSON_body_merged_with_a_flag: main = {code:1 stdout: stderr:Cannot merge flags with a body that is not a map: {"model":"test-model","input":"test input"}
YAML_body: request body = map[string]interface {}{"input":"test input", "\ufeffmodel":"test-model"}, want map[string]interface {}{"input":"test input", "model":"test-model"}
JSON_path_parameter: main = {code:1 stdout: stderr:Required flag "response-id" not set

Every BOM unit case fails the same way, apart from the in-value guard, which passes either way.

Validation

Linux (WSL Ubuntu 22.04) with Go 1.25.14, on a clean clone of this commit, all passing:

• go mod verify and go mod tidy -diff
• go vet ./...
• ./scripts/lint
• go test ./... -run '^$'
• go test ./internal/... and go test ./cmd/openai
• go test ./pkg/cmd -run 'Stdin|EmbedFiles|IsUTF8|Multipart', the existing piped stdin, file embedding, and multipart tests
• go test -race ./internal/requestflag ./cmd/openai
• go test -race -count=100 -shuffle=on ./internal/requestflag -run 'TestUnmarshalYAMLOrJSON|TestJSONExponentNumbersInFlagValues'
• go test -race -count=50 ./cmd/openai -run '^(TestMainStdinByteOrderMark|TestMainJSONExponentNumbers)$'
• gofmt -l on the changed files

On the same clone, with internal/requestflag/unmarshal.go restored from main, the new end-to-end test fails in all three cases as shown above.

I did not run the full ./scripts/test suite against the Steady mock server; that is left to CI. The before and after table above was captured on Windows with binaries built from main and from this branch.