Skip to content

query: resolve all-map Bloblang field paths without gabs - #472

Open
slachiewicz wants to merge 3 commits into
redpanda-data:mainfrom
slachiewicz:bloblang-eval-perf
Open

query: resolve all-map Bloblang field paths without gabs#472
slachiewicz wants to merge 3 commits into
redpanda-data:mainfrom
slachiewicz:bloblang-eval-perf

Conversation

@slachiewicz

Copy link
Copy Markdown

Field reads in a Bloblang mapping go through gabs.Wrap(target).S(path...).Data(), which builds and discards a *gabs.Container per path segment. Allocations therefore scale with the number of field reads in a mapping rather than staying flat. Walking map[string]any directly removes that.

Arrays are deliberately left on gabs. Container.S calls searchStrict with allowWildcard=true, so a * segment over an array is real behaviour; resolveFieldPath hands the whole path back to gabs the moment it meets anything that is not a map[string]any, rather than reimplementing it. Equivalence against gabs is asserted directly in TestResolveFieldPathMatchesGabs over missing keys, nil intermediates, scalars mid-path, negative and out-of-range indices, wildcards on both arrays and maps, and numeric-looking map keys.

The first commit adds benchmarks because there were none for this path — only the parser and iterator had any, so Executor.Query and Overlay were unmeasured.

benchstat, 10 runs per side, darwin/arm64 (M1):

                        │  before     │              after               │
                        │  allocs/op  │  allocs/op   vs base             │
ExecutorQuery/field_read    5.000        4.000       -20.00% (p=0.000)
ExecutorQuery/predicate     5.000        4.000       -20.00% (p=0.000)
ExecutorQuery/and2          6.000        4.000       -33.33% (p=0.000)
ExecutorQuery/and3          8.000        4.000       -50.00% (p=0.000)
ExecutorQuery/contains     13.00        12.00         -7.69% (p=0.000)
ExecutorQuery/exists        6.000        5.000       -16.67% (p=0.000)
ExecutorQuery/multi_assign 12.000        9.000       -25.00% (p=0.000)
ExecutorOverlay             8.000        7.000       -12.50% (p=0.000)

                        │  sec/op     │  sec/op      vs base             │
ExecutorQuery/field_read   147.0n       137.6n       ~ (p=0.118)
ExecutorQuery/predicate    178.6n       159.8n       -10.55% (p=0.000)
ExecutorQuery/and2         257.4n       229.4n       -10.90% (p=0.000)
ExecutorQuery/and3         370.1n       317.9n       -14.09% (p=0.000)
ExecutorQuery/contains     363.2n       353.4n        -2.70% (p=0.041)
ExecutorQuery/exists       172.0n       159.3n        -7.38% (p=0.000)
ExecutorQuery/multi_assign 433.2n       386.2n       -10.85% (p=0.000)
ExecutorOverlay            256.6n       236.5n        -7.82% (p=0.000)
geomean                    257.8n       234.5n        -9.01%   (B/op -14.10%)

and3 is the shape that shows the point: 8 allocations to 4, and constant from there regardless of how many fields the mapping reads. field_read's latency delta is not statistically significant at n=10 even though its allocation count drops, so I have not claimed it.

Two allocation sites remain on this path and are untouched here: the Vars: map[string]any{} built per call in public/bloblang/executor.go even when a mapping declares no variables, and FunctionContext.WithValue taking any by value and storing &value, which forces a heap escape per call. Both look addressable; I left them out to keep this reviewable, and am happy to follow up if the approach here is welcome.

Verified: make test and golangci-lint run internal/bloblang/query/... public/bloblang/... both clean (the two pre-existing govet inline hints in methods_strings.go are untouched).


Disclosure: this change was developed with AI assistance (Claude Code) and reviewed by me before submission. I saw .github/ai-opt-out only after the work was done — I am raising the PR rather than discarding it because the measurements and the equivalence test stand on their own, but please close it without ceremony if AI-assisted contributions are not wanted here, and I will not follow up with the remaining two optimisations. No hard feelings either way.

I also skipped the issue-first step in CONTRIBUTING.md, which I should have followed; happy to move the discussion to an issue if you would prefer that.

Only the parser and iterator had benchmarks, leaving Executor.Query and
Overlay - the hot path for embedded compile-once/evaluate-many use - with
no in-tree measurement.
Field reads went through gabs.Wrap(target).S(path...).Data(), which builds and
discards a Container per segment, so allocations scaled with the number of field
reads in a mapping. Walking map[string]any directly makes them constant.

Arrays stay on gabs deliberately: its Search permits a "*" wildcard segment, and
resolveFieldPath delegates the whole path back rather than reimplement that.
@CLAassistant

CLAassistant commented Aug 10, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Query and Overlay allocated a variables map on every call, including for the
majority of mappings that never assign a variable. AssignsVariables reports
whether any statement targets one, and mappings that do not share a single
empty map.

Maps applied with the apply method are not consulted, because that method
already replaces the variables map before invoking the target - a named map
cannot write to its caller's variables. The shared map is empty rather than
nil so that reading an undefined variable keeps reporting "variable 'x'
undefined" rather than "variables were undefined".
@slachiewicz

Copy link
Copy Markdown
Author

Added the variables-map commit mentioned above, and closing out the third idea as a dead end.

Variables map. AssignsVariables walks the statements' existing AssignmentTargets for a TargetVariable; mappings without one share a single empty map. Two things make this narrower than it first looked: apply already replaces ctx.Vars before invoking a named map (the // ISOLATED VARIABLES line in methods.go), so named maps cannot write to their caller's variables and need no analysis; and the map has to be empty rather than nil, because a nil Vars reports variables were undefined where an empty one reports variable 'x' undefined. Tests cover both, plus a race-detector run and a case asserting a var-assigning mapping still gets a private map.

Cumulative against the branch point, allocations per Query:

mapping before after
this.a."b"."c" != "x" 5 3
3-term && over 4 field reads 8 3
several root.x = assignments 12 8
Overlay 8 6

B/op on the simple predicate goes 120 → 56.

The dead end, for anyone who tries it next: FunctionContext.WithValue takes any by value and stores &value, which looked like a free heap escape to remove. It is not. Adding a WithValuePtr(*any) and passing &val from Query moves the escape rather than removing it — -gcflags=-m then reports moved to heap: val, and the benchmark is unchanged to the nanosecond (3 allocs, 56 B/op both ways). Removing that one needs FunctionContext to hold the value directly instead of behind *any, which changes Value() and every caller — not worth it for 16 bytes, in my view, but flagging it so the option is on the record.

On wall-clock I am claiming nothing for this commit: this machine sat at load average 8 on 8 cores while I measured, and an interleaved A/B put the latency delta inside the noise (p=0.33–0.65 at ±22–40% variance). The allocation counts are exact and reproducible; the timing needs a quiet machine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants