Skip to content

make the refusal of a filter that selects no case more readable and helpful - #119

Open
ev-br wants to merge 1 commit into
benchx-dev:mainfrom
ev-br:filter-refusal-hints
Open

ev-br wants to merge 1 commit into
benchx-dev:mainfrom
ev-br:filter-refusal-hints

Conversation

@ev-br

@ev-br ev-br commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

This is a small UI tweak, split off gh-118.

For parametrized benchmarks it's easy to mis-type a name, especially when selecting subsets of parameters. The resulting error on main is... unhelpful:

$ NUMPY=~/repos/numpy ./compare.sh -t "Partition\.time_argpartition\(\('sorted_block', 1000\), 'int16', 10\)" main

....

2.6.0.dev0+git20261026.6e1aca7 /home/ev-br/repos/numpy/build-6e1aca7eda-install/usr/lib/python3.12/site-packages/numpy/__init__.py
precision: 5 rounds x 3 repeat (config: /home/ev-br/proj/quansight/benchx_user/examples/numpy-revisions/benchx.conf.json)
refused: filter "Partition\\.time_argpartition\\(\\('sorted_block', 1000\\), 'int16', 10\\)" selects no case in /home/ev-br/repos/numpy/benchmarks/benchmarks; available: bench_alloc_cache.SmallArrayCreation.time_empty_loop, bench_alloc_cache.SmallArrayCreation.time_full_loop, bench_alloc_cache.SmallArrayCreation.time_ones_loop, bench_alloc_cache.SmallArrayCreation.time_zeros_loop, bench_app.LaplaceInplace.time_it, bench_app.MaxesOfDots.time_it, bench_array_coercion.ArrayCoercionSmall.time_array, bench_array_coercion.ArrayCoercionSmall.time_array_all_kwargs, bench_array_coercion.ArrayCoercionSmall.time_array_dtype_not_kwargs, bench_array_coercion.ArrayCoercionSmall.time_array_invalid_kwarg, … (481 more); the filter matches the name of a parametrized benchmark (bench_function_base.Partition.time_argpartition) but not its arguments, and those are searched as name(p0, p1), each as its repr: bench_function_base.Partition.time_argpartition(dtype, array_type, k) is searched as bench_function_base.Partition.time_argpartition('float64', ('random',), 10)

With this PR, the error message is a tad saner:

precision: 5 rounds x 3 repeat (config: /home/ev-br/proj/quansight/benchx_user/examples/numpy-revisions/benchx.conf.json)
refused: filter 'Partition\.time_argpartition\(\('sorted_block', 1000\), 'int16', 101\)' selects no case in /home/ev-br/repos/numpy/benchmarks/benchmarks
  the filter matches the name of a parametrized benchmark (bench_function_base.Partition.time_argpartition) but not its arguments.
  They are searched as name(p0, p1), each as its repr, in the order of param_names:
    bench_function_base.Partition.time_argpartition(dtype, array_type, k)
  for example:
    bench_function_base.Partition.time_argpartition('float64', ('random',), 10)
  similar: bench_function_base.Partition.time_argpartition, bench_function_base.Partition.time_partition

…elpful

A filter that named a parametrized benchmark but gave its arguments in the wrong
order was refused with the first ten benchmark names, on one long line, with the
filter shown as a Python string (`\.` as `\\.`).

The refusal is now one fact per line:

- the filter as typed;
- when the filter, up to its first `\(`, matches the name of a parametrized
  benchmark, how its cases are searched: `name(p0, p1)`, each as its repr, in the
  order of `param_names`, with the parameters of that benchmark and one case;
- the benchmarks similar to the filter (words of 3+ characters compared with the
  dotted parts of each name, difflib ratio >= 0.75, the names matching the most
  words first, 5 at most), else the first 5 available.

Only the text of the refusal changes: not what a filter selects, no schema, no new
option.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant