Add pytest.ini support for only_rerun - #369
Conversation
|
Added coverage for multiple |
icemac
left a comment
icemac
left a comment
There was a problem hiding this comment.
Thanks for picking this up! Precedence (CLI beats ini), pyproject.toml support, -o only_rerun= and --help rendering all check out. A few remarks inline, mostly about the sibling option, tests and docs.
— Comment created by Claude
| RERUNS_DELAY_BACKOFF_FACTOR_DESC, | ||
| type=arg_type, | ||
| ) | ||
| parser.addini( |
There was a problem hiding this comment.
Only only_rerun gets an ini option; its documented sibling rerun_except stays CLI-only. With rerun_except = ValueError in pytest.ini the setting is silently ignored, the only signal being PytestConfigWarning: Unknown config option: rerun_except in the warnings summary. Since the docs introduce the two as a mirrored pair, adding parser.addini("rerun_except", ..., type="linelist") here would keep them in sync and also lets the fallback below drop its hardcoded name check.
— Comment created by Claude
| ) | ||
| parser.addini( | ||
| "only_rerun", | ||
| "only rerun errors matching the regex provided.", |
There was a problem hiding this comment.
The other three addini calls share their help text with the CLI option via the *_DESC constants (RERUNS_DESC etc.) so the two cannot drift. This one inlines a paraphrase that drops the accumulation sentence, so pytest --help gives no hint that the ini form takes one regex per line (which docs/configuration.rst does explain). Suggest hoisting an ONLY_RERUN_DESC next to the other constants and using it for both --only-rerun and this addini.
— Comment created by Claude
| type=arg_type, | ||
| ) | ||
| parser.addini( | ||
| "only_rerun", |
There was a problem hiding this comment.
Not new to this PR, but worth considering now that the pattern can live in a checked-in file: an invalid regex such as only_rerun = [unclosed produces INTERNALERROR> re.error: unterminated character set at position 0 with a traceback into re/_parser.py and nothing naming only_rerun or the config file. A CLI typo breaks the one command you typed; a committed ini typo breaks every run in CI. check_options already raises pytest.UsageError for other option problems and would be the natural place to re.compile each entry.
— Comment created by Claude
| else: | ||
| regex = getattr(item.session.config.option, regex_name) | ||
| if regex is None and regex_name == "only_rerun": | ||
| regex = item.session.config.getini("only_rerun") |
There was a problem hiding this comment.
_get_rerun_filter_regex is generic over regex_name (it is called with both "only_rerun" and "rerun_except") and reads the CLI value generically via getattr, but this fallback hardcodes the name and re-types the literal in getini(). It also makes the return type differ per name ([] for only_rerun, None for rerun_except). If both names are registered with addini, this can simply be:
if regex is None:
regex = item.session.config.getini(regex_name)— Comment created by Claude
| ) | ||
|
|
||
|
|
||
| def test_only_rerun_ini(testdir): |
There was a problem hiding this comment.
The central claim of the PR, that the CLI still overrides the ini, has no test. The file pairs test_ini_file_parameters with test_ini_file_parameters_override for the existing ini options; this new option only gets the plain half. I confirmed manually that ini only_rerun = AssertionError plus --only-rerun ValueError reruns only the ValueError test, but a refactor that swaps the lookup order (or uses getattr(...) or getini(...)) would invert precedence and the suite would stay green. Also worth a test: marker only_rerun vs ini, and ini only_rerun combined with --rerun-except, which takes the four-way branch in _should_hard_fail_on_error.
— Comment created by Claude
| ``only_rerun`` | ||
| ^^^^^^^^^^^^^^ | ||
|
|
||
| - **Description**: Sets regular expressions for errors that should be rerun. Add one expression per line. |
There was a problem hiding this comment.
Two behaviours of the new ini tier are not stated anywhere: (a) --only-rerun on the command line replaces the ini list rather than accumulating with it, even though the CLI help says "pass this flag multiple times to accumulate"; (b) a repo-wide only_rerun also applies to tests marked plain @pytest.mark.flaky(reruns=2), so a test raising a non-matching error stops rerunning with no warning. Both match existing --only-rerun semantics, but a checked-in ini key becomes the permanent project default. The Priority sections in docs/mark.rst and README.rst still describe precedence as marker-vs-CLI only and should mention the ini tier.
— Comment created by Claude
| [pytest] | ||
| reruns_delay = 2.5 | ||
|
|
||
| ``only_rerun`` |
There was a problem hiding this comment.
README.rst is also the PyPI long description (see readme in pyproject.toml), and it is not touched here: the "Re-run all failures matching certain expressions" section still presents --only-rerun as CLI-only and the "Priority" section shows the pytest.ini layer holding only reruns = 3. Leaving it stale reproduces the confusion that led to #165. Previous option additions (e.g. --reruns-delay-backoff-factor) updated README together with the docs.
— Comment created by Claude
| [pytest] | ||
| reruns = 3 | ||
| reruns_delay = 2.0 | ||
| only_rerun = AssertionError |
There was a problem hiding this comment.
The example gained a third key, but the lead-in above ("for consistent retries and delays") and the "This setup ensures that:" list below still cover only two. A third bullet would help, especially since only_rerun restricts reruns rather than adding behaviour.
— Comment created by Claude
3 participants
Fixes #165.
Add the missing pytest.ini configuration for --only-rerun while preserving command-line precedence. The setting accepts one regex per line, matching the existing repeatable CLI option.
Included a regression test, configuration documentation, and a towncrier feature entry.