Skip to content

bench: add five curated getsentry/sentry-python corpus cases - #152

Merged
SutuSebastian merged 3 commits into
mainfrom
agent/corpus-sentry-python
Nov 15, 2026
Merged

SutuSebastian merged 3 commits into
mainfrom
agent/corpus-sentry-python

Conversation

@SutuSebastian

Copy link
Copy Markdown
Contributor

Part of STA-325. Five real benchmark cases from getsentry/sentry-python (Python observability SDK), the last repo of the v1 corpus.

case PR kind bug grounding
sentry-python-4121-logging-mapping-args getsentry/sentry-python#4121 mapping-args-sent-as-keys "params": record.args becomes a generator, so a dict logged against a %(name)s template ships its keys, not its values revert e71ccbf19, which adds tests asserting the dict
sentry-python-4133-debug-parent-logger getsentry/sentry-python#4133 debug-output-hijacked logger.hasHandlers() replaces not logger.handlers, so once the app configures the root logger, the SDK's own debug handler is never installed revert 5689bc09f
sentry-python-4281-threading-channels-version getsentry/sentry-python#4281 shadowed-module-crashes-setup channels.__version__ is read inside a try that catches only ImportError, so a shadowing channels module raises AttributeError out of init() fix a35050694 (#5299)
sentry-python-6168-span-batcher-flush-thread getsentry/sentry-python#6168 flusher-thread-death SpanBatcher._flush_loop calls _flush unguarded, so one failed send ends the drain thread for the life of the process fix e17291eea; pre-existing, re-authored (disclosed)
sentry-python-8119-clean-baggage-trailing-comma getsentry/sentry-python#8119 control a one-line guard on third-party baggage items; no later commit touches it —

Evidence

  • full.diff equals git diff base head, and it also passes git apply --check at its own base.
  • Every ground-truth line is an added line.
  • The keywords are graded against the runtime-filtered vocabulary. All 26 are unspelled by their diff, and the two-sided probe is pinned in TestCheckedInCorpusKeywordsGradeBothWays. A mutation in each direction makes the probe fail.

Verification

  • make test-race, golangci-lint (native and GOOS=windows), ./evals/one.sh bench-corpus and make schemas-check all pass.
  • The no-spend smoke loads all five cases.
  • This PR merges after its cross-family Gemini audit.
  • Data and test rows only, so there is no changeset.

Four real bug cases and one clean control from the Python SDK, every PR
merged inside the 24mo window:

- 4121 replaces `"params": record.args` with a generator over record.args,
  so a mapping logged against a %(name)s template ships its keys instead of
  its values. Reverted by e71ccbf19, which states the regression and adds two
  tests asserting the params are the dict.
- 4133 swaps `if not logger.handlers:` for `logger.hasHandlers()`, so once
  the application configures the root logger the SDK never installs its own.
  Reverted by 5689bc09f.
- 4281 reads channels.__version__ inside a try that only catches ImportError,
  so a shadowing `channels` module raises AttributeError out of init(); fixed
  by a35050694 (#5299), which splits the block and widens the handler.
- 6168 writes SpanBatcher its own _flush_loop with two unguarded _flush
  calls, so one failed send ends the drain thread for the process lifetime;
  fixed by e17291eea. The pattern predates the PR in Batcher, disclosed in
  provenance.note.
- 8119 is the control: a one-line guard on the third-party baggage items,
  exact because an empty field means there was nothing to append.

Ground truth per case is a line the reviewed PR added and the later commit
deletes or rewrites. None of the four kinds is in the published table, so
each vocabulary is its kind plus keywords, every keyword unspelled by the
case's own diff. Behaviour was reproduced by extracting the statement from
both commits and running it in a harness (the SDK cannot be imported on this
host); the probes are pinned in TestCheckedInCorpusKeywordsGradeBothWays.

Corpus: 44 cases (8 seeded, 27 real, 9 control).
@changeset-bot

changeset-bot Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: fcef4f0

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 31 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 4c1c07ca-2cbd-42b9-9b20-ab5d98080891
Commits

Reviewing files that changed from the base of the PR and between df0abb6 and fcef4f0.

Files ignored due to path filters: 5

  • bench/corpus/sentry-python-4121-logging-mapping-args/hunks.tsv is excluded by !**/*.tsv
  • bench/corpus/sentry-python-4133-debug-parent-logger/hunks.tsv is excluded by !**/*.tsv
  • bench/corpus/sentry-python-4281-threading-channels-version/hunks.tsv is excluded by !**/*.tsv
  • bench/corpus/sentry-python-6168-span-batcher-flush-thread/hunks.tsv is excluded by !**/*.tsv
  • bench/corpus/sentry-python-8119-clean-baggage-trailing-comma/hunks.tsv is excluded by !**/*.tsv
Files selected for processing: 11

  • bench/corpus/sentry-python-4121-logging-mapping-args/case.json
  • bench/corpus/sentry-python-4121-logging-mapping-args/full.diff
  • bench/corpus/sentry-python-4133-debug-parent-logger/case.json
  • bench/corpus/sentry-python-4133-debug-parent-logger/full.diff
  • bench/corpus/sentry-python-4281-threading-channels-version/case.json
  • bench/corpus/sentry-python-4281-threading-channels-version/full.diff
  • bench/corpus/sentry-python-6168-span-batcher-flush-thread/case.json
  • bench/corpus/sentry-python-6168-span-batcher-flush-thread/full.diff
  • bench/corpus/sentry-python-8119-clean-baggage-trailing-comma/case.json
  • bench/corpus/sentry-python-8119-clean-baggage-trailing-comma/full.diff
  • internal/core/corpus/keywords_test.go
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

The audit read the control clean and accepted the revert grounding, 6168's
pre-existing disclosure and 4281's trigger, but found two defects in the
vocabulary that the case-level probe could not see.

Three keywords described intent, not a failure, so a neutral or approving
comment graded as a hit: 4133's "any existing root setup stops the library from
installing its own output" and "a configured root makes the library skip its own
setup", and 4281's "the version read is assumed to exist on any module of that
name". They are gone, and the audit's sentences carrying them are now
must-grade-false rows, so re-adding any of the three reproduces the failure.

Every keyword was a 7-14 word clause, so a reviewer who words the same defect
differently graded as a miss. Each bug case gains 2-3 short directional phrases
instead — "keys instead of values", "debug=true fails", "crashes init", "kills
the background thread" — vetted one at a time against a plausible non-negated
neutral sentence; the audit's direction-free suggestions (uncaught exception,
host application's log, tuple of keys, never initialized) stay out for that
reason. Three of the audit's eight paraphrased findings now grade true; the
five that need an identifier or the mechanism are recorded as known misses,
not bought by loosening a keyword into a bare consequence.

Probe run over the runtime vocabulary with each case's full.diff, four
mutations observed failing in the intended direction, every file restored from
a byte copy.
PRs #151 (cli/cli) and #153 (urfave) landed five new keyword boards and
rewrote three urfave rows, in the same slice this branch appended four
sentry-python rows to, so keywords_test.go conflicted at the tail of
keywordCases.

Resolved as the union: origin/main's file verbatim with this branch's 98-line
block appended after its last row. Proven rather than eyeballed - removing that
block from the merge result reproduces origin/main byte-for-byte, every
origin/main row block is byte-identical in the result, and each of the four
appended rows is byte-identical to this branch's parent. Rows go from 26 to 30
(29 distinct boards: undici-4223 holds bugs 0 and 1).
@SutuSebastian
SutuSebastian merged commit 2f3b6c7 into main Nov 15, 2026
21 checks passed
@SutuSebastian
SutuSebastian deleted the agent/corpus-sentry-python branch November 15, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant