Repository navigation
bench: add five curated getsentry/sentry-python corpus cases - #152
Conversation
Four real bug cases and one clean control from the Python SDK, every PR merged inside the 24mo window: - 4121 replaces `"params": record.args` with a generator over record.args, so a mapping logged against a %(name)s template ships its keys instead of its values. Reverted by e71ccbf19, which states the regression and adds two tests asserting the params are the dict. - 4133 swaps `if not logger.handlers:` for `logger.hasHandlers()`, so once the application configures the root logger the SDK never installs its own. Reverted by 5689bc09f. - 4281 reads channels.__version__ inside a try that only catches ImportError, so a shadowing `channels` module raises AttributeError out of init(); fixed by a35050694 (#5299), which splits the block and widens the handler. - 6168 writes SpanBatcher its own _flush_loop with two unguarded _flush calls, so one failed send ends the drain thread for the process lifetime; fixed by e17291eea. The pattern predates the PR in Batcher, disclosed in provenance.note. - 8119 is the control: a one-line guard on the third-party baggage items, exact because an empty field means there was nothing to append. Ground truth per case is a line the reviewed PR added and the later commit deletes or rewrites. None of the four kinds is in the published table, so each vocabulary is its kind plus keywords, every keyword unspelled by the case's own diff. Behaviour was reproduced by extracting the statement from both commits and running it in a harness (the SDK cannot be imported on this host); the probes are pinned in TestCheckedInCorpusKeywordsGradeBothWays. Corpus: 44 cases (8 seeded, 27 real, 9 control).
|
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 31 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
The audit read the control clean and accepted the revert grounding, 6168's pre-existing disclosure and 4281's trigger, but found two defects in the vocabulary that the case-level probe could not see. Three keywords described intent, not a failure, so a neutral or approving comment graded as a hit: 4133's "any existing root setup stops the library from installing its own output" and "a configured root makes the library skip its own setup", and 4281's "the version read is assumed to exist on any module of that name". They are gone, and the audit's sentences carrying them are now must-grade-false rows, so re-adding any of the three reproduces the failure. Every keyword was a 7-14 word clause, so a reviewer who words the same defect differently graded as a miss. Each bug case gains 2-3 short directional phrases instead — "keys instead of values", "debug=true fails", "crashes init", "kills the background thread" — vetted one at a time against a plausible non-negated neutral sentence; the audit's direction-free suggestions (uncaught exception, host application's log, tuple of keys, never initialized) stay out for that reason. Three of the audit's eight paraphrased findings now grade true; the five that need an identifier or the mechanism are recorded as known misses, not bought by loosening a keyword into a bare consequence. Probe run over the runtime vocabulary with each case's full.diff, four mutations observed failing in the intended direction, every file restored from a byte copy.
PRs #151 (cli/cli) and #153 (urfave) landed five new keyword boards and rewrote three urfave rows, in the same slice this branch appended four sentry-python rows to, so keywords_test.go conflicted at the tail of keywordCases. Resolved as the union: origin/main's file verbatim with this branch's 98-line block appended after its last row. Proven rather than eyeballed - removing that block from the merge result reproduces origin/main byte-for-byte, every origin/main row block is byte-identical in the result, and each of the four appended rows is byte-identical to this branch's parent. Rows go from 26 to 30 (29 distinct boards: undici-4223 holds bugs 0 and 1).
Part of STA-325. Five real benchmark cases from getsentry/sentry-python (Python observability SDK), the last repo of the v1 corpus.
"params": record.argsbecomes a generator, so a dict logged against a%(name)stemplate ships its keys, not its valueslogger.hasHandlers()replacesnot logger.handlers, so once the app configures the root logger, the SDK's own debug handler is never installedchannels.__version__is read inside atrythat catches onlyImportError, so a shadowingchannelsmodule raisesAttributeErrorout ofinit()SpanBatcher._flush_loopcalls_flushunguarded, so one failed send ends the drain thread for the life of the processEvidence
full.diffequalsgit diff base head, and it also passesgit apply --checkat its own base.TestCheckedInCorpusKeywordsGradeBothWays. A mutation in each direction makes the probe fail.Verification
make test-race, golangci-lint (native andGOOS=windows),./evals/one.sh bench-corpusandmake schemas-checkall pass.