Add an eval suite for dimensional-analysis - #239
Open
mkzung wants to merge 1 commit into
Open
Conversation
Five cases, in Python rather than Solidity. references/bug-patterns.md says the patterns occur in any language doing arithmetic with mixed units, then carries 59 Solidity code blocks and no Python ones. Case 01 is a real bug and the four snippets are the upstream source verbatim. Four sibling executors report a return percentage; one divides a quote amount by a base amount, leaving a price where a dimensionless ratio belongs, so a round trip returning 0.27% reports 800%. Reported upstream as hummingbot#8408. Three of the four siblings agree, so "this one differs from the others" finds the right file without any dimensional reasoning; the grader asks for the unit argument and for the fix to land back in quote over quote. Case 05 is constructed: a feed conversion multiplying by ten to the feed's decimals instead of the gap between the two representations, 100x low. Case 03 carries the same conversion done right. Case 02 is case 01 after the fix, with the wrong fields still in scope. Case 04 asks for a docstring and fails if that turns into an audit, which SKILL.md already says to decline. Both negative cases have a second grader for whether a conclusion was reached at all, since a precision check alone is passable by silence. Two caveats. validate_reference_links skips any path with evals in it, so make validate does not cover this directory and the links were checked by hand. No case has been run against a judge yet; ground truth is arithmetic only.
mkzung
force-pushed
the
evals/dimensional-analysis
branch
from
August 6, 2026 08:31
701a966 to
cd4fb45
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five cases, in Python rather than Solidity. references/bug-patterns.md says
the patterns occur in any language doing arithmetic with mixed units, then
carries 59 Solidity code blocks and no Python ones, so nothing measures that.
Case 01 is a real bug rather than a constructed one, and the four snippets are
the upstream source verbatim. Four sibling executors report a return
percentage; one divides a quote amount by a base amount, leaving a price where
a dimensionless ratio belongs. A round trip returning 0.27% reports 800%, off
by the entry price. Reported upstream as hummingbot#8408.
Nothing annotates the numerator's dimension, so
executed_amount_base * average_executed_price has to be reduced to quote before
the division reads as wrong. Three of the four siblings agree, which means
"this one differs from the others" reaches the right file by counting rather
than by dimensional reasoning, so the recall grader requires the unit argument
and requires the proposed fix to land back in quote over quote.
Case 05 is the constructed one and puts the plugin's home turf in Python: a
feed conversion multiplying by ten to the feed's decimals instead of the gap
between the two representations, 100x low, with two siblings that use the gap
correctly. Case 03 carries the same conversion done right, so the pair is a
discrimination test: one shape, once as the bug and once as the control.
Case 02 is case 01's function after the fix, with the wrong fields still in
scope, and case 03 is five helpers that each resemble a named pattern from the
reference while being correct. Those two are the precision floor. Case 04 asks
for a docstring and fails if the skill turns that into an audit, which is the
shape SKILL.md's own "When NOT to Use" says to decline.
Both negative cases carry a second grader that asks only whether a conclusion
was reached. A precision check on its own is passable by silence: a response
that narrates the code and never commits has claimed no defect, so it clears
the bar without answering. Commitment and correctness are judged separately
because a run can fail one while passing the other.
Nine scored graders: two recall, four precision, two conclusion, one that
keeps an answer an answer. Less recall-weighted than the variant-analysis
suite, and the README says why: every helper here has units in it, so the
likelier failure on unfamiliar code is flagging all of them rather than
missing one.
Case 01 carries a display-only skill-fired grader with no arm and no weight.
The trigger rate is worth watching because the skill's description routes on
blockchain code while its own "When to Use" says financial code, and these
prompts name the domain rather than asking for a dimensional analysis.
Note that validate_reference_links skips any path with evals in it, so
make validate does not cover this directory; the links were checked by hand.
No case has been piloted against a live run, which the README says plainly.
Ground truth is checked by hand and by arithmetic; the rubrics have not met a
judge. make self-test, make validate and make lint pass. make python-tests
fails the same way on a clean checkout, in constant-time-analysis.