Skip to content

Add an eval suite for dimensional-analysis - #239

Open
mkzung wants to merge 1 commit into
trailofbits:mainfrom
mkzung:evals/dimensional-analysis
Open

Add an eval suite for dimensional-analysis#239
mkzung wants to merge 1 commit into
trailofbits:mainfrom
mkzung:evals/dimensional-analysis

Conversation

@mkzung

@mkzung mkzung commented Aug 5, 2026

Copy link
Copy Markdown

Five cases, in Python rather than Solidity. references/bug-patterns.md says
the patterns occur in any language doing arithmetic with mixed units, then
carries 59 Solidity code blocks and no Python ones, so nothing measures that.

Case 01 is a real bug rather than a constructed one, and the four snippets are
the upstream source verbatim. Four sibling executors report a return
percentage; one divides a quote amount by a base amount, leaving a price where
a dimensionless ratio belongs. A round trip returning 0.27% reports 800%, off
by the entry price. Reported upstream as hummingbot#8408.

Nothing annotates the numerator's dimension, so
executed_amount_base * average_executed_price has to be reduced to quote before
the division reads as wrong. Three of the four siblings agree, which means
"this one differs from the others" reaches the right file by counting rather
than by dimensional reasoning, so the recall grader requires the unit argument
and requires the proposed fix to land back in quote over quote.

Case 05 is the constructed one and puts the plugin's home turf in Python: a
feed conversion multiplying by ten to the feed's decimals instead of the gap
between the two representations, 100x low, with two siblings that use the gap
correctly. Case 03 carries the same conversion done right, so the pair is a
discrimination test: one shape, once as the bug and once as the control.

Case 02 is case 01's function after the fix, with the wrong fields still in
scope, and case 03 is five helpers that each resemble a named pattern from the
reference while being correct. Those two are the precision floor. Case 04 asks
for a docstring and fails if the skill turns that into an audit, which is the
shape SKILL.md's own "When NOT to Use" says to decline.

Both negative cases carry a second grader that asks only whether a conclusion
was reached. A precision check on its own is passable by silence: a response
that narrates the code and never commits has claimed no defect, so it clears
the bar without answering. Commitment and correctness are judged separately
because a run can fail one while passing the other.

Nine scored graders: two recall, four precision, two conclusion, one that
keeps an answer an answer. Less recall-weighted than the variant-analysis
suite, and the README says why: every helper here has units in it, so the
likelier failure on unfamiliar code is flagging all of them rather than
missing one.

Case 01 carries a display-only skill-fired grader with no arm and no weight.
The trigger rate is worth watching because the skill's description routes on
blockchain code while its own "When to Use" says financial code, and these
prompts name the domain rather than asking for a dimensional analysis.

Note that validate_reference_links skips any path with evals in it, so
make validate does not cover this directory; the links were checked by hand.
No case has been piloted against a live run, which the README says plainly.
Ground truth is checked by hand and by arithmetic; the rubrics have not met a
judge. make self-test, make validate and make lint pass. make python-tests
fails the same way on a clean checkout, in constant-time-analysis.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@CLAassistant

CLAassistant commented Aug 5, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Five cases, in Python rather than Solidity. references/bug-patterns.md says
the patterns occur in any language doing arithmetic with mixed units, then
carries 59 Solidity code blocks and no Python ones.

Case 01 is a real bug and the four snippets are the upstream source verbatim.
Four sibling executors report a return percentage; one divides a quote amount
by a base amount, leaving a price where a dimensionless ratio belongs, so a
round trip returning 0.27% reports 800%. Reported upstream as hummingbot#8408.
Three of the four siblings agree, so "this one differs from the others" finds
the right file without any dimensional reasoning; the grader asks for the unit
argument and for the fix to land back in quote over quote.

Case 05 is constructed: a feed conversion multiplying by ten to the feed's
decimals instead of the gap between the two representations, 100x low. Case 03
carries the same conversion done right.

Case 02 is case 01 after the fix, with the wrong fields still in scope. Case
04 asks for a docstring and fails if that turns into an audit, which SKILL.md
already says to decline. Both negative cases have a second grader for whether
a conclusion was reached at all, since a precision check alone is passable by
silence.

Two caveats. validate_reference_links skips any path with evals in it, so make
validate does not cover this directory and the links were checked by hand. No
case has been run against a judge yet; ground truth is arithmetic only.
@mkzung
mkzung force-pushed the evals/dimensional-analysis branch from 701a966 to cd4fb45 Compare August 6, 2026 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants