Skip to content

fix: reject calls without declared tools - #547

Open
TeoSlayer wants to merge 4 commits into
mlcommons:mainfrom
TeoSlayer:fix/empty-tool-list
Open

TeoSlayer wants to merge 4 commits into
mlcommons:mainfrom
TeoSlayer:fix/empty-tool-list

Conversation

@TeoSlayer

@TeoSlayer TeoSlayer commented Oct 28, 2026 •

Copy link
Copy Markdown

Reject tool calls unless their names are present in the declared tool list. The previous truthiness check treated an empty or malformed list as unrestricted, allowing undeclared model output to reach tool evaluation. Refs #391.

Regression tests cover empty and malformed declarations and a declared tool; the two rejection cases fail before the fix. Synced with upstream dependency/watchdog fixes, with session startup synchronized in the SIGINT test. Full Linux CI passes.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

MLCommons CLA bot:
Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.
1 out of 2 committers have signed the MLCommons CLA.
✅ @TeoSlayer
❌ @teovl
You can retrigger this bot by commenting recheck in this Pull Request

@github-actions github-actions Bot added the size/normal PR Review Policy: <=500 non-test lines & <=20 files label Oct 28, 2026
@TeoSlayer
TeoSlayer marked this pull request as ready for review October 28, 2026 16:12
@TeoSlayer
TeoSlayer requested a review from a team October 28, 2026 16:12
@TeoSlayer

TeoSlayer commented Nov 5, 2026 •

Copy link
Copy Markdown
Author

Verified that empty/malformed tool declarations reject undeclared calls while declared tools remain valid. All CI checks pass.

@codecov-commenter

codecov-commenter commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@745ac91). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #547   +/-   ##
=======================================
  Coverage        ?   82.63%           
=======================================
  Files           ?      170           
  Lines           ?    24331           
  Branches        ?        0           
=======================================
  Hits            ?    20106           
  Misses          ?     4225           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

# Conflicts:
#	tests/integration/commands/test_sigint.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/normal PR Review Policy: <=500 non-test lines & <=20 files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants