Skip to content

feat: add --toolset/-t option to vlmrun chat CLI#164

Merged
spillai merged 3 commits intomainfrom
devin/1773872705-add-toolsets-cli
Mar 19, 2026
Merged

feat: add --toolset/-t option to vlmrun chat CLI#164
spillai merged 3 commits intomainfrom
devin/1773872705-add-toolsets-cli

Conversation

@devin-ai-integration
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot commented Mar 18, 2026

Summary

Adds a repeatable --toolset / -t CLI option to vlmrun chat that lets users specify which tool categories the Orion agent should have access to during a chat session. The toolsets are validated against the AgentToolset literal type and passed via extra_body["toolsets"] to the OpenAI-compatible chat completions endpoint.

Also fixes the AgentToolset literal in types.py to use "world-gen" (hyphen) instead of "world_gen" (underscore), matching the API docs. The same rename is applied to the docstring in agent.py.

Bumps package version to 0.5.8.

Example usage:

vlmrun chat "Describe this" -i photo.jpg -t web -t image
vlmrun chat "Generate a chart" -t viz -t image-gen

Review & Testing Checklist for Human

  • Verify world-gen rename is backward-compatible: AgentToolset was changed from "world_gen" to "world-gen". Confirm the server accepts "world-gen" and that no other SDK consumers (e.g. users passing toolsets=["world_gen"] to client.agent.execute()) will break.
  • Test end-to-end: Run vlmrun chat "what can you do?" -t web -t core with a valid API key and confirm the toolsets appear in the request and are respected in the response behavior.
  • Verify AgentToolset.__args__ stability: The AVAILABLE_TOOLSETS list is derived from Literal.__args__, which works on Python 3.9–3.12 but is not a documented public API. Confirm this is acceptable or consider using get_args() from typing.

Notes

  • No new tests were added; the existing 296 tests continue to pass across Python 3.9–3.12.
  • The hardcoded toolset names in the --help text could drift from AgentToolset.__args__ if the type is updated — a minor maintenance concern.

Link to Devin session: https://app.devin.ai/sessions/74e0d8fb30974dc3938dd63f5eb22ae8
Requested by: @spillai

Add support for enabling a list of toolsets in the chat CLI command.
Users can now specify tool categories to enable via repeatable --toolset/-t flags.

Available toolsets: core, image, image-gen, world_gen, viz, document, video, web.

Example usage:
  vlmrun chat "Describe this" -i photo.jpg -t web -t image
  vlmrun chat "Generate a chart" -t viz -t image-gen

Co-Authored-By: Sudeep Pillai <sudeep.pillai@gmail.com>
@devin-ai-integration
Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@spillai
Copy link
Copy Markdown
Contributor

spillai commented Mar 19, 2026

Update the references for world_gen to world-gen and image-gen appropriately.

Co-Authored-By: Sudeep Pillai <sudeep.pillai@gmail.com>
@spillai
Copy link
Copy Markdown
Contributor

spillai commented Mar 19, 2026

Bump up the patch version.

Co-Authored-By: Sudeep Pillai <sudeep.pillai@gmail.com>
@spillai spillai merged commit 27659bf into main Mar 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant