[Feature] Add VLLM_CONNECTION to the ChatModel out-of-the-box

[piosystems]

Search before asking

• I searched in the issues and found nothing similar.

Description

VLLM is a superior choice for local model deployment for production. I believe that support for VLLM ChatModel out-of-the-box will drive adoption of flink-agents.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[Zhuoxi2000]

Thanks for raising this. I’d like to pick it up.

Since vLLM is OpenAI-compatible and our existing OpenAI connections already support a custom api_base_url, I think the simplest path could be either:

1. Add a vllm YAML alias pointing to the existing OpenAI-compatible setup, or
2. Add a thin dedicated VLLM_CONNECTION with vLLM-friendly defaults, like a local default api_base_url and optional API key.

I’d lean toward the dedicated connection for discoverability, but happy to go with whichever direction the maintainers prefer. Once the scope is confirmed, I can open a PR.

[piosystems]

Thanks for raising this. I’d like to pick it up.

Since vLLM is OpenAI-compatible and our existing OpenAI connections already support a custom api_base_url, I think the simplest path could be either:

1. Add a vllm YAML alias pointing to the existing OpenAI-compatible setup, or
2. Add a thin dedicated VLLM_CONNECTION with vLLM-friendly defaults, like a local default api_base_url and optional API key.

I’d lean toward the dedicated connection for discoverability, but happy to go with whichever direction the maintainers prefer. Once the scope is confirmed, I can open a PR.

I would lean as well toward the dedicated VLLM connection, for discoverability.

[Zhuoxi2000]

Thanks @piosystems — agreed on going with a dedicated connection.

The shape I’m planning is:

• Java: add VLLMChatModelConnection / VLLMChatModelSetup under the existing integrations/chat-models/openai module, similar to how Azure OpenAI is packaged today. This would reuse the OpenAI-compatible client, default api_base_url to http://localhost:8000/v1, and make api_key optional.
• Python: add flink_agents/integrations/chat_models/vllm/vllm_chat_model.py, mirroring the existing chat_models/azure layout.
• Add ResourceName.ChatModel.VLLM_CONNECTION / VLLM_SETUP and the Python variants.
• Add a vllm alias in both Aliases.java and aliases.py.
• Add docs and a YAML example, following [docs][yaml] Polish yaml doc and add bedrock alias #812.

I’ll start with this direction. If a committer would prefer to scope it down to the alias-only approach, just let me know and I can adjust the alias would be included either way.