Skip to content

feat: add LiteLLM as AI gateway provider#45

Open
RheagalFire wants to merge 1 commit into
elder-plinius:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as AI gateway provider#45
RheagalFire wants to merge 1 commit into
elder-plinius:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

Copy link
Copy Markdown

Summary

  • Adds LiteLLM as a first-class AI gateway provider, giving T3MP3ST access to 100+ LLM providers (AWS Bedrock, Azure, Vertex AI, Groq, Mistral, Cohere, etc.) through a single OpenAI-compatible proxy.
  • Additive only - zero changes to existing providers.

Changes

  • src/types/index.ts - Added 'litellm' to the LLMProvider union type
  • src/llm/index.ts - Added LiteLLMAdapter (extends OpenRouterAdapter since LiteLLM proxy speaks the same OpenAI-compatible wire protocol), factory function createLiteLLMBackbone(), registered in createAdapter() switch and createBestAvailableBackbone() priority chain
  • src/config/index.ts - Added LiteLLM config block (baseUrl, defaultModel), LITELLM_API_KEY/LITELLM_BASE_URL/LITELLM_MODEL env vars, AVAILABLE_MODELS entry, hasLiteLLMProxy() helper, and wired through all provider helper methods (getLLMConfig, getConfiguredProviders, buildFallbackChain, setDefaultProvider, setDefaultModel, exportConfig, createEnvTemplate)
  • src/setup.ts - Added interactive setupLiteLLMProxy() wizard (prompts for proxy URL, optional API key, default model; tests connection with save-anyway fallback)
  • src/index.ts - Exported createLiteLLMBackbone

Tests

1. TypeScript type check: npx tsc --noEmit passes clean (zero errors).

2. Full test suite: All 328 tests pass across 28 test files.

 Test Files  28 passed (28)
      Tests  328 passed (328)
   Duration  3.53s

3. Live E2E against LiteLLM proxy (Azure Foundry backend):

Proxy config: litellm --config config.yaml --port 4111 routing anthropic/claude-sonnet-4-6 through Azure Foundry.

=== T3MP3ST LiteLLM E2E Test ===
Provider: litellm
Model: anthropic/claude-sonnet-4-6
Validation: {"valid":true}

--- Test 1: Simple chat ---
Content: "4"
Model: anthropic/claude-sonnet-4-6
Usage: {"promptTokens":20,"completionTokens":5,"totalTokens":25}
Finish reason: stop

--- Test 2: Streaming ---
hello world
Stream content: "hello world"

--- Test 3: Tool calling ---
Content: "I'll look up all the DNS records for example.com..."
Tool calls: [
  { "id": "toolu_01C4B2vpP8AXiYinmZUDwqra", "name": "dns_lookup", "arguments": { "domain": "example.com", "type": "A" } },
  { "id": "toolu_014UHdpekiLj1NbiordiW8Bn", "name": "dns_lookup", "arguments": { "domain": "example.com", "type": "AAAA" } },
  { "id": "toolu_01HnyTDQHgsFjckXqyi2TT6w", "name": "dns_lookup", "arguments": { "domain": "example.com" } }
]
Finish reason: length

=== All tests passed ===

Full chain verified: LLMBackbone('litellm') -> LiteLLMAdapter -> fetch(proxy/v1/chat/completions) -> LiteLLM proxy -> Azure Foundry -> Claude Sonnet 4.6 -> parsed LLMResponse. Chat, streaming, and tool calling all work.

Risk / Compatibility

  • Additive only. Existing providers untouched.
  • LiteLLM is detected via LITELLM_BASE_URL env var or explicit config - no auto-detection from a default localhost URL (avoids false positives).
  • Sits in the createBestAvailableBackbone() priority chain between Venice and Anthropic.
  • Participates in the model fallback ladder when TEMPEST_MODEL_FALLBACK is enabled.

Example usage

Environment variables:

export LITELLM_BASE_URL=http://localhost:4000/v1
export LITELLM_API_KEY=sk-...        # optional, depends on proxy config
export LITELLM_MODEL=anthropic/claude-sonnet-4-6  # optional, defaults to gpt-4o

CLI:

npx t3mp3st --provider litellm --model anthropic/claude-sonnet-4-6

Programmatic:

import { createLiteLLMBackbone } from 't3mp3st';

const llm = createLiteLLMBackbone(
  'sk-my-proxy-key',              // optional API key
  'anthropic/claude-sonnet-4-6',  // model (any litellm model string)
  'http://localhost:4000/v1',     // proxy base URL
);

const response = await llm.prompt('Plan a recon sweep of example.com');

Setup wizard:

npx t3mp3st setup
# Select "LiteLLM Proxy" from the provider list
# Enter proxy URL, optional API key, and default model

@RheagalFire

Copy link
Copy Markdown
Author

cc @elder-plinius - would love your review when you get a chance!

@jmagly

jmagly commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Reviewed the wiring — it's logically sound (LiteLLM via the OpenRouter-compatible adapter, base URL http://localhost:4000/v1${baseUrl}/chat/completions, and treating the provider as configured by base-URL presence rather than a key is a nice touch). Main thing is it's currently conflicting — needs a rebase onto main. It also collides with the other provider PRs on llm/config/types (#55, and #57), so sequence it with whichever provider PR lands first and rebase so the union / Record<LLMProvider> picks up both keys.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants