Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 69 additions & 9 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,85 @@
# Architecture

`@quantum-l9/llm-router` is a reusable TypeScript routing library. The root `L9LLMRouter` is the supported execution surface.
`@quantum-l9/llm-router` is a reusable TypeScript routing library. `L9LLMRouter` is the supported production execution surface and the only component that composes routing, budget, resilience, and provider dispatch.

## Runtime flow

```text
validated TaskDescriptor
validated execution task
-> effective image set merged into task
-> pure route resolution
-> request identity and timestamp
-> atomic process-local budget reservation
-> provider-family-safe downgrade
-> per-provider circuit permit
-> provider dispatch
-> cost reconciliation and audit log
-> provider dispatch with explicit cancellation and timeout
-> aggregate execution accounting
-> budget reconciliation and audit log
```

Provider clients live under `src/providers/`. Production modules outside `src/index.ts` and `src/providers/` may not import them. Existing provider subpath exports remain available during the 1.x line for compatibility, but direct use is deprecated because it bypasses budget and circuit controls.
Route resolution is pure. Request IDs and timestamps are added afterward and do not participate in routing equivalence.

## State scope
## Module ownership

The built-in budget tracker and circuit breaker are process-local. They prevent concurrent overspend and provider stampedes inside one process. Distributed enforcement requires an external persistence adapter and is intentionally not claimed here.
```text
src/types.ts public legacy contracts
src/schemas.ts runtime validation for public legacy input
src/matrices/* deterministic model and search resolution
src/pricing.ts canonical OpenRouter price table
src/budget/* process-local admission and spend accounting
src/circuit-breaker.ts process-local provider health control
src/provider-errors.ts typed failure classification and redaction
src/providers/* provider I/O and SDK transport isolation
src/vision/* vision configuration and task planning
src/index.ts composition root and supported execution API
src/control-plane/* internal Phase 1 contract kernel
```

## Provider boundary

Provider clients live under `src/providers/`. Production modules outside `src/index.ts` and `src/providers/` may not import them. ESLint and a programmatic probe enforce the rule.

Existing provider subpath exports remain available during the 1.x line for compatibility. They are deprecated because direct use bypasses budget and circuit controls. Their removal requires a major version.

## Budget state

The built-in budget tracker owns committed spend and active reservations. Admission evaluates both, so concurrent requests inside one JavaScript process cannot all pass against the same unreserved ceiling.

Critical tasks retain the documented override but still reserve and record cost. Reservation identifiers must be non-empty and unique. Client and direct tracker configuration is runtime validated.

The tracker is not a distributed ledger. Multiple processes require an external atomic persistence adapter, which is outside this repository's current scope.

## Circuit state

The circuit breaker owns independent state per provider.

- Closed calls receive ordinary permits.
- The failure threshold opens the circuit.
- The cooldown permits exactly one half-open probe.
- Retryable network, timeout, rate-limit, and server failures count.
- Client, cancellation, budget, policy, and local validation failures do not count.
- Late results from calls acquired before a circuit opened cannot overwrite newer state.

## Provider execution

The OpenAI SDK is isolated behind `OpenAIChatTransport`. SDK retries are disabled. The router controls fallback order explicitly.

OpenRouter fallbacks advance only after retryable failures. Non-retryable client failures and cancellation terminate immediately.

Perplexity consensus executes configured variations in parallel. The selected content remains one successful candidate, while budget-facing cost and token accounting aggregate all successful variations.

## Image safety

Image execution accepts public HTTPS URLs and bounded supported image data URIs. Local, private, loopback, link-local, reserved, and local-domain targets are rejected before dispatch. Option-supplied images are validated and included in route selection before budget reservation.

## Control Plane Phase 1

The Control Plane kernel owns strict runtime contracts, canonical JSON, deterministic identity, immutable builders, policy interfaces, and provider-adapter interfaces. It does not own provider clients, network access, Gate ingress, TransportPacket authority, Graphiti, Neo4j, promotion, mutable global state, or legacy cutover.

The internal barrel `src/control-plane/index.ts` is a deliberate module boundary but is absent from package exports. The one-line pass-through type module was removed because it had no independent responsibility.

Route identity excludes request-specific values and explanatory prose, including route, budget, and provider-health reasons. Complete content hashing still protects those fields.

## Runtime floor
## Runtime support

The package runtime floor is Node 20.19.0. Type definitions track the Node 20 line so compilation cannot silently authorize APIs that are unavailable to supported consumers. CI also exercises the current active LTS as a secondary runtime.
The package preserves a Node 20.19.0 compatibility floor for the 1.x line. CI validates the floor plus maintained Node 22 and Node 24 LTS lines. Publish and supply-chain jobs run on Node 24 LTS.
135 changes: 130 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,145 @@
# @quantum-l9/llm-router

Shared deterministic model routing, budget enforcement, provider resilience, and visual QA for L9 bots.
`@quantum-l9/llm-router` is the shared TypeScript routing library for L9 applications. It validates task input, selects a provider and model deterministically, reserves budget before dispatch, applies provider-family-safe downgrades, controls provider failure pressure, executes through typed provider clients, and reconciles actual cost.

## What is implemented

- Deterministic routing for search, general, and vision task families
- Per-client and global process-local budget enforcement with pre-dispatch reservations
- Per-provider circuit breaking with one half-open recovery probe
- Explicit timeout, cancellation, retry, fallback, and provider-error classification
- OpenRouter and Perplexity clients through an OpenAI SDK transport boundary
- Bounded image URL and inline-image validation
- Runtime validation with Zod 4
- Internal Control Plane Phase 1 contracts, canonical hashing, builders, and boundaries
- Package, declaration, lint-boundary, audit, and isolated-consumer validation

## Installation

The package is published to GitHub Packages.

```bash
npm install @quantum-l9/llm-router
```

Configure the `@quantum-l9` registry and authentication through the consuming environment. Do not commit package tokens or provider credentials.

## Supported runtimes

The 1.x compatibility floor remains Node.js `20.19.0`. CI also validates maintained Node.js 22 and 24 LTS lines. Release and supply-chain jobs run on Node.js 24 LTS.

## Basic use

```ts
import { L9LLMRouter, TaskComplexity, TaskType } from '@quantum-l9/llm-router';
import {
L9LLMRouter,
TaskComplexity,
TaskType,
} from '@quantum-l9/llm-router';

const router = new L9LLMRouter({
perplexityApiKey: process.env.PERPLEXITY_API_KEY!,
openrouterApiKey: process.env.OPENROUTER_API_KEY!,
providerTimeoutMs: 60_000,
providerMaxRetries: 0,
});

router.initClient('tenant-a', {
monthlyBudgetPerClient: 200,
weeklyTarget: 50,
weeklyHardCeiling: 100,
});
router.initClient('tenant-a', { monthlyBudgetPerClient: 200 });

const result = await router.execute(
{ clientId: 'tenant-a', type: TaskType.CONTENT_GENERATION, complexity: TaskComplexity.MEDIUM },
{
clientId: 'tenant-a',
type: TaskType.CONTENT_GENERATION,
complexity: TaskComplexity.MEDIUM,
expectedOutputTokens: 1_500,
},
'You are a careful writer.',
'Draft the article.',
);
```

Direct imports from `./openrouter` or `./perplexity` are retained for 1.x compatibility but are deprecated. They bypass router-level budget and circuit controls.
`execute()` requires a non-empty `clientId`. The router rejects malformed execution input before allocating a request ID, reserving budget, or dispatching a provider call.

## Vision execution

Images supplied through execution options are merged into the validated task before routing. This ensures model selection and budget estimation use the same image count that reaches the provider.

```ts
const result = await router.execute(
{
clientId: 'tenant-a',
type: TaskType.SCREENSHOT_ANALYSIS,
complexity: TaskComplexity.MEDIUM,
},
'Inspect the screenshots.',
'Compare the layouts.',
{
images: [
'https://cdn.example.com/current.png',
'https://cdn.example.com/competitor.png',
],
},
);
```

Only HTTPS public URLs and bounded `data:image/*;base64` payloads are accepted. Private, loopback, link-local, reserved, local-domain, non-image, and oversized inline targets are rejected before provider dispatch.

## Search consensus

For eligible high-complexity Perplexity tasks, `{ consensus: true }` executes the configured variations in parallel. The returned content is selected from the successful responses, while token and cost fields represent the aggregate successful consensus execution so budget reconciliation does not undercount spend.

## Budget semantics

The built-in tracker is process-local.

1. Estimate the route cost.
2. Evaluate committed spend plus active reservations.
3. Reserve estimated cost before provider dispatch.
4. Release the reservation on confirmed unbilled failure.
5. Reconcile the reservation to actual reported cost on success.

This prevents concurrent overspend inside one process. It does not claim distributed enforcement across processes or machines.

## Provider failure behavior

Provider failures are classified as network, timeout, rate limit, server, client, cancellation, local, or unknown.

- Retryable provider failures may advance through an explicit fallback chain.
- Client errors, cancellation, and local validation failures stop immediately.
- Local validation and budget failures do not poison provider circuit health.
- A circuit permits only one half-open recovery probe.
- Late successes from older calls cannot close a circuit opened by newer failures.

The OpenAI SDK has hidden retries disabled. Every router-controlled fallback remains visible and bounded.

## Direct provider imports

These 1.x compatibility exports remain available:

```ts
import { OpenRouterClient } from '@quantum-l9/llm-router/openrouter';
import { PerplexityClient } from '@quantum-l9/llm-router/perplexity';
```

They are deprecated because they bypass router-level budget and circuit controls. Use `L9LLMRouter` for production execution. Removal requires a future major-version migration.

## Internal Control Plane kernel

Phase 1 Control Plane files are compiled but are not exposed through `package.json` exports. They define strict contracts, deterministic canonicalization, identity verification, immutable builders, policy interfaces, and provider-adapter interfaces. They perform no network calls and do not replace the legacy router.

See [`docs/control-plane-architecture.md`](docs/control-plane-architecture.md).

## Validation

```bash
npm ci
npm run verify:all
```

`verify:all` runs the production build, strict type verification, declaration-consumer compilation, ESLint, the provider-boundary probe, Vitest, production dependency audit, package allowlist inspection, isolated tarball installation, and package export smoke tests.

Operational procedures and failure recovery are documented in [`RUNBOOK.md`](RUNBOOK.md). Architecture and ownership boundaries are documented in [`ARCHITECTURE.md`](ARCHITECTURE.md).
Loading
Loading