Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 29 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ The official command-line interface for [ZeroGPU](https://zerogpu.ai) — run fa
- [Classification](#classification)
- [`classify_iab`](#classify_iab)
- [`classify_iab_enriched`](#classify_iab_enriched)
- [`classify_domain`](#classify_domain)
- [`classify_structured`](#classify_structured)
- [`classify_zero_shot`](#classify_zero_shot)
- [Extraction](#extraction)
Expand Down Expand Up @@ -122,16 +123,33 @@ zerogpu status

#### `chat`

Chat with the **LFM2.5-1.2B-Instruct** model.
Chat with a ZeroGPU text-generation model. Defaults to **LFM2.5-1.2B-Instruct**; use `--model` to reach for a larger reasoning model.

```bash
zerogpu chat "Explain edge inference in one sentence."
zerogpu chat "Translate to French: Good morning." -i "You are a precise translator."

# Frontier-grade reasoning, with the trace printed alongside the answer
zerogpu chat "Why does my API keep getting rate-limited?" -m gpt-oss-120b -r

# Multilingual reasoning
zerogpu chat "Explique la mise en cache en une phrase." -m qwen3-30b-a3b-fp8
```

| Option | Description |
|---|---|
| `-i, --instructions <text>` | System instructions that steer the assistant's behavior. |
| `-m, --model <model>` | Model to use (default `LFM2.5-1.2B-Instruct`). |
| `-r, --reasoning` | Also print the reasoning trace, for models that return one. |

| Model | Notes |
|---|---|
| `LFM2.5-1.2B-Instruct` | Default. Fast edge chat. |
| `LFM2.5-1.2B-Thinking` | Compact reasoning model. |
| `gpt-oss-120b` | 117B MoE, 131K context, reasoning + function calling. |
| `qwen3-30b-a3b-fp8` | 30.5B MoE, 100+ languages, reasoning + function calling. |

`qwen3-30b-a3b-fp8` is served by the Chat Completions API rather than the Responses API; the CLI routes it automatically.

#### `chat_thinking`

Expand Down Expand Up @@ -177,6 +195,16 @@ Same as `classify_iab` but returns enriched output: audience, topics, keywords,
zerogpu classify_iab_enriched "Best running shoes for marathon training."
```

#### `classify_domain`

Classify a **domain name** against the IAB taxonomy using the ZeroGPU domain edge model. Takes only the hostname — no page fetch or body text — so it suits bidstream enrichment and inventory-level targeting.

```bash
zerogpu classify_domain nytimes.com
```

Returns scored content categories plus topics, keywords, and inferred user intent for the domain. When you have the page text and need per-URL precision, use `classify_iab` instead.

#### `classify_structured`

Classify text against a **structured schema** of categories and allowed labels using GLiNER2.
Expand Down
7 changes: 5 additions & 2 deletions docs/ADDING_COMMANDS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ This guide explains how to add a new CLI command to the ZeroGPU CLI.

- `src/commands/` — one file per command, each exporting a `register<Name>Command(program)` function.
- `src/cli.ts` — wires every command into the root program.
- `src/lib/responses.ts` — shared `RESPONSES_ENDPOINT` and `ResponsesApiResponse` for `/v1/responses` calls.
- `src/lib/responses.ts` — shared `RESPONSES_ENDPOINT`, `ResponsesApiResponse`, and the `extractOutputText` / `extractReasoningText` helpers for `/v1/responses` calls.
- `src/lib/chatCompletions.ts` — the same for `/v1/chat/completions`, used by models the platform serves only there (currently `qwen3-30b-a3b-fp8`), plus `toResponsesUsage` to normalize token counts for savings tracking.
- `src/lib/auth.ts` — `getApiKey()` for authenticated requests.

## Steps
Expand All @@ -15,7 +16,9 @@ This guide explains how to add a new CLI command to the ZeroGPU CLI.
- Export `registerXxxCommand(program: Command): void`.
- Define the command name, args, and `.description(...)`.
- In `.action(...)`, validate auth, call the API, and print the result.
- For Responses API models, import `RESPONSES_ENDPOINT` and `ResponsesApiResponse` from `../lib/responses.js` — do not hardcode the endpoint or redefine the type.
- For Responses API models, import `RESPONSES_ENDPOINT` and `ResponsesApiResponse` from `../lib/responses.js` — do not hardcode the endpoint or redefine the type. Use `extractOutputText(data)` rather than reading `output[0]`: reasoning models put a `reasoning` item ahead of the assistant message.
- For Chat Completions models, import `CHAT_COMPLETIONS_ENDPOINT` and `ChatCompletionsApiResponse` from `../lib/chatCompletions.js`, and pass `toResponsesUsage(data.usage)` to `recordAndMaybeNotify`.
- Add the model's published input/output price to `ZGPU_PRICING` in `src/lib/savings.ts`, so savings are computed at the real rate rather than the fallback.
- Keep only command-specific constants local (e.g. `const MODEL = "..."`).

2. **Register it** in `src/cli.ts`:
Expand Down
82 changes: 76 additions & 6 deletions docs/DOCUMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,12 @@

## 1. Project Description

`zerogpu-cli` is the official command-line interface for [ZeroGPU](https://zerogpu.ai), a distributed / edge inference platform for small language models (SLMs) and nano language models. The CLI is a thin, OpenAI-compatible client around the ZeroGPU **Responses API** (`https://api.zerogpu.ai/v1/responses`) that lets you call a curated set of edge-optimized models directly from your terminal for common NLP workloads:
`zerogpu-cli` is the official command-line interface for [ZeroGPU](https://zerogpu.ai), a distributed / edge inference platform for small language models (SLMs) and nano language models. The CLI is a thin, OpenAI-compatible client around the ZeroGPU **Responses API** (`https://api.zerogpu.ai/v1/responses`) — and, for models served only there, the **Chat Completions API** (`https://api.zerogpu.ai/v1/chat/completions`) — that lets you call a curated set of edge-optimized models directly from your terminal for common NLP workloads:

- Conversational chat (`LFM2.5-1.2B-Instruct`, `LFM2.5-1.2B-Thinking`)
- IAB content/audience classification (`zlm-v1-iab-classify-edge`, `…-enriched`)
- Reasoning and tool-use chat (`gpt-oss-120b`, `qwen3-30b-a3b-fp8`)
- IAB content/audience classification (`zlm-v1-iab-classify-edge`, `zlm-v2-iab-classify-edge-enriched`)
- Domain-level IAB classification (`zlm-v1-iab-domain-classifier`)
- Zero-shot classification (`deberta-v3-small`)
- Structured / schema-driven classification and JSON extraction (`gliner2-base-v1`)
- Named-entity recognition with custom labels (`gliner2-base-v1`)
Expand Down Expand Up @@ -89,7 +91,7 @@ The CLI exposes the following commands:
|---|---|
| [`login`](#41-login) | Sign in and persist API key |
| [`status`](#42-status) | Show current sign-in status |
| [`chat`](#43-chat) | Chat with `LFM2.5-1.2B-Instruct` |
| [`chat`](#43-chat) | Chat with a text-generation model (default `LFM2.5-1.2B-Instruct`) |
| [`chat_thinking`](#44-chat_thinking) | Chat with the Thinking variant (returns reasoning) |
| [`classify_iab`](#45-classify_iab) | IAB taxonomy classification |
| [`classify_iab_enriched`](#46-classify_iab_enriched) | IAB enriched (topics / keywords / intent) |
Expand All @@ -101,6 +103,7 @@ The CLI exposes the following commands:
| [`extract_json`](#412-extract_json) | Schema-driven structured JSON extraction |
| [`summarize`](#413-summarize) | Summarize text with `llama-3.1-8b-instruct-fast` |
| [`generate_followups`](#414-generate_followups) | Generate follow-up questions |
| [`classify_domain`](#415-classify_domain) | Domain-level IAB classification |

### Common exit codes
| Code | Meaning |
Expand Down Expand Up @@ -182,11 +185,11 @@ zerogpu status

### 4.3 `chat`

Chat with the `LFM2.5-1.2B-Instruct` model.
Chat with a ZeroGPU text-generation model. Defaults to `LFM2.5-1.2B-Instruct`.

**Synopsis**
```

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add languages to the fenced code blocks. These fences fail markdownlint MD040.

  • docs/DOCUMENTATION.md#L191-L191: label the CLI synopsis fence as bash.
  • docs/DOCUMENTATION.md#L637-L637: label the CLI synopsis fence as bash.
  • docs/DOCUMENTATION.md#L685-L685: label the endpoint-contract fence as http or text.
🧰 Tools
🪛 markdownlint-cli2 (0.23.0)

[warning] 191-191: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

📍 Affects 1 file
  • docs/DOCUMENTATION.md#L191-L191 (this comment)
  • docs/DOCUMENTATION.md#L637-L637
  • docs/DOCUMENTATION.md#L685-L685
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/DOCUMENTATION.md` at line 191, Label the fenced code block at
docs/DOCUMENTATION.md lines 191-191 as bash, label the fenced code block at
lines 637-637 as bash, and label the endpoint-contract fence at lines 685-685 as
http or text to satisfy markdownlint MD040.

Source: Linters/SAST tools

zerogpu chat <text> [-i <instructions>]
zerogpu chat <text> [-i <instructions>] [-m <model>] [-r]
```

**Parameters**
Expand All @@ -195,11 +198,26 @@ zerogpu chat <text> [-i <instructions>]
|---|---|---|---|
| `text` (positional) | string | yes | The user message / prompt to send. |
| `-i`, `--instructions <instructions>` | string | optional | System instructions that steer the assistant's behavior. |
| `-m`, `--model <model>` | string | optional | Model to use. Default `LFM2.5-1.2B-Instruct`. Matched case-insensitively; an unknown id exits `1` and lists the valid ones. |
| `-r`, `--reasoning` | boolean | optional | Also print the reasoning trace, for models that return one. Ignored when the model returns none. |

**Models**

| Model | API | Notes |
|---|---|---|
| `LFM2.5-1.2B-Instruct` | Responses | Default. Fast edge chat. |
| `LFM2.5-1.2B-Thinking` | Responses | Compact reasoning model. |
| `gpt-oss-120b` | Responses | 117B MoE, 131K context, reasoning + function calling. |
| `qwen3-30b-a3b-fp8` | Chat Completions | 30.5B MoE, 100+ languages, reasoning + function calling. |

`qwen3-30b-a3b-fp8` has no Responses endpoint, so the CLI posts it to `/v1/chat/completions` instead, mapping `--instructions` to a `system` message and normalizing `prompt_tokens` / `completion_tokens` back to Responses token names for savings tracking. This routing is transparent — the command and its output are identical either way.

**Example**
```bash
zerogpu chat "Explain WebSockets in two sentences." \
-i "You are a concise technical writer."

zerogpu chat "Why is my p99 latency spiking?" -m gpt-oss-120b -r
```

**Expected output**
Expand Down Expand Up @@ -281,7 +299,7 @@ zerogpu classify_iab "The Lakers signed a new point guard ahead of the playoffs.

### 4.6 `classify_iab_enriched`

Classify text with the **enriched** IAB edge model (`zlm-v1-iab-classify-edge-enriched`) — returns audience categories plus topics, keywords, and inferred intent.
Classify text with the **enriched** IAB edge model (`zlm-v2-iab-classify-edge-enriched`) — returns audience categories plus topics, keywords, and inferred intent.

**Synopsis**
```
Expand Down Expand Up @@ -611,6 +629,47 @@ zerogpu generate_followups \

---

### 4.15 `classify_domain`

Classify a domain name against the IAB taxonomy using `zlm-v1-iab-domain-classifier`. It infers the categories that characterize a site as a whole from the hostname alone — no crawl, no page text — which keeps payloads roughly 10x smaller than page-level classification. Use it for bidstream enrichment, allow/deny-list scoring, and inventory-level targeting; when you have the page text and need per-URL precision, use [`classify_iab`](#45-classify_iab) instead.

**Synopsis**
```
zerogpu classify_domain <domain>
```

**Parameters**

| Name | Type | Required | Description |
|---|---|---|---|
| `domain` (positional) | string | yes | Bare domain name to classify, e.g. `nytimes.com`. |

**Example**
```bash
zerogpu classify_domain nytimes.com
```

**Expected output (illustrative)**
```json
{
"content": {
"iab_1_0": [{ "code": "IAB12", "name": "News", "score": 0.8124 }],
"iab_2_2": [{ "name": "News and Politics", "score": 0.8124 }]
},
"topics": [{ "name": "news", "score": 0.8124 }],
"keywords": ["news", "politics", "world", "business"],
"user_intent": {
"name": "reading news and current events",
"category": "informational",
"score": 0.5
}
}
```

**Outcomes** — same as the common table.

---

## 5. Network & API Contract

All inference commands POST to:
Expand All @@ -621,6 +680,14 @@ Content-Type: application/json
x-api-key: <ZEROGPU_API_KEY>
```

The one exception is `chat --model qwen3-30b-a3b-fp8`, which the ZeroGPU platform serves only through the OpenAI-compatible Chat Completions endpoint:

```
POST https://api.zerogpu.ai/v1/chat/completions
Content-Type: application/json
x-api-key: <ZEROGPU_API_KEY>
```

The request body is:
```jsonc
{
Expand All @@ -635,12 +702,15 @@ The CLI parses the response as:
```ts
interface ResponsesApiResponse {
output?: Array<{
type?: string;
content?: Array<{ type?: string; text?: string }>;
}>;
}
```
It picks the first `content` entry whose `type === "output_text"` (falling back to `content[0]`), then prints `text` — pretty-printed if it parses as JSON, otherwise as a raw string.

Reasoning models emit a `reasoning` output item **ahead of** the assistant message, so `chat` selects the item whose `type === "message"` rather than assuming `output[0]`, and reads the trace from the `reasoning` item when `--reasoning` is passed. On the Chat Completions path the answer is `choices[0].message.content` and the trace is `choices[0].message.reasoning`.

Comment on lines +705 to +713

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Align the common output contract with the new parser.

The earlier “Common output behavior” section still says every inference command reads output[0], which contradicts this message-item selection rule. Update that section to describe output_text extraction from the assistant message so integrations do not assume reasoning-first payloads are unsupported.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/DOCUMENTATION.md` around lines 705 - 713, Update the “Common output
behavior” documentation section to describe selecting the assistant message from
output and extracting its output_text content, rather than assuming output[0].
Ensure it reflects reasoning-first payloads and the existing
fallback/pretty-print behavior used by the parser.

---

## 6. Version Compatibility & Update Behavior
Expand Down
2 changes: 2 additions & 0 deletions src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { Command } from "commander";
import updateNotifier from "update-notifier";
import { registerChatCommand } from "./commands/chat.js";
import { registerChatThinkingCommand } from "./commands/chatThinking.js";
import { registerClassifyDomainCommand } from "./commands/classifyDomain.js";
import { registerClassifyIabCommand } from "./commands/classifyIab.js";
import { registerClassifyIabEnrichedCommand } from "./commands/classifyIabEnriched.js";
import { registerClassifyStructuredCommand } from "./commands/classifyStructured.js";
Expand Down Expand Up @@ -95,6 +96,7 @@ export function buildProgram(): Command {
registerStatusCommand(program);
registerClassifyIabCommand(program);
registerClassifyIabEnrichedCommand(program);
registerClassifyDomainCommand(program);
registerClassifyStructuredCommand(program);
registerClassifyZeroShotCommand(program);
registerGenerateFollowupsCommand(program);
Expand Down
Loading