diff --git a/AGENTS.md b/AGENTS.md index e266776dd..0e40ad83f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -93,7 +93,7 @@ Dataset Manager --> Load Generator --> Endpoint Client --> External Endpoint | **Endpoint Client** | `src/inference_endpoint/endpoint_client/` | Multi-process HTTP workers communicating via ZMQ IPC. `HTTPEndpointClient` is the main entry point | | **Dataset Manager** | `src/inference_endpoint/dataset_manager/` | Loads JSONL, HuggingFace, CSV, JSON, Parquet datasets. `Dataset` base class with `load_sample()`/`num_samples()` interface | | **Metrics Aggregator** | `src/inference_endpoint/async_utils/services/metrics_aggregator/` | Subprocess. Subscribes to events, aggregates per-sample metrics into a `MetricsRegistry` (counters + HDR-histogram series + raw values), publishes `MetricsSnapshot` over IPC PUB at a configurable cadence (`SessionState`: `INITIALIZE` → `LIVE` → `DRAINING` → {`COMPLETE` \| `INTERRUPTED`}). Final snapshot is atomically written to `final_snapshot.json` as the **primary** Report source; the terminal pub/sub frame is a TUI "run finished" signal only. | -| **Report** | `src/inference_endpoint/metrics/report.py` | `Report.from_snapshot(dict)` builds reports from the dict form (`snapshot_to_dict`). Reads `final_snapshot.json` directly via `json.loads` (no Struct decode). Sets `complete = (state == "complete" and n_pending_tasks == 0)` and renders an explicit warning for `INTERRUPTED` runs. Keeps the steady-window verdict on `Report.steady_state` and includes it in `to_json` (unlike `accuracy`). | +| **Report** | `src/inference_endpoint/metrics/report.py` | `Report.from_snapshot(dict, token_count_source=)` builds reports from the dict form (`snapshot_to_dict`). Reads `final_snapshot.json` directly via `json.loads` (no Struct decode). `complete` depends on the token-count source: server needs `state == "complete"`, valid accounting, no dropped samples and `n_samples_missing_usage == 0`; client needs `n_pending_tasks == 0` instead of the usage check. Headline ISL/OSL/TPOT/TPS are on the primary basis (`token_count_source`); the other basis sits beside them in `_`-suffixed blocks (`Report.other_basis`) and `osl_paired`. Renders an explicit warning for `INTERRUPTED` runs. Keeps the steady-window verdict on `Report.steady_state` and includes it in `to_json` (unlike `accuracy`). | | **Config** | `src/inference_endpoint/config/`, `endpoint_client/config.py` | Pydantic-based YAML schema (`schema.py`) — the declared user surface, distinct from the resolved `runtime_settings.py`; `Timeouts` (`config/schema.py` — all give-up deadlines in one frozen model at `settings.timeouts`, incl. the whole-run `run_timeout_s` watchdog; the workload duration `runtime.max_issue_duration_ms` stays on `settings.runtime`; client worker-lifecycle timeouts stay on `settings.client`), `HTTPClientConfig` (single Pydantic model for CLI/YAML/runtime), `RuntimeSettings` | | **CLI** | `src/inference_endpoint/main.py`, `commands/benchmark/cli.py` | cyclopts-based, auto-generated from `schema.py` and `HTTPClientConfig` Pydantic models. Flat shorthands via `cyclopts.Parameter(alias=...)` | | **Async Utils** | `src/inference_endpoint/async_utils/` | `LoopManager` (uvloop + eager_task_factory), ZMQ transport layer, generic `MessageCodec[T]`-parametrized pub/sub, event publisher | @@ -142,9 +142,14 @@ working in this layer: abnormal pipeline teardown). The aggregator **ignores SIGINT**: a terminal ^C reaches the whole foreground group, and the parent's ENDED path is authoritative. Drain timeout is detected by consumers as - `state == COMPLETE and n_pending_tasks > 0` — `run_benchmark` fails the run on - it (artifacts written with `complete: false`, then non-zero exit); interrupted - runs are detected as `state == INTERRUPTED` directly. + `state == COMPLETE and n_pending_tasks > 0`. `n_pending_tasks` counts the + client-basis tokenizer work. With a client primary `run_benchmark` fails the + run on it (artifacts written with `complete: false`, then non-zero exit). With + a server primary a drain timeout leaves the run passing and marks the + client-basis blocks partial (`Report.client_token_tasks_pending`, a logged + warning); there a tracked sample without server usage fails the run instead + (see **Token count source**). Interrupted runs are detected as + `state == INTERRUPTED` directly. - **Final delivery is dual-path with separated concerns**: `publish_final` atomically writes `final_snapshot.json` (`tmp + fsync(file) + rename + fsync(parent_dir)`) — this is the **primary** @@ -172,6 +177,63 @@ working in this layer: `metrics/early_stopping.py`; post-hoc recomputation from any run's `events.jsonl` via `scripts/early_stopping_estimate_from_events.py`. See docs/early_stopping.md. +- **Token count source**: a run has one primary basis, `server` (the endpoint's + per-request `usage`) or `client` (the local tokenizer). The primary basis + keeps the base names (the headline `isl`, `osl` and `tpot_ns` series; the + `tpot`, `input_sequence_lengths`, ... report keys) and feeds TPS and the + early-stopping and steady-state inputs. The other basis is recorded beside it + under a `_` suffix from `basis_key(base, basis, *, primary)` + (`core/types.py`): `isl_server`, `osl_server`, `tpot_ns_server` beside a + client primary, `isl_client`, `osl_client`, `tpot_ns_client` beside a server + primary. `session.py` lifts + `QueryResult.metadata["prompt_tokens"/"completion_tokens"]` onto + `EventRecord.prompt_tokens`/`completion_tokens` on COMPLETE (`UNSET` when + absent, as in archived logs). The server trigger in `metrics_table.py` + (`ServerUsageTrigger`) classifies the usage once per COMPLETE + (`classify_server_usage`) and records synchronously with no tokenizer, so it + adds nothing to `n_pending_tasks`. Server TPOT is + `(complete − first chunk) / (completion_tokens − 1)`, LoadGen's rule. No + server count series records a value below 1: server OSL skips 0, and server + TPOT skips `completion_tokens <= 1` and negative windows (a zero window + records 0, as on the client basis). + + The tokenizer triggers record into the client-basis series whenever a + tokenizer is configured (`--no-metrics-isl` drops ISL on both bases; only the + primary basis's TPOT is a tail-latency series). The `osl_paired_samples`, + `osl_paired_client_tokens` and `osl_paired_server_tokens` counters total both + OSLs over the samples counted on both bases (`Report.osl_paired`). On either + primary a tracked, successful sample whose COMPLETE carries no usable usage + (no usage, or 0 `completion_tokens` beside non-empty output) increments + `tracked_samples_missing_usage` (and + `tracked_samples_missing_usage_empty_output` when it has no usage and empty + output). The post-run counts (full-run OSL, accuracy OSL, the standalone + replays) apply the same rule. With a server primary any such sample makes + `Report.complete` false, and `run_benchmark` raises `ExecutionError` (exit 4) + with the cause-aware message of `Report.incomplete_reason()`; with a client + primary it only marks the server-basis blocks partial. + + `--token-count-source {server,client}` (`settings.token_count_source`, + `TokenCountSource` in `core/types.py`) is resolved once by + `commands/benchmark/pipeline.py:resolve_token_count_source`, the only reader + of the setting; api_type `videogen` resolves to `client`. The result lives on + `BenchmarkContext.token_count_source`, goes to the aggregator as + `--token-count-source`, and is recorded as the top-level `token_count_source` + of `result_summary.json` and `accuracy/accuracy_results.json`. Asking for + usage is a run setting, `settings.client.stream_include_usage` (on by default; + `--no-stream-include-usage` opts out), passed to every adapter as + `encode_query(query, *, include_usage=)`; the OpenAI adapters send + `stream_options: {"include_usage": true}` only on streaming requests with at + most one choice. A server primary rejects the opt-out on a streaming OpenAI + run and fails setup on a row with `n > 1` (a client primary logs a warning + instead). Standalone post-run tools (the steady-state replay CLI and the + early-stopping script) take the primary from + `metrics/token_count_basis.py:token_count_source_for_run(run_dir, *, summary_path=None)`: + the top-level `token_count_source` of the summary (`summary_path`, by default + `performance/result_summary.json` under `run_dir`), else `config.yaml`; a run + that records neither (archived runs) means `client`. Both go through + `replay_basis(run_dir, *, override, summary_path=None)`, which pairs that + primary with the basis to replay on (`--token-count-source` when given). + - **Steady-state detection (opt-in, off by default)**: Enable with `settings.steady_state.enabled` / `--steady-state`. The parent resolves a workload profile (`commands/benchmark/pipeline.py:steady_state_profile`) and @@ -180,37 +242,45 @@ working in this layer: The aggregator feeds a `SuperPassCollector` (`metrics/steady_state_diagnostics.py`) from the sample events it already - routes. It reuses `TpotTrigger`'s token counts, so nothing is tokenized twice. - Per-sample series are `array("d")`, ~34 bytes/sample. Bucket size arrives on - `PHASE_START` (`PhaseData.num_turns`) and is latched while the series is - empty. + routes. It reuses the headline TPOT token counts, so nothing is tokenized + twice: in server mode it is fed by `ServerUsageTrigger` with count + `completion_tokens − 1`, in client mode by `TpotTrigger`. Per-sample series + are `array("d")`, ~34 bytes/sample. Bucket size arrives on `PHASE_START` + (`PhaseData.num_turns`) and is latched while the series is empty. The collector holds no per-sample state. The super-pass index lives on - `SampleRow.sp_index`; the standalone parse keeps its own `_PendingRow` because - it has no table. This avoids holding the same in-flight sample in two uuid - maps. + `SampleRow.superpass_index`; the standalone parse keeps its own `_PendingRow` + because it has no table. This avoids holding the same in-flight sample in two + uuid maps. Two ordering rules preserve the index. First, the router reads the row **before** `table.set_field`, because COMPLETE drops it from the table and RECV_FIRST overwrites `recv_first_ns`. That overwrite is how a retry's second - first-chunk is told apart. Second, `TpotTrigger` reads `row.sp_index` **at - fire time** and carries the int in the recorder closure. The count arrives at - the drain flush, long after the row is gone. + first-chunk is told apart. Second, the client-mode `TpotTrigger` reads + `row.superpass_index` **at fire time** and carries the int in the recorder + closure. The count arrives at the drain flush, long after the row is gone. + `ServerUsageTrigger` has no such lag: it runs inside `set_field(COMPLETE)` + while the row is still in the table, reads `row.superpass_index` directly and + feeds the collector synchronously. `ISSUED` is also gated on `table.is_tracking`, because `set_field` returns early outside the window without dropping the row. The verdict is computed after token drain. It rides `MetricsSnapshot.steady_state` into the single `final_snapshot.json` write, then onto `Report.steady_state`. - The series describes only a run with terminal state `COMPLETE` and - `n_pending_tasks == 0`. This path is best-effort: no failure in steady-state - detection can fail a run. + The series describes only a run with terminal state `COMPLETE` whose TPOT and + OSL counts are whole. In server mode the verdict is withheld when any tracked + sample lacks usage (`tracked_samples_missing_usage > 0`); a pending client + tokenizer drain does not withhold it, since those counts never feed the + series. In client mode it is withheld on `n_pending_tasks > 0`. This path is + best-effort: no failure in steady-state detection can fail a run. Eligibility is read from `Profile.supported`; offline and agentic are `False`, and an unclassified load pattern resolves to no profile. This keeps the parent's gate and the child's profile selection in sync. The gate also - requires streaming, because without `TpotTrigger` every verdict would be - `found: false`; a resolved tokenizer; and a non-accuracy-only run. + requires streaming, because without a TPOT trigger every verdict would be + `found: false`; a resolved tokenizer **or** server token-count mode (server + TPOT needs none); and a non-accuracy-only run. Agentic is refused twice: by load-pattern name in the parent gate and by `Profile.supported` in the aggregator. Agentic runs do not get a steady-state @@ -314,8 +384,9 @@ docs/compliance_audit_plan.md ("Run ordering"). `run_audit`: 2. Runs each `AuditRunSpec` phase (from `AuditTest.plan_runs`) back-to-back under its own `/