add the prime-runs SDK - #856
Open
kcoopermiller wants to merge 29 commits into
Open
Conversation
prime-evals models a run as three stateless calls over Dict[str, Any], so
neither of its intended consumers uses it on their main path: verifiers
reimplemented create -> batch -> finalize inline in v1/utils/platform.py, and
prime-rl hits a different API family entirely from utils/monitor/prime.py.
Producers have objects, a long-running loop, steps, ranks, forks and crashes;
each of them solved that privately, twice.
prime-runs makes the run an object instead:
run = pr.init(name=..., environments=["gsm8k"], model=..., framework=...)
run.log_traces([episode]); run.log({"reward": r}, step=step)
run.finish(summary=pr.metrics.from_episodes(episodes))
Identity: init() is called before rollouts start and the ID it returns is the
run ID everywhere, local archive included. Producers already stamp the run onto
their traces, so nothing is re-stamped and no producer record is rewritten. The
join key is run.id inside the trace document -- an indexed ClickHouse column
with a delete-by-run path -- not an upload-scoped context key.
Backends and sinks are independent axes. Backends own lifecycle (EvalsBackend,
OfflineBackend); sinks own transport (TracesSink, plus EvalSamplesSink for the
viewer's flat table). Both sinks run during the transition, because Prime
Traces is gated to an account allowlist and a traces-only client would leave
everyone else with an empty dashboard. When the Viewer API reads traces
natively, the default sink list drops one entry and no producer changes.
The SDK owns the operational work: streaming instead of buffering, a bounded
upload queue that drops and counts rather than stalling a training run, fork
safety via register_at_fork, contained errors (on_error="warn" by default),
terminal status through the context manager / atexit / signals, rank awareness,
and an offline mode that is a real run -- which is what lets producers delete
their --no-push branching.
trace_to_sample / build_samples move here from verifiers: it is knowledge about
a platform wire format, and prime-rl currently reaches across a repo boundary
to import it from a module path that has already drifted.
Known platform gap: there is no producer-facing way to mark an evaluation
failed (finalize only goes PROCESSING -> COMPLETED, UpdateEvaluationRequest has
no status). EvalsBackend calls the status endpoint it needs, latches on 404 so
it probes once, and falls back to recording the terminal state in metadata
while warning that the run will keep showing as running. The fallback stops
firing on its own once the endpoint ships.
Leaf package by construction -- httpx, pydantic, tenacity, prime-traces and
nothing else -- because the prime CLI depends on verifiers, so verifiers can
never depend on prime. verifiers already takes prime-tunnel and
prime-sandboxes on the same terms.
107 tests, hermetic (httpx.MockTransport + tmp dirs).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kcoopermiller
requested review from
DamianB-BitFlipper,
JannikSt,
JohannesHa,
burnpiro,
kennethnym and
willccbb
as code owners
August 19, 2026 20:53
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0dcb62a816
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
All six were real. Verified against the backend where the claim depended on
service behaviour.
Run identity (high). init() exported PRIME_RUN_ID and then read it back, so a
second init() in the same process silently attached to the first run and never
created or finalized one of its own. Exports now record the exporting PID, and
only an ID from a *different* PID counts as inherited — a forked child sees the
parent's PID and joins, a re-init sees its own and opens a fresh run. Lifecycle
ownership follows intent rather than "was an ID present": an explicit id= is a
deliberate resume and finalizes, an ID picked up from the environment belongs
to whoever exported it. finish() also stops advertising the run it owned.
Fork safety (high). The child inherited the parent's httpx pools and buffered
file handles. Two processes writing one socket interleave into a single HTTP
stream, and a duplicated write buffer gets flushed twice. Connection and file
holders now reset in the child through a single process-wide hook in _fork.py:
dropped, never closed (closing sends close_notify down a socket the parent is
still reading) and never flushed (the buffer holds records the parent will
write itself). The hook is registered once rather than per instance, because
os.register_at_fork cannot be undone — the old per-worker registration pinned
every run the process ever opened.
Metadata replacement (medium). The service writes metadata with
{"$set": {"metadata": ...}}, a document-level replace. The failure fallback
PUT carried only {"prime_runs": ...}, erasing the config finish() had just
written. finalize() now receives the run's full config and merges into it.
Abandoned uploads (medium). finish() waited 60s for a flush, ignored the
result, then close() joined 30s — while a single sample POST is allowed 300s.
The budget is now derived from the upload timeout, a flush that does not drain
warns instead of passing silently, and close() leaves sinks open when the
thread is still alive rather than pulling a client out from under a live
request.
Signal status (medium). The handler reported FAILED while atexit, RunStatus and
the README all said CRASHED. Signals and Ctrl-C now report CRASHED: the
producer never said the run failed, it was stopped from outside its control
flow. FAILED stays for what the producer itself reports.
Environment version pinning (medium). EnvironmentRef accepted version_id and
dropped it, even though the API's EnvironmentReference carries it — the run
attached to whatever version the hub resolved that day.
Found while testing the signal fix: _handle_signal read the displaced handler
*after* finish(), which restores and clears that table, so chaining always fell
back to SIG_DFL — re-raising the signal at default disposition and killing the
process instead of running the handler the application installed. Captured
before finish() now.
122 tests (up from 107), including a real os.fork() end-to-end check that the
child joins the parent's run and no record is written twice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three findings the Bugbot pass did not cover. The other two Codex raised (environment version pinning, PRIME_RUN_ID re-init) were already fixed in 14bbac5, including its note that `setdefault` left descendants pointing at a stale ID when an explicit `id=` was supplied — `_announce` now assigns unconditionally. Non-idempotent retries. The client retried every method through ambiguous failures, so a lost response to POST /evaluations/ would create a second evaluation that the SDK never tracked, leaving an orphaned duplicate run. The module docstring even asserted this was safe "because run creation happens once per init()", which confuses the call site with the retry loop inside it. Retry safety is now decided per call. A failure is ambiguous when the request may already have been processed (gateway 502/504, read timeout, stream broken after the bytes went out); unambiguous when nothing reached the server (connect failure, 429 refused before any work). Unambiguous failures replay for every method. Ambiguous ones replay only when the caller declares `idempotent=True`, which defaults to `method != "POST"`. Run creation and sample appends keep the default; get-or-create, finalize and status writes declare themselves safe. Same classification prime-traces' client already uses. This also stops the samples sink duplicating rows on a lost response — duplicates silently skew every average on the dashboard, where a lost batch is at least recoverable. Uploader failures under on_error="raise". A sink fails on the uploader thread, where the raise went straight into the worker's own except and was discarded — so flush() and finish() returned success while records were being dropped, in exactly the mode documented as being for "tests and CI, where a silent upload failure is the bug". The failure is now held and re-raised at the next synchronization point the caller controls: flush(), or the very end of finish() so the run is still closed out properly first. The atexit and signal paths swallow it, since neither is a place to surface an exception. Signal handlers were never restored. `_restore_signal_handlers` compared `signal.getsignal(signum) is self._handle_signal`, but every access to a bound method builds a new object, so the identity check could never match. Handlers stayed installed for the life of the process: the finished Run was pinned, and the next run in that process saw a non-default handler and declined to install its own, leaving it unable to report signal termination. The bound method is created once in __init__ and compared against. 133 tests (up from 122). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two follow-up findings, both on the previous rounds' fixes. Offline records were buffered. reset_after_fork() dropped the inherited file handles, but on CPython the last reference going away closes them, and close() flushes — writing out the child's copy of the parent's buffer and duplicating every record still sitting in it. Dropping without closing is not expressible for a buffered writer, so the buffer is gone instead: records are written to an unbuffered append-mode handle, encoded here. Nothing is ever held in process memory, so a fork has nothing to copy and flush() has nothing to do. The earlier fork test passed on this path by luck. It forked before the uploader thread had opened the file, so no handle and no buffer existed in the child. Replaced with an assertion that records are readable through a separate handle with no flush and no close, which is what actually pins the property. Transient sample failures retired the sink. Making sample POSTs non-replayable (332fc21) was right on its own, but combined with the worker disabling a sink on any raise it meant a single 502 stopped every later batch — so one gateway blip could leave the rest of a run missing from the dashboard of exactly the accounts the v0 sample table exists to serve. That trade is worse than the duplicates it avoids. The worker now separates "this batch failed" from "this sink is finished". A permanent failure (gated account, rejected credential) will fail identically forever and still retires the sink immediately. A transient one gets three consecutive strikes, reset by any success, so a blip costs one batch and a sustained outage still stops the SDK re-attempting for hours. Dropped records are counted either way, so run.dropped_records reflects the loss. This applies to the traces sink too, which had the same all-or-nothing behaviour. 138 tests (up from 133). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit folded failed sink writes into `dropped`, which is documented and warned about as queue backpressure. Two things went wrong with that. A default online run writes to both the traces sink and the sample table, so one failed batch was counted twice. And a failure on one sink was counted at all even when the other sink stored the records, so `finish()` could warn about data missing from a run that has all of it. They are different losses and stay separate now. `dropped` counts records that reached no sink because the queue was full — the producer outran the uploader, and those records are stored nowhere. `failed_records` counts per sink, exposed as a mapping rather than a total, because summing it would recreate exactly the overstatement above. The finish warnings are phrased to match: one about records that reached nothing, one per sink about what that sink could not store. 139 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five points where prime-runs diverged from prime-traces, prime-evals, prime-sandboxes and prime-tunnel without meaning to: - Normalize an explicitly passed `base_url`. `Config` strips a trailing `/api/v1`; the `PlatformClient` constructor did not, so `pr.init( base_url=".../api/v1")` requested `/api/v1/api/v1` while the identical value in `PRIME_API_BASE_URL` worked. prime-traces carries the same helper for the same reason. - Map 403 to a typed `ForbiddenError`, matching `prime_traces.ForbiddenError` — which the traces sink already branches on for beta gating, while a 403 from the platform API collapsed into a generic `RunAPIError`. Behavior is unchanged (it was already classified permanent); callers can now branch. - Drop `pydantic` and `tenacity`. Neither was imported: this package models no response bodies and hand-rolls its retry loop. They do not belong in the dependency tree of a leaf package that lands inside verifiers. - Add the LICENSE file the other packages ship. - Declare 3.13, which CI has been testing all along. Also documents the three departures that are deliberate — private client instead of `core/`, dataclasses instead of pydantic models, no async client — and what to do about the three blocking calls when driving a run from async code, since every sibling SDK ships an async client and this one does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The platform's Config tab is unusable for both run kinds, and neither cause is the platform's: eval runs store a v0 blob of four keys, training runs store a fully-resolved dump where three chosen values sit under hundreds of defaults nobody picked. Both producers are now launched from one user-authored file — `uv run eval @ eval.toml`, `uv run rl @ train.toml` — and nothing captured it. Two changes, matching the two failures: `config_source=` takes the path to that file and stores it byte for byte, comments and section grouping intact. It rides inside the run's config under a reserved key, so every write that already carries the config carries the source too — create, the periodic update, finalize, the failure fallback, and the offline archive — with no extra plumbing and no chance of one path forgetting it. A str or Path is always a path, never inline text: guessing between them would turn a mistyped filename into a run whose config tab displays the filename. `config=` now accepts a pydantic model and dumps it with `exclude_unset=True`, so only fields somebody actually set are recorded. A mapping is still stored exactly as given, and a caller who wants every resolved default can still pass `cfg.model_dump()`. The asymmetry is deliberate — the shorter call should give the more useful answer. Note this is only the upload half. Both Config tabs are derived projections today (the eval tab filters through a 31-key allowlist, the training tab reconstructs TOML from stored fields), so rendering `metadata.config_source` verbatim is a separate frontend change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
config_source= was a second way to say the same thing. Collapse it into
config=, which now takes whichever form the caller has — the path to the
file the run was launched from, a mapping, or a pydantic model — the same
polymorphism environments= already has for slugs, dicts and
EnvironmentRefs.
The forms are distinguished by type, never by inspecting keys, so a
config that happens to carry a `text` field is not mistaken for a launch
file. Storage is unchanged: a path still lands under the reserved
config_source key inside the run's config, which stays the single key a
config-tab renderer has to know about.
A run launched from a file that also wants a derived value adds it with
run.update_config({...}) rather than the SDK carrying a second argument
for the uncommon case.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four bits of surface that were each a second way to say something the SDK already had a first way to say. dataset= is always the environment under a different name, and no UI reads the column. The API field is still populated, derived from the first environment — producers just no longer repeat themselves to fill it. summary= on init() asked for a run's *outputs* at the moment it opens, before it has any. finish(summary=) and run.summary already cover the real case, so RunSpec loses the field too and create() stops sending an empty metrics blob. sinks= was a third way to configure transport next to traces= and samples=, with no caller outside one of my own tests. Run(sinks=) stays, so tests still inject fakes and a future custom sink has somewhere to land. Re-adding a keyword argument is non-breaking; removing one is not, which is the argument for cutting it now rather than later. log_episodes was a third name for log_traces. log_samples stays — it is the method name prime-rl's Monitor ABC already uses, so the adapter it exists for is real. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_normalize_config guessed why model_dump() failed: any TypeError was read as "this callable has a different signature" and retried bare. Inferring a cause from an exception type is the problem, because the only recovery available — dumping every field, defaults included — is exactly the outcome passing a model was meant to avoid. A wrong guess therefore degrades silently, in the one direction that matters. Now the signature is inspected before the call, so a failure during serialization surfaces as itself, and the fallback fires only when the keywords genuinely are not accepted. When it does fire it logs, because silently recording a hundred defaults nobody chose is the behaviour this path exists to prevent. Note for the record: Bugbot reported this as PydanticSerializationError being caught by `except TypeError`. That specific claim is wrong — PydanticSerializationError subclasses ValueError, and pydantic wraps even a serializer's own TypeError into it, so the handler could only ever catch a signature mismatch. The underlying concern about guessing was still worth acting on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Design review found ~40% of the package serving scenarios with no backend
and no consumer: training runs, multi-rank lifecycle ownership, PRIME_RUN_ID
join, signal handling, a probe for a /status endpoint the platform does not
have, and a dict-based reimplementation of the v0 projection. verifiers —
the one producer — uses seven init() arguments, log_traces() and finish().
Removed:
- training / multi-rank / join machinery: RANK_ENV_VARS, is_primary,
PRIME_RUN_ID publish/retract, id= resume, attach(), log()/log_samples()/
update_config()/commit=, MetricItem/RunUpdateItem, supports_step_metrics,
the summary timer, RunKind/kind, RunHandle.raw
- signal handling (install/relinquish/chain protocol, _pending_signal);
atexit and the context manager still report CRASHED/FAILED
- the /evaluations/{id}/status probe; finalize() writes metadata.prime_runs
directly for non-COMPLETED runs
- projection.py's serialized-mapping path and the bare-Trace branch of
EvalSamplesSink; the sink takes Episode objects or v0 sample dicts
- init() params kind, id, traces_url, traces, samples, handle_signals,
queue_size, finish_timeout, and the pydantic-model form of config=
- _build(); the second PlatformClient (backend and samples sink share one)
- 20 names from __all__ (Backend, Sink, RunSpec, RunHandle, sink and
backend classes, RUN_ID_ENV)
Consolidated: is_episode/stamp_run live once in sinks/base.py; line_format
and step no longer thread through log_traces -> WriteItem -> Sink.write.
Docstrings trimmed to contracts; README rewritten for the reduced surface.
Kept offline mode, the fork hooks and the prime_runs-local HTTP/config/
exception plumbing — each is a separate discussion.
Source 4,105 -> 2,524 lines; tests 3,201 -> 2,492 (165 passing); init()
21 -> 13 parameters.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4S4DNVRX5BNtsLva7TZz1
exceptions.py re-exports the prime_traces API error family and keeps only the SDK-local errors (PrimeRunsError, ConfigurationError, EnvironmentResolutionError, RunFinishedError). RunAPIError is gone: it was prime_traces.APIError under another name. is_transient and the traces sink's gating check lose their try-import dance since there is one family. config.Config subclasses prime_traces.core.Config and adds frontend_url. _http.PlatformClient uses raise_for_response, retry_delay and AMBIGUOUS_TRANSPORT_ERRORS from prime_traces.core.client; the local error mapping, Retry-After parsing and backoff ladder are deleted. What stays local is the per-call idempotent= replay policy, encode_json and the non-JSON-body guard. Backoff is now the shared jittered schedule. Everything used is in the released prime-traces 0.0.2; the pin is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4S4DNVRX5BNtsLva7TZz1
Nothing consumed it: verifiers picks online or disabled. There was no sync path, and building one collides with the identity design — records carry the local run.id, which is the join key the traces service indexes, so a later upload would have to rewrite every record or the platform would have to accept client-issued IDs. If air-gapped evals are ever wanted, this comes back together with the sync command and that decision. Removes OfflineBackend, OfflineSink, dir=, PRIME_RUNS_DIR and mode="offline". A missing API key now resolves to disabled with a warning that the run will not be tracked, rather than silently writing a ./prime-runs/ directory into the cwd. Tests that used offline as a cheap real backend now use disabled or the online MockTransport fixture. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4S4DNVRX5BNtsLva7TZz1
Prime Traces is gated to an owner allowlist; in production that is three internal teams. For everyone else the traces sink's first upload returns 403 service_not_enabled, and the sink re-raised it so the worker counted the batch as lost: three warnings per run, "N failed via traces" in the verifiers footer, and a ForbiddenError out of finish() under on_error="raise" — for a run that stored everything it was asked to. The sink now distinguishes the two 403s. service_not_enabled retires the sink at INFO and returns: nothing was lost, there was never anywhere for the records to go. forbidden (a token without the traces scope) is something the caller can fix, so it still disables the sink and raises for loss accounting. A 403 with no recognised code is treated as the latter, so the failure mode of being wrong is the old behaviour, not silent loss. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4S4DNVRX5BNtsLva7TZz1
…caffolding The backends/ package was shaped for two implementations; after the offline-mode cut it held one. Backend (protocol), EvalsBackend and the disabled no-op now share backend.py, next to the contract they satisfy. Also removed: @runtime_checkable on both protocols (no isinstance check anywhere), the normalize_base_url helper (inlined into PlatformClient — the prime_traces equivalent is private), and the top-level build_samples / trace_to_sample aliases (prime_runs.projection is the spelling verifiers already uses). The pyproject dependency comment no longer claims a hand-rolled retry loop. No behaviour change; 161 tests unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4S4DNVRX5BNtsLva7TZz1
kcoopermiller
changed the title
kcoopermiller
changed the title
…to a retired sink Both found by the first live e2e (verifiers cooper/prime-runs -> prod). 1. The traces service derives run_id from trace.run.id only and never reads the episode envelope's run (ingestion/extract.py); verifiers records the run on the Episode and its Trace has no run field. The sink passed producer episodes through untouched, so every row of an episode upload landed with an empty run_id — unqueryable by run. stamp_run now also stamps members that lack a run, and TracesSink runs producer objects through to_record() itself so the members are reachable (same bytes the transport would have produced). 2. After a sink was retired by an error, later batches skipped it with no accounting: five episodes lost to the traces sink reported as "1 failed via traces". The worker now counts records that skip a sink it retired. A sink that switches itself off without raising (service_not_enabled) is still not counted — nothing was lost. Verified: run lkcw5sfzgb6jlqli6qnpggns, 5/5 traces and 5/5 episodes carry run_id on prime-traces.pintel.dev. 166 tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4S4DNVRX5BNtsLva7TZz1
kennethnym
previously approved these changes
Aug 22, 2026
kennethnym
left a comment
kennethnym
left a comment
There was a problem hiding this comment.
i don't have any blocking concern. the code can be tightened up though, but the api feels directionally right. the only debate i have is whether we should also have a run.log_episodes counterpart to run.log_traces
| self._owns_lifecycle = True | ||
|
|
||
| self.config: Dict[str, Any] = dict(spec.config) | ||
| self.summary: Dict[str, Any] = {} |
There was a problem hiding this comment.
maybe expose summary update as Run.update_summary? also i wonder if we should export a typed summary dict
…ate_summary Review by kennethnym on #856: - UploadWorker takes Sequence[Sink] instead of List[Any]; the getattr fallbacks for `name` / `enabled` go with it (worker, run). - The traces sink imports ErrorCode / LineFormat / TracesClient at the top; prime-traces is a hard dependency and exceptions.py already imports it eagerly, so the lazy imports and the ImportError branch were dead. - Run.log_episodes() as the counterpart to log_traces(); both share one submit path, the sink still infers the line format. verifiers' call sites pass episodes, so they move to log_episodes. - Run.update_summary() merges outputs before finish() with the same non-finite filtering; metrics.from_episodes() returns a RunSummary TypedDict naming the three keys the dashboard reads. 169 tests; ty back at its 15-diagnostic baseline (FakeSink's start() now takes Mapping, matching the protocol). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4S4DNVRX5BNtsLva7TZz1
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 34faa4a. Configure here.
| ``traces`` list. The episode's ``run`` reaches every member trace. | ||
| """ | ||
| self._require_live("log_episodes") | ||
| self._submit(episodes) |
There was a problem hiding this comment.
Episode mappings break samples sink
Medium Severity
log_episodes documents accepting plain JSON mappings with a traces list, but EvalSamplesSink only projects non-mapping episode objects (or sample_id dicts). Those mappings raise TypeError, and the worker retires the samples sink for the rest of the run. On accounts outside the traces beta that leaves the viewer with no samples, while the same call with episode objects would have succeeded.
Reviewed by Cursor Bugbot for commit 34faa4a. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Eval jobs need one call that opens a run on the platform, uploads the config they were actually launched with, streams traces while they execute, and marks the run finished, failed, or crashed when they stop. verifiers hand-rolls that lifecycle against the raw API today — which is why the Config tab shows a four-key v0 blob instead of anyone's real
eval.toml.What
A new leaf package,
prime-runs(httpx+prime-traces, nothing else), where the run is an object:Runexposesid,url,log_episodes()/log_traces(),update_summary(),flush(),finish(),fail(), a context manager, and loss counters (dropped_records,failed_records,errors).Every record is written to two places during the transition: Prime Traces (native format, the system of record) and the v0 sample table, which is what the dashboard's trace viewer still reads. Once the viewer reads traces natively, the sample sink is one line to drop and no producer changes.
Verifiers integration: verifiers#2415.
Known gaps
traces_urlfalls back tobase_url(api.primeintellect.ai), which does not route/api/v1/traces; prod traces is only onprime-traces.pintel.dev.finalizeonly moves PROCESSING → COMPLETED. The SDK records status/error undermetadata.prime_runsand warns; a platform status endpoint is the real fix.config_sourcewithout a frontend change (evaluationConfig.tsfilters through a hard-coded allowlist). This PR ships the upload half.Note
Medium Risk
New authenticated client that creates evaluations and dual-writes traces/samples, plus a PyPI release workflow. Isolated as a leaf package with hermetic tests, but retry/idempotency and lifecycle semantics affect live eval data.
Overview
Adds
prime-runs, a leaf SDK (httpx+prime-tracesonly) so producers caninit()an eval run, stream traces/episodes on a background uploader, andfinish()/fail()with dashboard summaries.init()resolves environments via the hub, stores launch config verbatim underconfig_source, and returns a handle withid/url. Online runs dual-write to Prime Traces (system of record; quietly off if the account is not on the allowlist) and the v0 sample table the current viewer still reads. v0 projection andmetrics.from_episodeslive here so verifiers does not need to own the wire format.Default
on_error="warn"keeps platform errors out of producer loops; uploads use a bounded queue with drop counters, fork-safe transports, and careful non-replay of non-idempotent POSTs. Failed/crashed evals are recorded inmetadata.prime_runsbecause the evaluations API cannot mark FAILED.CI now tests the package on 3.11–3.13, version-bump checks include it, and
release-runs.ymlpublishes to PyPI.Reviewed by Cursor Bugbot for commit 34faa4a. Bugbot is set up for automated code reviews on this repo. Configure here.