Skip to content

Publish a run's logs, whole-run node metrics and specs to the dashboard - #462

Merged
radiken merged 1 commit into
masterfrom
alan/dashboard-coverage
Nov 18, 2026
Merged

radiken merged 1 commit into
masterfrom
alan/dashboard-coverage

Conversation

@radiken

@radiken radiken commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Each run now also stores the nodes' metrics from every statefulset, container CPU, memory and network, and the pods' host and start time, per pod at 30 s over the whole run. It also stores the run's sends and deliveries (nim-libp2p, checked against the run's CSVs) and its events and deployed specs. Metrics that every pod reports with the same labels and value for the whole run go into the metadata instead (UNCHANGED_RULE). A 409 now checks the stored run and raises if it's short, but never deletes it, since it may still be processing or hold data VM no longer has; a publish only deletes what it created. About 300 MB as JSON for a 1k-node run.

@radiken radiken self-assigned this Nov 17, 2026
@radiken radiken added the enhancement New feature or request label Nov 17, 2026
@radiken radiken linked an issue Nov 17, 2026 that may be closed by this pull request
@radiken
radiken marked this pull request as ready for review November 17, 2026 12:07
@radiken
radiken force-pushed the alan/dashboard-fetch branch from b1ba943 to d85e33c Compare November 17, 2026 12:09
@radiken
radiken force-pushed the alan/dashboard-coverage branch from 78bc597 to 99fc708 Compare November 17, 2026 12:09
Base automatically changed from alan/dashboard-fetch to master November 17, 2026 19:58
… dashboard

Besides the 13 stable-window metrics, each run stores every metric the
nodes' own scrape job reported on any statefulset's first pod, plus
container CPU, throttling, memory and network and the pods' host and start
time, per pod at 30 s over the whole run. nim-libp2p runs also store their
sends and deliveries (the dashboard parses only that tracer), checked
against the run's summary CSVs; metadata adds the events and deployed specs.

A metric that every pod reports with the same labels and value for the
whole run is recorded in metadata.unchanged with the rule, not stored;
fetch.run_dataset returns it as one row per pod.

A publish deletes only an experiment it created and could not check. On a
409 the stored run is checked and an error raised if it is short, but it is
never deleted: it may still be processing, or hold data VM no longer has.
The loader reads datasets one at a time instead of all at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@radiken
radiken force-pushed the alan/dashboard-coverage branch from 99fc708 to 3220ef0 Compare November 18, 2026 01:20
@radiken
radiken merged commit 7e47d78 into master Nov 18, 2026
2 checks passed
@radiken
radiken deleted the alan/dashboard-coverage branch November 18, 2026 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

General tooling (recurring)

2 participants