Skip to content

Report: per-submission breakdown of measurements and checks #231

Description

@Unisay

As a report consumer, for each submission in a scenario's list I want to see a breakdown of its individual evaluations (measurements and checks) with their per-evaluation cost, without leaving the report.

Context

On a scenario report page each submission is a single aggregate bar per metric tab. Since #229 that aggregate covers the accepting measurements only; the remaining evaluations (failure-path checks and any pending measurements) are summarised in a footnote that ends with "see each submission's metrics.json for their cost". That footnote is where the UI currently gives up: to answer "what does this submission's double-satisfaction check cost" or "which evaluations were excluded", the reader has to leave the report and open the raw JSON.

The data needed to answer those questions in the report already exists. Each submission's metrics.json carries an evaluations[] array, and since #229 every entry has an included_in_aggregates flag plus its own cpu_units and memory_units. Nothing needs re-measuring.

Proposal

Give each submission its own detail page at its own URL, linked from the scenario page's submission row. The detail page lists every evaluation from that submission's metrics.json: the evaluation name, its CPU and memory units, and whether it feeds the aggregates. Measurements and checks are distinguished, and the excluded ones are the same set the footnote counts.

The scenario (aggregate) page stays a light comparison view: it does not embed per-evaluation data, it only links out to the detail pages. Moving the breakdown to a separate file per submission is what keeps the aggregate page light, rather than inlining every submission's evaluation list into it.

Acceptance criteria

  • Each submission has its own detail page at its own URL, reachable from the scenario page's submission row.
  • The detail page lists every evaluation by name, with its stored cpu_units and memory_units and its included_in_aggregates status. Measurements (included) and checks (excluded) are visually distinct, and the number of excluded rows reconciles with the footnote's count.
  • The aggregate scenario page does not embed per-evaluation data; it only links to the detail pages, so its weight does not grow with the number of evaluations.
  • The breakdown is sourced from each submission's metrics.json evaluations[]; no re-measuring and no new metric fields.

Implementation notes

  • Per-evaluation numbers are the stored cpu_units and memory_units. A derived per-evaluation fee is out of scope for now: fee is computed from protocol parameters against the aggregate and is not stored per evaluation.
  • URL and file layout are the implementer's choice, following the report's existing conventions (something along the lines of report/benchmarks/<scenario>/<submission_dir>.html, with the breadcrumb extended to the submission).
  • The generator emits one detail page per submission, reading that submission's evaluations[] from its metrics.json. The positional aggregate CSV that feeds the scenario page is not a fit for a variable-length evaluation list, so the detail pages read metrics.json directly.

Scope

Report pipeline only: scripts/cape-subcommands/submission/report.sh and the report templates (a new per-submission detail template alongside benchmark.html.tmpl). Depends on #229 (merged), which introduced the measurements/checks split and the per-evaluation included_in_aggregates flag.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions