Skip to content

Restricted the slicing performance-over-time plot to the largest slice and added change tables - #231

Open
oruebel wants to merge 3 commits into
mainfrom
slicing_perf_over_time_fix
Open

oruebel wants to merge 3 commits into
mainfrom
slicing_perf_over_time_fix

Conversation

@oruebel

@oruebel oruebel commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • The slicing performance-over-time plot (performance_over_time_remote_slicing.pdf) previously pooled all five slice sizes into each point. This mixed runtimes that differ several-fold, widened the error bars, and made the values incomparable with the method-ranking heatmap, which uses the largest slice. The plot and its summary table now use only the largest slice, and the figure caption says so.
  • generate_figures now also writes performance_over_time_remote_file_reading_changes.csv and performance_over_time_remote_slicing_changes.csv to the summary-tables directory. Each row compares one method and modality between two consecutive environment timepoints, with:
    • mean runtime, absolute change, percent change, and fold change
    • mean downloaded bytes and web packets from the network tracking runs, with their percent changes
    • rows sorted by the absolute log2 fold change, with rows whose runtimes are both below 10 ms (reads served from cache) placed last
  • Bumped the package version to 0.4.4.

These tables back the software-version analysis in the NWB benchmarks paper.

Testing

  • Ran nwb_benchmarks generate_figures --output-dir figures_benchmark --summary-tables-dir figures_benchmark_summary_tables. Apart from the slicing-over-time summary, all existing summary CSVs regenerated byte-identical.
  • Spot-checked the change tables against values computed independently from the per-environment summary tables.
  • pytest tests/test_incremental_visualization.py passes, and so do the pre-commit hooks.

🤖 Generated with Claude Code

oruebel and others added 3 commits November 16, 2026 21:17
…ables

The slicing performance-over-time plot pooled all five slice sizes into each
point, mixing runtimes that differ several-fold and making the values
incomparable with the method-ranking heatmap. It now uses the largest slice.

generate_figures also writes performance_over_<benchmark>_changes.csv tables
comparing each method and modality between consecutive environment
timepoints, with absolute, percent, and fold changes in runtime plus the
corresponding downloaded bytes and web packets from the network tracking
runs. Rows are sorted by the size of the change, with sub-10 ms cached reads
placed last.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Panel titles showed raw facet variables (e.g., "is_preloaded = False |
modality = Ecephys") and the legend title showed the internal column name
"benchmark_name_clean". Panels are now titled by modality, with "first read"
or "repeated read" for the slicing plot, and the legend is titled "Method".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant