Skip to content

Improve benchmark realism for production scenarios #80

Description

@Oaklight

Summary

Current benchmarks focus on mean execution time against reference libraries with synthetic data. While this is a solid baseline, it misses several dimensions that matter in production: tail latency, memory usage, concurrency, data realism, and scalability curves.

This is a tracking issue for a set of improvements to make zerodep benchmarks more representative of real-world workloads.

Current strengths

  • Apple-to-apple comparison against 30+ reference libraries
  • S/M/L data size tiers across most modules
  • Network modules (httpclient, websocket, httpserver) use real local server fixtures
  • readability uses real CNN/BBC/Guardian HTML pages

Gaps vs production reality

Gap Impact Description
Mean-only reporting Low Report shows only mean; min/max/stddev already collected but not displayed
No memory metrics Medium Peak RSS matters in containerized deployments; parsers (yaml, soup, multipart, protobuf) are most affected
No concurrency benchmarks High httpclient tested async but single-request only; no asyncio.gather or thread-pool contention
Synthetic data dominance Medium Most modules use generated data; real configs, dirty HTML, edge-case payloads are absent
No cold start measurement Low First-call overhead (import, regex compile, table init) hidden by warmup rounds
Fixed S/M/L, no scale curve Medium Three data points can't reveal complexity inflection points

Sub-issues

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions