Skip to content

Fix FastAPI benchmark: match baseline routing to fastapi-startkit#11

Merged
tmgbedu merged 12 commits into
fastapi-starkitfrom
task/1035-fastapi-benchmark-routing
Jul 15, 2026
Merged

Fix FastAPI benchmark: match baseline routing to fastapi-startkit#11
tmgbedu merged 12 commits into
fastapi-starkitfrom
task/1035-fastapi-benchmark-routing

Conversation

@tmgbedu

@tmgbedu tmgbedu commented Jul 15, 2026

Copy link
Copy Markdown

Problem

The raw-FastAPI baseline (python/fastapi/server.py) registered routes with
direct @app.get/@app.post decorators, while the fastapi-startkit app
registers the same three routes via APIRouter + app.include_router(...).
On FastAPI 0.139, include_router keeps a nested _IncludedRouter node on
app.router.routes, adding a per-request resolution layer that flat
add_api_route registration doesn't pay. So the benchmark was measuring
include_router's routing cost, not fastapi-startkit's framework overhead.

Fix

Register the baseline's three routes on an APIRouter, mounted via
app.include_router(router) — the only remaining difference between the two
apps is now the framework itself.

Route behavior is unchanged and verified:

  • GET /200, empty body
  • GET /user/123200, body 123
  • POST /user200, empty body

Benchmark: before vs after

Method: Docker python:3.14-slim, uvicorn --workers=$(nproc) (11 workers),
oha 1.14 with keepalive + --latency-correction, 8s duration × 4 reps,
concurrency 64 and 256 — run twice per cell (once target-sequential, once
target-interleaved per rep, to average out host-load drift), 8 reps/cell
total. Matched stack both sides: FastAPI 0.139.0 / Starlette 1.3.1
(fastapi-startkit pinned to 0.47.0).

Relative delta (fastapi-startkit vs fastapi baseline; negative = startkit slower):

Route Δ @ c64 before → after Δ @ c256 before → after
GET / −5.8% → −0.4% −9.3% → +0.3%
GET /user/0 −11.7% → −1.0% −6.3% → +8.4%
POST /user −9.8% → +1.4% −10.9% → −1.8%

Average delta across all 6 cells: −8.9% before → +1.1% after. The
framework-level overhead collapses to noise once both sides register routes
identically — confirming the prior gap was routing-mechanism cost
(include_router), not fastapi-startkit overhead.

Caveat: same-host macOS/OrbStack comparison, not official/authoritative
numbers (those require the Linux run.sh harness); host was under mixed
background load during measurement, hence the double run per cell.

tmgbedu and others added 12 commits July 6, 2026 10:14
The fastapi_startkit config pinned language.version: 3.13 while every other
Python framework (fastapi, django, flask, ...) uses the repo-wide default of
3.14. That runtime mismatch made head-to-head comparisons unfair: benchmarking
against fastapi was really comparing Python 3.13 vs 3.14 (and older Starlette),
not the framework itself.

Removing the override lets fastapi_startkit inherit the shared 3.14 default,
so it runs on the same interpreter as fastapi and the rest.

QA evidence:
- Correctness: real .spec/route_spec.rb rspec suite -> 6/6 pass on 3.14.
- Builds cleanly on python:3.14-slim (fastapi-startkit 0.46.0 from PyPI).
- Matched-Python (both 3.14) throughput delta vs fastapi collapses to +/-noise
  around zero, confirming the wrapper adds no meaningful overhead; the earlier
  apparent advantage was solely the 3.13-vs-3.14 skew.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fastapi already inherited the repo-wide 3.14 default, but declaring it
explicitly makes the fastapi vs fastapi_startkit comparison unambiguous:
both configs now state Python 3.14, so any benchmark delta reflects the
frameworks, not the runtime.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pin the exact known-good release so rebuilds resolve a reproducible
dependency instead of drifting within the 0.46.x range.
Stop local Claude Code config (.claude/, .mcp.json) from cluttering git
status. Also ignore the per-run cid-<engine>.txt / ip-<engine>.txt files
the docker harness generates (the existing rules only covered the older
cid.txt / ip.txt names).
Pin fastapi_startkit benchmark to fastapi-startkit==0.46.0
…files

Ignore Claude Code config and per-run container id/ip files
Align fastapi_startkit to Python 3.14 (match other Python frameworks)
The previous range resolved fastapi-startkit to 0.46.x -> FastAPI 0.124.4,
while the fastapi baseline entry runs FastAPI 0.139. Pin to 0.47.0 so both
Python entries benchmark on the same FastAPI/Starlette (0.139.0 / 1.3.1),
making the comparison apples-to-apples. Idiomatic include_router registration
is kept unchanged. route_spec.rb passes 6/6 on the 0.139 Docker build.
Keep the benchmark's reported version label in sync with the 0.47.0 pin.
…-0470

fastapi_startkit: pin to 0.47.0 (FastAPI 0.139) to match baseline
README: fastapi vs fastapi-startkit comparison
The raw-FastAPI baseline registered routes with direct @app.get/@app.post
decorators while the fastapi-startkit app registers the same routes via
APIRouter + include_router. That mismatch meant the benchmark was measuring
include_router's per-request routing cost, not framework overhead.

Register the baseline's routes on an APIRouter mounted with
app.include_router(router) so both sides use the identical routing
mechanism, isolating the comparison to actual framework overhead.
@tmgbedu
tmgbedu merged commit f75a843 into fastapi-starkit Jul 15, 2026

@tmgbedu tmgbedu left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Post-merge review note (agent code-reviewer, task the-benchmarker#1036):

This PR was already merged (merge commit f75a843, merged_at 2026-07-15T19:27:04Z by @tmgbedu) before this review could be submitted — 0 reviews and 0 CI checks (statusCheckRollup) recorded prior to merge, despite .github/workflows/ci.yml being configured to run on pull_request.

Technical assessment performed post-hoc against the merged content on fastapi-starkit:

  1. python/fastapi/server.py: routes now registered on APIRouter() and mounted via app.include_router(router). All 3 routes (GET /, GET /user/{id}, POST /user) preserved with identical handlers/behavior. Correct and matches the stated goal (match fastapi_startkit's include_router usage).
  2. Diff scope: the PR's file list (vs its stale merge-base) also touched .gitignore, python/fastapi/config.yaml, python/fastapi_startkit/config.yaml, python/fastapi_startkit/pyproject.toml — but diffing PR head against the branch tip at merge time shows these were already identical (applied via earlier merged PRs #4/#9/etc.), so the actual merged delta was server.py only. No unrelated/corrupted content (no README changes).
  3. Base branch: confirmed fastapi-starkit is this fork's actual default branch (fastapi-startkit/web-frameworks), not upstream the-benchmarker/web-frameworks. isCrossRepository=false.
  4. Benchmark numbers in the PR description (avg delta -8.9% -> +1.1%) are self-reported/manual (no CI benchmark job ran); methodology is documented but unverified by automation.

No functional objections to the code change itself. Flagging the merge-without-review/CI as a process gap for the team, not a defect in the diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant