Skip to content

docs: record the layered backend - #89

Merged
dimslaev merged 2 commits into
ci/deploy-app-onlyfrom
docs/backend-layers
Oct 19, 2026
Merged

dimslaev merged 2 commits into
ci/deploy-app-onlyfrom
docs/backend-layers

Conversation

@dimslaev

Copy link
Copy Markdown
Owner

Stacked on #88 (step 8 of the backend migration plan, the last one). Docs only, plus comment and docstring path fixes.

What changed

  • New docs/adr/0017-layered-features.md. It records the three layers (entry, features, core), that features never import each other, tables in core/db, shared mechanism flat in core, tests beside their module, the five import-linter contracts, and the deploy shipping only what the box runs. It supersedes ADR 1 and ADR 4, and says what changed in ADR 7 (one engine from Settings, per-source knobs still read os.environ) and ADR 8 (the stage-type chain splits at the queue). ADRs 1, 4, 7 and 8 are unchanged.
  • docs/style.md: the backend rules now describe layers, feature folder names, core/db, flat core, adapters in entry/, and tests beside their module. The old domain-folder, tests/ and pipeline-exception rules are gone. The frontend and repo-wide rules stay, except the two that pointed at ADR 1 and ADR 4.
  • docs/backend.md: setup uses the root .venv (uv workspace), plus a layout section, tests beside their module, migrations in backend/migrations with [tool.alembic], email templates in features/audience and features/newsletter, and the pre-commit hooks (mypy/ty strict, lint-imports). Dropped the VS Code line: there is no .vscode/ in the repo.
  • docs/glossary.md: every code path points at its new home. No definition changed.
  • docs/product.md: "Under the hood" names the layers and the collect command. The pointers to backend/README.md and frontend/README.md, which don't exist, now point at docs/backend.md and docs/frontend.md.
  • deploy/README.md: checked both env tables against the code.
    • The newsletter table now says subscribe and send_mail read Resend vars from os.environ, and adds app/core/email.py to the rows.
    • The pipeline table only had 2 of the vars the code reads. It now has IMAP, HN, Tavily, GitHub, the proxy and kDrive too.
  • Stale paths fixed in .env.dev, collect/hn.py, curation/test_tags.py, frontend/scripts/blog-posts.ts, frontend/scripts/prerender-blog.ts, and one comment in baml_src/extract_items.baml, with baml_client regenerated (one string in inlinedbaml.py).
  • CLAUDE.md and README.md needed nothing: CLAUDE.md already had the MCP path, and README.md names no backend paths. .claude/skills had no stale paths.

Checks

  • prek run on the changed files: typos, ruff, mypy, ty, import-linter, biome, banned names and the client generation all pass. The frontend client is unchanged.
  • Stale-path grep (app/platform, pipeline/, app/catalog, app/audience, app/newsletter, app/analytics, app/mcp, backend/scripts, backend/tests, alembic.ini, app/alembic) finds nothing outside migrations, the generated client and the ADRs. ADR 17's Context names the old paths on purpose.
  • No docstring that reaches OpenAPI or the MCP tool list changed. The FastMCP instructions still say "article pipeline". That text is part of the fixed tools/list, so I left it.

Review

I ran the code-review skill at high effort on the diff. It confirmed the five contracts pass, every named path exists, and the baml edit matches baml_src. Fixed:

  • style.md and ADR 17 said flat core modules import only stdlib and third-party packages. They import each other (extract -> http, github, urls). Now: stdlib, third-party and other flat modules, never core.db.
  • record_reject was the example of a table rule two features need, but only collect calls it. Replaced with like_counts_subquery, which catalog and newsletter both read.
  • The newsletter env note left out send_mail, which reads RESEND_API_KEY and EMAILS_FROM_EMAIL from os.environ. Added, and the RESEND_API_KEY row now says an unset key means the daily mail is only logged.
  • style.md said a handler calls one feature verb, while ADR 17 says entry/api composes features (sign-up). style.md now says the handler composes them.
  • The glossary's Story entry pointed at stories/service.py for the Story table. It now names core/db/story.py and the feature folder.
  • "Read beside their one consumer" was wrong for the proxy and the Tavily key, which sit in shared core modules. Now: "in the module that uses them".
  • The proxy row said feeds only. core/extract.py uses it too.
  • The HN row now says a first-party URL skips the gate and HN_MIN_POINTS=0 turns it off.
  • backend.md said ruff runs over backend/app. It runs on every Python file.

Not fixed:

  • The glossary still says "the four shapes" and "each stage extends the one before", but Summarized sits between Scored and Persisted, and Scored no longer subclasses Candidate. These are definitions, which this step doesn't change. Listed below.

Follow-ups

  • min_score on GET /articles is capped at le=10, but scores are 1-100. Raising the cap changes the OpenAPI document and the generated client.
  • article.url and publisher.slug have no unique index.
  • send_mail in app/core/email.py reads os.environ beside settings, and so does subscribe in features/newsletter/service.py.
  • Contract 5 checks import paths, not names. A name re-exported from a feature's __init__.py would get past it.
  • No OnFailure= alerting on the units. A failed report unit sends nothing.
  • uv sync --package app installs the dev groups on the box. --no-dev would drop them.
  • BAML removal (the plan's "Out of scope" notes): only SelectNotableItems and SelectProductLink are still called, both in the newsletter source. ClassifyProfiles has no caller. NVIDIA_NIM_*/MISTRAL_AI_* are in neither .env.dev nor deploy/README.
  • Glossary definitions of the stage types (four shapes, each extends the one before) no longer match curation/model.py (ADR 17, change to ADR 8).
  • The FastMCP instructions say "article pipeline". Changing them changes tools/list.
  • README.md's "How it works" still says an LLM scores articles in the pipeline (ADR 9 changed that). It's not a path, so I left it.
  • .claude/skills/curate/regression.md has two typos hits from truncated titles (also noted on ci: deploy only what the box runs #88).

Generated by Claude Code

* fix: configure logging for the api process

* fix: let a publisher's link claim a platform subdomain

* fix: fail a link loudly on a non-numeric jev answer

* docs: point stale docstrings at where the code moved

---------

Co-authored-by: Claude <noreply@anthropic.com>
@dimslaev
dimslaev merged commit e7ef258 into ci/deploy-app-only Oct 19, 2026
6 checks passed
@dimslaev
dimslaev deleted the docs/backend-layers branch October 19, 2026 22:37
dimslaev added a commit that referenced this pull request Oct 19, 2026
* docs: point the deploy README's env table at app/features

* ci: deploy only what the box runs

* test: check the deploy filter ships every path the units and the job use

* docs: record the layered backend (#89)

* docs: record the layered backend

* fix: address the backend migration review (#91)

* fix: configure logging for the api process

* fix: let a publisher's link claim a platform subdomain

* fix: fail a link loudly on a non-numeric jev answer

* docs: point stale docstrings at where the code moved

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
dimslaev added a commit that referenced this pull request Oct 19, 2026
* fix: run mypy and ty in pre-commit with the backend config

* ci: enforce backend import layers

* fix: name the exact claude-code-action tag in its pin comments

* ci: deploy only what the box runs (#88)

* docs: point the deploy README's env table at app/features

* ci: deploy only what the box runs

* test: check the deploy filter ships every path the units and the job use

* docs: record the layered backend (#89)

* docs: record the layered backend

* fix: address the backend migration review (#91)

* fix: configure logging for the api process

* fix: let a publisher's link claim a platform subdomain

* fix: fail a link loudly on a non-numeric jev answer

* docs: point stale docstrings at where the code moved

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
dimslaev added a commit that referenced this pull request Oct 19, 2026
* test: colocate backend tests beside their modules

* test: cover the curation tools, sign-in refusals, publisher lookups and the known-URL filter

* fix: fail the daily report when its mail does not send

* test: pin that the report reaches the send, and clean up the filter's publisher

* ci: enforce backend import layers (#87)

* fix: run mypy and ty in pre-commit with the backend config

* ci: enforce backend import layers

* fix: name the exact claude-code-action tag in its pin comments

* ci: deploy only what the box runs (#88)

* docs: point the deploy README's env table at app/features

* ci: deploy only what the box runs

* test: check the deploy filter ships every path the units and the job use

* docs: record the layered backend (#89)

* docs: record the layered backend

* fix: address the backend migration review (#91)

* fix: configure logging for the api process

* fix: let a publisher's link claim a platform subdomain

* fix: fail a link loudly on a non-numeric jev answer

* docs: point stale docstrings at where the code moved

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
dimslaev added a commit that referenced this pull request Oct 19, 2026
* refactor: expose what entry needs through each feature's service

* refactor: split the mcp tools into app/entry/mcp and fold the router into main

* refactor: move scripts and the timer entrypoints into app/entry/cli

* test: cover the mcp token gate end to end over http

* test: cover the cli entrypoints' targets and exit codes

* fix: fail the backup when pg_dump writes nothing

* refactor: make app/entry pass strict mypy

* fix: keep sentry off in development

* chore: name the seed module in restore-db's hint without backticks

* docs: say what tests-start.sh does with its argument

* test: return the tool result's error flag and text from the gate tests' helper

* fix: stop importing the email module from configuring the root logger

* fix: require the alembic that reads its config from pyproject

* test: check the collect run leaves the root logger alone

* docs: say where command-line entrypoints live

* refactor: name the newsletter service alike in every mcp tool

* test: colocate backend tests beside their modules (#86)

* test: colocate backend tests beside their modules

* test: cover the curation tools, sign-in refusals, publisher lookups and the known-URL filter

* fix: fail the daily report when its mail does not send

* test: pin that the report reaches the send, and clean up the filter's publisher

* ci: enforce backend import layers (#87)

* fix: run mypy and ty in pre-commit with the backend config

* ci: enforce backend import layers

* fix: name the exact claude-code-action tag in its pin comments

* ci: deploy only what the box runs (#88)

* docs: point the deploy README's env table at app/features

* ci: deploy only what the box runs

* test: check the deploy filter ships every path the units and the job use

* docs: record the layered backend (#89)

* docs: record the layered backend

* fix: address the backend migration review (#91)

* fix: configure logging for the api process

* fix: let a publisher's link claim a platform subdomain

* fix: fail a link loudly on a non-numeric jev answer

* docs: point stale docstrings at where the code moved

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
dimslaev added a commit that referenced this pull request Oct 19, 2026
* refactor: split backend domains into app/features

* refactor: type app/features for mypy strict

* fix: keep links on a shared platform's subdomains with the aggregator

* fix: drop a newsletter product whose picked link is past the search results

* test: cover approve and reject against the schema, and publisher resolution edges

* chore: point the end-of-file exclude at the moved email templates

* refactor: tidy comments and test helpers from the features split

* refactor: move mcp tools and scripts into app/entry (#85)

* refactor: expose what entry needs through each feature's service

* refactor: split the mcp tools into app/entry/mcp and fold the router into main

* refactor: move scripts and the timer entrypoints into app/entry/cli

* test: cover the mcp token gate end to end over http

* test: cover the cli entrypoints' targets and exit codes

* fix: fail the backup when pg_dump writes nothing

* refactor: make app/entry pass strict mypy

* fix: keep sentry off in development

* chore: name the seed module in restore-db's hint without backticks

* docs: say what tests-start.sh does with its argument

* test: return the tool result's error flag and text from the gate tests' helper

* fix: stop importing the email module from configuring the root logger

* fix: require the alembic that reads its config from pyproject

* test: check the collect run leaves the root logger alone

* docs: say where command-line entrypoints live

* refactor: name the newsletter service alike in every mcp tool

* test: colocate backend tests beside their modules (#86)

* test: colocate backend tests beside their modules

* test: cover the curation tools, sign-in refusals, publisher lookups and the known-URL filter

* fix: fail the daily report when its mail does not send

* test: pin that the report reaches the send, and clean up the filter's publisher

* ci: enforce backend import layers (#87)

* fix: run mypy and ty in pre-commit with the backend config

* ci: enforce backend import layers

* fix: name the exact claude-code-action tag in its pin comments

* ci: deploy only what the box runs (#88)

* docs: point the deploy README's env table at app/features

* ci: deploy only what the box runs

* test: check the deploy filter ships every path the units and the job use

* docs: record the layered backend (#89)

* docs: record the layered backend

* fix: address the backend migration review (#91)

* fix: configure logging for the api process

* fix: let a publisher's link claim a platform subdomain

* fix: fail a link loudly on a non-numeric jev answer

* docs: point stale docstrings at where the code moved

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
dimslaev added a commit that referenced this pull request Oct 19, 2026
* fix: percent-encode database credentials in the dsn

* fix: declare the indexes and column types the migrations built

* refactor: move tables into app/core/db

* test: cover the schema against migrations, the engine and ledger writes

* refactor: split backend domains into app/features (#84)

* refactor: split backend domains into app/features

* refactor: type app/features for mypy strict

* fix: keep links on a shared platform's subdomains with the aggregator

* fix: drop a newsletter product whose picked link is past the search results

* test: cover approve and reject against the schema, and publisher resolution edges

* chore: point the end-of-file exclude at the moved email templates

* refactor: tidy comments and test helpers from the features split

* refactor: move mcp tools and scripts into app/entry (#85)

* refactor: expose what entry needs through each feature's service

* refactor: split the mcp tools into app/entry/mcp and fold the router into main

* refactor: move scripts and the timer entrypoints into app/entry/cli

* test: cover the mcp token gate end to end over http

* test: cover the cli entrypoints' targets and exit codes

* fix: fail the backup when pg_dump writes nothing

* refactor: make app/entry pass strict mypy

* fix: keep sentry off in development

* chore: name the seed module in restore-db's hint without backticks

* docs: say what tests-start.sh does with its argument

* test: return the tool result's error flag and text from the gate tests' helper

* fix: stop importing the email module from configuring the root logger

* fix: require the alembic that reads its config from pyproject

* test: check the collect run leaves the root logger alone

* docs: say where command-line entrypoints live

* refactor: name the newsletter service alike in every mcp tool

* test: colocate backend tests beside their modules (#86)

* test: colocate backend tests beside their modules

* test: cover the curation tools, sign-in refusals, publisher lookups and the known-URL filter

* fix: fail the daily report when its mail does not send

* test: pin that the report reaches the send, and clean up the filter's publisher

* ci: enforce backend import layers (#87)

* fix: run mypy and ty in pre-commit with the backend config

* ci: enforce backend import layers

* fix: name the exact claude-code-action tag in its pin comments

* ci: deploy only what the box runs (#88)

* docs: point the deploy README's env table at app/features

* ci: deploy only what the box runs

* test: check the deploy filter ships every path the units and the job use

* docs: record the layered backend (#89)

* docs: record the layered backend

* fix: address the backend migration review (#91)

* fix: configure logging for the api process

* fix: let a publisher's link claim a platform subdomain

* fix: fail a link loudly on a non-numeric jev answer

* docs: point stale docstrings at where the code moved

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
dimslaev added a commit that referenced this pull request Oct 19, 2026
* refactor: move shared backend mechanism into app/core

* refactor: reword the check_link module docstring

* fix: read a feed date with no timezone as utc

* test: cover token rejection, the daily mail send and github lookups

* test: spell unparsable the way typos expects

* refactor: move tables into app/core/db (#83)

* fix: percent-encode database credentials in the dsn

* fix: declare the indexes and column types the migrations built

* refactor: move tables into app/core/db

* test: cover the schema against migrations, the engine and ledger writes

* refactor: split backend domains into app/features (#84)

* refactor: split backend domains into app/features

* refactor: type app/features for mypy strict

* fix: keep links on a shared platform's subdomains with the aggregator

* fix: drop a newsletter product whose picked link is past the search results

* test: cover approve and reject against the schema, and publisher resolution edges

* chore: point the end-of-file exclude at the moved email templates

* refactor: tidy comments and test helpers from the features split

* refactor: move mcp tools and scripts into app/entry (#85)

* refactor: expose what entry needs through each feature's service

* refactor: split the mcp tools into app/entry/mcp and fold the router into main

* refactor: move scripts and the timer entrypoints into app/entry/cli

* test: cover the mcp token gate end to end over http

* test: cover the cli entrypoints' targets and exit codes

* fix: fail the backup when pg_dump writes nothing

* refactor: make app/entry pass strict mypy

* fix: keep sentry off in development

* chore: name the seed module in restore-db's hint without backticks

* docs: say what tests-start.sh does with its argument

* test: return the tool result's error flag and text from the gate tests' helper

* fix: stop importing the email module from configuring the root logger

* fix: require the alembic that reads its config from pyproject

* test: check the collect run leaves the root logger alone

* docs: say where command-line entrypoints live

* refactor: name the newsletter service alike in every mcp tool

* test: colocate backend tests beside their modules (#86)

* test: colocate backend tests beside their modules

* test: cover the curation tools, sign-in refusals, publisher lookups and the known-URL filter

* fix: fail the daily report when its mail does not send

* test: pin that the report reaches the send, and clean up the filter's publisher

* ci: enforce backend import layers (#87)

* fix: run mypy and ty in pre-commit with the backend config

* ci: enforce backend import layers

* fix: name the exact claude-code-action tag in its pin comments

* ci: deploy only what the box runs (#88)

* docs: point the deploy README's env table at app/features

* ci: deploy only what the box runs

* test: check the deploy filter ships every path the units and the job use

* docs: record the layered backend (#89)

* docs: record the layered backend

* fix: address the backend migration review (#91)

* fix: configure logging for the api process

* fix: let a publisher's link claim a platform subdomain

* fix: fail a link loudly on a non-numeric jev answer

* docs: point stale docstrings at where the code moved

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants