[dimslaev]

Step 1 of backend-architecture.md: the core/ layer.

What moved

• app/platform/ to app/core/: settings, logging, security, dates, email, db, email-templates.
• pipeline/fetching/http.py to core/http.py. tavily_search is split out into core/web_search.py.
• pipeline/fetching/extract_content.py to core/extract.py. It also takes outbound_links, group_links and the link host sets from pipeline/fetching/page_facts.py.
• pipeline/github.py to core/github.py. It also takes repo_for, has_readme and account_type from pipeline/sources/github_stars.py.
• pipeline/urls.py and pipeline/first_party.py merge into core/urls.py.
• pipeline/embedding.py to core/embedding.py. It also takes SNIPPET_CAP from pipeline/steps/__init__.py and cosine_distances, nearest, story_groups from pipeline/curation.py.
• pipeline/jev.py and pipeline/llm_text.py move to core/ unchanged.
• parse_date moves from pipeline/freshness.py to core/dates.py.
• send_mail moves from pipeline/report.py to core/email.py.

Every import, plus the monkeypatch targets in tests, now points at the new paths.

Differences from the plan's tree

• core/db.py is the old platform/db.py, kept as a flat module. Step 2 replaces it with the core/db/ package.
• The email templates and the sign-in/welcome builders stay in core/ for now. Step 3 moves them to audience/ and newsletter/.
• pipeline/sources/github_stars.py is deleted. It held only the three API helpers, which now live in core/github.py, so the plan's collect/github_stars.py has nothing left to hold.
• covering() stays in pipeline/curation.py, because curation is its only caller.
• pipeline/fetching/page_facts.py keeps check_link (the future curation/links.py). It imports HF_HOSTS, HF_SITE_PAGES and GITLAB_SITE_PAGES from core/extract.py.
• Existing tests stay where they are, with updated imports. Step 5 moves them.

Behaviour changes

• parse_date reads a date with no offset, or RFC 2822's -0000, as UTC. Before, it returned a naive datetime. is_within_window then read that datetime in the server's local time, and published_at was stored in the session timezone. On a UTC box the values are the same. The fix is its own commit (fix: read a feed date with no timezone as utc).
• The report process now builds Settings(), because core/email.py imports it. See below.
• One substack log line now says "unparsable" instead of "unparseable".

Tests added

• Access tokens: an expired token, a token signed with another key, and a tampered token each get 403 from /login/test-token.
• send_mail: with no key it logs instead of sending, it sends to PIPELINE_ALERT_EMAIL with the body escaped, and a failed send does not raise.
• core/github.py: account_type and repo_for return None on a 404, a rate limit, a non-JSON body, a body of the wrong shape, or a failed request.
• parse_date reads a date with no offset as UTC.

tests/platform/ is renamed to tests/core/, where the new core tests live. Step 5 flattens it.

Other changes beyond moving

• The moved modules now pass mypy --strict. jev.py uses Question and Json aliases in place of Any and bare dict. web_search.py and http.py got typed dicts. embedding.py takes and returns NDArray[np.float32]. The values are the same: np.asarray does not copy a float32 array, and 2.0**attempt gives the same backoff numbers.
• http.py builds its client as httpx.Client(follow_redirects=True, proxy=proxy or None) instead of building a kwargs dict. The behaviour is the same.
• Added the types-regex dev dependency so mypy can check llm_text.py.
• send_mail still reads os.environ (ADR 7). Importing core/email.py builds Settings(), so the report process now builds it too. The report unit already loads the full /opt/agentique/.env, so nothing changes on the box.
• unparseable became unparsable in pipeline/freshness.py and pipeline/sources/substack.py. CI runs typos on changed files, and both files are in this diff.
• Path references updated in .pre-commit-config.yaml (template exclude), deploy/README.md, docs/backend.md, docs/style.md and docs/glossary.md. ADRs are left as written.

No systemd unit, workflow or script path changes. pipeline.run and pipeline.report keep their module paths, so the units don't need reinstalling.

Note for later steps

Pre-commit runs uv run mypy backend/app backend/scripts from the repo root. From there mypy doesn't read backend/pyproject.toml's [tool.mypy], so CI type-checks non-strict. With the strict config (run from backend/), the count goes from 144 errors on master to 126 here, with none left in app/core.

Verification

• ruff check, ruff format, mypy and ty over backend/app and backend/scripts, typos and the banned-names hook: all pass (prek run on the changed files).
• Backend tests against local Postgres 16 with pgvector (prestart.sh, then tests-start.sh): 592 passed, 1 failed. The failure is test_read_endpoints_are_public[/search?q=agents], which downloads the embedding model from Hugging Face. The sandbox proxy blocks that download, and the same test fails the same way on master here.
• scripts/generate-client.sh: frontend/src/client is unchanged.
• MCP tools/list dump from master and from this branch: identical.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Tprah8Se8dG6ykxn1hV2C9

[claude]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.