Skip to content

feat: unified operations-core architecture (0.3.0) + MCP extra - #40

Merged
EvanOman merged 24 commits into
mainfrom
restructure
Jul 15, 2026
Merged

EvanOman merged 24 commits into
mainfrom
restructure

Conversation

@EvanOman

Copy link
Copy Markdown
Owner

Restructures readwise-plus around a single async operations core with the SDK, CLI, and MCP server as thin adapters over it, following Michael Feathers' Working Effectively with Legacy Code discipline throughout (characterization tests first; every stage behavior-preserving and just fc-green; public contract preserved via shims).

Highlights

  • New primary API: Readwise / AsyncReadwise with .documents/.highlights/.books/.tags/.digests/.sync + .raw.v2/.raw.v3. Old ReadwiseClient/.v2/.v3/managers/workflows/contrib still work (compat shims, unchanged behavior).
  • MCP server folded in as the readwise-plus[mcp] extra (9 tools; byte-for-byte identical output, proven by characterization contract tests).
  • CLI rebuilt over the operations core: per-resource command modules, --output table|json|jsonl, reader deprecation aliases.
  • ~1,770 net lines of duplicated sync/async code deleted; all modules now type-checked (ty), py.typed shipped, core import pulls no typer/rich/mcp.
  • Test suite 542 → 836 non-live passing; 69 characterization tests unchanged across all stages.

See MIGRATION.md for the old→new mapping and deprecation timeline (removal at 1.0). Full stage-by-stage history in the commits.

🤖 Generated with Claude Code

EvanOman and others added 24 commits July 3, 2026 18:06
The Readwise export API returns integer `nextPageCursor` values, but
`parse_pagination_cursor` called `.startswith("http")` on the raw
value, crashing with `AttributeError` on non-string cursors. Add
`str()` coercion before the string check.

This was previously worked around by a monkey-patch in highlight_helper.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fold the standalone readwise-mcp server into readwise-plus as an optional
interface, alongside the existing [cli] extra.

- New readwise_sdk.mcp package (server + console entry point) exposing 9
  agent-friendly tools over the SDK: documents (v3), highlights + books (v2).
- New `mcp` optional-dependency and `readwise-mcp` console script
  (also runnable as `python -m readwise_sdk.mcp`).
- Tests under tests/mcp/ (28 unit tests, mocked HTTP; live tests gated on
  READWISE_API_KEY). Graceful collect_ignore + importorskip when the extra
  isn't installed, mirroring the cli pattern. mcp added to the dev group so
  CI exercises it; ty excludes the FastMCP module like it does for cli.
- README: MCP Server section, install extra, ToC entry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Design (Codex gpt-5.6 oracle) for the unified operations-core restructure,
plus the standing Working-Effectively-With-Legacy-Code contract every stage
must follow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage 1 of the 0.3.0 restructure (Feathers step 1: cover before you move).
69 characterization tests pinning today's observable behavior before any
code moves — public exports, client/context-manager contracts, the three
pagination cursor styles, manager return shapes, sync-state JSON schemas,
CLI stdout/stderr/exit codes, and all nine MCP tools' exact JSON envelopes.

Includes tests that pin known-buggy behavior AS-IS (e.g. save_to_reader
drops `category`) so a later, separate commit can fix it deliberately.

Tests only; no src/ changes. `just fc` green (605 passed). Generated by
Codex gpt-5.6-sol under REFACTORING_DISCIPLINE.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behavior-preserving. Adds immutable config.py (ClientConfig) and canonical
errors.py; exceptions.py re-exports the same classes (identity-preserving,
verified `exceptions.ReadwiseError is errors.ReadwiseError`). Sync/async
clients delegate to ClientConfig with all defaults carried over verbatim
(timeout 30.0, retries 3, backoff 0.5, v2/v3 base URLs). No behavior change;
characterization + full suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behavior-preserving. Moves retry classification/backoff and the pagination
engine out of client.py/_utils.py into transport/{retry,pagination,errors}.py,
with endpoint-aware page decoders (standard v2 `next`, v2 export integer
cursors, v3 `nextPageCursor`). Retry semantics (max_retries+1 attempts,
retry_backoff * 2**attempt, same retryable conditions) and cursor behavior
carried over verbatim. ReadwiseClient/AsyncReadwiseClient.paginate() remain
compatibility wrappers with unchanged signatures. Characterization + full
suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behavior-preserving. Adds canonical async transport (transport/async_.py) and
a resources/ layer (v2 highlights/books/tags/export/review, v3 documents/tags)
that build endpoint requests and decode models with no local search/format/
workflow logic. AsyncReadwiseV2Client/AsyncReadwiseV3Client now delegate to
resources with unchanged public methods, signatures, and return types. Sync
clients untouched. New tests cover the resources/transport surface; existing
characterization suite unchanged. Non-live gate green (605 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive. New models/ package: operation input models (search/create/update
inputs owning limit bounds + normalization), shared read/summary models
(DocumentSummary/HighlightSummary/BookSummary matching current adapter
projections), and result types (DocumentSearchResult, generic BulkResult,
OperationFailure, unified SyncCheckpoint/SyncResult). Not yet wired into any
call site. New unit tests for validation/normalization. Nothing existing
changed; gate green (621 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive. New operations/documents.py + service wiring: canonical document
operations (search, get, save, update, delete, move, tag add/remove/set,
inbox/archive/later views, statistics) built on resources/ + stage-5 models.
Semantics ported verbatim from DocumentManager, ReadingInbox, DocumentImporter,
the MCP document tools, and CLI Reader commands — quirks carried as-is
(list_documents single-tag; save_to_reader category still dropped). Operations
own intent only (no formatting/protocol). MCP/CLI/managers NOT rewired yet.
New operation tests; characterization unchanged; gate green (634 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive; completes the operations core. New operations/highlights.py and
operations/books.py (wired into service): highlight search/list/get/create/
update/delete, bulk tag/untag, export, truncation + partial-failure policy;
book search/list/get, book-with-highlights, reading stats. Semantics ported
verbatim from the sync/async managers, HighlightPusher, CLI, and MCP tools
(text-substring search, 1-200/1-100 limit clamps, text[:8191] truncation,
BookCategory validation incl. supplementals, grouped export). BulkResult
retains original exceptions to reproduce legacy dict/list/bool returns later.
MCP/CLI/managers not rewired yet. Gate green (657 passed); characterization
unchanged; uv.lock unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behavior-preserving adapter rewire. Each MCP tool now does: arg adaptation →
one operations service call → existing JSON rendering → existing error
translation. Presenters (_doc_summary/_highlight_summary/_book_summary/
_json_result + error rendering) moved to mcp/output.py, behavior unchanged.
The MCP characterization contract (test_mcp_contracts.py, 17 tests) passes
UNCHANGED — byte-for-byte output preserved (tool names, compact JSON, None
omission, not-found text, invalid-enum envelopes, limit clamps, lenient
datetime, and the still-dropped save_to_reader category). No CLI/manager/
operation changes. Gate green (657 passed); uv.lock unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive. New sdk/async_.py: AsyncReadwise async context manager composed from
the operations service, exposing concept attributes (.documents/.highlights/
.books) plus a .raw.v2/.raw.v3 low-level escape hatch. Exported from the
package root alongside the unchanged AsyncReadwiseClient and all existing
imports. Facade tests cover construction, lifecycle, delegation, and .raw.
Nothing existing changed; gate green (665 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive. New sdk/sync.py: Readwise sync context manager that runs the async
operations service through a dedicated AnyIO blocking portal on its own
event-loop thread, mirroring AsyncReadwise's surface and returning bounded
results. The facade owns and closes its portal + async client. Adds anyio>=4.0
as a direct dependency. Exported from the package root; ReadwiseClient and all
existing imports unchanged. Tests cover ordinary use, use from a thread with a
running event loop (no deadlock), lifecycle, repeated ops, and exceptions
crossing the portal. Gate green (677 passed); characterization unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cli extra (typer/rich) was not in the dev environment, so the CLI
characterization + shell tests were silently skipped and the CLI wasn't
type-checked in CI. Add them to the dev group (as done for mcp) so the CLI is
a genuinely exercised surface during the restructure. No runtime deps change
(typer/rich remain the optional [cli] extra for end users).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ore (stage 11)

Behavior-preserving adapter rewire. cli/main.py split into cli/commands/
{documents,highlights,books}.py + cli/context.py (service lifecycle + credential
discovery) + cli/output.py (renderer). Each migrated command calls exactly one
operation. Adds a global --output table|json|jsonl whose default reproduces
today's Rich output byte-for-byte (machine data → stdout, notices → stderr).
Existing command names/options/output/exit codes unchanged; test_cli_contracts.py
and test_cli.py pass unchanged. cli/ stays ty-excluded until PR17. Gate green
(767 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive. New commands (each one operation): document get/update/delete/move/
tag; highlight create/update/delete/tag/untag. Adds --output jsonl streaming
and global --no-color/--quiet; JSON/JSONL output stays free of color codes and
notices. Deprecated `reader ...` aliases forward to `documents ...` with a
stderr-only deprecation notice. Existing command behavior/output/exit codes
unchanged (test_cli_contracts.py + test_cli.py green). digest custom / sync
status+reset deferred to stages 13-14 (their operations don't exist yet).
Gate green (807 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e 13)

Behavior-preserving. New operations/tags.py + operations/digests.py own tag
and digest data/selection/grouping; new presenters/ (json/text/markdown/csv)
own digest rendering. TagWorkflow and DigestBuilder become thin shims
delegating to them with unchanged public methods, return types, and output —
their existing workflow tests pass unchanged. AsyncReadwise/Readwise now expose
.tags and .digests; new CLI `digest custom` command added. Characterization +
workflow tests unchanged; gate green (821 passed); uv.lock unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behavior-preserving. New operations/sync.py (full/incremental/poll-once/batch/
background scheduling) + state/ (StateStore protocol + JsonFileStateStore).
SyncManager, BatchSync, and BackgroundPoller become shims delegating to the
unified sync operation, preserving public methods, return types, callback
(and exception-suppression) semantics, and — critically — the on-disk JSON
formats (state-contract tests green). New writes carry a version field;
migration readers load all legacy formats; malformed state is never silently
reset. .sync exposed on both facades; new `sync status`/`sync reset` commands
(existing sync full/incremental unchanged). Characterization unchanged; gate
green (835 passed); uv.lock unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…uplication (stage 15)

Behavior-preserving. HighlightManager/BookManager/DocumentManager,
async_managers, HighlightPusher, DocumentImporter, and ReadingInbox now
delegate to the operations layer; ~1,770 net lines of duplicated legacy
implementation (incl. the ~800-line parallel async manager copy) deleted.
Legacy return shapes preserved exactly — lists, lazy iterators, dict[id,bool],
dataclasses, None, formatted output, broad exception suppression, and
partial-failure behavior (BulkResult converted back to legacy boolean maps).
No direct .v2/.v3 calls remain in these layers. Manager/contrib/async +
characterization tests pass UNCHANGED; gate green (835 passed); uv.lock unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behavior-preserving. Model definitions moved to models/readwise.py (v2
concepts) and models/reader.py (v3 concepts); v2/models.py and v3/models.py
become identity-preserving re-export shims (verified
v2.models.HighlightCreate is models.readwise.HighlightCreate, and the v3
equivalent). Internal imports across resources/operations/sdk and mcp/server.py
now use canonical paths. Every legacy model import path still resolves to the
same class object. Characterization unchanged; gate green (835 passed);
deps/uv.lock/Justfile untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes the cli/ and mcp/ (and stale test) ty exclusions — the `type` recipe
is now `uv run ty check .`, checking every module; the restructure left cli/
and mcp/ type-clean, so zero fixes were needed (verified ty is not no-opping).
Adds a core-import isolation test proving `import readwise_sdk` pulls no
typer/rich/mcp (subprocess + meta_path blocker to simulate a core-only install
past httpx's own optional-CLI fallback). Adds py.typed (shipped in the wheel)
and .gitattributes (eol=lf). Wheel builds with py.typed + both console scripts
(readwise, readwise-mcp). Gate green (836 passed); deps/uv.lock unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(stage 18)

Rewrites README around the new Readwise/AsyncReadwise facade (.documents/
.highlights/.books/.tags/.digests/.sync + .raw escape hatch), with accurate
CLI (expanded tree + --output) and MCP sections and a Compatibility APIs
section for the still-supported ReadwiseClient/managers/workflows/contrib.
Adds MIGRATION.md (old→new mapping + deprecation timeline to 1.0) and a short
docs/architecture.md contributor note; refreshes llms.txt/llms-full.txt. Every
snippet was executed under respx mocks, which also corrected several pre-existing
doc inaccuracies. Removes the refactor working docs (ARCHITECTURE_PLAN.md,
REFACTORING_DISCIPLINE.md). Docs only; gate green (836 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
save_to_reader accepted a `category` argument but never forwarded it — the MCP
tool didn't parse it, and the document save operation deliberately stripped it
to preserve the pre-refactor bug. Now the tool validates category (returning an
"Invalid category" envelope on bad input) and passes it through; the save
operation forwards it unchanged. This is a deliberate, isolated behavior change:
the three tests that pinned the dropped-category behavior are flipped to assert
the fix (plus a new invalid-category rejection test). Gate green (837 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_typer_parse_error_is_exit_two_and_routed_to_stderr asserted a plain
substring, but typer/rich colorizes its error box when color is forced (GitHub
Actions enables it; a plain terminal doesn't), which split "--bad-option" with
escape codes and failed only on CI's 3.14 job. Strip ANSI before the content
assertions — the exit code, empty stdout, and stderr routing contract is
unchanged; only the incidental coloring is normalized. Gate green (837 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@EvanOman
EvanOman merged commit 3159ad4 into main Jul 15, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant